Model training method, image processing method and device

Abstract

The invention provides a model training method and device and an image processing method and device, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, AI medical treatment and the like. The specific implementation scheme of the model training method comprises the following steps: detecting a sample medical image by using a visual large model, and generating first position information for representing a sample abnormal object in the sample medical image; detecting the sample medical image by using the target model, and generating second position information representing a sample abnormal object; and based on the target loss function, taking the visual large model as a teacher model, taking the target model as a student model, and training the target model by using the first position information and the second position information to obtain a trained target model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, AI medical treatment and the like, and specifically to a model training method, an image processing method and device. BACKGROUND

[0002] Medical image segmentation is an important part of clinical auxiliary diagnosis, and is widely used in the scene of early disease screening. Due to the high noise of medical images, the neural network model based on feature modeling has low accuracy in outputting results when performing medical image segmentation tasks, and it is difficult to meet the detection needs of medical images in actual application scenarios. SUMMARY

[0003] The present disclosure provides a model training method, an image processing method and device.

[0004] According to an aspect of the present disclosure, a model training method is provided, comprising: detecting a sample medical image using a visual large model to generate first position information for representing a sample abnormal object in the sample medical image; detecting the sample medical image using a target model to generate second position information representing the sample abnormal object; and based on a target loss function, using the visual large model as a teacher model and the target model as a student model, training the target model using the first position information and the second position information to obtain a trained target model.

[0005] According to another aspect of the present disclosure, an image processing method is provided, comprising: detecting a to-be-processed medical image using a trained target model to generate position information of a target abnormal object in the to-be-processed medical image; wherein the trained target model is obtained based on the model training method described above; and labeling the to-be-processed medical image based on the position information to generate a target image.

[0006] According to another aspect of the present disclosure, a model training device is provided, comprising: a first detection module, a second detection module and a training module.

[0007] The first detection module is configured to detect a sample medical image using a visual large model to generate first position information for representing a sample abnormal object in the sample medical image.

[0008] The second detection module is configured to detect the sample medical image using a target model to generate second position information representing the sample abnormal object.

[0009] The training module is configured to, based on a target loss function, use the visual large model as a teacher model and the target model as a student model, train the target model using the first position information and the second position information to obtain a trained target model.

[0010] According to another aspect of the present disclosure, an image processing apparatus is provided, comprising: a third detection module and a marking module.

[0011] The third detection module is configured to detect the medical image to be processed by using the trained target model, to generate position information of the target abnormal object in the medical image to be processed; wherein the trained target model is obtained by using the model training method described above.

[0012] The marking module is configured to mark the medical image to be processed based on the position information, to generate a target image.

[0013] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0014] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method described above.

[0015] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method described above.

[0016] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0018] Figure 1 An exemplary system architecture to which the model training method and apparatus according to embodiments of the present disclosure can be applied is schematically shown;

[0019] Figure 2 A flowchart of the model training method according to embodiments of the present disclosure is schematically shown;

[0020] Figure 3 A schematic diagram of the model training method according to embodiments of the present disclosure is schematically shown;

[0021] Figure 4 A schematic diagram of generating first position information by using a visual large model according to embodiments of the present disclosure is schematically shown;

[0022] Figure 5A FIG. 1 schematically illustrates a schematic diagram of screening sample labels for training a target model according to an embodiment of the present disclosure;

[0023] Figure 5B FIG. 1 schematically illustrates a schematic diagram of screening sample labels for training a target model according to an embodiment of the present disclosure;

[0024] Figure 5C FIG. 1 schematically illustrates a schematic diagram of screening sample labels for training a target model according to an embodiment of the present disclosure;

[0025] Figure 5D FIG. 1 schematically illustrates a schematic diagram of screening sample labels for training a target model according to an embodiment of the present disclosure;

[0026] Figure 6A FIG. 1 schematically illustrates a schematic diagram of training a target model with a visual large model as a teacher model according to an embodiment of the present disclosure;

[0027] Figure 6B FIG. 1 schematically illustrates a schematic diagram of training a target model with a visual large model as a teacher model according to an embodiment of the present disclosure;

[0028] Figure 6C FIG. 1 schematically illustrates a schematic diagram of training a target model with a visual large model as a teacher model according to an embodiment of the present disclosure;

[0029] Figure 7 FIG. 1 schematically illustrates a schematic diagram of training a target model with a visual large model as a teacher model according to an embodiment of the present disclosure;

[0030] Figure 8 FIG. 1 schematically illustrates a flowchart of an image processing method according to an embodiment of the present disclosure;

[0031] Figure 9 FIG. 1 schematically illustrates a schematic diagram of an image processing method according to an embodiment of the present disclosure;

[0032] Figure 10 FIG. 1 schematically illustrates a block diagram of a model training apparatus according to an embodiment of the present disclosure;

[0033] Figure 11 FIG. 1 schematically illustrates a block diagram of an image processing apparatus according to an embodiment of the present disclosure; and

[0034] Figure 12 FIG. 1 schematically illustrates a block diagram of an electronic device suitable for implementing a model training method or an image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary. Therefore, it should be understood that various changes and modifications to the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted herein.

[0036] When facing the image segmentation task, some related examples usually utilize deep neural networks to achieve image segmentation through cross-modal or cross-time sequence modeling. For example: multi-modal input is adopted, modal feature fusion is performed through a Transformer encoder, and semi-supervised image segmentation is achieved by combining optical flow modeling or a twin network.

[0037] This feature-based image segmentation method has obvious advantages in feature representation, cross-modal fusion, and detailed feature capture. These advantages often rely on a large number of labeled samples or multi-modal data input and complex network architecture. However, in actual application scenarios, the clarity of medical images is uneven, and labeled samples are scarce, resulting in fluctuations in the accuracy of the model output. Secondly, the operation of a complex network architecture requires a large amount of computing resources, thus limiting the deployment of this solution in edge devices or real-time diagnosis and treatment scenarios.

[0038] Some other related examples mainly use data augmentation and weak supervision methods to alleviate the impact of the scarcity of labeled samples on model training. However, although data augmentation and weak supervision methods improve the generalization ability of the model, they lack prior global semantics and knowledge guidance, resulting in low detection accuracy for complex lesion images or edge-blurred lesion images.

[0039] Therefore, the model training method provided in the embodiments of the present disclosure uses a visual large model as a teacher model to guide the training of a target model. The visual large model provides prior global semantics and knowledge guidance for the target model, improves the detection accuracy of medical images with low clarity or edge blur, reduces the dependence of the detection accuracy of the target model on computing resources, and improves the feasibility of deploying the target model in edge devices or real-time diagnosis and treatment scenarios. In this way, high-precision detection of the position of an abnormal object in a medical image can be achieved with less computing resource occupation.

[0040] Figure 1 An exemplary system architecture to which the model training method or the image processing method and device according to the embodiments of the present disclosure can be applied is schematically shown.

[0041] It should be noted that, Figure 1The shown is only an example of a system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be applied to other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the model training method or image processing method and device can be applied can include a terminal device, but the terminal device can not need to interact with a server to implement the model training method or image processing method and device provided by the embodiments of the present disclosure.

[0042] As shown in Figure 1 The system architecture 100 according to the embodiment can include a terminal device 101 and a terminal device 102.

[0043] First, the terminal device 101 can load a visual large model, and use the visual large model to detect a sample medical image 110 to generate first position information for representing a sample abnormal object in the sample medical image 110.

[0044] Then, the terminal device 102 can load a target model, which can be any lightweight neural network model. The target model is used to detect the sample medical image 110 to generate second position information for representing the sample abnormal object in the sample medical image 110.

[0045] Next, the first position information can be used as a sample label, the visual large model can be used as a teacher model, and the target model can be used as a student model. The target model is trained by knowledge distillation, so that the loss value between the second position information output by the target model and the first position information output by the visual large model converges, and a trained target model is obtained.

[0046] It should be noted that the model training method or image processing method provided by the embodiments of the present disclosure can generally be executed by the terminal devices 101 and 102. Correspondingly, the model training method or image processing device provided by the embodiments of the present disclosure can also be arranged in the terminal devices 101 and 102.

[0047] It should be understood that Figure 1 The number of terminal devices in the above

[0048] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information comply with relevant laws and regulations, necessary security measures are taken, and public order and good customs are not violated.

[0049] In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before the user's personal information is acquired or collected.

[0050] Figure 2 A flowchart of a model training method according to an embodiment of the present disclosure is schematically shown.

[0051] As shown in the method 200 includes operations S210-S230. Figure 2

[0052] At operation S210, the sample medical image is detected by using a visual large model to generate first position information for representing a sample abnormal object in the sample medical image.

[0053] At operation S220, the sample medical image is detected by using a target model to generate second position information for representing the sample abnormal object.

[0054] At operation S230, based on a target loss function, the target model is trained by using the first position information and the second position information, with the visual large model as a teacher model and the target model as a student model, to obtain a trained target model.

[0055] According to an embodiment of the present disclosure, the sample medical image can be an image collected by a medical device for detecting a disease, for example, an image collected by a magnetic resonance imaging device, an image collected by a computed tomography device, an image collected by various spectrum devices, etc. It can also be an image collected by an image collection device, for example, an image obtained by photographing the skin surface by using an image collection device.

[0056] It should be noted that the acquisition of the sample medical image in the embodiments of the present disclosure is agreed by the collected object, complies with the relevant legal regulations, takes necessary security measures, and does not violate public order and good customs.

[0057] In some embodiments, in order to further improve the generalization ability of the model, in the case that the collected sample medical images are limited, some sample medical images with blurred edges or low clarity can also be generated by using a multi-modal large model.

[0058] The visual large model refers to a model pre-trained based on large-scale visual data (for example, images, videos) and multi-modal data (such as image-text pairs). It learns general visual features and semantic concepts from massive data, has strong visual perception, understanding, generation and reasoning capabilities, and can adapt to various downstream visual tasks through fine-tuning, prompting and other mechanisms.

[0059] The target model can be any lightweight neural network model. Illustratively, the specific type of the lightweight neural network model can be determined based on the state of the computing resources possessed by the region to be deployed, which is not described herein.

[0060] ​In some embodiments, the sample medical image is detected by using the visual large model to generate first position information for representing a sample abnormal object in the sample medical image. The sample abnormal object can represent a sample disease diagnosed based on the sample medical image, for example, folliculitis. The first position information can represent a lesion area of the sample disease in the sample medical image output by the visual large model. For example, the lesion area can be a polygon capable of surrounding the lesion position, and the first position information can be represented as the coordinates of the vertices in the polygon.

[0061] In some embodiments, the sample medical image is detected by using the target model to generate second position information for representing a sample abnormal object in the sample medical image. The second position information can represent a lesion area of the sample disease in the sample medical image output by the target model.

[0062] In the embodiments of the present disclosure, the shape of the polygon for marking the lesion area can be pre-configured, for example, it can be a rectangle, and then the first position information and the second position information can be represented as the coordinates of the four vertices in the matrix.

[0063] When the sample medical image is collected by using a medical device, the image sizes collected by different medical devices are different, and the first position information output by the visual large model and the second position information output by the target model can be normalized respectively to reduce the influence of the size difference of the sample medical image on the training.

[0064] In some embodiments, the first position information can be used as a sample label, and a target loss function can be used to calculate a loss value between the first position information and the second position information. The target loss function can be, for example, a cross-entropy loss function. Then, the model parameters of the target model are adjusted based on the loss value to obtain a trained target model. This synchronous training method uses the visual large model as a teacher model, which is equivalent to providing prior global semantics and knowledge guidance for the target model. In the training process, the visual perception, understanding and detection ability of the visual large model on the sample medical image are distilled to the target model, and the detection accuracy of the target model on the sample medical image is further improved.

[0065] In some embodiments, the visual large model can be used to detect a plurality of sample medical images to generate a plurality of first position information respectively corresponding to the plurality of sample medical images. Then, the plurality of sample medical images and the plurality of first position information are used as training samples to train the target model. Since the visual large model requires more computing resources during operation, this asynchronous training method can train the target model with less computing resources.

[0066] The embodiments of the present disclosure take a visual large model as a teacher model to guide the training of a target model, and the visual large model provides prior global semantics and knowledge guidance for the target model, thereby improving the detection accuracy of medical images with low clarity or blurred edges. At the same time, the dependence of the detection accuracy of the target model on computing resources is reduced, and the feasibility of deploying the target model in edge devices or real-time diagnosis and treatment scenarios is improved, so as to realize high-precision detection of the position of an abnormal object in a medical image while occupying fewer computing resources.

[0067] Figure 3 A schematic diagram of a model training method according to an embodiment of the present disclosure is schematically shown.

[0068] As shown in Figure 3 The sample medical image 301 in this embodiment 300 can be a skin image with a folliculitis lesion site. The sample medical image 301 is input into the visual large model 310, and the first position information 311 is output. Exemplarily, the first position information 311 can be the coordinates of the four vertices of a rectangular box that can surround the folliculitis lesion area detected by the visual large model 310.

[0069] In this embodiment 300, a synchronous training method can be used, or an asynchronous training method can be used. The sample medical image 301 is input into the target model 320, and the second position information 321 is output. Exemplarily, the second position information 321 can be the coordinates of the four vertices of a rectangular box that can surround the folliculitis lesion area detected by the target model 320.

[0070] Then, the first position information 311 and the second position information 321 can be processed by using a target loss function to obtain a loss value 302. Based on the loss value 302, the model parameters of the target model 320 are adjusted by back propagation until the loss value 302 converges, and a trained target model is obtained.

[0071] In actual application scenarios, the skin image in the medical image often has a blurred edge problem, which may cause the first position information output by the visual large model for such difficult samples to also have a certain deviation. Therefore, in order to further improve the output accuracy of the visual large model, the embodiments of the present disclosure can include the following operations when the visual large model is used to detect the sample medical image to generate the first position information for representing the sample abnormal object in the sample medical image: using the visual large model to recognize the sample medical image to generate a sample text for describing abnormal features; and using the visual large model to detect the sample medical image based on the sample text to generate the first position information.

[0072] Figure 4 A schematic diagram of generating first position information by using a visual large model according to an embodiment of the present disclosure is schematically shown.

[0073] As Figure 4 shown, in this embodiment 400, the sample medical image 301 can be input into the visual large model 310, and the sample text 401 is output. The sample text 401 is used to describe the abnormal features of the sample abnormal object in the sample medical image 301. For example, the sample abnormal object can be folliculitis, and the abnormal features can be the state features used to describe the lesion site of folliculitis, such as: red, solid, irregular-shaped papules at the follicle orifice, yellow-white pus at the top of the red papules, etc.

[0074] Then, the sample text and the sample medical image are input into the visual large model, the abnormal features described in the sample text are used to provide reference information for the visual large model, the understanding degree of the visual large model for the lesion area is further improved, and the accuracy of the first position output by the visual large model is further improved, so that the target model can obtain more accurate sample labels in the process of knowledge distillation.

[0075] Since the visual large model has certain randomness when facing difficult samples, the visual large model can be used to detect the same sample medical image multiple times, so as to be able to screen out samples for training the target model.

[0076] The following will be based on Figure 5A to Figure 5D the embodiments of screening sample labels for training the target model.

[0077] In some embodiments, detecting the sample medical image by using the visual large model to generate the first position information for representing the sample abnormal object in the sample medical image can further include the following operations: extracting a first local image from the sample medical image based on the first position information; identifying the first local image by using the visual large model to generate a description text for describing the abnormal features in the local image; and determining that the first position information is used for training the target model in response to determining that the sample text matches the description text.

[0078] Figure 5A An illustrative diagram of screening sample labels for training the target model according to an embodiment of the present disclosure is schematically shown.

[0079] As Figure 5A shown, the difference between this embodiment 500A and the embodiment 400 is that after the visual large model 310 generates the first position information 311, a first local image 501 is extracted from the sample medical image based on the first position information 311.

[0080] Then, the first local image 501 is identified by using the visual large model 310 to generate a description text 502 for describing the abnormal features in the local image.

[0081] Finally, operation S510 is performed to determine whether the sample text 401 matches the description text 502. If yes, operation S520 is performed to retain the first position information for training the target model. If not, operation S530 is performed to discard the first position information.

[0082] Exemplarily, it can be determined whether the sample text 401 matches the semantics of the description text 502, or whether the disease type described by the sample text 401 matches the disease type described by the description text 502. For example, it can be determined whether the texts match by calculating the text similarity between the sample text 401 and the description text 502, or the sample text 401 and the description text 502 can be input into a large language model to determine whether the sample text 401 and the description text 502 describe the same type of disease by using the natural language understanding capability of the large language model.

[0083] Since the lesion area in the sample medical image may only occupy a small part of the sample medical image, or the lesion area in the sample medical image may have a blurred edge with other non-lesion areas, these will reduce the understanding degree of the visual large model to the image. Therefore, extracting the local image of the lesion area can help the visual large model exclude interference factors and only describe the sample abnormal object according to the abnormal features presented in the local image.

[0084] In some embodiments, the visual large model is used to detect the sample medical image to generate first position information for representing a sample abnormal object in the sample medical image, and the operation can further include the following operations: transforming the sample medical image along a first direction to generate a to-be-processed image; using the visual large model to detect the to-be-processed image to generate an initial position of the sample abnormal object in the to-be-processed image; transforming the initial position along a second direction to generate a target position; and in response to determining that the difference between the target position and the first position information is less than a predetermined threshold, determining that the first position information is used to train the target model.

[0085] Exemplarily, the first direction is opposite to the second direction. The sample medical image is transformed along the first direction, for example, the first direction can be a clockwise direction, and the transformation processing can be rotation processing. The initial position is transformed along the second direction, for example, the second direction can be an anticlockwise direction, and the transformation processing can be rotation processing.

[0086] In addition to rotation, the to-be-processed image can also be obtained by performing stretching, mirroring and other transformation operations on the sample medical image. It should be noted that the transformation operation performed on the sample medical image is the same as the transformation operation performed on the initial position, for example, a 45° rotation operation, but the rotation direction is opposite.

[0087] Figure 5BThe illustration shows a schematic diagram of screening sample labels for training a target model according to an embodiment of the present disclosure.

[0088] like Figure 5B As shown, in Example 500B, the sample medical image 301 is rotated 90 degrees clockwise to obtain the image to be processed 510. The visual large model 310 is used to detect the image to be processed 510, and the generated initial position 511 can be represented as the coordinates of the four vertices of the rectangle abcd.

[0089] At this point, the rectangle abcd used to mark the initial position can be rotated 90 degrees counterclockwise to obtain the target position 512, which can be represented as the coordinates of the four vertices of the rectangle A'B'C'D'.

[0090] Then, the difference between the coordinates of the four vertices of the rectangle ABCD used to represent the first position information and the coordinates of the four vertices of the rectangle A'B'C'D' used to represent the target position is calculated. Operation S540 is then executed to determine if the difference is less than a predetermined threshold. If yes, operation S520 is executed; otherwise, operation S530 is executed.

[0091] If the detected initial position can be aligned with the detection result of the original image after inverse spatial transformation by performing spatial transformation on the sample medical image, it indicates that the output result of the large visual model has high stability. Therefore, the output result can be retained for training the target model. By screening out the output result with high stability through multi-view consistency, the accuracy of the sample data is further improved.

[0092] In some embodiments, the detection of sample medical images using a large visual model to generate first location information for characterizing abnormal objects in the sample medical images may further include the following operations: extracting multiple candidate local images from the sample medical images based on multiple candidate locations; using a large visual model to filter the multiple candidate local images to determine a target local image from the multiple candidate local images; and in response to determining that the target local image matches the first local image, determining the first location information for training the target model.

[0093] For example, multiple candidate locations can represent the vertex coordinates of multiple randomly generated detection boxes, and the shape of the detection boxes can be any polygon. It is understood that multiple candidate local images are extracted from the sample medical image at arbitrary locations. Then, a large visual model is used to select the local image that best represents the lesion region from the multiple candidate local images as the target local image. If the target local image matches the first local image, for example, if the pixel similarity is greater than a preset pixel threshold, it can be determined that the output of the large visual model is stable, and the first position information of this output can be used to train the target model.

[0094] Figure 5C A schematic diagram of screening sample labels for training a target model is shown according to an embodiment of the present disclosure.

[0095] As Figure 5C shown, a plurality of local images 521 are extracted from the sample medical image 301 based on a plurality of candidate positions 520, respectively. Then, the plurality of local images 521 are input into the visual large model 310, and a target local image 522 is output.

[0096] In some embodiments, in order to improve the understanding ability of the visual large model 310, the plurality of local images 521 can also be input into the visual large model 310 together with the sample text 401, so that the visual large model 310 can filter out the target local image 522 matching the lesion characteristics from the plurality of local images according to the lesion characteristics described in the sample text 401.

[0097] Then, the first local image 501 extracted from the sample medical image based on the first position information 311 is compared with the target local image, operation S550 is performed, and it is determined whether the first local image 501 matches the target local image 522. If yes, operation S520 is performed, and if not, operation S530 is performed.

[0098] By using the screening ability of the visual large model on images, the image that best reflects the lesion characteristics is filtered out. By comparing the first local image extracted from the sample medical image based on the first position information with the filtered target local image, the sample when the visual large model is stable can be filtered out, the fluctuation of the detection accuracy of the visual large model when processing sample medical images with different clarity is reduced, and the accuracy of the sample data used for training the target model is further improved.

[0099] In some embodiments, the visual large model is used to detect the sample medical image to generate first position information for representing a sample abnormal object in the sample medical image. The method can further include the following operations: extracting a first local image from the sample medical image based on the first position information; using a multi-modal large model to generate a sample local image based on the sample text and the sample medical image; and in response to determining that the sample local image matches the first local image, determining that the first position information is used for training the target model.

[0100] In an embodiment of the present disclosure, the multi-modal large model refers to an artificial intelligence model capable of processing, understanding and generating multiple types of information. The multiple types of information include but are not limited to images, texts, audios, videos, etc.

[0101] Since the visual large model mainly focuses on obtaining a deep understanding of pixel data and generation capability, the multi-modal large model mainly focuses on understanding the correspondence between different modal information. Therefore, when the sample local image of the output of the multi-modal large model matches the first local image, it can be determined that the output result of the visual large model is stable, and the first position information output this time can be used for training the target model.

[0102] Figure 5D An illustrative diagram of screening sample labels for training a target model according to an embodiment of the present disclosure is shown.

[0103] As Figure 5D shown, in this embodiment 500D, the sample text 401 and the sample medical image 301 can be input into the multi-modal large model 530, and on the basis of the multi-modal large model 530 deeply understanding the correspondence between the sample text 401 and the sample medical image 301, a sample local image 531 matching the sample text can be generated. The sample local image 531 can represent a local image for showing a lesion area where the sample abnormal object is located.

[0104] Then, the first local image 501 extracted from the sample medical image based on the first position information is compared with the sample local image 531, operation S550 is performed, and it is determined whether the first local image 501 matches the sample local image 531, for example: whether the image similarity is less than a predetermined similarity threshold. If yes, operation S520 is performed. If not, operation S530 is performed.

[0105] Using the sample local image generated by the multi-modal large model on the basis of deeply understanding the correspondence between the abnormal features described by the sample text and the sample medical image, for screening the first position information, can reduce the influence of single model fluctuation on sample quality, thereby further improving the training efficiency.

[0106] In some embodiments, the sample local image can also be detected by the visual large model to generate label position information for training the target model.

[0107] Since the sample local image is generated by the multi-modal large model on the basis of deeply understanding the text and image for describing the sample abnormal object, the result is more stable than the output of the visual large model. Therefore, the sample local image can be detected by the visual large model, and the label position information for training the target model can be generated by using the understanding ability of the large model for pixel data.

[0108] This asynchronous training method can reduce the computing resources consumed when training the target model, and at the same time, it also solves the problem of sample annotation scarcity in actual application scenarios.

[0109] The very small difference in grayscale / color between the diseased tissue and the surrounding healthy tissue leads to blurred boundaries; the same tissue exhibits different intensity values ​​in different areas of the image; and the location of the lesion varies greatly among different individuals, and even among different times of the same individual, all of which affect the accuracy of image segmentation.

[0110] Therefore, in addition to directly utilizing positional differences for knowledge distillation, the embodiments of this disclosure can also perform knowledge distillation by extracting local images and aligning multimodal information, thereby further improving distillation efficiency.

[0111] The following is combined Figure 6A to Figure 6C A detailed description of an implementation of the training target model is provided.

[0112] Based on the target loss function, using a large visual model as the teacher model and the target model as the student model, the target model is trained using first and second location information to obtain a trained target model. This can include the following operations: extracting a first local image from the sample medical image based on the first location information; extracting a second local image from the sample medical image based on the second location information; and training the target model using the first and second local images based on the target loss function.

[0113] Since lesion areas may only occupy a small portion of a sample medical image, class imbalance needs to be addressed when performing image segmentation tasks. Therefore, the objective loss function can be a loss function that optimizes the degree of overlap between the segmented region and the ground truth region, such as the Desce similarity coefficient loss function.

[0114] Figure 6A The illustration shows a schematic diagram of training a target model using a large visual model as a teacher model according to an embodiment of the present disclosure.

[0115] like Figure 6A As shown, in this embodiment 600A, the first location information 311 can be the coordinates of the first polygonal region where the abnormal object of the sample medical image 301 is located, generated based on the detection of the sample medical image 301 by the visual large model 310. The second location information 321 can be the coordinates of the second polygonal region where the abnormal object of the sample is located, generated based on the detection of the sample medical image 301 by the target model 320.

[0116] Then, the first local image 501 can represent the image corresponding to the first polygonal region. The second local image 601 can represent the image corresponding to the second polygonal region.

[0117] In some embodiments, the target loss function can include a first loss function, which can be a mean square error loss function for calculating pixel loss, or a structural similarity index loss function. The type of loss function is not specifically limited in the embodiments of the present disclosure.

[0118] According to an embodiment of the present disclosure, based on the target loss function, the visual large model is taken as a teacher model, the target model is taken as a student model, the first local image and the second local image are used to train the target model, and a trained target model can include the following operations: the first local image and the second local image are processed by using the first loss function to generate a first loss value; and based on the first loss value, the model parameters of the target model are adjusted to obtain the trained target model.

[0119] Exemplarily, based on the first loss value 302A, the model parameters of the target model can be adjusted by using the back propagation method until the first loss value converges or reaches a maximum iteration number, and the trained target model is generated.

[0120] By extracting the local image for knowledge distillation, the difficulty caused by the low clarity or small area of the sample abnormal object in the sample medical image to the image segmentation task can be reduced, and the efficiency of distillation is further improved.

[0121] In some embodiments, the visual large model can also be used to generate the abnormal features of the sample abnormal object in the first local image and the second local image based on the depth understanding ability of the pixel data in the image, and knowledge distillation can be performed based on the text description function and the respective description texts.

[0122] Exemplarily, the target loss function includes a second loss function; the second loss function can be a function for calculating text loss.

[0123] According to an embodiment of the present disclosure, based on the target loss function, the visual large model is taken as a teacher model, the target model is taken as a student model, the first local image and the second local image are used to train the target model, and a trained target model can include the following operations: the first local image is identified by using the visual large model to generate a first text; the first text is used to describe the abnormal features in the first local image; the second local image is identified by using the visual large model to generate a second text; the second text is used to describe the abnormal features in the second local image; the first text and the second text are processed by using the second loss function to generate a second loss value; and based on the second loss value, the model parameters of the target model are adjusted to obtain the trained target model.

[0124] Figure 6B A schematic diagram of training a target model by using a visual large model as a teacher model according to an embodiment of the present disclosure is schematically shown.

[0125] As shown in Figure 6B The embodiment 600B is different from the embodiment 600A in that, based on the deep understanding ability of the visual large model 310 to the pixel data in the sample medical image 301, the first text 610 is generated based on the first local image 501 and the second text 620 is generated based on the second local image 502. The first text 610 describes the abnormal features in the lesion area corresponding to the first position information. The second text 620 describes the abnormal features in the lesion area corresponding to the second position information. If the abnormal features described by the first text 610 and the second text 620 are similar, it indicates that the position of the sample abnormal object in the sample medical image detected by the target model is close to the position of the sample abnormal object in the sample medical image detected by the visual large model, so as to convert the image loss into the text loss, perform knowledge distillation, reduce the influence of the large position difference caused by the edge blur of the sample medical image on the distillation efficiency, and further improve the distillation efficiency.

[0126] Then, the second loss value 302B is generated by processing the first text 610 and the second text 620 by using the second loss function, and the trained target model is generated by training the target model 320 based on the second loss value 302B.

[0127] For example, the second loss function can be a function for calculating the text loss, such as a cross-entropy loss function.

[0128] In some embodiments, when training the target model 320, the model parameters of the target model 320 can be continuously adjusted based on the second loss value 302B until the second loss value 302B converges or reaches a maximum number of iterations, and the trained target model is obtained.

[0129] In some embodiments, knowledge distillation can also be performed in a combination of image loss and text loss.

[0130] For example, the target loss function can also include a first text loss function, i.e., a function for calculating the pixel loss. Based on the second loss value, adjusting the model parameters of the target model to obtain the trained target model can include the following operations: based on the second loss value, adjusting the model parameters of the target model to obtain an intermediate model; detecting the sample medical image by using the intermediate model to generate third position information for representing the sample abnormal object in the sample medical image; extracting a third local image from the sample medical image based on the third position information; generating a third loss value by processing the first local image and the third local image by using the first loss function; and based on the third loss value, adjusting the model parameters of the intermediate model to obtain the trained target model.

[0131] In some embodiments, the maximum number of iterations can be pre-configured, for example, can be N, N is an integer greater than 1, and then the target model is trained indirectly using the text loss and the image loss.

[0132] For example, in the n th training process, the model parameters of the n-1 th intermediate model obtained by the n-1 th training can be adjusted based on the second loss value to generate an n th intermediate model. Then the n+1 th training is performed. The n th intermediate model is used to detect the sample medical image to generate third position information. The third position information is represented as the coordinates of a polygon capable of enclosing the position region of the sample abnormal object in the sample medical image. n is an integer greater than or equal to 2 and less than or equal to N-1.

[0133] It should be noted that the objects of adjusting the model parameters described above are all lightweight target models, and the model parameters of the visual large model are unchanged.

[0134] Then, the third local image is extracted from the sample medical image based on the third position information. The model parameters of the n th intermediate model are adjusted based on the image loss between the third local image and the first local image to generate an n+1 th intermediate model. In this way, until the loss converges or the maximum number of iterations is reached, the trained target model is obtained.

[0135] The distillation method combining the image loss and the text loss makes full use of the deep understanding ability of the visual large model for pixel data, and further improves the training precision of the target model.

[0136] In some embodiments, based on the target loss function, the visual large model is used as the teacher model, the target model is used as the student model, the target model is trained using the first position information and the second position information, and the trained target model can include the following operations: using the visual large model, identifying the sample medical image based on the second position information to generate a third text; using the visual large model, identifying the sample medical image based on the first position information to generate a label text for describing the sample abnormal object; and using the visual large model as the teacher model, the target model as the student model, based on the target loss function, using the third text and the label text to train the target model to obtain the trained target model.

[0137] In addition to extracting local images to reduce the interference of background images on the identification of lesion regions, the generation ability of the visual large model can also be used to generate description texts corresponding to the regions represented by the position information output by each model.

[0138] Figure 6C A schematic diagram of training a target model using a visual large model as a teacher model according to an embodiment of the present disclosure is schematically shown.

[0139] AsFigure 6C As shown, the embodiment 600C is different from the embodiment 600B in that the operation of local image extraction is not performed, but the generation capability of the visual large model 310 is utilized to respectively generate the label text 630 and the third text 640 based on the first position information 311 and the second position information 321 respectively. The label text 630 is used to describe the abnormal features of the region in the sample medical image 301 associated with the first position information 311. The third text 640 is used to describe the abnormal features of the region in the sample medical image 301 associated with the second position information 321.

[0140] In some embodiments, the target loss function can include a second loss function. Based on the target loss function, the target model is trained by using the third text and the label text with the visual large model as the teacher model and the target model as the student model, and a trained target model can include the following operations: processing the third text and the label text by using the second loss function to generate a text loss value; and adjusting the model parameters of the target model based on the text loss value to obtain the trained target model.

[0141] If the abnormal features described by the label text 630 and the third text 640 are similar, it indicates that the second position information 321 detected by the target model is close to the first position information 311 detected by the visual large model 310. The training accuracy is further improved by training the target model 320 based on the text loss between the label text and the third text.

[0142] As Figure 6C shown, processing the third text 640 and the label text 630 by using the second loss function can obtain a loss value 302C. The loss value 302C can represent a text loss value. Then, the model parameters of the target model 320 can be adjusted based on the loss value 302C until the loss value converges or the maximum number of iterations is reached to obtain a trained target model.

[0143] In addition, distillation can also be performed by combining the text loss and the position loss, and in this case, the loss value 302C can represent the weighted sum of the text loss value and the position loss value.

[0144] In some embodiments, the above method can further include the following operations: processing the first position information and the second position information by using the first loss function to generate a position loss value; and adjusting the model parameters of the target model based on the text loss value and the position loss value to obtain a trained target model.

[0145] For example, the target loss function can include a first loss function and a second loss function. Weights can be pre-configured for each loss function, and the target loss function value can represent the weighted sum of all loss values. Alternatively, the weights of each loss function can be configured as learnable parameters, allowing the target model to dynamically adapt the learnable weights based on the sample input during training, further improving the model's generalization ability.

[0146] For example, the weights of the first loss function can be w1, and the weights of the second loss function can be w2, where w1 + w2 = 1. First, the second loss function is used to process the third text and the label text to generate a text loss value. Then, the first loss function is used to process the first and second positional information to generate a positional loss value. Next, the text loss value and the positional loss value are weighted and summed according to their respective weights to obtain a loss value of 302C. Finally, the model parameters of the target model are adjusted based on the loss value 302C to generate the trained target model.

[0147] By using the text loss between the labeled text and the third text, combined with the positional loss to train the target model 320, and by incorporating dynamically learnable weights, the training accuracy and the generalization ability of the trained model can be further improved.

[0148] Figure 7 The illustration shows a schematic diagram of training a target model using a large visual model as a teacher model according to an embodiment of the present disclosure.

[0149] like Figure 7 As shown, in this embodiment 700, the sample medical image 711 can be randomly sampled from the medical image database 710.

[0150] First, the sample medical image 711 is input into the large visual model 310, and the lesion description text T1 712 is output. Then, the lesion description text T1 712 and the sample medical image 711 are input into the large visual model 310 together, and the lesion location 713 is output. Next, a local image 714 is extracted from the sample medical image 711 based on the lesion location 713. The local image 714 is then input into the large visual model 310, and the lesion description text T2 715 is output. Then, operation S710 is performed to determine whether the lesion description text T1 712 and the lesion description text T2 715 are consistent. If they are inconsistent, they are discarded. If they are consistent, the sample medical image 711 and the lesion location 713 are used as training samples 716 for training the target model 320.

[0151] The embodiments of the present disclosure improve the reliability of the training samples and the adaptability to small-scale samples through the cross-modal image-text fusion and iterative quality control mechanism, and enhance the stability of the trained target model in heterogeneous data, noisy images and actual application scenarios. On the basis of occupying less computing resources, the precise positioning of the lesion position is realized, which provides a solution for the clinical deployment of intelligent auxiliary diagnosis and treatment equipment.

[0152] Figure 8 A flowchart of an image processing method according to an embodiment of the present disclosure is schematically shown.

[0153] As shown in Figure 8 the image processing method 800 can include operations S810-S820.

[0154] In operation S810, the trained target model is used to detect the to-be-processed medical image to generate position information of a target abnormal object in the to-be-processed medical image.

[0155] In operation S820, the to-be-processed medical image is labeled based on the position information to generate a target image.

[0156] According to an embodiment of the present disclosure, the trained target model is trained by using the model training method of any of the preceding embodiments. Exemplarily, the visual large model is used as the teacher model, the target model is used as the student model, and the target model is trained based on a loss value between first position information output by the visual large model by detecting a sample medical image and second position information output by the target model by detecting the sample medical image.

[0157] The to-be-processed medical image has the same definition range as the sample medical image described in the preceding embodiments, and the target abnormal object has the same definition range as the sample abnormal object, which will not be repeated here.

[0158] In an embodiment of the present disclosure, the position information can represent vertex coordinates of a polygon capable of surrounding a region of the target abnormal object recognized by the trained target model from the to-be-processed medical image.

[0159] Finally, the to-be-processed medical image can be labeled based on the position information, for example, the region where the target abnormal object is located is labeled in the to-be-processed medical image in the form of a polygon to generate a target image.

[0160] The trained target model can accurately locate the position of the target abnormal object in the to-be-processed image, and the positioning accuracy is similar to that of the visual large model. The calculation resources consumed by deploying the trained target model are less than those consumed by deploying the visual large model. The target object whose position has been labeled can be shown to the user, further improving the visualization of auxiliary diagnosis and treatment.

[0161] Figure 9 A schematic diagram of an image processing method according to an embodiment of the present disclosure is shown schematically.

[0162] As shown in the embodiment 900, the image to be processed 901 is input into the trained target model 910, and target position information 902 is output. Then, the image to be processed 901 is labeled based on the target position information 902, and a target image 903 in which the position of the target abnormal object is labeled is obtained. Figure 9

[0163] A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown schematically. Figure 10 As shown in the embodiment 900, the image to be processed 901 is input into the trained target model 910, and target position information 902 is output. Then, the image to be processed 901 is labeled based on the target position information 902, and a target image 903 in which the position of the target abnormal object is labeled is obtained.

[0164] Figure 10 The model training apparatus 1000 can include a first detection module 1010, a second detection module 1020, and a training module 1030.

[0165] The first detection module 1010 is configured to detect the sample medical image by using a visual large model, and generate first position information for representing a sample abnormal object in the sample medical image.

[0166] The second detection module 1020 is configured to detect the sample medical image by using a target model, and generate second position information for representing the sample abnormal object.

[0167] The training module 1030 is configured to train the target model based on a target loss function, take the visual large model as a teacher model, and take the target model as a student model, and train the target model by using the first position information and the second position information, to obtain a trained target model.

[0168] According to an embodiment of the present disclosure, the training module 1030 can include a first extraction sub-module, a second extraction sub-module, and a first training sub-module.

[0169] The first extraction sub-module is configured to extract a first local image from the sample medical image based on the first position information.

[0170] The second extraction sub-module is configured to extract a second local image from the sample medical image based on the second position information.

[0171] The first training sub-module is configured to train the target model based on the target loss function, take the visual large model as the teacher model, and take the target model as the student model, and train the target model by using the first local image and the second local image, to obtain the trained target model.

[0172] According to an embodiment of the present disclosure, the target loss function includes a first loss function; and the first training sub-module can include a first processing unit and a first training unit. ​

[0173] The first processing unit is configured to process the first local image and the second local image by using the first loss function to generate a first loss value.

[0174] The first training unit is configured to adjust the model parameters of the target model based on the first loss value to obtain a trained target model.

[0175] According to an embodiment of the present disclosure, the target loss function comprises a second loss function. The training module 1030 can comprise a first identification sub-module, a second identification sub-module, a first processing sub-module, and a first training sub-module.

[0176] The first identification sub-module is configured to identify the first local image by using the visual large model to generate a first text; the first text is used to describe the abnormal features in the first local image.

[0177] The second identification sub-module is configured to identify the second local image by using the visual large model to generate a second text; the second text is used to describe the abnormal features in the second local image.

[0178] The first processing sub-module is configured to process the first text and the second text by using the second loss function to generate a second loss value.

[0179] The first training sub-module is configured to adjust the model parameters of the target model based on the second loss value to obtain a trained target model.

[0180] According to an embodiment of the present disclosure, the target loss function further comprises a first loss function. The first training sub-module comprises a second training unit, a first detection unit, a first extraction unit, a second processing unit, and a third training unit.

[0181] The second training unit is configured to adjust the model parameters of the target model based on the second loss value to obtain an intermediate model.

[0182] The first detection unit is configured to detect the sample medical image by using the intermediate model to generate third position information used to represent the sample abnormal object in the sample medical image.

[0183] The first extraction unit is configured to extract a third local image from the sample medical image based on the third position information.

[0184] The second processing unit is configured to process the first local image and the third local image by using the first loss function to generate a third loss value.

[0185] The third training unit is configured to adjust the model parameters of the intermediate model based on the third loss value to obtain a trained target model.

[0186] According to an embodiment of the present disclosure, the training module 1030 can include a third identification sub-module, a fourth identification sub-module, and a second training sub-module.

[0187] The third identification sub-module is configured to identify the sample medical image based on the second position information by using the visual large model, to generate a third text; the third text is used to describe an abnormal feature of a region associated with the second position information in the sample medical image.

[0188] The fourth identification sub-module is configured to identify the sample medical image based on the first position information by using the visual large model, to generate a label text used to describe the sample abnormal object.

[0189] The second training sub-module is configured to take the visual large model as a teacher model, take a target model as a student model, and train the target model based on a target loss function by using the third text and the label text, to obtain a trained target model.

[0190] According to an embodiment of the present disclosure, the target loss function includes a second loss function. The second training sub-module includes a third processing unit and a third training unit.

[0191] The third processing unit is configured to process the third text and the label text by using the second loss function, to generate a text loss value.

[0192] The third training unit is configured to adjust a model parameter of the target model based on the text loss value, to obtain the trained target model.

[0193] According to an embodiment of the present disclosure, the target loss function further includes a first loss function; and the second training sub-module further includes a fourth processing unit and a fourth training unit.

[0194] The fourth processing unit is configured to process the first position information and the second position information by using the first loss function, to generate a position loss value.

[0195] The fourth training unit is configured to adjust the model parameter of the target model based on the text loss value and the position loss value, to obtain the trained target model.

[0196] According to an embodiment of the present disclosure, the training device 1000 can further include a first identification module and a first generation module.

[0197] The first identification module is configured to identify the sample medical image by using the visual large model, to generate a sample text used to describe the abnormal feature.

[0198] The first generation module is configured to detect the sample medical image based on the sample text by using the visual large model, to generate the first position information.

[0199] According to an embodiment of the present disclosure, the training apparatus 1000 can further include a first extraction module, a second identification module, and a first determination module.

[0200] The first extraction module is configured to extract a first local image from the sample medical image based on first position information.

[0201] The second identification module is configured to identify the first local image by using a visual large model to generate a description text for describing an abnormal feature in the local image.

[0202] The first determination module is configured to determine that the first position information is used for training the target model in response to determining that the sample text matches the description text.

[0203] According to an embodiment of the present disclosure, the training apparatus 1000 can further include a first transformation module, a fourth detection module, a second transformation module, and a second determination module.

[0204] The first transformation module is configured to perform transformation processing on the sample medical image along a first direction to generate a to-be-processed image.

[0205] The fourth detection module is configured to perform detection on the to-be-processed image by using the visual large model to generate an initial position of a sample abnormal object in the to-be-processed image.

[0206] The second transformation module is configured to perform transformation processing on the initial position along a second direction to generate a target position.

[0207] The second determination module is configured to determine that the first position information is used for training the target model in response to determining that a difference degree between the target position and the first position information is less than a predetermined threshold.

[0208] According to an embodiment of the present disclosure, the training apparatus 1000 can further include a second extraction module, a screening module, and a third determination module.

[0209] The second extraction module is configured to extract a plurality of candidate local images from the sample medical image based on a plurality of candidate positions, respectively.

[0210] The screening module is configured to determine a target local image from the plurality of candidate local images by screening the plurality of candidate local images by using the visual large model.

[0211] The third determination module is configured to determine that the first position information is used for training the target model in response to determining that the target local image and the first local image match.

[0212] According to an embodiment of the present disclosure, the training apparatus 1000 can further include a third extraction module, a generation module, and a fourth determination module.

[0213] The third extraction module is configured to extract a first local image from the sample medical image based on the first position information.

[0214] The first generation module is configured to generate a sample local image based on the sample text and the sample medical image by using the multi-modal large model.

[0215] The fourth determination module is configured to determine that the first position information is used for training the target model in response to determining that the sample local image matches the first local image.

[0216] According to an embodiment of the present disclosure, the apparatus further includes a second generation module configured to detect the sample local image by using a visual large model, and generate label position information used for training the target model.

[0217] Figure 11 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is schematically shown.

[0218] As shown in Figure 11 The image processing apparatus 1100 can include a third detection module 1110 and a marking module 1120.

[0219] The third detection module 1110 is configured to detect a to-be-processed medical image by using the trained target model, and generate position information of a target abnormal object in the to-be-processed medical image; wherein the trained target model is obtained by using any one of the training methods described above.

[0220] The marking module 1120 is configured to mark the to-be-processed medical image based on the position information, and generate a target image.

[0221] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0222] According to an embodiment of the present disclosure, an electronic device includes at least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0223] According to an embodiment of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the method described above.

[0224] According to an embodiment of the present disclosure, a computer program product includes a computer program, and the computer program is executed by a processor to implement the method described above.

[0225] Figure 12A block diagram of an electronic device 1200 suitable for implementing a model training method or an image processing method according to embodiments of the present disclosure is shown schematically. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit the implementations of the present disclosure described and / or claimed in this document.

[0226] As shown in Figure 12 The device 1200 includes a computing unit 1201 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0227] Various components in the device 1200 are connected to the I / O interface 1205, including an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; the storage unit 1208, such as a magnetic disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0228] The computing unit 1201 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs various methods and processes described above, such as a model training method or an image processing method. For example, in some embodiments, the model training method or the image processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded onto the RAM 1203 and executed by the computing unit 1201, one or more steps of the model training method or the image processing method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 can be configured to perform the model training method or the image processing method by any other suitable means, such as by means of firmware.

[0229] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0230] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0231] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0232] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0233] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0234] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0235] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the flow. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0236] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model training method, comprising: detecting a sample medical image using a visual large model to generate first position information for representing a sample abnormal object in the sample medical image; detecting the sample medical image using a target model to generate second position information for representing the sample abnormal object; and training the target model based on a target loss function, taking the visual large model as a teacher model and the target model as a student model, using the first position information and the second position information, to obtain a trained target model. The training of the target model based on the target loss function, taking the visual large model as the teacher model and the target model as the student model, using the first position information and the second position information, to obtain the trained target model, comprises:

2. The method of claim 1, wherein, extracting a first local image from the sample medical image based on the first position information; extracting a second local image from the sample medical image based on the second position information; and training the target model based on the target loss function, taking the visual large model as the teacher model and the target model as the student model, using the first local image and the second local image, to obtain the trained target model. The target loss function comprises a first loss function; 3. The method of claim 2, wherein, The training of the target model based on the target loss function, taking the visual large model as the teacher model and the target model as the student model, using the first local image and the second local image, to obtain the trained target model, comprises: processing the first local image and the second local image using the first loss function to generate a first loss value; and adjusting model parameters of the target model based on the first loss value to obtain the trained target model. The target loss function comprises a second loss function; 4. The method of claim 2, wherein, The training of the target model based on the target loss function, taking the visual large model as the teacher model and the target model as the student model, using the first local image and the second local image, to obtain the trained target model, comprises: identifying the first local image using the visual large model to generate a first text; the first text is used to describe abnormal features in the first local image; identifying the second local image using the visual large model to generate a second text; the second text is used to describe abnormal features in the second local image; processing the first text and the second text using the second loss function to generate a second loss value; and adjusting model parameters of the target model based on the second loss value to obtain the trained target model. The target loss function further comprises a first loss function; 5. The method of claim 4, wherein, The adjusting of the model parameters of the target model based on the second loss value to obtain the trained target model comprises: adjusting model parameters of the target model based on the second loss value to obtain an intermediate model; ​ detecting the sample medical image by using the intermediate model to generate third position information for representing a sample abnormal object in the sample medical image; extracting a third local image from the sample medical image based on the third position information; processing the first local image and the third local image by using the first loss function to generate a third loss value; and adjusting model parameters of the intermediate model based on the third loss value to obtain the trained target model.

6. The method of claim 1, wherein, The target loss function is used to train the target model by taking the visual large model as a teacher model and the target model as a student model, and using the first position information and the second position information, to obtain a trained target model, including: using the visual large model to identify the sample medical image based on the second position information to generate a third text; the third text is used to describe abnormal features of a region associated with the second position information in the sample medical image; using the visual large model to identify the sample medical image based on the first position information to generate a label text for describing a sample abnormal object; and taking the visual large model as a teacher model and the target model as a student model, and using the third text and the label text to train the target model based on the target loss function to obtain the trained target model.

7. The method of claim 6, wherein, The target loss function includes a second loss function; taking the visual large model as a teacher model and the target model as a student model, and using the third text and the label text to train the target model based on the target loss function to obtain the trained target model, including: processing the third text and the label text by using the second loss function to generate a text loss value; and adjusting model parameters of the target model based on the text loss value to obtain the trained target model.

8. The method of claim 7, wherein the target loss function further includes a first loss function; and the method further includes: processing the first position information and the second position information by using the first loss function to generate a position loss value; and adjusting model parameters of the target model based on the text loss value and the position loss value to obtain the trained target model. The method further includes:

9. The method of claim 1, wherein, using the visual large model to identify the sample medical image to generate a sample text for describing abnormal features; and using the visual large model to detect the sample medical image based on the sample text to generate the first position information. The method further includes:

10. The method of claim 9, wherein, extracting a first local image from the sample medical image based on the first position information; ​ recognize the first local image by using the visual large model to generate a description text for describing an abnormal feature in the local image; and in response to determining that the sample text matches the description text, determine that the first position information is used to train the target model.

11. The method of claim 9, wherein, The detecting the sample medical image by using the visual large model to generate the first position information for representing a sample abnormal object in the sample medical image further includes: transforming the sample medical image along a first direction to generate a to-be-processed image; detecting the to-be-processed image by using the visual large model to generate an initial position of the sample abnormal object in the to-be-processed image; transforming the initial position along a second direction to generate a target position, the second direction being opposite to the first direction; and in response to determining that a difference between the target position and the first position information is less than a predetermined threshold, determining that the first position information is used to train the target model.

12. The method of claim 9, wherein, The detecting the sample medical image by using the visual large model to generate the first position information for representing a sample abnormal object in the sample medical image further includes: extracting a plurality of candidate local images from the sample medical image based on a plurality of candidate positions respectively; determining a target local image from the plurality of candidate local images by screening the plurality of candidate local images by using the visual large model; and in response to determining that the target local image matches the first local image, determining that the first position information is used to train the target model.

13. The method of claim 9, wherein, The detecting the sample medical image by using the visual large model to generate the first position information for representing a sample abnormal object in the sample medical image further includes: extracting a first local image from the sample medical image based on the first position information; generating a sample local image based on the sample text and the sample medical image by using a multi-modal large model; and in response to determining that the sample local image matches the first local image, determining that the first position information is used to train the target model.

14. The method of claim 13, further comprising: detecting the sample local image by using the visual large model to generate label position information for training the target model.

15. An image processing method, comprising: detecting a to-be-processed medical image by using a trained target model to generate position information of a target abnormal object in the to-be-processed medical image; wherein the trained target model is obtained by using the training method of any one of claims 1-14; and labeling the to-be-processed medical image based on the position information to generate a target image.

16. A model training apparatus, comprising: a first detection module configured to detect a sample medical image by using a visual large model to generate first position information for representing a sample abnormal object in the sample medical image; a second detection module configured to detect the sample medical image by using a target model to generate second position information for representing the sample abnormal object; and a determination module configured to determine whether the first position information is used to train the target model based on a comparison between the first position information and the second position information. The training module is configured to train the target model based on a target loss function, with the visual large model as a teacher model and the target model as a student model, and by using the first position information and the second position information, to obtain a trained target model.

17. An image processing apparatus, comprising: a third detection module configured to detect a target abnormal object in a to-be-processed medical image by using the trained target model to generate position information of the target abnormal object in the to-be-processed medical image; wherein the trained target model is obtained by using the training method in any one of claims 1-14; and a marking module configured to mark the to-be-processed medical image based on the position information to generate a target image.

18. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method in any one of claims 1-15.

19. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method in any one of claims 1-15.

20. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-15.

Images