Method and device for generating endoscope image quality information

Abstract

The present invention provides a method and device for generating endoscopic image quality information, the method comprising: using a knowledge embedding module to perform feature conversion on multiple target category text embeddings respectively to obtain multiple first category perception prompt vectors; using an image encoder to encode a target endoscopic image to obtain a target image feature vector; using a shared network to perform feature extraction on the target image feature vector to obtain a target image prompt vector; fusing each first category perception prompt vector with a target image prompt vector and an automatic learning prompt vector to obtain multiple target fused prompt vectors; using a first text encoder to encode multiple target fused prompt vectors respectively to obtain multiple target category prompt vectors; calculating the similarity between multiple target category prompt vectors and the target image feature vector respectively to obtain multiple target image category similarities; generating an image quality evaluation result according to the multiple target image category similarities.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a method and device for generating endoscopic image quality information. Background Art

[0002] With the advancement of surgical techniques, minimally invasive surgery has become more and more popular. Endoscopes, as commonly used medical devices in minimally invasive surgery, play an important role in minimally invasive surgery. During minimally invasive surgery, the images taken by the endoscope can provide doctors with a clear internal view, thereby helping them to make accurate diagnosis and treatment. Therefore, high-quality endoscopic images are crucial to the success of minimally invasive surgery. How to evaluate the quality of endoscopic images has become a crucial link.

[0003] However, existing image or video quality assessment methods mainly focus on technical indicators such as peak signal-to-noise ratio, edge structure and blur. Although these methods can evaluate some basic image quality, due to the particularity and professional requirements of endoscopic images, these methods are not completely suitable for endoscopic image quality assessment. Therefore, a method specifically for endoscopic image quality assessment is urgently needed. Summary of the invention

[0004] In view of this, the present invention provides a method and device for generating endoscopic image quality information.

[0005] One aspect of the present invention provides a method for generating endoscopic image quality information, comprising: using a knowledge embedding module to perform feature conversion on multiple target category text embeddings respectively to obtain multiple first category perception prompt vectors, wherein the multiple target category text embeddings are text features corresponding to multiple preset text prompt words one by one, and the multiple preset text prompt words are prompt words related to the endoscopic image quality and have opposite meanings; using an image encoder to encode the target endoscopic image to obtain a target image feature vector; using a shared network to perform feature extraction on the target image feature vector to obtain a target image prompt vector; each of the multiple first category perception prompt vectors is converted into a first category perception prompt vector; Fusing with the target image prompt vector and the automatic learning prompt vector, a plurality of target fused prompt vectors are obtained, wherein the shared network, the automatic learning prompt vector and the knowledge embedding module are all trained based on a plurality of endoscopic image samples and a plurality of groups of text prompt words reflecting image quality and having opposite meanings corresponding to the plurality of endoscopic image samples; encoding the plurality of target fused prompt vectors respectively by using a first text encoder to obtain a plurality of target category prompt vectors; calculating the similarities between the plurality of target category prompt vectors and the target image feature vectors respectively to obtain a plurality of target image category similarities; generating an image quality evaluation result based on the plurality of target image category similarities.

[0006] Another aspect of the present invention provides an endoscope image quality information generating device, comprising: a feature conversion unit, a first encoding unit, a feature extraction unit, a prompt vector fusion unit, a second encoding unit, a calculation unit and a generation unit.

[0007] A feature conversion unit is used to use a knowledge embedding module to perform feature conversion on multiple target category text embeddings respectively to obtain multiple first category perception prompt vectors, wherein the above-mentioned multiple target category text embeddings are text features corresponding one-to-one to multiple preset text prompt words, and the above-mentioned multiple preset text prompt words are prompt words related to the endoscopic image quality and have opposite meanings.

[0008] The first encoding unit is used to encode the target endoscopic image using an image encoder to obtain a target image feature vector.

[0009] The feature extraction unit is used to extract features from the target image feature vector using a shared network to obtain a target image prompt vector.

[0010] A prompt vector fusion unit is used to fuse each of the above-mentioned multiple first-category perception prompt vectors with the above-mentioned target image prompt vector and the automatic learning prompt vector to obtain multiple target fused prompt vectors, wherein the above-mentioned shared network, the above-mentioned automatic learning prompt vector and the above-mentioned knowledge embedding module are all trained based on multiple endoscopic image samples and multiple groups of text prompt words reflecting image quality and having opposite meanings corresponding to the above-mentioned multiple endoscopic image samples.

[0011] The second encoding unit is used to use the first text encoder to encode the multiple target fusion prompt vectors respectively to obtain multiple target category prompt vectors.

[0012] The calculation unit is used to calculate the similarities between the above-mentioned multiple target category prompt vectors and the above-mentioned target image feature vectors respectively, so as to obtain multiple target image category similarities.

[0013] The generating unit is used to generate an image quality evaluation result according to the similarities of the above-mentioned multiple target image categories.

[0014] According to the embodiment of the present invention, since the shared network, the automatic learning prompt vector and the knowledge embedding module are all trained based on multiple endoscopic image samples and multiple groups of text prompt words corresponding to the multiple endoscopic image samples that reflect image quality and have opposite meanings, the knowledge embedding module is used to perform feature conversion on multiple target category text embeddings respectively to obtain multiple first category perception prompt vectors, and a category perception prompt vector that accurately represents the quality of the endoscopic image and is closer to the subjective feeling can be obtained. Then, the target endoscopic image is encoded by an image encoder to obtain a target image feature vector, and the target image feature vector is extracted by the shared network to obtain a target image prompt vector. Each of the first category perception prompt vectors in the multiple first category perception prompt vectors is fused with the target image prompt vector and the automatic learning prompt vector to obtain multiple target fused prompt vectors. The multiple target fused prompt vectors are respectively encoded by the first text encoder to obtain multiple target category prompt vectors, and a conditional prompt vector that fuses the subjective feeling and the actual target endoscopic image quality features can be dynamically generated. Then, the similarities between multiple target category prompt vectors and the target image feature vectors are calculated to obtain multiple target image category similarities, and the similarities of image indicators that accurately reflect the target endoscopic image quality and subjective needs are obtained. Then, when the image quality evaluation results are generated according to the multiple target image category similarities, the evaluation results that accurately reflect whether the endoscopic image quality meets the subjective needs can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0016] Figure 1 An application scenario diagram of the method for generating endoscope image quality information according to an embodiment of the present invention is shown;

[0017] Figure 2 A flow chart of a method for generating endoscopic image quality information according to an embodiment of the present invention is shown;

[0018] Figure 3 A schematic diagram showing a neural network model used in a method for generating endoscopic image quality information according to an embodiment of the present invention is shown;

[0019] Figure 4 A flow chart showing a model training method of a neural network model used in a method for generating endoscopic image quality information according to an embodiment of the present invention is shown;

[0020] Figure 5 A schematic diagram showing an image quality evaluation result of an endoscope image quality information generating method according to an embodiment of the present invention;

[0021] Figure 6 A block diagram showing an endoscope image quality information generating device according to an embodiment of the present invention; and

[0022] Figure 7 A block diagram of an electronic device suitable for implementing the above-described method for generating endoscopic image quality information according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage and other aspects of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain user personal information security and network security. In the embodiments of the present invention, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0028] Due to the particularity and professional requirements of endoscopic images, the methods used in related technologies are not fully applicable to the quality assessment of endoscopic images. Therefore, there is an urgent need for a method specifically for evaluating the quality of endoscopic images. This method should not only consider the basic technical indicators of the image, but also combine the unique requirements of endoscopic images. For example, when evaluating endoscopic images, it is necessary to focus on factors such as the image's detail presentation, color reproduction, and contrast. Because these factors directly affect the doctor's identification and judgment of lesions during minimally invasive surgery, which has a significant impact on the success of minimally invasive surgery.

[0029] With the widespread application of endoscopic technology in the medical field, it is becoming increasingly urgent to establish an evaluation method for endoscopic image quality. This can not only improve the diagnostic accuracy of endoscopic images, but also greatly improve the overall quality and safety of endoscopic surgery. Through such an evaluation method, doctors can better perform standardized surgical operations, thereby providing patients with better medical services.

[0030] In view of this, an embodiment of the present invention provides a method for generating endoscopic image quality information, including: using a knowledge embedding module to perform feature conversion on multiple target category text embeddings respectively, to obtain multiple first category perception prompt vectors; using an image encoder to encode the target endoscopic image to obtain a target image feature vector; using a shared network to perform feature extraction on the target image feature vector to obtain a target image prompt vector; fusing each of the multiple first category perception prompt vectors with the target image prompt vector and the automatic learning prompt vector to obtain multiple target fused prompt vectors; using a first text encoder to encode the multiple target fused prompt vectors respectively, to obtain multiple target category prompt vectors; calculating the similarity between the multiple target category prompt vectors and the target image feature vector respectively, to obtain multiple target image category similarities; generating an image quality evaluation result based on the multiple target image category similarities.

[0031] Figure 1 A diagram showing an application scenario of the method for generating endoscopic image quality information according to an embodiment of the present invention is shown.

[0032] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0033] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0035] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0036] It should be noted that the endoscopic image quality information generation method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the endoscopic image quality information generation device provided in the embodiment of the present invention can generally be set in the server 105. The endoscopic image quality information generation method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the endoscopic image quality information generation device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] Figure 2 A flow chart of a method for generating endoscopic image quality information according to an embodiment of the present invention is shown.

[0039] like Figure 2 As shown, the method for generating endoscope image quality information of this embodiment may include operations S201 to S207.

[0040] In operation S201, a knowledge embedding module is used to perform feature conversion on multiple target category text embeddings respectively to obtain multiple first category perception prompt vectors, wherein the multiple target category text embeddings are text features corresponding one-to-one to multiple preset text prompt words, and the multiple preset text prompt words are prompt words related to the endoscopic image quality and have opposite meanings.

[0041] According to an embodiment of the present invention, the plurality of preset text prompt words may be prompt words related to quality, brightness, noise, chroma, clarity or contrast and having opposite meanings. These prompt words provide a baseline for the algorithm to evaluate image quality.

[0042] For example, multiple preset text prompt words can be: prompt words related to quality: "good picture" and "bad picture", prompt words related to brightness: "bright picture" and "dark picture", prompt words related to noise: "clean picture" and "noisy picture", prompt words related to chroma: "bright picture" and "dim picture", prompt words related to clarity: "clear photo" and "blurred photo" or prompt words related to contrast: "high contrast photo" and "low contrast photo".

[0043] According to the embodiment of the present invention, when it is necessary to evaluate other evaluation indicators of the endoscopic image, relevant prompt words with opposite meanings can be customized.

[0044] In operation S202, the target endoscopic image is encoded using an image encoder to obtain a target image feature vector.

[0045] In operation S203, a shared network is used to extract features from a target image feature vector to obtain a target image prompt vector.

[0046] According to an embodiment of the present invention, in the related art, a fixed artificial template, such as "a picture of [preset type]", is usually used for text embedding. However, the ability of fixed artificial prompt words to describe downstream tasks is very poor, so it is necessary to tune the text prompts to enhance the text embedding by automatically learning text prompts. Therefore, unlike using fixed text prompts of artificial templates to guide the matching between images and text prompt words, the present invention introduces a lightweight shared network that can dynamically generate a conditional prompt vector for the input target endoscopic image, and can generate more accurate prompts for the specific content of each input target endoscopic image, thereby improving the generalization ability of the model on unknown categories or new data sets. The structural design of the shared network can improve the model performance, so the shared network can use a multi-layer perceptron with a simple structure, easy implementation and powerful nonlinear fitting capabilities.

[0047] According to an embodiment of the present invention, a target image feature vector is extracted using a shared network to obtain a target image prompt vector, which can generate a unique conditional prompt for each input target endoscopic image, so that the model is no longer focused on a specific category in the training set, but is more focused on the characteristics of the input target endoscopic image itself. Therefore, it can help to reduce overfitting and improve the generalization performance of the model on a wider range of categories.

[0048] In operation S204, each of the multiple first-category perception prompt vectors is fused with the target image prompt vector and the automatic learning prompt vector to obtain multiple target fused prompt vectors, wherein the shared network, the automatic learning prompt vector and the knowledge embedding module are all trained based on multiple endoscopic image samples and multiple groups of text prompt words reflecting image quality and having opposite meanings corresponding to the multiple endoscopic image samples.

[0049] According to an embodiment of the present invention, since the shared network, the automatic learning prompt vector and the knowledge embedding module are all trained based on multiple endoscopic image samples and multiple groups of text prompt words reflecting image quality and having opposite meanings corresponding to the multiple endoscopic image samples, the shared network, the automatic learning prompt vector and the knowledge embedding module can obtain prompt vectors that accurately reflect the endoscopic image quality category.

[0050] In operation S205, a first text encoder is used to encode the plurality of target fusion prompt vectors respectively to obtain a plurality of target category prompt vectors.

[0051] In operation S206, similarities between the plurality of target category prompt vectors and the target image feature vectors are calculated to obtain a plurality of target image category similarities.

[0052] For example, multiple target image category similarities may be obtained by calculating the cosine similarities between multiple target category prompt vectors and target image feature vectors respectively.

[0053] According to an embodiment of the present invention, cosine similarity, as an indicator for measuring the directional similarity of two vectors, can be used to quantify the degree of match between the image content, i.e., the target image feature vector, and the text prompt, i.e., the target category prompt vector. The higher the similarity value, the more the image meets the quality standard described by the text prompt. Otherwise, it indicates that the image quality is relatively low.

[0054] For example, the contrastive learning module in the multimodal large model can be used to compare multiple target category hint vectors with the target image feature vectors, capture the complex semantic associations between images and texts, and obtain the similarities of multiple target image categories. Among them, the multimodal large model performs efficient contrastive learning on a large amount of image-text pair data, which can capture the complex semantic associations between images and texts.

[0055] In operation S207, an image quality evaluation result is generated according to the similarities of the multiple target image categories.

[0056] According to the embodiment of the present invention, since the shared network, the automatic learning prompt vector and the knowledge embedding module are all trained based on multiple endoscopic image samples and multiple groups of text prompt words corresponding to the multiple endoscopic image samples that reflect image quality and have opposite meanings, the knowledge embedding module is used to perform feature conversion on multiple target category text embeddings respectively to obtain multiple first category perception prompt vectors, and a category perception prompt vector that accurately represents the quality of the endoscopic image and is closer to the subjective feeling can be obtained. Then, the target endoscopic image is encoded by an image encoder to obtain a target image feature vector, and the target image feature vector is extracted by the shared network to obtain a target image prompt vector. Each of the first category perception prompt vectors in the multiple first category perception prompt vectors is fused with the target image prompt vector and the automatic learning prompt vector to obtain multiple target fused prompt vectors. The multiple target fused prompt vectors are respectively encoded by the first text encoder to obtain multiple target category prompt vectors, and a conditional prompt vector that fuses the subjective feeling and the actual target endoscopic image quality features can be dynamically generated. Then, the similarities between multiple target category prompt vectors and the target image feature vectors are calculated to obtain multiple target image category similarities, and the similarities of image indicators that accurately reflect the target endoscopic image quality and subjective needs are obtained. Then, when the image quality evaluation results are generated according to the multiple target image category similarities, the evaluation results that accurately reflect whether the endoscopic image quality meets the subjective needs can be obtained.

[0057] The method for generating endoscopic image quality information provided by an embodiment of the present invention quantifies the quality of the image by measuring the semantic match between the target endoscopic image content and a given preset text prompt word. It not only breaks through the limitation of traditional image quality assessment technology that relies on pixel-level or structure-level features, but also introduces semantic understanding at the human perception level, making image quality evaluation closer to subjective feelings.

[0058] Figure 3 A schematic diagram of a neural network model used in a method for generating endoscopic image quality information according to an embodiment of the present invention is shown.

[0059] like Figure 3 As shown, the neural network model includes: an image encoder 310, a shared network 320, a knowledge embedding module 330, a first text encoder 340 and a second text encoder 350.

[0060] Figure 3 The image encoder 310 is an image encoder in a pre-trained multimodal model, and the first text encoder 340 and the second text encoder 350 are both text encoders in a pre-trained multimodal model.

[0061] According to an embodiment of the present invention, since the image encoder 310 is an image encoder in a pre-trained multimodal model, the first text encoder 340 and the second text encoder 350 are both text encoders in a pre-trained multimodal model, after the target fusion prompt vector is input into the text encoder of the multimodal large model, and the target endoscopic image to be evaluated is input into the image encoder of the multimodal large model, the model can extract high-dimensional feature vectors of the image and text through a complex neural network structure, and these feature vectors contain rich semantic information.

[0062] like Figure 3 As shown, for Figure 2 The operation S202 shown, using the image encoder 310 to encode the target endoscopic image to obtain a target image feature vector, may include the following operations: performing image segmentation on the target endoscopic image to obtain multiple segmented images; using multiple image encoders 310 to encode the multiple segmented images in parallel to obtain multiple segmented image features; performing feature fusion on the multiple segmented image features to obtain a target image feature vector.

[0063] For example, Figure 2 The neural network model used in the method for generating endoscope image quality information in the embodiment further includes a fusion module 360. The fusion module 360 ​​can be used to perform channel splicing on multiple segmented image features to obtain a target image feature vector X. The fusion module 360 ​​can also be used to perform self-attention operation on multiple segmented image features to obtain a target image feature vector X.

[0064] According to an embodiment of the present invention, when processing a target endoscopic image using the image encoder 310, the image processing can be accelerated by using parallel computing technology. By dividing the target endoscopic image into a plurality of small blocks, and then processing the small blocks in parallel, and finally splicing the processing results together, the efficiency of the target endoscopic image processing can be greatly improved.

[0065] exist Figure 3 In FIG. 3 , the shared network 320 sequentially includes a first linear layer, a nonlinear activation function layer, and a second linear layer.

[0066] According to an embodiment of the present invention, the specific structure of the shared network 320 includes a two-layer bottleneck structure. First, the shared network 320 performs a preliminary transformation on the input target image feature vector through a first linear layer, followed by a ReLU activation function to increase nonlinearity, and finally maps the feature to the target dimension through another second linear layer to obtain a target image prompt vector. The design of the shared network 320 achieves a significant reduction in the dimension of the input target image feature vector, which helps to reduce the amount of calculation, prevent overfitting, and can prompt the model to learn a more compact and robust feature representation.

[0067] like Figure 3 As shown, before the knowledge embedding module 330 is used to perform feature conversion on multiple target category text embeddings to obtain multiple first category perception prompt vectors, Figure 2 The method for generating endoscopic image quality information also includes: using a second text encoder 350 to encode multiple preset text prompt words respectively to obtain multiple target category text embeddings.

[0068] According to an embodiment of the present invention, the traditional multimodal model pre-trained text encoder generates a specific category text embedding by combining text prompts with general category tags, but these text encoders have poor generalization performance on unseen categories. Therefore, the general text knowledge extracted by the text encoder can be used to create discriminative prior knowledge of new categories, thereby enhancing the generalization of the automatically learned prompts.

[0069] According to an embodiment of the present invention, the knowledge embedding module 330 has an advantage over unseen categories because it can generate prompts for specific categories, and explicitly combine the perceptual prompts of visible categories to enhance the discrimination ability of unseen categories. Therefore, the advantages of the general text knowledge of visible and unseen categories can be borrowed, and the knowledge embedding module 330 can be used to convert the general category level text embedding, i.e., multiple target category text embeddings, into the first category perceptual prompt vector, which is then combined with the automatically learned prompt vector and the target image prompt vector output by the shared network 320 to obtain a prompt vector that integrates subjective feelings and actual target endoscopic image quality features. For example, a target category prompt vector T1 and a target category prompt vector T2 are obtained.

[0070] like Figure 3 As shown, after obtaining the target image feature vector X and the target category prompt vector T1 and the target category prompt vector T2, Figure 2 The neural network model used in the endoscope image quality information generation method can calculate the cosine similarity X between the target image feature vector X and the target category prompt vector T1 . T1, and calculate the cosine similarity X between the target image feature vector X and the target category prompt vector T2 . T2.

[0071] like Figure 3 As shown, Figure 2 The neural network model used in the method for generating endoscopic image quality information in the embodiment further includes: a normalization module 370. The normalization module 370 is used to implement the normalization function of the normalized exponential function softmax function. For example, the normalization module 370 can normalize the cosine similarity X.T1 and the cosine similarity X.T2 based on the normalized exponential function softmax function.

[0072] For example Figure 2 Operation S207 shown, generating an image quality evaluation result based on multiple target image category similarities, may include the following operations: normalizing the multiple target image category similarities using a normalized exponential function to obtain multiple normalized category similarities corresponding to multiple preset text prompt words; generating an image quality evaluation result based on the multiple normalized category similarities.

[0073] According to an embodiment of the present invention, in order to obtain a comprehensive and comparable image quality score, the model will further process all calculated similarity values, that is, perform a normalization step. The softmax function can be used to convert the original similarities of multiple target image categories into a probability distribution form, so that the quality scores of different images can be directly compared. After such processing, the model can output a quality score between 0 and 1 for each target endoscopic image. The higher the score, the better the image quality. This result provides a new method for image quality assessment that is closer to people's subjective feelings.

[0074] like Figure 3 As shown, the present invention can evaluate multiple evaluation indicators of endoscopic images based on the same multimodal model, and can be independent of the pixel-level or structure-level features of the image, but can quantitatively output the subjective feelings of people.

[0075] like Figure 3 The neural network model shown uses the vector that combines the category-aware prompts and the automatically learned prompt vectors and the target image prompt vector output by the shared network as the new input to the text encoder, which can obtain more accurate results and has better generalization for unseen evaluation prompt words.

[0076] like Figure 3 The neural network model shown utilizes parallel computing technology to accelerate image processing when using an image encoder to process images, and divides the image into windows as input of the image encoder, which can improve the operating efficiency of the algorithm.

[0077] Figure 3 The model training method of the neural network model in Figure 4 shown.

[0078] Figure 4 A flowchart of a model training method for a neural network model used in a method for generating endoscopic image quality information according to an embodiment of the present invention is shown.

[0079] According to an embodiment of the present invention, Figure 3 The shared network 320, the automatic learning hint vector and the knowledge embedding module 330 are all trained according to the following operations S401 to S407.

[0080] In operation S401, for each endoscopic image sample among the multiple endoscopic image samples, the currently trained knowledge embedding module is used to perform feature conversion on the multiple sample category text embeddings corresponding to the endoscopic image sample to obtain multiple second category perception prompt vectors, wherein the multiple sample category text embeddings are text features that correspond one-to-one to the multiple text prompt words for the endoscopic image sample.

[0081] According to an embodiment of the present invention, operation S401 and Figure 2 The implementation method of operation S201 in is similar and will not be repeated here for the sake of simplicity.

[0082] In operation S402, an image encoder is used to encode an endoscopic image sample to obtain a sample image feature vector.

[0083] According to an embodiment of the present invention, operation S402 is performed with Figure 2 The implementation method of operation S202 in is similar and will not be repeated here for the sake of simplicity.

[0084] In operation S403, feature extraction is performed on the sample image feature vector using the currently trained shared network to obtain a sample image prompt vector.

[0085] According to an embodiment of the present invention, the sample image features extracted from the image encoder are passed as input to the shared network, and the lightweight shared network can dynamically generate a unique sample image prompt vector based on the specific content of the image. This vector is not static, but changes with the changes in the image content. It contains specific information about the image content and can guide subsequent processing to understand the image more accurately. Through the processing of the shared network, each image is given an exclusive, highly customized conditional tag, which enhances the flexibility and adaptability of the model.

[0086] According to an embodiment of the present invention, operation S403 and Figure 2 The implementation method of operation S203 in is similar and will not be repeated here for the sake of simplicity.

[0087] In operation S404, each second category perceptual prompt vector in the plurality of second category perceptual prompt vectors is fused with the sample image prompt vector and the currently trained automatic learning prompt vector to obtain a plurality of sample fused prompt vectors.

[0088] According to an embodiment of the present invention, when combining the image prompt vector generated by the shared network with the category-aware prompt vector and the automatically learned prompt vector, the combination process can use vector concatenation or other forms of fusion operations to further strengthen the potential association between image features and text prompts. The combined vector not only contains the feature information of the image itself, but also incorporates the text context information obtained through learning, providing comprehensive and rich input information for the subsequent text encoder.

[0089] According to an embodiment of the present invention, operation S404 is performed with Figure 2 The implementation method of operation S204 in is similar and will not be repeated here for the sake of brevity.

[0090] In operation S405, a first text encoder is used to encode the plurality of sample fusion hint vectors respectively to obtain a plurality of sample category hint vectors.

[0091] According to an embodiment of the present invention, the combined vector, i.e., the sample fusion hint vector, is sent to the first text encoder for processing to generate a text representation closely related to the image, i.e., the sample category hint vector. The text representation is a high-dimensional vector that can capture the correspondence between the image and the text in the semantic space.

[0092] According to an embodiment of the present invention, operation S405 is Figure 2 The implementation method of operation S205 in is similar and will not be repeated here for the sake of simplicity.

[0093] In operation S406, similarities between the plurality of sample category hint vectors and the sample image feature vectors are calculated to obtain a plurality of sample image category similarities, and image quality prediction results are generated according to the plurality of sample image category similarities.

[0094] According to an embodiment of the present invention, by calculating the similarity, the closeness between the image and different categories can be evaluated. The similarity value can be used to calculate the final image quality score.

[0095] According to an embodiment of the present invention, cosine similarity values ​​can be calculated, and all calculated cosine similarity values ​​are normalized using a softmax function. The softmax function can map a set of real numbers to the interval (0,1), and the sum of all output values ​​is 1, so that each text prompt word corresponds to a probability value or score indicating its degree of matching. Based on the normalized score, it can be determined which text prompt word the image matches better. Generally, images with a high degree of matching with positive prompts such as "good" are considered to have a higher quality, while images with a high degree of matching with negative prompts such as "bad" are considered to have a lower quality. Therefore, the quality level of the image can be evaluated by comparing the scores corresponding to different text prompts.

[0096] According to an embodiment of the present invention, operation S406 is Figure 2 The implementation method of operation S206 in is similar and will not be repeated here for the sake of simplicity.

[0097] In operation S407, based on the image quality prediction results and category labels corresponding to multiple endoscopic image samples, the values ​​of the currently trained knowledge embedding module, the network parameters of the currently trained shared network, and the currently trained automatic learning prompt vector are iteratively updated, and when the image quality prediction accuracy is within a preset range, the currently trained shared network, automatic learning prompt vector, and knowledge embedding module are respectively determined as the shared network, automatic learning prompt vector, and knowledge embedding module.

[0098] According to an embodiment of the present invention, based on image quality prediction results and category labels corresponding to multiple endoscopic image samples, the currently trained knowledge embedding module, the network parameters of the currently trained shared network, and the value of the currently trained automatic learning prompt vector are iteratively updated, including: calculating loss and back propagation.

[0099] According to an embodiment of the present invention, calculation loss and back propagation are important parts of the algorithm, which directly determine the direction and speed of improving the model performance.

[0100] For example, the loss function can be calculated based on the difference between the true category label and the probability distribution predicted by the model. The loss function can reflect the degree of inconsistency between the probability distribution predicted by the model and the true distribution. When the two are exactly the same, the loss value is zero, indicating that the model prediction is completely accurate. When calculating the loss, first traverse the prediction results of each sample, compare it with the corresponding true category label, and then calculate the loss value of the sample according to the loss function. Finally, average the loss values ​​of all samples to get the total loss of the entire training batch.

[0101] After obtaining the loss value, the back propagation algorithm is used to optimize the model parameters to reduce the loss value and improve the model's predictive ability. The back propagation algorithm is an efficient method for calculating gradients. It uses the chain rule to propagate the gradient of the loss function to the model parameters layer by layer to the front end of the network. The algorithm uses the Adam optimizer to guide the update direction of the parameters. The Adam optimizer is an adaptive learning rate optimization algorithm based on first-order moment estimation and second-order moment estimation. It can automatically adjust the learning rate according to the update history of the parameters and the size of the gradient, so that the model can converge quickly during the training process and avoid falling into the local optimal solution or gradient disappearance / explosion.

[0102] In the back-propagation process, the gradient of the loss function to the model output layer parameters is first calculated, and then it is propagated forward layer by layer to calculate the gradient of each layer parameter in turn. In particular, for the shared network, automatic learning prompt vector and knowledge embedding module, their corresponding gradients need to be calculated respectively. The gradient value indicates how to adjust these parameters in the current state to reduce the loss function value. Finally, based on the calculated gradient value, the optimization algorithm is used to update the parameters of the shared network and knowledge embedding module and the value of the automatic learning prompt, in the hope of obtaining a smaller loss value and more accurate prediction results in the next round of iterations. By continuously iterating this process, the model gradually learns how to extract useful information from the input image and accurately map it to the corresponding classification.

[0103] In order to continuously improve the performance of the model, it is necessary to repeat a series of steps such as forward propagation, loss calculation, back propagation and optimization, and perform multiple iterative training. Each iteration represents a learning and adaptation process of the model to the given data. As the number of iterations increases, the model gradually learns more complex and abstract feature representations from the data, and its predictive ability is enhanced accordingly. However, the more iterations, the better, because when the model reaches an overfitting state on the training set, its performance on the test set will decrease. Therefore, in practical applications, it is necessary to set an appropriate number of iterations according to the specific situation to find the best balance between training efficiency and model performance.

[0104] After back-propagation and optimization, the accuracy of the evaluation algorithm will be improved, and it can better capture the image quality characteristics and give evaluation results that are closer to human perception. It can also maintain good evaluation performance on unseen data and have better generalization capabilities.

[0105] According to an embodiment of the present invention, the trained model can be used to evaluate the quality of the endoscopic image by inputting pairs of opposite prompt words, such as "good picture" and "bad picture", and then calculating the similarity between the endoscopic image and the opposite prompt words, and determining which prompt word the endoscopic image is more similar to through the similarity, thereby outputting an evaluation score.

[0106] According to an embodiment of the present invention, Figure 4 Before training the model in , we need to collect endoscopic image data. Then we construct text prompt words and filter endoscopic image data. Then we build training and test datasets.

[0107] According to an embodiment of the present invention, when collecting endoscopic image data, the original image data can be first collected through an endoscopic camera. In order to improve data processing efficiency and result quality, the original image data is screened and preprocessed as necessary to eliminate some data that does not meet the requirements.

[0108] According to an embodiment of the present invention, in operation S401, for each endoscopic image sample among a plurality of endoscopic image samples, a plurality of sample category text embeddings corresponding to the endoscopic image sample are respectively subjected to feature conversion using the currently trained knowledge embedding module to obtain a plurality of second category perception prompt vectors, and further includes: constructing a plurality of groups of text prompt words reflecting image quality and having opposite meanings; acquiring a plurality of endoscopic image samples; and constructing, based on the plurality of groups of text prompt words, text prompt words reflecting image quality and having opposite meanings corresponding to each endoscopic image sample.

[0109] For example, multiple groups of text prompt words that reflect image quality and have opposite meanings can be: prompt words related to quality: "good picture" and "bad picture", prompt words related to brightness: "bright picture" and "dark picture", prompt words related to noise: "clean picture" and "noisy picture", prompt words related to chroma: "bright picture" and "dim picture", prompt words related to clarity: "clear photo" and "blurred photo" and prompt words related to contrast: "high contrast photo" and "low contrast photo".

[0110] According to an embodiment of the present invention, when constructing text prompts, multiple pairs of text prompts with opposite meanings can be designed. These prompts are used to evaluate the overall quality or specific attributes (such as brightness, clarity, etc.) of the image. When the collected endoscopic image data is screened, the images can be divided into two groups according to the corresponding image descriptions, for example, divided into "good" and "bad" groups according to the image quality, and the higher quality images are divided into the "good" group. The "good" group image standards include high definition, accurate color reproduction, rich details and no obvious distortion. These images can truly reflect the actual situation of the surgical related area. The images with poor quality are divided into the "bad" group. The "bad" group contains images that are blurred, unclear, color distorted, and have serious artifacts or occlusions. These images cannot provide effective image information for various reasons.

[0111] According to an embodiment of the present invention, when dividing the test set and the training set, in order to verify the effectiveness and accuracy of the subsequent image processing and analysis algorithm, it is necessary to select a certain number of images from the two sets of data as the test set. These test set images should be representative and can fully cover images of different quality levels and feature types. Finally, the remaining large number of images are used as the training set to train and optimize the model, providing reliable technical support for subsequent practical applications.

[0112] According to an embodiment of the present invention, after constructing the training and test data sets, Figure 4 Before training the model in , you need to load the pre-trained multimodal model to obtain Figure 4The image encoder, the first text encoder, and the second text encoder to be used in the image encoder. For example, the pre-trained multimodal model can be a large multimodal model based on vision and language that has been fully pre-trained and has excellent performance.

[0113] According to an embodiment of the present invention, in the process of training the shared network, automatically learning the prompt vector and the knowledge embedding module, the network parameters of the image encoder, the first text encoder and the second text encoder remain unchanged.

[0114] According to an embodiment of the present invention, in order to utilize the general visual and language representation capabilities learned by the pre-trained multimodal model on a large amount of data, while avoiding the introduction of unnecessary noise in subsequent specific task training or causing the model to forget the learned knowledge, it is necessary to fix its parameters when loading the model to ensure that all weights and parameters remain unchanged from the state when the training is completed. By fixing the parameters, it can be ensured that the pre-trained multimodal model acts as a stable feature extractor in subsequent processing, providing high-quality visual and language representation for image evaluation.

[0115] According to an embodiment of the present invention, the role of the automatically learned cue vector is to capture and enhance the key information in the text cue words so as to better guide the model to understand and respond to a given visual task. A random initialization method or an initialization strategy based on some prior knowledge can be used to assign initial values ​​to the automatically learned cue vector. As the training proceeds, the automatically learned cue vector will be continuously optimized through the back propagation algorithm to learn how to more effectively extract the most helpful information for image evaluation from the text cue.

[0116] According to an embodiment of the present invention, in order to further enhance the flexibility and adaptability of the neural network model used in the method for generating endoscopic image quality information of the embodiment of the invention, the model introduces a lightweight neural network as a shared network. The shared network can dynamically generate input conditional tokens based on the specific content of each input endoscopic image (or endoscopic image sample), and then the tokens are used as additional information to guide the model to better understand the image content and make more accurate predictions. In the initialization stage, the optimal initialization strategy is also used to set the parameters of the shared network to ensure that it starts learning from a good starting point. As training progresses, the shared network will learn how to automatically adjust the tokens it generates based on image features, thereby achieving accurate modeling and response to different image contents.

[0117] Figure 5 A schematic diagram showing an image quality evaluation result of an endoscope image quality information generating method according to an embodiment of the present invention.

[0118] like Figure 5As shown, the method for generating endoscopic image quality information according to the embodiment of the present invention can evaluate multiple evaluation indicators of endoscopic images and has high practical value.

[0119] It should be noted that, unless it is explicitly stated that there is a sequence of execution between different operations shown in the flowchart in the embodiments of the present invention, or different operations have a sequence of execution in technical implementation, otherwise, the execution order of multiple operations may not be particular, and multiple operations may also be executed simultaneously.

[0120] Based on the above-mentioned endoscopic image quality information generation method, the embodiment of the present invention also provides an endoscopic image quality information generation device. Figure 6 The device is described in detail.

[0121] Figure 6 A block diagram of an endoscope image quality information generating device according to an embodiment of the present invention is shown.

[0122] like Figure 6 As shown, the endoscope image quality information generating device 600 of this embodiment includes a feature conversion unit 610, a first encoding unit 620, a feature extraction unit 630, a hint vector fusion unit 640, a second encoding unit 650, a calculation unit 660 and a generation unit 670.

[0123] The feature conversion unit 610 is used to use the knowledge embedding module to perform feature conversion on multiple target category text embeddings respectively to obtain multiple first category perception prompt vectors, wherein the multiple target category text embeddings are text features corresponding one-to-one to multiple preset text prompt words, and the multiple preset text prompt words are prompt words related to the endoscopic image quality and have opposite meanings.

[0124] The first encoding unit 620 is used to encode the target endoscopic image using an image encoder to obtain a target image feature vector.

[0125] The feature extraction unit 630 is used to extract features from the target image feature vector using a shared network to obtain a target image prompt vector.

[0126] The prompt vector fusion unit 640 is used to fuse each first-category perception prompt vector of the multiple first-category perception prompt vectors with the target image prompt vector and the automatic learning prompt vector to obtain multiple target fused prompt vectors, wherein the shared network, the automatic learning prompt vector and the knowledge embedding module are all trained based on multiple endoscopic image samples and multiple groups of text prompt words corresponding to the multiple endoscopic image samples that reflect image quality and have opposite meanings.

[0127] The second encoding unit 650 is used to encode the multiple target fusion prompt vectors respectively by using the first text encoder to obtain multiple target category prompt vectors.

[0128] The calculation unit 660 is used to calculate the similarities between the multiple target category prompt vectors and the target image feature vectors respectively, to obtain multiple target image category similarities.

[0129] The generating unit 670 is used to generate an image quality evaluation result according to the similarities of multiple target image categories.

[0130] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0131] For example, any multiple of the feature conversion unit 610, the first encoding unit 620, the feature extraction unit 630, the hint vector fusion unit 640, the second encoding unit 650, the calculation unit 660, and the generation unit 670 may be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units may be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of the modules / units / sub-units may be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the feature conversion unit 610, the first encoding unit 620, the feature extraction unit 630, the hint vector fusion unit 640, the second encoding unit 650, the calculation unit 660 and the generation unit 670 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the feature conversion unit 610, the first encoding unit 620, the feature extraction unit 630, the hint vector fusion unit 640, the second encoding unit 650, the calculation unit 660 and the generation unit 670 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function may be performed.

[0132] It should be noted that the part of the endoscopic image quality information generating device in the embodiment of the present invention corresponds to the part of the endoscopic image quality information generating method in the embodiment of the present invention. The description of the part of the endoscopic image quality information generating device specifically refers to the part of the endoscopic image quality information generating method, which will not be repeated here.

[0133] Figure 7 A block diagram of an electronic device suitable for implementing the above-described method for generating endoscopic image quality information according to an embodiment of the present invention is shown. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0134] like Figure 7As shown, the electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0135] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0136] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0137] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0138] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0139] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0140] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .

[0141] An embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0142] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0143] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0144] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0146] The embodiments of the present invention have been described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment has been described above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present invention is limited by the attached embodiments and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for generating endoscopic image quality information, characterized in that: include: Using a knowledge embedding module to perform feature conversion on multiple target category text embeddings respectively, to obtain multiple first category perception prompt vectors, wherein the multiple target category text embeddings are text features corresponding to multiple preset text prompt words one by one, and the multiple preset text prompt words are prompt words related to the endoscopic image quality and have opposite meanings; Encode the target endoscopic image using an image encoder to obtain a target image feature vector; Extracting features from the target image feature vector using a shared network to obtain a target image prompt vector; fusing each of the plurality of first-category perception cue vectors with the target image cue vector and the automatic learning cue vector to obtain a plurality of target fused cue vectors, wherein the shared network, the automatic learning cue vector and the knowledge embedding module are all trained based on a plurality of endoscopic image samples and a plurality of groups of text cue words corresponding to the plurality of endoscopic image samples that reflect image quality and have opposite meanings; Using a first text encoder to encode the multiple target fusion prompt vectors respectively to obtain multiple target category prompt vectors; Calculating the similarities between the plurality of target category prompt vectors and the target image feature vectors respectively to obtain a plurality of target image category similarities; An image quality evaluation result is generated according to the similarities of the multiple target image categories.

2. The method according to claim 1, characterized in that The method of encoding the target endoscopic image using an image encoder to obtain a target image feature vector includes: Performing image segmentation on the target endoscopic image to obtain a plurality of segmented images; Using a plurality of the image encoders to encode the plurality of segmented images in parallel to obtain a plurality of segmented image features; The multiple segmented image features are fused to obtain the target image feature vector.

3. The method according to claim 1 or 2, characterized in that: Before using the knowledge embedding module to perform feature conversion on multiple target category text embeddings respectively to obtain multiple first category perception prompt vectors, the method further includes: The plurality of preset text prompt words are respectively encoded using a second text encoder to obtain the plurality of target category text embeddings.

4. The method according to claim 1 or 2, characterized in that: The shared network, the automatically learned hint vector and the knowledge embedding module are all trained according to the following operations: For each endoscopic image sample among the multiple endoscopic image samples, using the currently trained knowledge embedding module, perform feature conversion on multiple sample category text embeddings corresponding to the endoscopic image sample to obtain multiple second category perception prompt vectors, wherein the multiple sample category text embeddings are text features corresponding one-to-one to multiple text prompt words for the endoscopic image sample; Encoding the endoscopic image sample using the image encoder to obtain a sample image feature vector; Using the currently trained shared network to perform feature extraction on the sample image feature vector to obtain a sample image prompt vector; Fusing each of the plurality of second category perception prompt vectors with the sample image prompt vector and the currently trained automatic learning prompt vector to obtain a plurality of sample fused prompt vectors; Using the first text encoder to encode the multiple sample fusion prompt vectors respectively to obtain multiple sample category prompt vectors; Calculating the similarities between the plurality of sample category hint vectors and the sample image feature vectors respectively to obtain a plurality of sample image category similarities, and generating an image quality prediction result according to the plurality of sample image category similarities; According to the image quality prediction results and category labels corresponding to the multiple endoscopic image samples, the currently trained knowledge embedding module, the network parameters of the currently trained shared network and the value of the currently trained automatic learning prompt vector are iteratively updated, and when the image quality prediction accuracy is within a preset range, the currently trained shared network, the automatic learning prompt vector and the knowledge embedding module are respectively determined as the shared network, the automatic learning prompt vector and the knowledge embedding module.

5. The method according to claim 4, characterized in that Before embedding multiple sample category texts corresponding to each of the multiple endoscopic image samples using the currently trained knowledge embedding module and performing feature conversion respectively to obtain multiple second category perception prompt vectors for each endoscopic image sample in the multiple endoscopic image samples, the method further includes: Construct multiple groups of text prompt words with opposite meanings that reflect image quality; acquiring a plurality of endoscopic image samples; According to the plurality of groups of text prompt words, text prompt words corresponding to each endoscopic image sample and reflecting the image quality and having opposite meanings are constructed.

6. The method according to claim 3, characterized in that The image encoder is an image encoder in a pre-trained multimodal model; The first text encoder and the second text encoder are both text encoders in the pre-trained multimodal model.

7. The method according to claim 3, characterized in that During the process of training the shared network, the automatic learning hint vector and the knowledge embedding module, the network parameters of the image encoder, the first text encoder and the second text encoder remain unchanged.

8. The method according to claim 1 or 2, characterized in that: Generating an image quality evaluation result according to the similarities of the multiple target image categories includes: Normalizing the multiple target image category similarities using a normalized exponential function to obtain multiple normalized category similarities corresponding to the multiple preset text prompt words; The image quality evaluation result is generated according to the multiple normalized category similarities.

9. The method according to claim 1 or 2, characterized in that: The shared network sequentially includes a first linear layer, a nonlinear activation function layer, and a second linear layer.

10. An endoscope image quality information generating device, characterized in that: include: a feature conversion unit, configured to perform feature conversion on a plurality of target category text embeddings respectively using a knowledge embedding module to obtain a plurality of first category perception prompt vectors, wherein the plurality of target category text embeddings are text features corresponding one-to-one to a plurality of preset text prompt words, and the plurality of preset text prompt words are prompt words related to the endoscopic image quality and having opposite meanings; A first encoding unit, configured to encode a target endoscopic image using an image encoder to obtain a target image feature vector; A feature extraction unit, used to extract features from the target image feature vector using a shared network to obtain a target image prompt vector; a prompt vector fusion unit, configured to fuse each of the plurality of first-category perceptual prompt vectors with the target image prompt vector and the automatic learning prompt vector to obtain a plurality of target fused prompt vectors, wherein the shared network, the automatic learning prompt vector and the knowledge embedding module are all trained based on a plurality of endoscopic image samples and a plurality of groups of text prompt words corresponding to the plurality of endoscopic image samples that reflect image quality and have opposite meanings; A second encoding unit, configured to respectively encode the plurality of target fusion prompt vectors using the first text encoder to obtain a plurality of target category prompt vectors; A calculation unit, used for calculating the similarities between the plurality of target category prompt vectors and the target image feature vectors respectively, to obtain a plurality of target image category similarities; A generating unit is used to generate an image quality evaluation result according to the similarities of the multiple target image categories.

Images