An image diagnosis learning recommendation method and device and a storage medium

Abstract

Embodiments of the present specification provide an image diagnosis learning recommendation method and device and a storage medium. The method comprises: determining at least one of a first feature and a second feature that a target user needs to learn, wherein the first feature is related to a target report detail in an image report of the target user, and the second feature is related to a target image detail in an image; determining a target feature vector based on at least one of the first feature and the second feature; and determining recommended learning content of the target user based on the target feature vector.

Description

Technical Field

[0001] This manual pertains to the medical field, and in particular to a recommended method and system for learning diagnostic imaging. Background Technology

[0002] In the medical field, doctors need extensive experience to interpret medical images (such as mammograms) and write corresponding reports, making the learning process quite costly. Current training methods to improve doctors' medical image interpretation skills generally involve reading classic cases or textbooks, or organizing workshops where typical cases are selected and shared for discussion based on experience. However, the number of cases doctors can actually learn from during their training is limited and lacks specificity, making it difficult to address the different problems doctors encounter when interpreting images.

[0003] Therefore, there is a need for a method, device, and storage medium for learning diagnostic imaging, which can analyze the reports written by doctors and the images they read, and provide targeted learning content. Summary of the Invention

[0004] This specification provides one or more embodiments of an image diagnostic learning recommendation method, the image diagnostic learning recommendation method comprising: determining at least one of a first feature and a second feature that a target user needs to learn, wherein the first feature is related to target report details in an image report of the target user, and the second feature is related to target image details in an image; determining a target feature vector based on at least one of the first feature and the second feature; and determining recommended learning content for the target user based on the target feature vector.

[0005] This specification provides one or more embodiments of an image diagnostic learning recommendation apparatus, the apparatus including a processor for executing an image diagnostic learning recommendation method.

[0006] This specification provides one or more embodiments of a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions from the storage medium, the computer executes an image diagnostic learning recommendation method. Attached Figure Description

[0007] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0008] Figure 1 This is a schematic diagram illustrating an application scenario of an image diagnostic learning recommendation system based on some embodiments of this specification;

[0009] Figure 2This is an exemplary block diagram of an image diagnostic learning recommendation system according to some embodiments of this specification;

[0010] Figure 3 This is an exemplary flowchart of an image diagnostic learning recommendation method according to some embodiments of this specification;

[0011] Figure 4 This is an exemplary flowchart of a training feature alignment model according to some embodiments of this specification. Detailed Implementation

[0012] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0013] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0014] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0015] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0016] When interpreting medical images and writing corresponding reports, doctors may make inaccurate assessments due to a lack of experience. Currently, the common approach to improve doctors' ability to interpret medical images and write reports is through studying typical cases. However, the number of cases doctors study may be limited, and some cases may be vaguely described (e.g., in breast reports, terms like "tendency of fine calcifications" or "faint nodular shadows at the edge of the gland") or have low confidence levels, making them difficult to categorize. This results in a high learning cost and fails to address the problems encountered by each individual doctor when interpreting images.

[0017] In view of this, some embodiments of this specification provide an image diagnosis learning recommendation method, device, and storage medium, which can analyze the reports written by doctors and the images they read, discover and summarize common problems in doctors' image reading, and based on these problems, provide personalized recommendations to doctors by querying similar cases, thereby helping doctors improve their ability to read images.

[0018] Figure 1 This is a schematic diagram illustrating an application scenario of an image diagnostic learning recommendation system based on some embodiments of this specification.

[0019] like Figure 1 As shown, the image diagnosis learning recommendation system 100 (hereinafter referred to as system 100) involved in the embodiments of this specification includes a processor 110, a terminal 120, a network 130, a storage device 140, image data 150, and recommended learning content 160.

[0020] Processor 110 is capable of processing information and / or data related to system 100 to perform one or more functions described in this specification. For example, processor 110 may be used to determine at least one of a first feature and a second feature that a target user needs to learn; determine a target feature vector based on at least one of the first feature, the second feature, and an image; and determine recommended learning content for the target user based on the target feature vector. As another example, processor 110 may analyze and process image data 150 to summarize problems encountered by the user (such as inaccuracies in image reports) and / or provide recommended learning content 160. Image data 150 may be medical images read by the user.

[0021] Image data 150 may include image data obtained through techniques such as angiography, computed tomography, mammography, magnetic resonance imaging, positron emission tomography, and medical ultrasound. In some embodiments, image data 150 may also include image reports written by users based on the images. Users can input image data 150 into terminal 120 and send it to processor 110 and / or storage device 140 via a network.

[0022] Recommended learning content 160 can refer to user-recommended learning materials related to improving film viewing skills. Recommended learning content 160 may include recommended learning video 161 and corresponding recommended learning materials 162. For more information on recommended learning content, please refer to [link to relevant documentation]. Figure 3 .

[0023] In some embodiments, processor 110 includes one or more processing engines (e.g., a single-chip processing engine or a multi-chip processing engine). By way of example only, processor 110 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction processor (ASIP), a graphics processing unit (GPU), a physical processor (PPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), a microprocessor, or any combination thereof.

[0024] Terminal 120 can provide functional components related to user interaction and enable user interaction functions (such as providing or displaying information and data to the user). The user can refer to medical personnel or others who need to improve their image interpretation skills. As an example only, terminal 120 can be one or any combination of mobile devices, tablet computers, laptop computers, desktop computers, and other devices with input and / or output functions. As an example only, output functions include, but are not limited to, one or more combinations of sound output such as voice, display screen, haptic transmission such as vibration, and electromagnetic wave signals such as light. As an example only, input functions can include, but are not limited to, one or more combinations of keyboard input, touchscreen input, voice input, motion event input such as device tilt / shake / rotation / swing, and electromagnetic wave signal input such as light.

[0025] In some embodiments, a user can input information and / or data through terminal 120, and can also obtain information and / or data through terminal 120. For example, a user can input image data through terminal 120; or, for example, a user can obtain recommended learning content through terminal 120.

[0026] Network 130 may include any suitable network that facilitates the exchange of information and / or data between system 100. In some embodiments, one or more components of system 100 (e.g., processor 110, terminal device 120, storage device 140) may communicate information and / or data with one or more other components of system 100 via network 130. In some embodiments, network 130 may be and / or include public networks, private networks, wide area networks (WANs), wired networks, wireless networks, cellular networks, frame relay networks, virtual private networks, satellite networks, telephone networks, routers, hubs, switches, and any combination thereof. In some embodiments, network 130 may include one or more network access points. For example, network 130 may include wired and / or wireless network access points such as base stations and / or internet switching points, through which one or more components of system 100 may connect to network 130 to exchange data and / or information.

[0027] Storage device 140 is capable of storing data, instructions, and / or any other information. In some embodiments, storage device 140 may store data acquired from terminal 120 and / or processor 110, such as image data input by a user. In some embodiments, storage device 140 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), or any combination thereof. In some embodiments, storage device 140 may execute on a cloud platform. In some embodiments, storage device 140 may be connected to network 130 to communicate with one or more other components of system 100 (e.g., processor 110, terminal 120). In some embodiments, storage device 140 may be part of processor 110.

[0028] It should be noted that the above description is provided for illustrative purposes only and is not intended to limit the scope of this specification. Various changes and modifications can be made by those skilled in the art based on the content of this specification. Features, structures, methods, and other features of the exemplary embodiments described herein can be combined in various ways to obtain other and / or alternative exemplary embodiments. However, these changes and modifications will not depart from the scope of this specification.

[0029] Figure 2 This is an exemplary block diagram of an image diagnostic learning recommendation system according to some embodiments of this specification. Figure 2 As shown, the processor 110 of the image diagnosis learning recommendation system may include a first determining module 111, a second determining module 112, and a third determining module 113.

[0030] The first determining module 111 can be used to determine at least one of a first feature and a second feature that the target user needs to learn, wherein the first feature is related to the target report details in the image report of the target user, and the second feature is related to the target image details in the image.

[0031] In some embodiments, the image includes a target image, and the image report includes a target report corresponding to the target image. The first determining module 111 is further configured to acquire the target image and the target report of the target user; analyze and process the target image and the target report based on a verification model, and determine at least one of a first feature in the target report and a second feature in the target image, wherein the feature verification model is a machine learning model.

[0032] In some embodiments, the image report includes historical image reports. The first determining module 111 is further configured to acquire multiple historical image reports of the target user; determine at least one target report detail of the target user based on the multiple historical image reports; cluster the at least one target report detail to determine at least one cluster center; and determine a first feature based on the at least one cluster center.

[0033] In some embodiments, the first determining module 111 is further configured to acquire target user information of the target user; determine target user characteristics of the target user based on the target user information; determine the user category of the target user through a user database based on the target user characteristics; determine target report details and determine a first feature based on reference report details associated with the user category; and / or determine target image details and determine a second feature based on reference image details associated with the user category.

[0034] The second determining module 112 is used to determine the target feature vector based on at least one of the first feature and the second feature.

[0035] In some embodiments, the second determining module 112 may be further configured to encode the first feature to determine the first encoding; encode the second feature by an image encoder to determine the second encoding; and generate a target feature vector based on at least one of the first encoding and the second encoding.

[0036] In some embodiments, the image encoder obtains the image through a trained feature alignment model.

[0037] In some embodiments, the feature alignment model is used to align images and associated features in the corresponding image reports.

[0038] The third determining module 113 can be used to determine the recommended learning content for the target user based on the target feature vector.

[0039] In some embodiments, the recommended learning content includes recommended learning images and recommended learning materials associated with the recommended learning images. The third determining module 113 can be further configured to perform a search in the reference database based on the target feature vector, determine reference feature vectors whose similarity to the target feature vectors meets preset conditions, determine the reference images associated with the reference feature vectors as recommended learning images, and obtain the recommended learning materials corresponding to the recommended learning images.

[0040] In some embodiments, the processor 110 of the image diagnosis learning recommendation system may further include a training module 114.

[0041] The training module 114 is used to obtain a feature alignment model through training. The training includes: acquiring multiple sets of training samples, wherein each set of training samples includes a sample image and a sample report corresponding to the sample image.

[0042] For any set of training samples, the training module 114 can be further used to: determine at least one sample report feature in the sample report based on the sample report and identify it as a positive sample; adjust the at least one sample report feature to obtain a preset number of negative samples; and train an initial feature extraction model through comparative learning based on the sample images, positive samples and negative samples corresponding to multiple sets of training samples to obtain a feature alignment model.

[0043] For a detailed description of each module of the processor 110 of the image diagnostic learning recommendation system shown above, please refer to the flowchart section of this manual, for example, Figure 3 , Figure 4 And related explanations.

[0044] It should be noted that the above description of the image diagnostic learning recommendation system and its modules is for convenience only and should not be construed as limiting this specification to the scope of the embodiments described. It is understood that those skilled in the art, after understanding the principles of the system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. In some embodiments, Figure 2 The first determining module 111, the second determining module 112, the third determining module 113, and the training module 114 disclosed herein can be different modules within a single system, or a single module can implement the functions of two or more of the aforementioned modules. For example, the modules can share a single storage module, or each module can have its own separate storage module. Such variations are all within the scope of protection of this specification.

[0045] Figure 3This is an exemplary flowchart of an image diagnostic learning recommendation method according to some embodiments of this specification. In some embodiments, process 300 may be executed by processor 110. Figure 3 As shown, process 300 includes the following steps.

[0046] Step 310: Determine at least one of the first feature and the second feature that the target user needs to learn.

[0047] The target users are those who need to improve their image interpretation skills. For example, target users could be doctors with less experience (e.g., less than 5 years of work experience).

[0048] The first feature refers to the feature that is related to the target user's target report details in the image report.

[0049] The target report details can include details corresponding to errors made by the target user in the imaging report. Errors include content omitted by the target user and / or inaccurate descriptions in the imaging report. For example, the first feature could include the feature omitted by the target user: "small clusters of calcifications near high-density glands." Alternatively, the first feature could also include the feature described by the target user as "multiple fine, faint punctate calcifications in both breasts, with relatively regular morphology" as "multiple fine, faint punctate calcifications in both breasts, with irregular morphology" (i.e., an inaccurate description by the target user).

[0050] The target report details may also include details from the imaging report that the target user wants to learn. Correspondingly, the first feature may also include features that the target user wants to learn. For example, the first feature may include features corresponding to details related to lung imaging reports (or other medical imaging reports, without limitation) that the target user wants to learn.

[0051] An imaging report is a report related to medical imaging. For example, imaging reports can include angiography reports, computed tomography (CT) scan reports, mammography reports, etc.

[0052] Imaging reports can be based on images. For example, doctors can write imaging reports based on images.

[0053] Medical imaging refers to patient-related medical images acquired using medical equipment. Examples include angiography, computed tomography (CT) scans, and mammography. Medical equipment used to acquire these images can include, but is not limited to, X-ray scanners, computed tomography (CT) scanners, magnetic resonance imaging (MRI) scanners, positron emission tomography (PET) scanners, optical coherence tomography (OCT) scanners, ultrasound scanners, and intravascular ultrasound scanners. There is a correspondence between imaging reports and images. For example, one imaging report can correspond to one image.

[0054] The second feature refers to the feature related to the details of the target image in the image.

[0055] The target image details can include details in the image corresponding to the errors made by the target user in the image report. For example, if the target user's error is omitting "small clusters of calcifications near high-density glands," the second feature can include features corresponding to the high-density gland area in the image. As another example, if the target user's error is incorrectly describing the feature "multiple fine, faint punctate calcifications in both breasts, with relatively regular shapes," the second feature can include features corresponding to the breast area in the image.

[0056] The target image details can also include details in the image that the target user wants to learn. Correspondingly, the second feature can also include features corresponding to the details in the image that the target user wants to learn. For example, features corresponding to details related to lung images (or other medical images, without limitation) that the target user wants to learn.

[0057] In some embodiments, the features that the target user needs to learn may include only one of the first feature or the second feature. For example, when the target user lacks the ability to distinguish locations (such as breasts), the target user's location discrimination ability can be improved by learning from the image report corresponding to the first feature or the image corresponding to the second feature, and only one of the first feature or the second feature needs to be acquired.

[0058] In some embodiments, the features that the target user needs to learn may include both a first feature and a second feature. For example, the target user can simultaneously learn from the image report corresponding to the first feature and the image corresponding to the second feature. When both a first feature and a second feature are included, the first feature and the second feature can be a pair of related features or unrelated features. For example, the first feature and the second feature can both be related features such as "spot-like" or "short-striped". As another example, the first feature could be the "spot-like" or "short-striped" feature in the image report, while the second feature could be the "isodensity" feature in the image.

[0059] In some embodiments, the image may include a target image, and the image report may include a target report corresponding to the target image. The target image may refer to an image that the target user will read and analyze, or it may refer to an image that the target user wants to learn from. The target report may include an image report written by the target user based on the target image, or it may refer to an image report that the target user wants to learn from.

[0060] In some embodiments, the processor may acquire a target image and a target report from a target user; analyze and process the target image and target report based on a verification model to determine at least one of a first feature in the target report and a second feature in the target image.

[0061] The processor can acquire target images and target reports based on input from the target user. For example, the target user can input target images and target reports into terminal 120, and the processor can acquire the target images and target reports from terminal 120 via network 130.

[0062] The verification model is used to determine the first feature and / or the second feature.

[0063] In some embodiments, the verification model can be a deep learning model, such as a convolutional neural network model. In some embodiments, the verification model can also share the image encoder and text encoder with the feature alignment model. Further details about the feature alignment model can be found in the relevant sections below.

[0064] When the verification model is a machine learning model, its input may include the target image and the target report, and its output may include at least one of the first feature in the target report and the second feature in the target image.

[0065] Accordingly, when the verification model is a machine learning model, its training samples can include historical images and corresponding historical reports. Labels can include manually annotated historical images and omissions or inaccurate descriptions in the corresponding historical reports. The training samples and labels for the verification model can be obtained based on historical image reading data.

[0066] In some preferred embodiments, the verification model can also be a language model (LM). For example, the verification model can be a BERT (Bidirectional Encoder Representations from Transformers) model, a GPT (Generative Pre-Trained Transformer) model, etc. When the verification model is a language model, it can be obtained through pre-training.

[0067] In some embodiments of this specification, the verification model can be a pre-trained LM model (such as a GPT model), which can quickly and accurately understand the natural language of the image report, eliminating the need for a complex natural language understanding training process, and more quickly and accurately determine the error content of the target user, and extract its corresponding first and second features.

[0068] Taking the GPT model as an example, the target user can input the target image, target report, and prompt into the verification model. The verification model outputs at least one of the corresponding first feature and second feature. The output of the verification model is related to the prompt. For example, the verification model can determine the output of the first feature and / or the second feature based on the prompt.

[0069] The prompt is a prompt related to determining the first feature and the second feature. The prompt can be determined through target user input. For example, when the prompt includes "Based on the input image and image report, find the areas in the image report that are not accurately identified," the verification model outputs the corresponding first feature; as another example, when the prompt includes "Based on the input image and image report, find the image regions corresponding to the areas in the image report that are not accurately identified," the verification model outputs the corresponding second feature; as yet another example, when the prompt includes "Based on the input image and image report, find the areas in the image report that are not accurately identified and the corresponding image regions," the verification model outputs the corresponding first feature and second feature.

[0070] Understandably, by inputting prompts into the verification model, the model can understand the needs of the target user (such as finding errors), thereby enabling the model's output to meet the target user's needs.

[0071] The verification model can identify target images and target reports based on prompts, and determine areas in the target report that are "not accurately identified" (i.e., target report details). For example, the verification model can find discrepancies between the content described in the target report and the target image, and identify these discrepancies as target report details provided by the target user in the image report. For instance, if the target image actually shows "multiple fine, faint punctate calcifications in both breasts, with relatively regular morphology," while the target report should describe "multiple fine, faint punctate calcifications in both breasts, with irregular morphology," then the verification model can determine that the target user's description of the breasts is a target report detail.

[0072] Furthermore, the verification model can identify target report details in the target report as the first feature. The verification model can also identify target image details in the corresponding image as the second feature.

[0073] It is understandable that certain details in an imaging report may constitute a small percentage of the overall report and are easily overlooked or ignored during learning, or there may be few relevant cases, resulting in low learning efficiency. For example, a common problem for a doctor is overlooking suspicious calcifications in high-density glands. However, "suspicious calcifications in high-density glands" may represent a small percentage of suspicious calcifications. If recommendations are made to the doctor solely based on suspicious calcifications, targeted learning of "suspicious calcifications in high-density glands" cannot be achieved. In some embodiments of this specification, by using machine learning models to determine features in imaging reports (and / or images), the detailed problems encountered by the target user can be more accurately identified, and targeted learning content recommendations can be made for these detailed problems, improving learning effectiveness.

[0074] The target user may also only need to learn a specific portion of the target image. In some embodiments, the target user can directly mark on the target image (e.g., draw a circle or describe the corresponding location in words) to determine the second feature corresponding to the marked area in the target image. In some embodiments of this specification, the target user can directly specify the content to be learned, avoiding the need for the verification model to process the entire target image and target report, thus improving efficiency.

[0075] In some embodiments, the image report may also include a historical image report. A historical image report refers to an image report written by the target user during the viewing of historical images.

[0076] In some embodiments, the processor may also acquire multiple historical image reports of the target user; determine at least one target report detail of the target user based on the multiple historical image reports; cluster the at least one target report detail to determine at least one cluster center; and determine a first feature based on the at least one cluster center.

[0077] In some embodiments, the target user's historical image report can be stored in storage device 140, and the processor can retrieve the target user's historical image report from storage device 140.

[0078] In some embodiments, the processor can determine at least one target report detail for a target user based on multiple historical image reports. The historical image reports may include evaluation information, which may refer to the evaluation of the writing quality of the historical image report by relevant experts, and the evaluation information may include the target report details of that historical image report. In some embodiments, the target report details in the historical image reports can also be determined by a verification model. For example, the processor can process the historical image reports and corresponding historical images based on a verification model to determine the target report details corresponding to the historical image reports. More information about verification models can be found in the relevant sections above this specification.

[0079] In some embodiments, the processor may further cluster at least one target report detail of the target user to determine at least one cluster center; and determine a first feature based on the at least one cluster center. For example, the processor may determine the target report detail corresponding to the cluster center and the content corresponding to its modified correct description as the first feature. The clustering method may include, but is not limited to, K-means clustering algorithm, K-centroid clustering algorithm, Gaussian Mixture Model (GMM) clustering, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), etc.

[0080] Furthermore, the processor can also determine the region of the historical image corresponding to the cluster center based on the target report details, and determine the image content corresponding to that region as the second feature.

[0081] In some other embodiments of this specification, by analyzing the target user's historical video reports, common problems encountered during the viewing process can be identified, and personalized recommendations can be made for these common problems to specifically improve the target user's weak points in viewing and enhance learning effectiveness.

[0082] In some embodiments, the processor may acquire target user information of the target user; determine target user characteristics of the target user based on the target user information; determine the user category of the target user through a user database based on the target user characteristics; determine target report details and a first feature based on reference report details associated with the user category; and / or determine target image details and a second feature based on reference image details associated with the user category.

[0083] Target user information refers to basic information related to a target user. For example, target user information includes, but is not limited to, one or more of the following: the target user's name, age, gender, alma mater, education level, length of employment, hospital, and department. Target user information can be obtained based on user input.

[0084] The processor can determine the characteristics of the target user based on the target user information. For example, the processor can vectorize the target user information and use it as target user features.

[0085] The user database stores a large number of reference user features and their corresponding reference user categories. In some embodiments, the user database may also store reference report details and / or reference image details associated with user categories. The processor can determine the reference user category of the target user based on vector retrieval, using the reference user category corresponding to the reference user feature with the smallest distance (or less than a threshold) to the target user feature. Reference user features include user features corresponding to reference users (such as a large number of medical workers in a certain region).

[0086] User categories refer to the categories that correspond to different users. User categories can be represented using vectors or other methods.

[0087] Reference report details refer to the image report details that various users need to study and refer to. Different user categories can correspond to different reference report details. The reference report details corresponding to different user categories can be obtained based on statistical data. For example, for each user category, image reports of users of that category in a certain region over a period of time (such as 1 month or 1 year) can be statistically analyzed, and the image report details with errors can be identified as reference report details. Another example is that for each user category, search or learning information of users of that category in a certain region over a period of time can be statistically analyzed, and the image report details studied by the most users can be identified as reference report details.

[0088] The processor can identify the reference report details associated with the user category corresponding to the target user as the first feature.

[0089] Reference image details refer to the image details that various users need to learn from. Similarly, the processor can determine the reference image details associated with each user category, and based on the reference image details associated with the user category, determine the target image details and the second feature.

[0090] Step 320: Determine the target feature vector based on at least one of the first feature and the second feature.

[0091] The target feature vector is the vector used to match the recommended learning content. For a detailed explanation of the recommended learning content, please refer to step 330.

[0092] In some embodiments, the processor may encode a first feature to determine a first encoding; encode a second feature by an image encoder to determine a second encoding; and generate a target feature vector based on at least one of the first encoding and the second encoding.

[0093] The first code refers to the code corresponding to the first feature in the image report. The processor processes the first feature using a first text encoder to obtain the corresponding first code. The aforementioned text encoder can be obtained through a feature alignment model or other methods.

[0094] The second encoding refers to the encoding corresponding to the second feature in the image. In some embodiments, the processor processes the second feature through an image encoder to obtain the corresponding second encoding. In some embodiments, the image encoder can be obtained through a trained feature alignment model.

[0095] Feature alignment models are used to align images with associated features in their corresponding image reports (e.g., the feature "patchy" in the image report). Feature alignment models are machine learning models, such as deep neural network models or other custom models. In some embodiments, the feature alignment model includes at least a text encoding layer and an image encoding layer.

[0096] The feature alignment model takes as input an image and its corresponding image report, and outputs aligned feature vectors from the image and its corresponding image report. The model extracts features from the image and its corresponding image report, aligns corresponding features, and then encodes and combines them to obtain at least one set of aligned feature vectors. Each set of aligned feature vectors includes a pair of associated features from the image and its corresponding image report.

[0097] Feature alignment models can be obtained through training. For more information on training feature alignment models, please refer to [link to relevant documentation]. Figure 4 .

[0098] In some embodiments of this specification, the feature alignment model can align the image with the image report, thereby enabling the image encoder in the feature alignment model to accurately identify and encode the various features in the image.

[0099] The processor can generate a corresponding target feature vector based on at least one of the first encoding and the second encoding. For example, it can generate a corresponding target feature vector based on either the first encoding or the second encoding. Another example is generating a corresponding target feature vector based on a combination of the first encoding and the second encoding.

[0100] In some embodiments, the processor further determines a target feature vector based on the image corresponding to the second feature, the first feature, and at least one of the second feature. The image encoder can also encode the image corresponding to the second feature to obtain a third code, and determine the target feature vector based on at least one of the third code, the first code, and the second code. The aforementioned third code refers to the code corresponding to each feature in the image. Some embodiments of this specification, by encoding the image corresponding to the second feature, can make the target feature vector possess information about the entire image, making the recommended learning content retrieved based on the target feature vector more relevant to the requirements.

[0101] Step 330: Based on the target feature vector, determine the recommended learning content for the target user.

[0102] In some embodiments, the processor can perform modeling or employ various data analysis algorithms, such as regression analysis and discriminant analysis, to analyze and process the target feature vector and determine the recommended learning content for the target user.

[0103] In some embodiments, the processor searches a reference database based on the target feature vector to determine a reference feature vector whose similarity to the target feature vector meets a preset condition.

[0104] A reference database is a database that provides a reference for target users' learning. A reference database can include a large number of reference feature vectors.

[0105] A reference feature vector is a vector used to match a target feature vector. The processor can extract features from each reference image and / or its corresponding reference image report, determine the corresponding reference feature vector, and store it in a reference database. A reference image refers to an image associated with the first feature and / or the second feature. A reference image report refers to the image report corresponding to the reference image.

[0106] The processor can match reference feature vectors that meet preset conditions from a reference database based on the target user's target feature vector. These preset conditions may include minimizing the distance between the reference feature vector and the target feature vector (e.g., cosine distance, Euclidean distance, Manhattan distance, Chebyshev distance, etc.) or meeting a preset threshold.

[0107] In some embodiments, the processor determines the reference image associated with the reference feature vector as the recommended learning image and obtains the recommended learning materials corresponding to the recommended learning image.

[0108] The reference database may also include reference learning content associated with the reference feature vector. Reference learning content refers to the learning content associated with the reference feature vector. Reference learning content includes reference images and reference learning materials associated with the reference images. Reference learning materials include, but are not limited to, the corresponding image reports and pathology reports of the reference images, and may also include image information from other modalities (such as ultrasound, MRI, etc.).

[0109] Recommended learning content refers to learning materials recommended to the target user to improve relevant abilities (such as improving film viewing skills). Recommended learning content includes at least recommended learning videos and related learning materials. Recommended learning videos and recommended learning materials are, respectively, the videos recommended to the target user and their corresponding learning resources.

[0110] Users can learn from the recommended learning videos and materials to identify their mistakes in the viewing process and make improvements.

[0111] In some embodiments, the processor may send recommended learning content to the terminal 120 for the target user to learn.

[0112] The image diagnosis learning recommendation method described in some embodiments of this specification can achieve at least the following effects: (1) It can effectively utilize a large number of image reports and image data within the hospital to recommend learning content to target users; (2) By analyzing the image reports written and images read by doctors with limited experience, it can discover and summarize common problems in their image reading process, and make personalized learning content recommendations to doctors based on the discovered problems, thereby achieving better recommendation results; (3) By encoding the doctor's mistakes and querying similar cases, the summary and analysis of problems can be made more intelligent and refined, and the recommendation of learning content for doctors will be more accurate and effective, thereby helping doctors improve their image reading ability.

[0113] Figure 4 This is an exemplary flowchart of a training method for a feature alignment model according to some embodiments of this specification. In some embodiments, process 400 may be executed by processor 110. Figure 4 As shown, process 400 may include the following steps.

[0114] Step 410: Obtain multiple sets of training samples.

[0115] Each of the multiple training samples includes a sample image and a corresponding sample report.

[0116] Sample images include images used for training the feature alignment model. For example, sample images may include historical images. For example, sample reports include image reports used as samples, corresponding to the sample images. Sample reports may include historical image reports corresponding to historical images. The sample reports may be error-free image reports issued by senior experts.

[0117] Sample images and sample reports can be obtained based on historical image reading data, and there is a corresponding relationship between sample images and sample reports. For example, the processor can collect historical images and their corresponding historical image reports, and determine a set of training samples for each historical image and its corresponding historical image report. For any set of training samples, the processor can determine positive samples and a preset number of negative samples based on steps 420 to 430.

[0118] Step 420: Based on the sample report, identify at least one sample report feature in the sample report and designate it as a positive sample.

[0119] Sample report features refer to the various features in a sample report, which can be obtained by processing the sample report. For multiple sets of training samples, the processor can process the sample reports in at least one set of training samples (such as performing semantic analysis) to obtain at least one sample report feature.

[0120] Sample report features are detailed characteristics that describe aspects of an image report. Sample report features can include, but are not limited to, one or more of the following: central description, location, morphological features, and distribution features. Central description refers to the specific evaluation of the object being described in the image; location refers to the part of the image being described; morphological features refer to the morphological characteristics of the image; distribution features refer to the distribution of abnormal (or normal) images within the image. For example, in an image report describing "multiple fine, faint punctate calcifications in both breasts, irregular in shape, with some fine calcifications showing a tendency to cluster," the central description could be "calcifications," the location could be "both breasts," the morphological feature could be "relatively regular in shape," and the distribution feature could be "some fine calcifications showing a tendency to cluster," etc.

[0121] The processor can obtain sample report features by parsing the image report, such as through token classification methods.

[0122] In some embodiments, the processor may identify a positive sample based on sample report features determined from the sample report.

[0123] Step 430: Adjust at least one sample report feature to obtain a preset number of negative samples.

[0124] The processor can adjust the sample report features to obtain a preset number of negative samples. For example, the processor can randomly select one of the sample report features (such as center description, location, morphological features, or distribution features). If the distribution feature is selected, the processor can randomly select N-1 other descriptions from the N possible descriptions of the distribution feature, and replace the distribution feature in the sample report as the adjusted sample report feature. For example, the distribution feature may include four possible descriptions: clustering trend distribution, diffuse distribution, regional distribution, and clustered distribution. If the distribution feature in the sample report is a clustering trend distribution, the processor can select one other description (such as regional distribution or clustered distribution) each time to replace the distribution feature in the sample report, obtaining three adjusted sample report features. Here, N is a manually set parameter that can correspond to one actual sample and N-1 virtual samples. Virtual samples refer to samples generated by replacing one or more of the sample report features in the actual sample.

[0125] The processor can identify positive samples adjusted in the above manner as negative samples. For each negative sample, the negative sample includes at least one adjusted sample report feature from the sample report, as well as other unadjusted sample report features.

[0126] Step 440: Based on the sample images, positive samples and negative samples corresponding to multiple sets of training samples, an initial feature alignment model is trained through contrastive learning to obtain the feature alignment model.

[0127] In some embodiments, the processor can train an initial feature alignment model based on positive samples corresponding to multiple sets of training samples and a preset number of negative samples through contrastive learning to obtain the feature alignment model.

[0128] The initial feature alignment model refers to the feature alignment model without parameters set (or with default values).

[0129] For example, contrastive learning can be performed through the following steps A1-A3.

[0130] Step A1: Obtain the sample vectors of positive samples, negative samples, and sample images.

[0131] The positive and negative samples are input into the pre-trained text encoder to obtain at least one sample report vector corresponding to each feature in the positive and negative samples; the sample images are input into the initial image encoder to obtain at least one sample image vector corresponding to each feature in the sample images.

[0132] Step A2: Generate sample vector pairs.

[0133] The processor generates sample vector pairs from a sample report vector and a sample image vector. For example, a sample vector pair might consist of a sample image vector representing "spot-like or short-striped" patterns from a sample image and a sample report vector representing "spot-like or short-striped" patterns from a positive sample. In this case, the aforementioned sample report vector and the aforementioned sample image vector represent the same feature attributes and have the same meaning; this sample vector pair is a correlated sample vector pair. As another example, a sample vector pair might consist of a sample image vector representing "spot-like or short-striped" patterns from a sample image and a sample report vector representing "patchy" patterns from a negative sample. In this case, the aforementioned sample report vector and the aforementioned sample image vector represent the same feature attributes but have different meanings. Yet another example is a sample vector pair might consist of a sample image vector representing "spot-like or short-striped" patterns from a sample image and a sample report vector representing "iso-density" patterns from a negative sample. In this case, the aforementioned sample report vector and the aforementioned sample image vector represent different feature attributes and have different meanings.

[0134] Step A3: Minimize the loss.

[0135] The processor can maximize the similarity of associated sample vector pairs by minimizing the contrastive loss function. For example, the Normalized Temperature-Scaled Cross-Entropy Loss (NT-Xent) can be selected as the contrastive loss function. The contrastive loss function is updated based on all sample vectors until conditions such as the contrastive loss function being less than a threshold, convergence, or the training period reaching a threshold are met, thus obtaining a trained feature alignment model. The aforementioned trained feature alignment model includes a trained image encoder.

[0136] A well-trained feature alignment model can select an image report that matches the description of the image from multiple image reports, and can align the image with each feature in the corresponding image report (for example, a well-trained feature alignment model can align the image with the "spot-like" or "short-striped" feature in the image report) and generate a target feature vector.

[0137] The trained feature alignment model includes an image encoder and a text encoder. The image encoder learns how to extract and encode features from an image, while the text encoder learns how to extract and encode features from an image report.

[0138] In one or more embodiments of this specification, by modifying positive samples to generate a preset number of negative samples, the features corresponding to the positive samples can be replaced with features of the same attributes but different meanings. The feature alignment model can select the image report that best matches the corresponding image description from the positive and negative samples, thereby enabling the feature alignment model to better learn to distinguish the various features in the image and the correspondence between the image report and the various features in the image.

[0139] It should be noted that the above descriptions of the various processes are for illustrative purposes only and do not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the processes under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0140] Some embodiments of this specification also provide an image diagnostic learning recommendation apparatus, the apparatus including at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least a portion of the computer instructions to implement the image diagnostic learning recommendation method as described in any of the above embodiments.

[0141] This specification provides one or more embodiments of a computer-readable storage medium that stores computer instructions that, when executed by a processor, implement the image diagnostic learning recommendation method as described in any of the above embodiments.

[0142] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0143] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0144] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of embodiments that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0145] Similarly, it should be noted that, in order to simplify the descriptions disclosed herein and thus aid in the understanding of one or more embodiments, the foregoing description of embodiments in this specification sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

[0146] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0147] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0148] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and are considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. An image diagnostic learning recommendation method, characterized in that, The method includes: Determine at least one of a first feature and a second feature that the target user needs to learn, wherein the first feature is related to the target report details of the target user in the image report, and the second feature is related to the target image details in the image; Based on at least one of the first feature and the second feature, a target feature vector is determined; Based on the target feature vector, the recommended learning content for the target user is determined; The image includes a target image, the image report includes a target report corresponding to the target image, and determining at least one of the first feature and the second feature that the target user needs to learn includes: Obtain the target image and the target report of the target user; The target image and the target report are analyzed and processed based on the verification model to determine at least one of the first feature in the target report and the second feature in the target image. The verification model is used to find the places where the content described in the target report does not match the target image and to take them as the target report details. The verification model is also used to determine the target report details as the first feature and to determine the target image details in the image corresponding to the target report details as the second feature.

2. The method according to claim 1, characterized in that, Determining the target feature vector based on at least one of the first feature and the second feature includes: Encode the first feature to determine the first code; The second feature is encoded using an image encoder to determine the second code; The target feature vector is generated based on at least one of the first encoding and the second encoding.

3. The method according to claim 2, characterized in that, The image encoder is obtained through a trained feature alignment model.

4. The method according to claim 3, characterized in that, The feature alignment model is used to align the image and the associated features in the corresponding image report. The feature alignment model is trained and obtained in the following way: Obtain multiple sets of training samples, wherein each set of training samples includes a sample image and a sample report corresponding to the sample image. For any set of training samples: Based on the sample report, at least one sample report feature in the sample report is determined and identified as a positive sample; Adjust the at least one sample report feature to obtain a preset number of negative samples; Based on the sample images, positive samples, and negative samples corresponding to the multiple sets of training samples, an initial feature alignment model is trained through contrastive learning to obtain the feature alignment model.

5. The method according to claim 1, characterized in that... The verification model is a machine learning model.

6. The method according to claim 1, characterized in that, The image report includes historical image reports, and the first feature that the target user needs to learn includes: Obtain multiple historical image reports from the target user; Based on the multiple historical image reports, at least one target report detail for the target user is determined; Cluster the at least one target report detail to determine at least one cluster center; The first feature is determined based on the at least one cluster center.

7. The method according to claim 1, characterized in that, The determination of at least one of the first feature and the second feature that the target user needs to learn includes: Obtain the target user information of the target user; Based on the target user information, the target user characteristics of the target user are determined; Based on the target user characteristics, the user category of the target user is determined through the user database; Based on reference report details associated with the user category, determine the target report details and determine the first feature; and / or Based on the reference image details associated with the user category, the target image details are determined and the second feature is determined.

8. The method according to claim 1, characterized in that, The recommended learning content includes recommended learning images and recommended learning materials associated with the recommended learning images. The step of determining the recommended learning content for the target user based on the target feature vector includes: Based on the target feature vector, a search is performed in the reference database to determine reference feature vectors whose similarity to the target feature vector meets preset conditions; The reference image associated with the reference feature vector is determined as the recommended learning image, and the recommended learning materials corresponding to the recommended learning image are obtained.

9. An image diagnostic learning recommendation device, characterized in that, The device includes a processor for executing the image diagnostic learning recommendation method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions. When the computer reads the computer instructions in the storage medium, the computer executes the image diagnosis learning recommendation method as described in any one of claims 1 to 8.

Images