A large model construction method for audiology vertical field

Abstract

The embodiment of the application discloses a large model construction method for an audiology vertical field, comprising: acquiring an audiology multi-modal data set; using the multi-modal data set, performing audiology field enhancement pre-training on a basic large model; using an instruction fine-tuning data set to fine-tune the pre-trained large model to obtain an audiology vertical field large model with interaction capability; wherein the instruction fine-tuning data set comprises at least one of audiology single-round dialogue data, audiology multi-round dialogue data, audiology NLP task instruction data and teaching interaction instruction data. The embodiment can provide a large model with audiology professional capability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for constructing large models for the vertical field of audiology. Background Technology

[0002] Audiology applications include hearing loss assessment, rehabilitation intervention, and teaching and training. However, interpreting hearing data (pure-tone audiometry, acoustic impedance, etc.) is highly specialized, identifying different types of hearing loss (conductive, sensorineural, etc.) is complex, rehabilitation plans need to be personalized, and traditional audiology teaching relies on static atlases and passive lectures, lacking immersive interaction and standardized training. These difficulties mean that using existing general-purpose models for interaction in the field of audiology presents at least one of the following problems: insufficient audiological knowledge, inadequate hearing data fusion capabilities, poor adaptability to teaching scenarios, and inability to conduct effective proactive questioning. Summary of the Invention

[0003] This invention provides a method for constructing large models for the vertical field of audiology, in order to solve at least one of the above problems.

[0004] In a first aspect, embodiments of the present invention provide a method for constructing large models for the vertical field of audiology, including: Obtain an audiological multimodal dataset, wherein the multimodal dataset includes at least one or more of text data and hearing test image data; Using the aforementioned multimodal dataset, the basic large model is pre-trained for audiological enhancement. The pre-trained large model was fine-tuned using the instruction fine-tuning dataset to obtain a large model for the audiology vertical domain with interactive capabilities. The instruction fine-tuning dataset includes at least one of audiology single-turn dialogue data, audiology multi-turn dialogue data, audiology NLP task instruction data, and teaching interaction instruction data.

[0005] In a second aspect, embodiments of the present invention provide an electronic device, the electronic device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the large model building method for the vertical field of audiology as described in any embodiment.

[0006] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the large model construction method for the vertical field of audiology as described in any embodiment.

[0007] In summary, this embodiment provides a method for constructing large models in the vertical field of audiology, which can achieve at least one of the following beneficial effects: 1. Enhances the professionalism of audiological interaction. By deeply integrating multimodal audiological data with professional knowledge graphs, it accurately achieves hearing loss classification and personalized rehabilitation plan recommendations, with a professional accuracy rate superior to general large models; 2. Able to conduct effective proactive questioning. Based on the attention heatmap, identify key information to ensure the effective completion of the task, and proactively ask questions based on this key information. Prioritize proactive questioning from the global backbone information and global logic, and adopt different follow-up questioning strategies based on the attention weight of supplementary content and the questioning effect. Differentiate the direction of follow-up questioning to detailed information in this modality or cross-modal information to further improve the effectiveness of proactive questioning and improve the quality of task completion; 3. Adaptable to core teaching scenarios: Through multi-role intelligent agent collaboration, it supports closed-loop teaching processes such as simulated consultation, practical guidance, and assessment feedback, solving the pain points of single interaction and scarce cases in traditional audiology teaching; 4. Flexible deployment: It adopts a lightweight pre-training and LoRA fine-tuning strategy, which is suitable for low computing power environments in universities, training bases, and primary rehabilitation institutions, and supports plug-in integration with existing teaching and medical systems.

[0008] 5. Wide range of applications: It takes into account the needs of clinical hearing assessment, rehabilitation guidance and standardized teaching, and can cover scenarios such as medical student training, resident physician training and primary care physician capacity building. Attached Figure Description

[0009] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a method for constructing a large model for the vertical field of audiology, provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0012] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0013] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0014] Figure 1 This is a flowchart illustrating a method for constructing large models in the vertical field of audiology, provided by an embodiment of the present invention. This method can be executed by electronic devices, or by a combination of electronic devices and professionals. Figure 1 As shown, the method specifically includes: S110. Obtain the audiological multimodal dataset.

[0015] This embodiment uses a multimodal audiology dataset as the foundation for constructing a large-scale model in the vertical field of audiology. Optionally, the multimodal audiology data may include the following modalities: Text data includes audiological electronic medical records, hearing rehabilitation guidelines, professional literature, teaching cases, etc. Functional test data: including pure tone audiogram (also known as audiogram), acoustic impedance, otoacoustic emissions, auditory brainstem response and other data and standardized labels; Three-dimensional dynamic data: three-dimensional point cloud models of anatomical structures such as the cochlea and auditory nerve, and dynamic simulation data of the pathological evolution of hearing loss (such as cochlear nerve degeneration and fixation of the ossicular chain); Teaching interaction data: simulated consultation dialogues, clinical reasoning training cases, and data on the practical process of hearing assessment.

[0016] S120. Preprocess the multimodal dataset.

[0017] In one specific implementation, format standardization can be performed first. Structured data such as audiograms and unstructured data such as medical records and teaching cases are uniformly converted into JSON format, and word segmentation and entity alignment are achieved through an audiology professional dictionary (including domain terminology).

[0018] Then, data cleaning is performed. A combination of rule-driven and machine learning methods is used to filter outliers in the hearing data (such as abnormal audiometry thresholds) and verify the logical consistency of the text data (such as matching diagnostic results with audiogram results).

[0019] Further data augmentation is then performed. Generative AI is used to synthesize diverse audiograms and case variations to construct a dataset linking "hearing loss type - etiology - rehabilitation plan". Optionally, ≥500 cases are required for each disease.

[0020] Finally, knowledge is structured. An audiology knowledge graph is constructed, covering core knowledge nodes such as hearing anatomy, testing indicators, disease classification, and rehabilitation interventions (hearing aid fitting, cochlear implantation), enabling knowledge association and reasoning.

[0021] S130. Using the multimodal dataset, perform audiological domain enhancement pre-training on the basic large model.

[0022] This embodiment focuses on the selection of a basic model and domain pre-training. Optionally, a lightweight multimodal Chinese large-scale model (such as Qianwen) can be selected as the basic framework. This model has natural language interaction capabilities and can balance model performance and deployment costs through knowledge distillation technology.

[0023] In one specific implementation, audiological domain-enhanced pre-training of the base large model may include at least one of the following aspects: Internalization of hearing knowledge: Through text pre-training tasks, the basic large model learns audiological terminology, rules for interpreting test indicators (such as the correspondence between pure tone audiometry thresholds and the degree of hearing loss), and the adaptation logic of rehabilitation plans.

[0024] Data Feature Fusion: This involves introducing audiogram feature extraction and text association tasks to enhance the basic large-scale model's ability to interpret hearing test data, achieving a logical connection between "audiogram - pathological diagnosis - rehabilitation suggestions." Optionally, audiograms, hearing test data, pathological diagnosis text, and rehabilitation suggestion text for the same patient can be extracted from a preprocessed multimodal dataset to form a multimodal sample, which can then be used to train the basic large-scale model. Then, for each multimodal sample, based on automatic retrieval and enhanced generation technology, the basic large model is instructed to retrieve knowledge related to audiograms, hearing test data, and pathological diagnosis tasks from the knowledge base (such as text fragments related to pathological diagnosis in guidelines and literature; or relevant subgraphs in the knowledge graph; or first retrieve relevant subgraphs in the knowledge graph, and then retrieve relevant text fragments based on the subgraphs, using the relevant subgraphs and relevant text fragments together as relevant knowledge, etc.). Based on the relevant knowledge, the specific audiograms and hearing test data in the sample are analyzed, and pathological diagnosis text is output. After the large model outputs the results, the parameters of the basic large model are adjusted according to the difference between the model's output and the actual pathological diagnosis text in the sample, so that the model output continuously approaches the actual pathological diagnosis text. Then, the basic large model is instructed to retrieve knowledge related to audiograms, hearing test data, pathological diagnosis content, and rehabilitation suggestion tasks, and analyze specific audiograms, hearing test data, and pathological diagnosis texts based on the knowledge, and provide corresponding rehabilitation suggestion texts; after the large model outputs the results, the parameters of the basic large model are adjusted according to the difference between the model output and the actual rehabilitation suggestion texts in the samples, so that the model output continuously approaches the actual rehabilitation suggestion texts.

[0025] Injection of teaching knowledge: Through pre-training with teaching cases, the basic large model learns the teaching logic of audiology (such as explanation of anatomical knowledge, guidance of practical steps, and correction of common errors), laying the foundation for adaptation to teaching scenarios.

[0026] After pre-training, a basic audiological model is obtained that has a foundation of audiological knowledge, but lacks professional interactive capabilities.

[0027] S140. Fine-tune the pre-trained large model using the instruction fine-tuning dataset to obtain a large model for the audiology vertical domain with interactive capabilities.

[0028] This step utilizes an interactive instruction dataset to fine-tune the basic audiology model, resulting in a large-scale audiology model that possesses both a foundation in audiology expertise and professional interactive capabilities. The instruction fine-tuning dataset includes at least one of the following: audiology single-turn dialogue data, audiology multi-turn dialogue data, audiology NLP (Natural Language Processing) task instruction data, and teaching interactive instruction data.

[0029] In one specific embodiment, the process may include the following steps: First, construct the instruction fine-tuning dataset. Optionally, the instruction fine-tuning dataset adopts the Alpaca dialogue format and can cover at least one of the following four core instruction data categories: Audiology single-turn dialogue data: can come from the ChatMed-TCM audiology subset and real clinical consultation data, focusing on scenarios such as hearing loss assessment, hearing aid fitting, and rehabilitation training guidance.

[0030] Audiology multi-turn dialogue data: You can select the multi-turn question and answer dataset of the audiology department in CMtMedQA, which includes a complete interactive process such as medical history collection, symptom in-depth exploration, and treatment plan adjustment.

[0031] Audiology NLP task instruction data: It can convert tasks such as hearing condition classification (referred to as hearing classification, such as hearing loss type classification), rehabilitation plan recommendation, and teaching knowledge point questioning into instruction format, thereby enhancing the model's adaptability to multiple types of audiological tasks.

[0032] Teaching interaction instruction data: Includes interaction data for three roles: “Teacher Assistant”, “Student Simulator”, and “Patient Simulator”, covering teaching scenarios such as simulated consultation, case discussion, and practical assessment.

[0033] Then, the basic audiology model is supervisedly fine-tuned using the instruction fine-tuning dataset. Optionally, LoRA fine-tuning technology can be used to fine-tune only the parameters of the model's adaptation layer, preserving the core capabilities of the basic model and reducing training and deployment costs. In one specific implementation, taking the hearing classification task in audiology NLP tasks as an example, the instruction data for the hearing classification task includes symptom information (such as symptom description text, or audiograms, or symptom description text + audiograms) and hearing classification labels (such as hearing loss type labels). The audiograms are mapped to a representation that the model can process through a pre-trained feature extraction module. The symptom information and hearing classification labels of the same patient can constitute a hearing classification sample. In supervised fine-tuning, symptom information from the samples can be input into the pre-trained large model. Hearing classification instructions are used to instruct the pre-trained large model to classify the patient's hearing condition based on the symptom information and output the confidence score for each classification type. The classification type with the highest confidence score and its corresponding confidence score represent the model's classification result and its confidence score. The parameters of the pre-trained large model can be fine-tuned based on the difference between the model's classification result and the actual classification, resulting in a large model with hearing classification capabilities. In subsequent embodiments, unless otherwise specified, the model's classification result and confidence score refer to the final output of the classification type with the highest confidence score and its corresponding confidence score.

[0034] Furthermore, large-scale models can leverage retrieval-enhanced generation techniques to connect audiological knowledge graphs with the latest rehabilitation guidelines in real time, ensuring the accuracy of responses.

[0035] Furthermore, in another specific implementation, to enable the large model to possess effective proactive inquiry capabilities, this embodiment also constructs a proactive inquiry module, which, together with the fine-tuned large model, constitutes the final audiology vertical domain large model. Taking the hearing classification task as an example, the proactive inquiry module is used to perform the following operations: Step 1: If the confidence level of a large model with hearing classification ability based on a certain symptom information is less than a set threshold, extract the historical symptom information from historical hearing classification tasks with a classification confidence level greater than or equal to the threshold and extract the self-attention heatmaps of each layer in the large model.

[0036] After fine-tuning in S140, the large model now possesses hearing classification capabilities. When new symptom information to be identified is input into the large model, it can output a hearing classification result and confidence level based on that information. However, if the confidence level of the classification result is too low (e.g., below 0.5), the classification may be inaccurate. In this embodiment, an active inquiry module is used to improve this issue. The active inquiry module operates based on a preset large model attention analysis strategy, guiding the large model to continue active inquiry to improve classification confidence. Optionally, historical classification tasks with high classification confidence (e.g., above 0.9) can be extracted from historical data. For ease of distinction and description, the new symptom information to be identified is referred to as the current symptom information, and the symptom information extracted from the tasks is referred to as historical symptom information. Each historical classification task corresponds to one historical symptom information (these historical symptom information can all obtain high classification confidence). The similarity between each historical symptom information and the current symptom information is calculated. The historical symptom information with the highest similarity is selected, and the self-attention heatmap and cross-modal attention heatmap output by each layer of the large model when the historical symptom information is input into the large model for classification are extracted as reference information for subsequent active queries.

[0037] For example, taking the Qianwen Multimodal Large Model as an example, the Transformer layer of this model consists of pairs of self-attention layers and cross-attention layers. From each Transformer layer, self-attention heatmaps and cross-attention heatmaps (also known as cross-modal attention heatmaps) can be extracted. The self-attention heatmaps include text self-attention heatmaps and image self-attention heatmaps. The text self-attention heatmap corresponds to the self-attention distribution of each token in the input text; the diagonal elements correspond to the degree of attention the algorithm pays to each token itself, and the other elements represent the degree of association between each pair of tokens. The image self-attention heatmap corresponds to the self-attention distribution of each patch in the input image; the diagonal elements correspond to the degree of attention the algorithm pays to each patch itself, and the other elements represent the degree of association between each pair of patches. The cross-modal attention heatmap corresponds to the cross-modal attention distribution between each token in the input text and each patch in the input image; each element represents the degree of association between each pair of tokens and the path. Attention weights are used to characterize the degree of attention the model pays to the corresponding input features during the current inference process. Generally, the higher the attention level within the same layer, the more important the corresponding token, patch, or token-patch information becomes in classification. Optionally, attention heatmaps for each layer of a large model can be extracted using directives such as `output_attentions=True` in the model's native class.

[0038] In practical applications, it is not necessary to store all historical data. For each classification type, some computational data from historical tasks with classification confidence exceeding a threshold can be pre-accumulated. The embedded representations of historical symptom information under each classification are then clustered, and the embedded representations of the historical symptom information closest to each cluster center, along with the attention heatmaps at each layer, are stored. These embedded representations can be used to calculate the similarity to subsequent symptom information to be classified, while these attention heatmaps serve as reference information for subsequent proactive inquiries.

[0039] Step 2: Based on the high-level self-attention map, determine the global self-attention hotspots of the historical symptom information, instruct the large model to actively inquire about the entities and relationships in the global self-attention hotspots, and reclassify the patient's hearing condition based on the supplementary information provided in response to the inquiries.

[0040] The greater the model depth, the higher the number of layers. Taking the layer distribution of Qianwen 3-VL (7B / 14B) as an example, layers 0-8 can be considered low layers, and layers 8 and above can be considered high layers. Other models can be divided into low and high layers as needed. In this embodiment, the global self-attention hotspots in the historical symptom information are first determined based on the high-level self-attention map. These hotspots cover a relatively large range of text or image information and often correspond to important global information, such as the core backbone information of the world, i.e., the logical relationships between these information. If the text or image information corresponding to these global hotspots is missing in the current symptom information, it is very likely to affect the classification confidence. Therefore, the large model can be guided to actively query these information.

[0041] Optionally, taking the symptom description text in the symptom information as an example, the matrix dimensions in the text self-attention heatmap extracted in the previous step correspond to the tokens in the referenced historical symptom description text. Therefore, token fragments can be deduced from the heatmap distribution. A certain high-level self-attention map can be selected, and regions in the map whose self-attention is higher than the average level of the entire map by a set multiple (e.g., 2 times, 3 times), whose historical symptom information is covered by more than a set proportion of the total information (e.g., more than 50% of the number of tokens), and whose self-attention is stable across layers (i.e., the self-attention of the tokens covered in this region is higher than the average level of the entire map by a certain multiple in several consecutive layers). The text fragments corresponding to these regions are the global self-attention hotspots. For example, the historical symptom text information used as a reference is: hearing loss in the left ear for 3 weeks, accompanied by a feeling of fullness in the ear and persistent, low-pitched tinnitus, without ear pain or purulent discharge; pure-tone audiometry showed an air conduction threshold of 45 dBHL and a bone conduction threshold of 10 dBHL in the left ear, resulting in an air-bone conduction difference of 35 dBHL; all hearing indicators in the right ear were normal; tympanic endoscopy revealed a small central perforation in the left tympanic membrane with smooth edges. Therefore, the global self-attention hotspots can be the three text fragments: "air-bone conduction difference 35 dBHL," "small central perforation in the left tympanic membrane," and "air conduction threshold of 45 dBHL and bone conduction threshold of 10 dBHL in the left ear." Each of these three fragments, as well as the fragments themselves and each other, has a relatively high self-attention weight.

[0042] Then, prompts can be used to instruct the large model to continue querying for missing content based on the relationships between entities in the global self-attention hotspot. For example, if the correlation between the three text fragments "air-bone conduction difference 35 dBHL," "small central perforation of the left tympanic membrane," and "left ear air conduction hearing threshold 45 dBHL, bone conduction hearing threshold 10 dBHL" in the global self-attention hotspot is relatively high, the following prompt can be input into the large model: "Based on historical queries, the correlation between the three text fragments 'air-bone conduction difference 35 dBHL,' 'small central perforation of the left tympanic membrane,' and 'left ear air conduction hearing threshold 45 dBHL, bone conduction hearing threshold 10 dBHL' is relatively high in hearing loss classification. Please supplement the query with the missing content in the current symptom description based on these correlations." After analysis, if the large model finds that the current symptom text information contains air-bone conduction difference but lacks features such as unilateral tympanic membrane morphology and unilateral hearing threshold, it might proactively query as follows: "Please describe the unilateral air conduction and bone conduction hearing threshold detection results, as well as the tympanic membrane mirror examination results."

[0043] When prompted by the large model, the user provides additional information, such as "there is a large difference between air conduction and bone conduction hearing thresholds, and there is also tympanic membrane lesion." The large model then reclassifies the information based on the additional information and context.

[0044] Similarly, the same operation can be performed on symptom images in the symptom information, using high-level self-attention heatmaps of the image modality to guide the large model to actively query.

[0045] Step 3: Based on the reclassification results and the position of the supplementary content in the reclassified self-attention heatmap, instruct the large model to continue active querying. Optionally, this embodiment can set the following two conditions based on the reclassification results and the position of the supplementary content in the reclassified self-attention heatmap: Condition 1: The supplementary content is a hotspot in the reclassified self-attention heatmap; Condition 2: The confidence level of the reclassification is greater than or equal to the threshold.

[0046] Based on conditions one and two, the reclassification can be divided into the following four cases, each of which can guide the large model row to adopt different proactive query strategies: If both Condition 1 and Condition 2 are met simultaneously, meaning the supplemented content is a hotspot in the reclassified self-attention heatmap and the confidence level of the reclassification is greater than or equal to the threshold, then Condition 1 indicates that the supplemented content is valid and has received sufficient attention in the reclassification, or has provided important verification information; Condition 2 indicates that the confidence level of the reclassification has met the requirements. Therefore, in this case, the hearing classification task ends, and no further follow-up questions are needed.

[0047] Scenario 2: Condition 1 is met, but Condition 2 is not. That is, the supplemented content is a hotspot in the reclassified self-attention heatmap, but the confidence level of the reclassification is still below the threshold. This indicates that the supplemented content is effective, has received sufficient attention in the reclassification, or provides important verification information, but its improvement on classification confidence is limited, requiring further inquiry. In this case, we can look for other modal information that is a hotspot in the reclassified self-attention heatmap but is not a hotspot in the cross-modal attention heatmap of the supplemented content. Specifically, we can look for other modal information that is a hotspot in the cross-modal attention heatmap of the referenced historical symptom information but is not a hotspot in the cross-modal attention heatmap of the supplemented content. It should be noted that in this embodiment, each modality has its own self-attention heatmap; therefore, the self-attention heatmap mentioned in the previous sentence refers to the self-attention heatmap corresponding to the other modal information. Taking image-text multimodal information as an example, if the inquiry in step 2 is for text information, then the other modalities here refer to image modal information. The information found in this step has high self-attention, but the correlation with the supplementary content is not fully explored, leading to ambiguity or errors in macroscopic logic. This is also a significant reason for low classification confidence. Therefore, the large model can be instructed to actively query based on the correlation between the other modal information and the supplementary content. For example, if the supplementary text is "small central perforation of the left ear tympanic membrane," and the found image patch is the tympanic membrane region in an otoscope image, the large model can be prompted: the correspondence between the tympanic membrane region image and "small central perforation of the left ear tympanic membrane" is unclear; please continue to ask questions to clarify this relationship. After analysis, if the large model finds an inconsistency, it may actively query as follows: "The current otoscope image is inconsistent with the state of the left ear tympanic membrane in the symptom description. Please confirm whether the input otoscope image is an otoscope image of the left ear or the right ear," to guide the user to provide more accurate information.

[0048] Similarly, the same operation can be performed on symptom images in symptom information, using cross-modal attention heatmaps to guide large models to actively ask questions.

[0049] Case 3: Conditions 1 and 2 are not met, meaning the supplemented content is not a hotspot in the reclassified self-attention heatmap, and the confidence level of the reclassification is still below the threshold. This indicates that the supplemented content is not effective and does not help the classification confidence level. It is very likely that the core information and logic focused on by the global attention hotspots are insufficient for correct classification, and more detailed information needs to be mined. In this case, the local attention hotspots of the historical symptom information can be determined based on the low-level self-attention graph corresponding to the referenced historical symptom information, and the large model can be instructed to actively query the missing content based on the entities and attributes in the local attention hotspots. Taking text information as an example, local self-attention focuses on the importance and correlation of local tokens, and the tokens covered are relatively short, often appearing as local sharp peaks in the low-level self-attention heatmap. Therefore, the maximum points in the graph can be determined first, and then, with each maximum point as the center, the peak areas with a peak ratio greater than a certain threshold and a coverage width within a certain range can be determined as sharp peak areas. The historical symptom information corresponding to these areas are the local self-attention hotspots. Optionally, using the above example, the historical symptom text information for reference is: hearing loss in the left ear for 3 weeks, accompanied by ear fullness, low-pitched persistent tinnitus, without ear pain or purulent discharge; pure-tone audiometry showed a left ear air conduction threshold of 45 dBHL, bone conduction threshold of 10 dBHL, and air-bone conduction difference of 35 dBHL, while the right ear hearing indicators were normal; tympanic endoscopy revealed a small central perforation in the left tympanic membrane with smooth edges. Local attention hotspots can include text fragments such as "air-bone conduction difference," "35 dBHL," "low-pitched," "tinnitus," "central small perforation," "smooth edges," "left ear air conduction threshold," "45 dBHL," "bone conduction threshold," and "10 dBHL," among which the correlation between two local hotspots belonging to the same short phrase is relatively high. Therefore, the following prompt can be input into the large model: In the historical question {historical symptom text information}, the local attention hotspots include {the above local attention hotspots}. Please analyze which entity and attribute information is still missing in the current symptom description based on this information. After large-scale model analysis, the following proactive questions were asked: "Please tell me the specific values ​​of your left ear air conduction hearing threshold and bone conduction hearing threshold, do you have tinnitus, and is the pitch of the tinnitus high or low?" When prompted by the large model, the user provides additional information, such as "the air conduction hearing threshold in the left ear is xx dBHL, the bone conduction hearing threshold is xx dBHL, there is tinnitus, and the tinnitus pitch is high." The large model then reclassifies the audio based on the additional information and the context.

[0050] Similarly, the same operation can be performed on symptom images in the symptom information, using low-level attention heatmaps of the image modality to guide the large model to actively ask questions.

[0051] Scenario 4: Condition 1 is not met, but Condition 2 is met. That is, the supplementary content is not a hotspot in the reclassified self-attention heatmap, but the reclassification confidence is greater than or equal to the threshold. This indicates that the supplementary content has not received much attention within its own modality, and it is likely that the classification confidence has been improved through combination with information from other modalities. In this case, we can look for other modal information in the reclassified cross-modal attention heatmap that are hotspots in cross-modal attention with the supplementary content, and instruct the large model to pay special attention to the association between the supplementary content and this other modal information in subsequent tasks. Specifically, in this case, since the self-attention of the supplementary content is not significant, its cross-modal attention relationship is also easily overlooked. Therefore, we can prompt the large model to pay special attention to mining the association between the two when the content and this other modal information appear simultaneously in subsequent tasks, in order to obtain effective information.

[0052] Optionally, in the above four cases, the layer from which each attention heatmap in the reclassification originates can be consistent with the layer from which the corresponding heatmap was extracted in the original classification. That is, the self-attention heatmap in the original classification originates from the same layer as the self-attention heatmap in the reclassification. The layer of cross-modal attention heatmaps is similar. It can also be adjusted according to specific circumstances.

[0053] Of course, only one, two, or three of the above four scenarios can be retained; this embodiment does not impose specific limitations. Optionally, after scenarios two and three are executed, the supplemented complete symptom information can be treated as a new question for reclassification. The active inquiry module then re-extracts the historical symptom information most similar to the new question and re-executes the subsequent processes within the module. This process is repeated until a set number of times is reached, or the classification confidence can no longer be improved.

[0054] Furthermore, the above method uses hyperparameters such as setting multiples and ratios when judging global hotspots, and hyperparameters such as peak ratio and coverage width when judging local hotspots. The specific values ​​of these hyperparameters can be determined manually or optimized using the grid search method. The optimization goal is to minimize the number of active queries required to finally satisfy the confidence requirement for classification, that is, to minimize the number of active queries required to finally satisfy the confidence threshold condition in each classification task on average.

[0055] Furthermore, the aforementioned active classification task can be replaced with other tasks, and the classification confidence can be replaced with the accuracy requirements of other tasks, thereby extending the above active inquiry method to other tasks of large models.

[0056] Furthermore, in another specific implementation, in step one, historical symptom information from historical hearing classification tasks with classification confidence greater than or equal to the threshold can be extracted into self-attention heatmaps and cross-modal attention heatmaps at each layer of the large model. A hotspot knowledge graph is then constructed based on the entities, attributes, and relationships within the hotspots in each attention heatmap, ensuring that the hotspot knowledge graph covers as much of the historical symptom information as possible, maintaining the comprehensiveness of the hotspot knowledge. Optionally, the method described in the above embodiments can be used to determine global attention hotspots (including global self-attention hotspots and global cross-modal attention hotspots) and local attention hotspots (i.e., local self-attention hotspots). Knowledge (tokens or patches) with high self-attention inherent in each global attention hotspot is used as nodes in the hotspot knowledge graph. If the relationships between these knowledge points also belong to global attention hotspots, an edge is constructed between them. Simultaneously, the specific attributes of each knowledge point in the local self-attention hotspot are used as the attributes of the nodes, and the modality from which each node originates is also labeled.

[0057] Accordingly, in the subsequent second step, the large model can be instructed to actively query for missing content based on the entities and relationships in the hotspot knowledge graph. For example, the following prompt can be used: "Based on historical queries, the global hotspot knowledge subgraph for the current hearing classification task includes {a subgraph consisting of nodes belonging to the same modality as the currently processed modality in the complete hotspot knowledge graph and the edges between these nodes}. Please supplement the query for missing content in the current symptom description based on the entities and relationships in this subgraph. Note that this supplementary query focuses on entities and the relationships between entities, without considering the specific attributes of the entities." Correspondingly, in the subsequent step three, scenario three, the large model can also be instructed to actively query for missing content based on the entities and attributes in the hotspot knowledge graph. For example, the following prompt can be used: "Based on historical queries, the local hotspot knowledge sub-graph for the current hearing classification task includes {a sub-graph composed of nodes in the complete hotspot knowledge graph that belong to the same modality as the currently processed modality and have specific attributes, and the edges between these hotspots}. Please analyze which entities and which attribute information are still missing in the current symptom description based on this information." The remaining steps are similar to the aforementioned specific implementation method. The hotspot knowledge graph referenced in this implementation method provides more comprehensive coverage of hotspot information for hearing classification tasks, facilitating the completion of more information at once. This is suitable for tasks with relatively fixed classification criteria (i.e., the information / indicators used for each classification are relatively fixed). In contrast, the aforementioned specific implementation method references hotspot information from historical symptom descriptions that are closer to the current symptom description. Since this hotspot information has been proven to have high confidence, it can maintain a smaller volume while ensuring confidence (sufficient symptom information is added each time, without needing to add the most complete symptom information each time). This is suitable for tasks with more flexible classification criteria (i.e., the information used for each classification can be different, as long as effective results are obtained). In practical applications, the appropriate implementation method can be selected according to the specific circumstances.

[0058] In summary, this embodiment provides a method for constructing large models in the vertical field of audiology, which can achieve at least one of the following beneficial effects: 1. Enhances the professionalism of audiological interaction. By deeply integrating multimodal audiological data with professional knowledge graphs, it accurately achieves hearing loss classification and personalized rehabilitation plan recommendations, with a professional accuracy rate superior to general large models; 2. Able to conduct effective proactive questioning. Based on the attention heatmap, identify key information to ensure the effective completion of the task, and proactively ask questions based on this key information. Prioritize proactive questioning from the global backbone information and global logic, and adopt different follow-up questioning strategies based on the attention weight of supplementary content and the questioning effect. Differentiate the direction of follow-up questioning to detailed information in this modality or cross-modal information to further improve the effectiveness of proactive questioning and improve the quality of task completion; 3. Adaptable to core teaching scenarios: Through multi-role intelligent agent collaboration, it supports closed-loop teaching processes such as simulated consultation, practical guidance, and assessment feedback, solving the pain points of single interaction and scarce cases in traditional audiology teaching; 4. Flexible deployment: It adopts a lightweight pre-training and LoRA fine-tuning strategy, which is suitable for low computing power environments in universities, training bases, and primary rehabilitation institutions, and supports plug-in integration with existing teaching and medical systems.

[0059] 5. Wide range of applications: It takes into account the needs of clinical hearing assessment, rehabilitation guidance and standardized teaching, and can cover scenarios such as medical student training, resident physician training and primary care physician capacity building.

[0060] It should be noted that all user data involved in this application is information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0061] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 2 As shown, the device includes a processor 60, a memory 61, an input device 62, and an output device 63; the number of processors 60 in the device can be one or more. Figure 2 Taking a processor 60 as an example; the processor 60, memory 61, input device 62, and output device 63 in the device can be connected via a bus or other means. Figure 2 Taking the example of a connection between China and Israel via a bus.

[0062] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the large model construction method for the vertical field of audiology in this embodiment of the invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 61, thereby realizing the aforementioned large model construction method for the vertical field of audiology.

[0063] The memory 61 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 61 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include memory remotely located relative to the processor 60, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0064] Input device 62 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 63 may include display devices such as a display screen.

[0065] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a large model construction method for the vertical field of audiology according to any embodiment.

[0066] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0067] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0068] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0069] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing large-scale models for the vertical field of audiology, characterized in that, include: Obtain an audiological multimodal dataset, wherein the multimodal dataset includes at least one or more of text data and hearing test image data; Using the aforementioned multimodal dataset, the basic large model is pre-trained for audiological enhancement. The pre-trained large model was fine-tuned using the instruction fine-tuning dataset to obtain a large model for the audiology vertical domain with interactive capabilities. The instruction fine-tuning dataset includes at least one of audiology single-turn dialogue data, audiology multi-turn dialogue data, audiology NLP task instruction data, and teaching interaction instruction data.

2. The method according to claim 1, characterized in that, The audiology NLP task instruction data includes hearing classification instructions; Accordingly, the fine-tuning of the pre-trained large model using the instruction fine-tuning dataset includes: The symptom information from the hearing classification samples is input into the pre-trained large model, and the hearing classification instructions are used to instruct the pre-trained large model to classify the patient's hearing condition into at least one predefined hearing category based on the symptom information. Based on the difference between the model's classification results and the actual classification, the pre-trained large model is fine-tuned to obtain a large model with hearing classification capabilities.

3. The method according to claim 1, characterized in that, After fine-tuning the pre-trained large model using the instruction fine-tuning dataset to obtain a large audiology vertical domain model with interactive capabilities, the following steps are also included: An active inquiry module is introduced as part of a larger model in the vertical field of audiology, wherein the active inquiry module is used for: If the confidence level of a large model with hearing classification ability based on certain symptom information is less than a set threshold, extract the self-attention heatmap of each layer of the large model from historical symptom information in historical hearing classification tasks with a classification confidence level greater than or equal to the threshold. Based on the high-level self-attention map, the global self-attention hotspots of the historical symptom information are determined, and the large model is instructed to actively ask questions based on the entities and relationships in the global self-attention hotspots, and to reclassify the patient's hearing condition based on the supplementary content in response to the questions. Based on the reclassification results and the inclusion of the supplementary information in the reclassified self-attention heatmap, the large model is instructed to initiate active queries.

4. The method according to claim 3, characterized in that, Based on the reclassification results and the inclusion of the supplementary content in the reclassified self-attention heatmap, the large model is instructed to actively query, including: If both conditions one and two are met, stop actively querying. If condition one is met but condition two is not met, search for other modal information that is a hotspot in the reclassified self-attention heatmap but is a non-hotspot in the cross-modal attention heatmap related to the supplementary content, and instruct the large model to actively query based on the association between the other modal information and the supplementary content. If neither condition one nor condition two is met, the local attention hotspots of the historical symptom information are determined based on the low-level self-attention map, and the large model is instructed to actively query based on the entities and attributes in the local attention hotspots. If condition one is not met but condition two is met, look for other modal information that belongs to the hotspot of cross-modal attention of the supplementary content in the reclassified cross-modal attention heatmap, and instruct the large model to pay attention to the correlation between the other modal information and the supplementary content in subsequent tasks. Condition one is that the supplementary content is a hotspot in the reclassified self-attention heatmap, and condition two is that the confidence level of the reclassification is greater than or equal to the threshold.

5. The method according to claim 3, characterized in that, The step of determining the global self-attention hotspots of the historical symptom information based on the high-level self-attention map includes: In a high-level self-attention map, if the self-attention of a certain region is higher than the average level by a set multiple, and the historical symptom information it covers exceeds the set proportion of the total, and the self-attention is stable across layers, then the historical symptom information fragment corresponding to that region is determined as a global self-attention hotspot.

6. The method according to claim 5, characterized in that, Before determining the global self-attention hotspots of the historical symptom information based on the high-level self-attention map, the method further includes: The values ​​of the set multiple and set ratio are determined by using a grid search method. The search objective is to minimize the number of active queries required to ultimately meet the confidence threshold in each classification task on average.

7. The method according to claim 1, characterized in that, The process of using the multimodal dataset to perform audiological domain enhancement pre-training on the basic large model includes at least one of the following steps: Through text pre-training tasks, the basic large model learns audiological terminology, rules for interpreting test indicators, and logic for adapting rehabilitation plans. By introducing audiogram feature extraction and text association tasks, the basic large model's ability to interpret hearing test data is enhanced, and the logical connection between "audiogram - pathological diagnosis - rehabilitation suggestions" is realized. Pre-training with teaching cases enables students to learn the teaching logic of audiology from a basic, large-scale model.

8. The method according to claim 7, characterized in that, The introduction of audiogram feature extraction and text association tasks enhances the basic large-scale model's ability to interpret hearing test data, realizing the logical connection between "audiogram - pathological diagnosis - rehabilitation suggestions," including: Based on the multimodal dataset, a multimodal sample including audiograms, hearing test data, pathological diagnosis text, and rehabilitation suggestion text is constructed; Based on the retrieval enhancement generation technology, the basic large model is instructed to retrieve relevant knowledge of pathological diagnosis, and the pathological diagnosis text is given according to the relevant knowledge, as well as the audiogram and hearing test data; Adjust the parameters of the base model based on the differences between the model output and the pathological diagnosis texts in the multimodal samples; Based on the retrieval enhancement generation technology, the basic large model is instructed to retrieve relevant knowledge of rehabilitation suggestions, and based on the relevant knowledge, as well as the audiogram, hearing test data and pathological diagnosis text, rehabilitation suggestion text is given; Adjust the parameters of the base model based on the differences between the model output and the rehabilitation suggestion text in the multimodal samples.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the large model construction method for the vertical field of audiology as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the large model construction method for the vertical field of audiology as described in any one of claims 1-8.

Images