Model decision-making method and device, electronic equipment and program product

Abstract

The embodiment of the invention discloses a model decision method and device, electronic equipment and a program product. The method comprises the following steps: acquiring one or more target literatures; extracting key information related to the health analysis model from each target document; based on a model evaluation rule, according to the key information corresponding to each target document, evaluating the health analysis model in each target document; determining one or more candidate health analysis models according to the evaluation result corresponding to each health analysis model; on the basis of the verification data set, verifying the model performance of each candidate health analysis model to obtain first performance information corresponding to each candidate health analysis model; and determining a target health analysis model according to the first performance information corresponding to each candidate health analysis model. In the embodiment of the invention, the subjectivity of manual selection can be avoided, the model performance and reliability of the decided health analysis model are ensured, and the screening efficiency of the health analysis model is improved.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, specifically to a model-based decision-making method, apparatus, electronic device, and program product. Background Technology

[0002] In fields such as healthcare and bioinformatics, the development and application of health analysis models has become a research hotspot. Currently, health analysis models typically rely on subjective selection by humans, which cannot guarantee the reliability of the applied health analysis models. Summary of the Invention

[0003] This application discloses a model decision-making method, apparatus, electronic device, and program product, which can avoid the subjectivity of manual selection, ensure the model performance and reliability of the health analysis model, and improve the screening efficiency of the health analysis model.

[0004] This application discloses a model-based decision-making method applied to electronic devices, the method comprising: Obtain one or more target documents, which include health analysis models; Extract key information related to the health analysis model from each of the target documents; Based on the model evaluation rules, the health analysis models in each of the target documents are evaluated according to the key information corresponding to each target document, and the evaluation results corresponding to each health analysis model are obtained. Based on the evaluation results corresponding to each of the health analysis models, one or more candidate health analysis models are determined. Based on the validation dataset, the model performance of each candidate health analysis model is validated, and the first performance information corresponding to each candidate health analysis model is obtained. Based on the first performance information corresponding to each candidate health analysis model, a target health analysis model is determined from the one or more candidate health analysis models.

[0005] This application discloses a model decision-making device, the device comprising: The document acquisition module is used to acquire one or more target documents, including health analysis models. The information extraction module is used to extract key information related to the health analysis model from each of the target documents; The evaluation module is used to evaluate the health analysis models in each of the target documents based on the model evaluation rules and the key information corresponding to each target document, and to obtain the evaluation results corresponding to each health analysis model. The evaluation module is also used to determine one or more candidate health analysis models based on the evaluation results corresponding to each health analysis model. The verification module is used to verify the model performance of each candidate health analysis model based on the verification dataset, and obtain the first performance information corresponding to each candidate health analysis model. The decision module is used to determine the target health analysis model from the one or more candidate health analysis models based on the first performance information corresponding to each candidate health analysis model.

[0006] This application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method described above.

[0007] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to implement the method described above.

[0008] This application discloses a computer program product, including a computer program, which, when executed by a processor, causes the processor to implement the method described above.

[0009] The model decision-making method, apparatus, electronic device, and program product disclosed in this application acquire one or more target documents, extract key information related to health analysis models from each target document, evaluate the health analysis models in each target document based on model evaluation rules and the key information corresponding to each target document, obtain evaluation results for each health analysis model, determine one or more candidate health analysis models based on the evaluation results, verify the model performance of each candidate health analysis model based on a validation dataset, obtain first performance information for each candidate health analysis model, and determine the target health analysis model from the one or more candidate health analysis models based on the first performance information. In this application embodiment, the model evaluation of health analysis models in each target document is performed using model evaluation rules executable by an electronic device, which avoids the subjectivity and randomness of manual selection, ensuring the quality of the selected candidate health analysis models. Furthermore, the validation dataset is used to validate the candidate health analysis models, combining theoretical rules with model validation, ensuring the model performance and reliability of the determined target health analysis model, and improving the efficiency of health analysis model selection. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a diagram illustrating an application scenario of the model decision-making method in one embodiment; Figure 2 Here is a flowchart of the model decision-making method in one embodiment; Figure 3 This is a flowchart illustrating the evaluation of a health analysis model in one embodiment; Figure 4 Here is a flowchart of the model decision-making method in another embodiment; Figure 5 This is a flowchart illustrating a decision target health analysis model in one embodiment; Figure 6 This is a flowchart illustrating the decision target health analysis model in another embodiment; Figure 7 This is a flowchart illustrating the generation of a validation dataset in one embodiment; Figure 8 This is a block diagram of a model decision-making device in one embodiment; Figure 9 This is a structural block diagram of an electronic device in one embodiment. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0014] It is understood that the terms "first," "second," etc., used in this application may be used to describe various elements herein, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of this application, a first evaluation rule may be referred to as a second evaluation rule, and similarly, a second evaluation rule may be referred to as a first evaluation rule. Both the first evaluation rule and the second evaluation rule are model evaluation rules, but they are not the same model evaluation rule. The term "multiple" as used in this application refers to two or more. The term "and / or" as used in this application refers to one of the solutions, or any combination of multiple solutions.

[0015] Currently, most health analysis models used in applications are sourced from scattered sources, relying on the model research capabilities and knowledge reserves of the R&D team. Furthermore, the selection of health analysis models for applications largely depends on human decision-making, which involves subjectivity and chance. This results in low efficiency in the screening of health analysis models and severely limits the accuracy and reliability of health analysis models used in practical applications.

[0016] This application discloses a model decision-making method, apparatus, electronic device, and program product, which can avoid the subjectivity of manual selection, ensure the model performance and reliability of the health analysis model, improve the screening efficiency of health analysis models, and realize standardized and automated model decision-making schemes, which is conducive to the promotion and use of decision-making schemes and improves the applicability of the schemes.

[0017] Figure 1 This is a diagram illustrating an application scenario of the model decision-making method in one embodiment. For example... Figure 1 As shown, electronic device 110 may include, but is not limited to, mobile phones, wearable devices (such as smartwatches, smart glasses, etc.), tablets, laptops, PCs (Personal Computers), or servers.

[0018] Electronic device 110 can communicate with one or more database servers 120, and electronic device 110 can access the databases in database servers 120, retrieve and download the required documents from the databases.

[0019] A model-based decision-making system can be constructed and can run in electronic device 110.

[0020] In some embodiments, the model decision system may include a database construction module. The database construction module may be used to communicate with one or more database servers 120 and retrieve one or more target documents from the one or more database servers 120 using constructed search queries.

[0021] As an optional implementation, the database building module can periodically perform retrieval tasks, periodically detecting target documents from the database server 120, thereby building a dynamically updated and comprehensive knowledge base.

[0022] In some embodiments, the model decision-making system may include a model screening and evaluation module. This module may have a pre-defined set of model evaluation rules based on clinical epidemiology, statistics, and evidence-based medicine. The module can screen and evaluate health analysis models from one or more target documents based on these rules, thereby identifying one or more candidate health analysis models.

[0023] As an optional implementation, the model screening and evaluation module can score each health analysis model in one or more target documents based on model evaluation rules, and rank each health analysis model according to the score. For example, the higher the score, the better the model performance, so the health analysis models can be ranked in descending order of score to select candidate health analysis models.

[0024] As an alternative implementation, the model evaluation rules used to screen and evaluate health analysis models may differ for different health research topics, and / or the scores of the model evaluation rules used to screen and evaluate health analysis models may differ, thereby meeting the differentiated needs of different health research topics and improving the accuracy of model screening and evaluation.

[0025] In some embodiments, the model decision system may include an external model validation and selection module. This module may access one or more independent external validation databases, obtain validation datasets from these databases, validate the performance of each candidate health analysis model based on these datasets, and select the optimal champion model, i.e., the target health analysis model, based on the validated performance of each candidate health analysis model.

[0026] As an optional implementation, the model external validation and optimization module can use automated scripts to batch calculate the performance indicators of multiple candidate health analysis models on the validation dataset, compare the performance indicators of multiple candidate health analysis models on the validation dataset, and thus decide on the target health analysis model.

[0027] As an optional implementation, the model external validation and optimization module can also output a model report, which provides various parameters of the target health analysis model, such as model parameters and performance indicators.

[0028] Electronic device 110 acquires one or more target documents, extracts key information related to the health analysis model from each target document, and evaluates the health analysis model in each target document based on the model evaluation rules and the key information corresponding to each target document, obtaining the evaluation results for each health analysis model. Based on the evaluation results for each health analysis model, electronic device 110 can determine one or more candidate health analysis models, and then verify the model performance of each candidate health analysis model based on a validation dataset, obtaining the first performance information corresponding to each candidate health analysis model. Based on the first performance information corresponding to each candidate health analysis model, electronic device 110 can determine the target health analysis model from the one or more candidate health analysis models.

[0029] It should be noted that the model decision-making method disclosed in this application can be executed on a single electronic device 110 or by multiple electronic devices 110 working together. For example, the process of evaluating health analysis models in various target documents based on model evaluation rules to obtain one or more candidate health analysis models, and the process of validating the model performance of each candidate health analysis model based on a validation dataset to determine the target health analysis model, can be implemented separately on different electronic devices 110. This application does not strictly limit the executing entity of the overall solution.

[0030] like Figure 2 As shown, in some embodiments, a model decision-making method is provided, which can be applied to the above-mentioned electronic device. The method includes the following steps: Step 210: Obtain one or more target documents, which include health analysis models.

[0031] In some embodiments, the electronic device may access one or more databases, which may be databases used to store documents such as papers, lab reports, patents, and academic conference proceedings. The electronic device may retrieve one or more target documents from the databases.

[0032] The target literature can be literature that studies health analysis models. Health analysis models refer to models used to analyze users' physical health status. For example, health analysis models may include, but are not limited to, disease prediction models (such as diabetes prediction models, hypertension prediction models, breast cancer prediction models, etc.), medical diagnostic models (such as disease diagnosis models based on medical images, disease diagnosis models based on medical samples, disease diagnosis models based on gene fragments, etc.), medical data analysis models (such as examination report analysis models, blood test report analysis models, gene test report analysis models, and imaging analysis models), and analysis models of user vital sign data (such as pulse data, heart rate data, exercise data, respiratory data, and / or sleep data, etc.).

[0033] It should be noted that health analysis models can be used not only for analyzing the physical health of humans, but also for analyzing the health of other living organisms (such as animals). A target document may contain one or more health analysis models.

[0034] The types of health analysis models may include, but are not limited to, one or more of the following: AI (Artificial Intelligence) models, ML (Machine Learning Model) models, statistical models, and simulation models.

[0035] In some embodiments, an electronic device may retrieve one or more target documents from a database based on a target search expression. The target search expression refers to an expression used to retrieve target documents from a database, and may consist of one or more of search terms, logical operators, and restrictive characters.

[0036] For example, the search term can be keywords related to the target health research topic, and / or keywords related to health analysis models, etc. Logical operators may include, but are not limited to, one or more of the following: "AND," "OR," and "NOT." "AND" is used to detect documents that simultaneously contain the connected search terms; "OR" is used to detect documents that contain either of the connected search terms; and "NOT" is used to retrieve documents that do not contain a certain search term, etc. Restrictors can be used to limit the scope of the retrieved documents. For example, they can be used to limit the time frame of the document's publication, the geographical scope of the research team or publication medium (such as journals or newspapers), the document category, and the position of the search term in the document (such as title, abstract, body text, or author fields).

[0037] Optionally, a search strategy can be pre-defined, and a target search query can be constructed based on the search strategy. This search strategy can be used to define the conditions that the target documents to be retrieved must meet. For example, it can define one or more of the following: the target health research topic (e.g., diabetes research topic, hypertension research topic, cardiovascular disease research topic, etc.), the publication time of the target documents, the document category of the target documents, and the research team of the target documents, but is not limited to these.

[0038] Based on the retrieval strategy, a target search formula can be constructed to retrieve target documents that meet the retrieval strategy. For example, if the retrieval strategy defines the target health research topic corresponding to the target documents to be retrieved, then a search formula containing the keywords of the target health research topic can be constructed; if the retrieval strategy defines the publication time (or time range) corresponding to the target documents to be retrieved, then a search formula containing the restriction symbols corresponding to the publication time (or time range) can be constructed, and so on.

[0039] In some embodiments, a target search formula corresponding to a specific health research topic can be constructed. This target health research topic may refer to a health research topic that researchers are interested in or need to apply. Based on the target search formula corresponding to the target health research topic, one or more target documents related to the target health research topic can be retrieved from the database. By evaluating the health analysis models in each target document related to the target health research topic, candidate health analysis models that match the target health research topic can be selected. Different target search formulas can be constructed for different target health research topics, thereby improving the accuracy and efficiency of literature retrieval.

[0040] In the above implementation, one or more target documents can be retrieved from the database based on the target search formula, thereby improving the accuracy and efficiency of the target document retrieval.

[0041] In some embodiments, multiple different search methods can be defined to trigger the electronic device to retrieve target documents. For example, one or more of the following search methods can be used: (1) Actively triggering the search method.

[0042] Researchers (referring to those who retrieve literature) can proactively trigger electronic devices to detect target documents as needed. Furthermore, researchers can adjust the target search formula each time they proactively trigger a search, or they can continue to use a previously used target search formula. The electronic device can respond to the triggered search command and retrieve one or more target documents from the database based on the target search formula.

[0043] (2) Periodically triggering the search.

[0044] Electronic devices can periodically perform retrieval tasks, retrieving one or more target documents from a database based on a preset retrieval cycle and the target search query. This retrieval cycle can be pre-set, for example, one month, 20 days, one week, or two months, but is not limited to these. The retrieval cycle can be set according to actual needs. Optionally, different retrieval cycles can be set for different health research topics. For example, a shorter retrieval cycle can be set for health research topics where health analysis models are updated rapidly, and a longer retrieval cycle can be set for health research topics where health analysis models are updated slowly, thus adapting to the speed of technological updates in different health research topics.

[0045] In some embodiments, the electronic device may retrieve one or more target documents published within the current retrieval period from the database based on the target search query, according to a retrieval period. The current retrieval period may refer to the time since the last retrieval, and the duration of the current retrieval period is equal to the duration of a preset retrieval period. For example, if the preset retrieval period is one month, the current retrieval period may refer to one month after the last retrieval.

[0046] Furthermore, based on the time of the last search and the preset search period, the time range corresponding to the current search period can be determined. A target search formula can then be constructed based on this time range. This target search formula can then be used to retrieve one or more target documents from the database whose publication date falls within the current search period. For example, if the last search was on March 1, 2025, and the preset search period is 20 days, then the current search date is March 21, 2025. The time range corresponding to the current search period is from March 2, 2025 to March 21, 2025. Target documents whose publication date falls within this time range can be detected using the target search formula.

[0047] In the above implementation, periodically retrieving target literature from the database facilitates the construction of a dynamically updated and comprehensive knowledge base. This ensures the timeliness and effectiveness of the subsequently acquired health analysis models, further improving the reliability of the resulting health analysis models. Furthermore, when periodically retrieving target literature, only those published within the current retrieval period can be searched, improving retrieval efficiency and avoiding duplicate searches.

[0048] Step 220: Extract key information related to the health analysis model from each target document.

[0049] Electronic devices can identify each acquired target document and extract key information related to the health analysis model from each document. Key information refers to information in the target document that is related to the health analysis model. For example, key information may include, but is not limited to, one or more of the following: model name, development cohort, study type, sample size, predictor factors, model type, and performance indicators.

[0050] The development queue refers to the dataset used to develop (or train) health analysis models.

[0051] Research type can refer to the design and methodology of a study, describing how the study is conducted. For example, research types may include, but are not limited to, one or more of the following: prospective study (a study method that recruits participants at the beginning of the study and follows them up for a period of time to observe the results), retrospective study (a study method that uses existing data for analysis), randomized controlled trial (a study method that randomly assigns participants to different treatment groups to evaluate the effects of different measures), cross-sectional study (a study method that collects data at a specific point in time to describe the characteristics of the participants), and cohort study (a study method that tracks a specific group of participants over a long period of time to observe the relationship between their exposure factors and outcomes).

[0052] Sample size refers to the number of samples contained in the dataset used to develop and / or validate the model.

[0053] Predictors are variables used by health analysis models to determine outcomes; they are also called features. Predictors are typically used as inputs to health analysis models.

[0054] Model type can refer to categories of health analysis, such as linear regression, random forest, Cox proportional hazards regression model, neural network, or support vector machine (SVM) model types.

[0055] Performance metrics can refer to quantitative standards used to evaluate the performance of a health analysis model. These performance metrics may include, but are not limited to, one or more of the following: AUC (Area Under Curve, the area under the ROC curve and the coordinate axis), sensitivity, specificity, C-index, calibration slope, accuracy, and precision.

[0056] The AUC value is the area under the ROC (Receiver Operating Characteristic Curve) curve and the coordinate axis. The ROC curve is a two-dimensional curve with the false positive rate (FPR) on the horizontal axis and the true positive rate (TPR) on the vertical axis.

[0057] Sensitivity (also known as the true positive rate) refers to the proportion of samples that a health analysis model correctly predicts as positive out of the actual positive samples. It is used to measure the health analysis model's ability to detect positive samples.

[0058] Specificity (also known as the true negative rate) refers to the proportion of samples that a health analysis model correctly predicts as negative out of the actual negative samples. It is used to measure the health analysis model's ability to detect negative samples.

[0059] The C-index is a metric used to evaluate the performance of survival prediction models. It measures the model's ability to rank survival times, representing the probability that the model correctly predicts the order of survival times of two individuals.

[0060] The calibration slope is used to evaluate the linear relationship between the predicted results and actual values ​​output by the health analysis model, and is used to measure the deviation between the predicted results and actual values ​​of the health analysis model.

[0061] Accuracy refers to the proportion of samples correctly predicted by a health analysis model out of the total number of samples.

[0062] Precision rate refers to the proportion of samples that a health analysis model predicts to be positive, but which are actually positive.

[0063] Furthermore, the electronic device can employ natural language processing (NLP) technology to identify the acquired target documents and extract key information relevant to the health analysis model from each document. For example, key information can be extracted through a NLP model by inputting each target document into the model, analyzing its content, and extracting key information relevant to the health analysis model. This NLP model can include, but is not limited to, models based on the Transformer architecture, models based on the CNN (Convolutional Neural Network) architecture, models based on the RNN (Recurrent Neural Network), and Large Language Models (LLM), but is not limited to these.

[0064] In some embodiments, after the electronic device extracts key information related to the health analysis model from various target documents, it can generate structured data based on the key information, which can improve the efficiency of subsequent evaluation of the health analysis model using model evaluation rules.

[0065] Step 230: Based on the model evaluation rules, evaluate the health analysis models in each target document according to the key information corresponding to each target document, and obtain the evaluation results corresponding to each health analysis model.

[0066] Model evaluation rules refer to the standards used to evaluate health analysis models. Multiple model evaluation rules can be pre-set. For example, methods used to evaluate research results in one or more systematic research methods such as clinical epidemiology, statistics, evidence-based medicine, and meta-analysis can be transformed into model evaluation rules that can be automatically executed by computers. Furthermore, the methods and experiences used by experts in various fields to evaluate research results can be extracted and transformed into model evaluation rules that can be automatically executed by computers. This enables the constructed model evaluation rules to have expert decision-making capabilities and to more accurately select reliable health analysis models.

[0067] Electronic devices can evaluate health analysis models in various target documents from multiple perspectives based on model evaluation rules and key information corresponding to each target document. Based on this key information, it can determine whether the health analysis models in each target document conform to the model evaluation rules, thereby obtaining the evaluation results for each health analysis model. For example, the evaluation results for a health analysis model can characterize whether the health analysis model conforms to all model evaluation rules. Furthermore, the evaluation results can also characterize the number of model evaluation rules that the health analysis model conforms to and / or the number of model evaluation rules that it does not conform to.

[0068] Step 240: Based on the evaluation results corresponding to each health analysis model, determine one or more candidate health analysis models.

[0069] Electronic devices can determine one or more candidate health analysis models from multiple health analysis models based on the evaluation results corresponding to each health analysis model. The candidate health analysis models can serve as alternatives to the health analysis models used for decision-making applications, and the target health analysis model for the application can be determined from the candidate health analysis models.

[0070] As one implementation method, health analysis models that meet all model evaluation rules can be selected as candidate health analysis models based on the evaluation results corresponding to each health analysis model.

[0071] As one implementation method, the health analysis models can be sorted according to the number of model evaluation rules they meet, from most to least, based on the evaluation results corresponding to each health analysis model. The top N health analysis models are then selected as candidate health analysis models, where N can be a positive integer.

[0072] As one implementation method, the health analysis models can be sorted according to the number of model evaluation rules that the health analysis models do not meet, from the smallest to the largest, based on the evaluation results corresponding to each health analysis model, and the top N health analysis models can be selected as candidate health analysis models.

[0073] In one implementation approach, the evaluation results for each health analysis model may include an evaluation score for that model. For each model's evaluation rules, a corresponding score can be set. The electronic device can accumulate the scores of each model's compliance with the evaluation rules to obtain the evaluation score for that health analysis model. The health analysis models can be sorted from highest to lowest evaluation score, and the top N models can be selected as candidate health analysis models.

[0074] Step 250: Based on the validation dataset, validate the model performance of each candidate health analysis model and obtain the first performance information corresponding to each candidate health analysis model.

[0075] In related technologies, when researchers select health analysis models for application, they rely solely on data from target literature and often directly apply models that perform well on development datasets. However, when these models are deployed to different real-world application environments, their performance may significantly decline, indicating insufficient generalization ability.

[0076] In this embodiment, the electronic device can use a validation dataset to validate various candidate health analysis models. The electronic device can acquire the validation dataset, which can be data collected with user consent from a real-world application environment, or data from publicly available databases such as UKB (UK Biobank) and CKB (China Kadoorie Biobank). Based on the validation dataset, the model performance of each candidate health analysis model can be validated, obtaining the first performance information corresponding to each candidate health analysis model.

[0077] The first performance information may refer to the performance metrics of the candidate health analysis model determined based on the validation dataset. The first performance information may include, but is not limited to, one or more of the following: AUC value, sensitivity, specificity, C-index, calibration slope, accuracy, and precision of the candidate health analysis model, determined based on the validation dataset.

[0078] For example, multiple sample data from the validation dataset can be input into the candidate health model. The candidate health model processes and analyzes the input sample data to obtain the prediction results output by the candidate health model. Based on the prediction results output by the candidate health model and the actual results corresponding to the sample data, the first performance information corresponding to the candidate health model is determined.

[0079] Optionally, different validation datasets can be constructed for different target health research topics. Furthermore, a different validation dataset can be constructed based on the dataset used in the target literature for the candidate health analysis model (such as the development and / or validation dataset used in the research phase), thereby further improving the accuracy of performance validation of the candidate health model.

[0080] Step 260: Determine the target health analysis model from one or more candidate health analysis models based on the first performance information corresponding to each candidate health analysis model.

[0081] When multiple candidate health analysis models exist, the first performance information corresponding to each candidate health analysis model can be compared, and the target health analysis model can be determined based on the comparison results.

[0082] As an optional implementation, a target performance indicator can be identified. Based on a validation dataset, the indicator value corresponding to each candidate health analysis model and the target performance indicator can be determined. Multiple candidate health analysis models can be compared with their respective indicator values ​​to select the candidate health analysis model with the optimal indicator value as the target health analysis model. For example, the Brier score can be used as the target performance indicator. The Brier score can be used to evaluate the calibration of a probabilistic prediction model. It measures the mean squared error between the probability of an event related to the target health research topic predicted by the health analysis model and the actual outcome (e.g., 1 for actual occurrence, 0 for actual non-occurrence). Based on the validation dataset, the Brier score corresponding to each candidate health analysis model can be determined, and the candidate health analysis model with the lowest Brier score can be selected as the target health analysis model. Alternatively, the AUC value can be used as the target performance indicator. Based on the validation dataset, the AUC value corresponding to each candidate health analysis model can be determined, and the candidate health analysis model with the highest AUC value can be selected as the target health analysis model.

[0083] As one implementation method, the first performance information corresponding to the candidate health analysis model may include the index values ​​of multiple performance indicators. Then, a weighted calculation (such as weighted sum calculation, weighted average calculation, etc.) can be performed on each candidate health analysis model and the index values ​​corresponding to the multiple performance indicators to obtain the performance score corresponding to each candidate health analysis model. The candidate health analysis model with the highest performance score is selected as the target health analysis model in descending order of performance score.

[0084] Utilizing external validation datasets to verify the performance of candidate health analysis models can connect the model development and model validation stages, thus forming a complete closed-loop feedback system of "retrieval-evaluation-validation-selection". This ensures that the selected target health analysis model can be tested in practice, rather than just having high theoretical model quality, thereby improving the reliability and credibility of the results output by the target health analysis model.

[0085] In some embodiments, the performance of candidate health analysis models can be combined with business requirements to jointly determine the target health analysis model for application. These business requirements may refer to the needs of the real-world application environment in which the health analysis model is actually used, such as the need to apply the health analysis model to wearable devices or to hospital terminals.

[0086] Electronic devices can determine the degree of matching between each candidate health analysis model and business requirements. This degree of matching refers to the adaptability of the candidate health analysis model to the business requirements. A higher degree of matching makes the candidate health analysis model easier to apply in the required real-world application environment; conversely, a lower degree of matching makes it more difficult to apply the candidate health analysis model in the required real-world application environment and increases the likelihood of problems after application. Electronic devices can determine the target health analysis model based on the degree of matching between each candidate health analysis model and the business requirements, as well as the corresponding primary performance information.

[0087] For example, the first candidate health analysis model can be selected from the first performance information corresponding to each candidate health analysis model, where the target performance indicator value is better than the indicator threshold or the performance score is greater than the score threshold. The multiple first candidate health analysis models are then sorted in descending order of their matching degree with business requirements, and the first candidate health analysis model with the highest matching degree with business requirements is selected as the target candidate health analysis model.

[0088] For example, a first weight corresponding to the matching degree with business requirements and a second weight corresponding to the first performance information can be assigned. Based on the first weight, the second weight, the matching degree between each candidate health analysis model and business requirements and the first performance information corresponding to each candidate health analysis model, the priority of each candidate health analysis model is determined, and the candidate health analysis model with the highest priority is selected as the target health analysis model.

[0089] In the above implementation, by comprehensively considering the first performance information of each candidate health analysis model and its matching degree with business requirements, the performance and quality of the target health analysis model are guaranteed, as well as its adaptability in the real application environment, thus meeting the actual business needs and improving the feasibility of the target health analysis model.

[0090] In this embodiment of the application, the health analysis models of each target document are evaluated using model evaluation rules executable by electronic devices. This avoids the subjectivity and randomness of manual selection, ensures the quality of the selected candidate health analysis models, and uses a validation dataset to validate the candidate health analysis models. By combining theoretical rules with model validation, the model performance and reliability of the selected target health analysis models are guaranteed, and the screening efficiency of health analysis models is improved.

[0091] In some embodiments, model evaluation rules may include evaluation rules of one or more rule categories. These rules can be used to evaluate health analysis models from multiple aspects, thereby selecting high-quality health analysis models. For example... Figure 3 As shown, in some embodiments, the steps are based on model evaluation rules, and the health analysis models in each first target document are evaluated according to the key information corresponding to each first target document to obtain the evaluation results corresponding to each health analysis model. This may include the following steps: Step 302: Based on the key information corresponding to each target document, select the second target document that contains key information corresponding to one or more rule categories respectively.

[0092] In some embodiments, the model evaluation rules may include evaluation rules from one or more of the following rule categories: I. First evaluation rule related to the research subjects.

[0093] The first evaluation rule is used to assess whether the research subjects used in developing (or training, validating, etc.) a health analysis model are appropriate. For example, the first evaluation rule may include, but is not limited to, one or more of the following: (1) Whether the health analysis model uses an appropriate dataset. For example, health analysis models using prospective and retrospective development cohort studies can be given priority, while health analysis models using randomized controlled trials, nested case-control studies, and cross-sectional studies can be temporarily disregarded because they are based on subjects with known health conditions. If the health analysis model uses an appropriate dataset, it meets this evaluation rule.

[0094] (2) Whether the inclusion and exclusion criteria of the research subjects in the health analysis model are reasonable. Inclusion criteria refer to the conditions that the included research subjects must meet, and exclusion criteria refer to the conditions that the excluded research subjects must meet. This assessment can evaluate whether the inclusion criteria of the research subjects corresponding to each health analysis model can represent the target group, and can also determine whether the exclusion criteria of the research subjects corresponding to each health analysis model are too strict or too lenient, thereby ensuring the reasonableness of the dataset used in the health analysis model. If the inclusion and exclusion criteria of the research subjects in the health analysis model are reasonable, then it meets the evaluation rules.

[0095] For example, in a health analysis model for hypertension research, it can be determined whether the age of the research subjects in the dataset is within the target range (e.g., whether they are older than 18 years old) and whether people with serious diseases have been excluded.

[0096] II. Second evaluation rule related to predictors in health analysis models.

[0097] The second evaluation rule is used to assess whether the predictive factors of the health analysis model are reasonable. For example, the second evaluation rule may include, but is not limited to, one or more of the following: (1) Whether all subjects in the health analysis model have defined and / or evaluated predictors using similar methods. Each predictor in the health analysis model should have a clear definition and evaluation method (also known as a measurement method or data collection method). It is necessary to ensure that all subjects in the health analysis model use the same evaluation method to obtain data on the same predictor, and that the definition of the same predictor is the same. If all subjects in the health analysis model have defined and / or evaluated predictors using similar methods, then this evaluation rule is met.

[0098] For example, if a health analysis model analyzes health status based on gene sequencing data, and its predictive factor is a target gene biomarker, then all research subjects need to use the same or similar gene detection methods to detect the target gene biomarker.

[0099] (2) Whether the evaluation of predictive factors of the health analysis model was conducted without knowledge of clinical outcome data. The evaluation of predictive factors of the health analysis model must be conducted without knowledge of clinical outcome data to ensure the objectivity of the study and avoid bias. If the evaluation of predictive factors of the health analysis model was conducted without knowledge of clinical outcome data, it complies with this evaluation rule.

[0100] (3) When applying the health analysis model, can the predictive factors of the health analysis model be obtained?

[0101] In some real-world applications, the availability of predictive factors for health analysis models may be limited. For example, if a health analysis model is applied to wearable devices and its predictive factors include genetic markers, these markers may not be available. Alternatively, it's necessary to ensure that all predictive factors for the health analysis model are available before the application date. If all predictive factors for the health analysis model are available at the time of application, it meets the evaluation criteria. This ensures the feasibility of the chosen health analysis model and allows for cost control.

[0102] (4) Whether the ratio between the sample size and the number of predictors in the health analysis model is reasonable. If the ratio between the sample size and the number of predictors in the health analysis model is too small, it may lead to overfitting of the health analysis model. Therefore, it can be determined whether the ratio between the sample size and the number of predictors in the health analysis model is greater than the preset value, such as whether it is greater than 10:1, 20:1, etc. If the ratio between the sample size and the number of predictors in the health analysis model is reasonable, it meets the evaluation rule.

[0103] III. Third assessment rules related to clinical outcomes.

[0104] The third assessment rule can be used to evaluate the effectiveness and / or utility of the clinical outcomes of a health analysis model. For example, the third assessment rule may include, but is not limited to, one or more of the following: (1) Whether the clinical outcome of the health analysis model is clearly defined. The clinical outcome of the health analysis model must have a clear and operational definition to ensure that the developers and researchers have a consistent understanding of the clinical outcome. If the clinical outcome of the health analysis model is clearly defined, it meets the evaluation rule.

[0105] (2) Whether similar methods are used to assess and determine clinical outcomes for all subjects in the health analysis model. For all subjects in the health analysis model, a unified assessment and determination method must be used to determine clinical outcomes. For example, assessment should be conducted uniformly using imaging examinations, laboratory tests, and medical records. It should be noted that different assessment and determination methods may be used for different types of clinical outcomes, but for the same type of clinical outcome, all subjects in the health analysis model must use the same or similar assessment and determination methods. If all subjects in the health analysis model use similar methods to assess and determine clinical outcomes, then this assessment rule is met.

[0106] (3) Whether the predictors of the health analysis model have been excluded from the clinical outcome. Since the predictors of the health analysis model are usually obtained before the clinical outcome occurs, they need to be excluded from the clinical outcome to avoid bias and overfitting in the health analysis model. If the predictors of the health analysis model are not present in the clinical outcome, then this assessment rule is met.

[0107] (4) Whether the clinical outcomes of the health analysis model were determined without knowledge of the predictors. During the development of a health analysis model, it is necessary to ensure that the study subjects and / or researchers assess the clinical outcomes of the model without knowledge of the predictors. This avoids affecting the clinical outcomes and improves their reliability. If the clinical outcomes of the health analysis model were determined without knowledge of the predictors, then this assessment rule applies.

[0108] (5) Whether the time interval between the measurement time of the predictor and the assessment time of the clinical outcome is appropriate. It can be determined whether the time interval between the measurement time of the information corresponding to the predictor in the health analysis model and the assessment time of the clinical outcome is within the set time range. For example, the measurement time should be within 3-6 months before the assessment time, or the measurement time should be within 1 month before the assessment time, thus assessing the reasonableness of the time interval. If the time interval corresponding to the health analysis model is appropriate, it meets the assessment rules.

[0109] IV. The fourth evaluation rule related to data analysis.

[0110] The fourth evaluation rule can be used to assess the reasonableness of the data analysis process related to the health analysis model in the target literature. For example, the fourth evaluation rule may include, but is not limited to, one or more of the following: (1) Whether the number of study subjects with specific clinical outcome events in the target literature is reasonable. Study subjects with specific clinical outcome events refer to those who, during the study, experienced a specific clinical outcome event. This specific clinical outcome event can be a predefined clinical outcome with clear criteria. It can be determined whether the number of study subjects with specific clinical outcome events exceeds a quantitative threshold, and / or whether the proportion of study subjects with specific clinical outcome events to the sample size exceeds a proportional threshold. This avoids overfitting of the health analysis model due to an insufficient number of study subjects with specific clinical outcome events or an excessively small proportion of study subjects with specific clinical outcome events to the sample size. If the number of study subjects with specific clinical outcome events is reasonable, it meets the evaluation rules.

[0111] (2) Whether the missing data in the research subjects was handled appropriately. It can be determined whether the missing data in the target literature was handled, for example, whether the missing data was deleted and only the samples without missing data were used to develop the health analysis model; or, whether the missing data was filled using methods such as mean imputation or multiple imputation; or, whether the missing data was handled using methods such as maximizing the likelihood function or Bayesian algorithms. Proper handling of missing data can ensure the accuracy of the research results of the health analysis model. If the missing data in the research subjects was handled appropriately, it meets the evaluation rules.

[0112] (3) Whether the continuous and categorical predictors used in the health analysis model have been properly processed. Continuous predictors refer to variables that take values ​​within a certain range, such as age, blood pressure, and blood glucose levels; categorical predictors refer to variables that take a finite number of discrete values, such as gender and disease stage. Different data analysis methods can be used for continuous and categorical predictors respectively. For example, continuous predictors can be processed using normal distribution analysis and grouping; categorical predictors can be processed using label coding and one-hot coding. If the continuous and categorical predictors used in the health analysis model have been properly processed, then it meets the evaluation rules.

[0113] (4) Whether data from all subjects in the health analysis model were included in the data analysis. It is necessary to ensure that data from all subjects in the health analysis model were included in the data analysis to guarantee the completeness of the research results. If data from all subjects in the health analysis model were included in the data analysis, then it meets this evaluation rule.

[0114] (5) Whether univariate analysis was avoided in screening predictors for the health analysis model. This can be assessed by determining whether the research process for the health analysis model considered only the independent relationship between each predictor and the clinical outcome, neglecting the interactions between multiple predictors. For example, it can be determined whether predictors were screened based on statistically significant indicators, ignoring important predictors; or whether multivariate regression analysis or other analytical methods were used to consider the impact of multiple predictors on the clinical outcome. If univariate analysis was not used to screen predictors for the health analysis model, then this assessment rule is met.

[0115] (6) Whether the predictors and their weights in the health analysis model are consistent with the results of the multivariate analysis. Multivariate analysis is used to simultaneously assess the impact of multiple predictors on clinical outcomes. When selecting predictors for the health analysis model, it is necessary to ensure that they are consistent with the predictors identified in the multivariate analysis. The weight of each predictor can be determined through multivariate analysis, such as by determining the weight of each predictor through the regression coefficients in multivariate regression analysis. If the predictors and their weights in the health analysis model are consistent with the results of the multivariate analysis, then they meet the evaluation rules and can ensure the accuracy and reliability of the health analysis model.

[0116] (7) Whether the complex issues arising in the dataset have been appropriately considered and addressed, such as censored data, competing risk data, and sampling of control group subjects. Censored data refers to incomplete clinical outcomes for some subjects during the study. It can be determined whether censored data has been handled using semi-parametric models, multiple imputation, or other processing methods. Competing risk data refers to the possibility that the occurrence of one event during the study may prevent the occurrence of other events. It can be determined whether competing risk data has been handled using cumulative occurrence functions, regression models for handling competing risks, or other methods. Sampling of control group subjects refers to the method used to select the control group during the study. It can be determined whether appropriate sampling methods such as random sampling or matched sampling have been used to select the control group. If the complex issues arising in the dataset have been appropriately considered and addressed during the research process of the health analysis model, then it meets the evaluation rules.

[0117] (8) Whether the performance metrics of the health analysis model were appropriately evaluated. This can be determined by whether a dataset was used to evaluate the model's performance during the research process, for example, by using a developed dataset to evaluate the model's performance. Optionally, it can also be determined whether the performance metrics used in the health analysis model are appropriate. For example, for a health analysis model belonging to the classification category, specificity and AUC values ​​can be used; for a health analysis model belonging to the survival category, C-index and calibration slope can be used. Optionally, it can be determined whether multiple performance metrics were used to evaluate the model's performance to avoid the limitations of a single performance metric. If the performance metrics of the health analysis model were appropriately evaluated, then this evaluation rule is met.

[0118] (9) Whether the overfitting and performance bias of the health analysis model have been appropriately addressed. It can be determined whether appropriate measures have been taken to reduce overfitting, such as using regularization methods or validating the model with a new dataset. It can also be determined whether performance bias has been corrected, for example, by validating the model with a new dataset to reduce existing performance bias, or by adjusting the model performance based on a calibration curve. If the overfitting and performance bias of the health analysis model have been appropriately addressed, then the evaluation rule is met.

[0119] It should be noted that the model evaluation rules used for health analysis models on different health research topics can be the same or different; and / or, the scores corresponding to the same model evaluation rule used for health analysis models on different health research topics can be the same or different. This is to meet the differentiated needs of different health research topics for health analysis models.

[0120] In the embodiments of this application, a comprehensive evaluation of the health analysis model can be carried out through evaluation rules related to multiple aspects such as research subjects, predictive factors, clinical outcomes and data analysis. This can accurately identify whether the health analysis model has a risk of bias, exclude health analysis models with a risk of bias, and ensure the model performance and reliability of the health analysis model that is decided.

[0121] In some embodiments, before evaluating the health analysis model using model evaluation rules, the electronic device can first filter out second target documents containing key information corresponding to each rule category based on the key information corresponding to each target document. It can be determined whether the key information corresponding to each target document contains key information corresponding to each rule category. For example, it can be determined whether the key information corresponding to the target document contains key information corresponding to the research subjects, key information corresponding to predictors, key information corresponding to clinical outcomes, and / or key information corresponding to data analysis, thereby filtering out second target documents containing key information corresponding to each rule category. By eliminating target documents with incomplete key information, the number of documents subsequently evaluated for model evaluation can be reduced, further improving the efficiency of model evaluation.

[0122] Step 304: Based on the evaluation rules corresponding to each rule category, and according to the key information corresponding to each second target literature and each rule category, evaluate the health analysis model in each second target literature to obtain the evaluation results corresponding to each health analysis model.

[0123] Electronic devices can determine whether each secondary target document conforms to the evaluation rules corresponding to each rule category based on the key information corresponding to each secondary target document and each rule category, thereby obtaining the evaluation results corresponding to each health analysis model. For example, based on the key information corresponding to the research subjects, it can determine whether the secondary target document conforms to the first evaluation rule related to the research subjects; based on the key information corresponding to the predictors, it can determine whether the secondary target document conforms to the second evaluation rule related to the predictors of the health analysis model, and so on. These are not listed here.

[0124] For example, electronic devices can use evaluation tools such as PROBAST to assess the risk of bias in each health analysis model included in each second-target literature based on model evaluation rules and key information corresponding to each second-target literature, and determine whether each health analysis model conforms to the evaluation rules of each rule category. If a health analysis model does not conform to the evaluation rules, it can be determined that the health analysis model has a risk of bias, and a health analysis model without a risk of bias can be identified as a candidate health analysis model. That is, a candidate health analysis model can be a health analysis model that conforms to all model evaluation rules.

[0125] In this embodiment, second target documents containing key information corresponding to each rule category can be screened first, and then the health analysis model of the second target documents can be evaluated according to the evaluation rules corresponding to each rule category. This approach takes into account both model evaluation efficiency and model evaluation effect, providing a more efficient and accurate screening scheme for health analysis models.

[0126] like Figure 4 As shown, in some embodiments, a model decision-making method is provided, which can be applied to the above-mentioned electronic device. The method may include the following steps: Step 402: Obtain one or more target documents, which include health analysis models.

[0127] Step 404: Extract key information related to the health analysis model from each target document.

[0128] The descriptions of steps 402 to 404 can be found in the descriptions of steps 210 to 220 in the above embodiments, and will not be repeated here.

[0129] Step 406: Based on the key information corresponding to each target document, select the first target document that matches the target health research topic.

[0130] After extracting key information from each target document, it can be determined whether each target document matches the target health research topic based on the key information in each target document.

[0131] Furthermore, key information may include keywords in the title and / or abstract of the target documents. The keywords in the title and / or abstract of each target document can be matched against the target health research topic to determine whether the title and / or abstract of each target document contains keywords matching the target health research topic. If the title and / or abstract of a target document contains keywords matching the target health research topic, then that target document can be used as the first target document for subsequent model evaluation. If the title and / or abstract of a target document does not contain keywords matching the target health research topic, it indicates that the target document is not related to the target health research topic or has a weak correlation, and therefore that target document can be eliminated.

[0132] Step 408: Based on the model evaluation rules, evaluate the health analysis models in each first target literature according to the key information corresponding to each first target literature, and obtain the evaluation results corresponding to each health analysis model.

[0133] Electronic devices can evaluate the health analysis models in each primary target literature based on model evaluation rules and the key information corresponding to each primary target literature.

[0134] In some embodiments, the model evaluation rules may include evaluation rules for one or more rule categories. The electronic device may, based on the key information corresponding to each first target document, select second target documents from the first target documents that contain key information corresponding to each rule category. Then, based on the evaluation rules corresponding to each rule category, it may evaluate the health analysis models in each second target document according to the key information corresponding to each rule category, thereby obtaining the evaluation results corresponding to each health analysis model.

[0135] It should be noted that the specific methods for screening second target literature and evaluating the health analysis models in each second target literature according to the evaluation rules corresponding to each rule category can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0136] Step 410: Based on the evaluation results corresponding to each health analysis model, determine one or more candidate health analysis models.

[0137] In some embodiments, the key information extracted from the target literature includes second performance information corresponding to the health analysis model. The second performance information may refer to the performance indicators of the health analysis model disclosed in the target literature. Candidate health analysis models can be determined based on the evaluation results and the second performance information corresponding to each health analysis model. These candidate health analysis models can be those that meet all model evaluation rules and whose corresponding second performance information satisfies the performance requirements.

[0138] For example, a candidate health analysis model can be a health analysis model that meets all model evaluation rules and whose corresponding AUC value is greater than a preset AUC threshold (such as 0.7, 0.8, 0.75, etc.).

[0139] In the above implementation, both the bias risk and performance indicators of the candidate health analysis models are taken into account, ensuring that the selected candidate health analysis models are free from bias risk and exhibit discriminative power on the development dataset, thereby improving the model quality and robustness of the candidate health analysis models.

[0140] Step 412: Based on the validation dataset, validate the model performance of each candidate health analysis model and obtain the first performance information corresponding to each candidate health analysis model.

[0141] Step 414: Determine the target health analysis model from one or more candidate health analysis models based on the first performance information corresponding to each candidate health analysis model.

[0142] The descriptions of steps 412 to 414 can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0143] For example, Figure 5 This is a flowchart illustrating the decision-making objective health analysis model in one embodiment. For example... Figure 5 As shown, the electronic device constructs a target search query based on a retrieval strategy, retrieves multiple target documents from the database based on the target search query, extracts key information from each target document, and performs three rounds of screening. In the first round of screening, target documents whose titles and / or abstracts contain keywords matching the target health research topic are selected. In the second round of screening, the completeness of key information (i.e., whether it contains key information corresponding to each rule category) of the target documents obtained in the first round is checked, and target documents with complete key information are selected. Optionally, the target documents obtained in the second round can be downloaded locally or to a server. In the third round of screening, the health analysis models in the target documents obtained in the second round are precisely screened based on model evaluation rules to obtain candidate health analysis models that meet the model evaluation rules. Then, the candidate health analysis models are validated based on a validation dataset to select the optimal target health analysis model, which is then developed and applied. By conducting multiple rounds of screening of the retrieved target literature, the efficiency of screening health analysis models can be improved, as well as the performance and quality of the selected health analysis models. In addition, a standardized model screening process can be established, expanding the applicability of the solution and lowering the technical threshold for model screening.

[0144] It should be noted that a manual review process can be incorporated into the three rounds of screening described above. For example, after the first round of screening, each target document selected in the first round can be reviewed manually to ensure that the title and / or abstract of each target document matches the target health research topic; and / or, after the second round of screening, domain experts corresponding to the target health research topic can review the model evaluation rules to ensure that the selected model evaluation rules are reasonable. Combining automation and manual processes can save labor costs and reduce subjective screening, while further ensuring the quality of the health analysis model resulting from the decision.

[0145] In this embodiment of the application, before evaluating the health analysis model of each target document based on the model evaluation rules, target documents that do not match the target health research topic can be eliminated first, which improves both the accuracy of the subsequent screening of health analysis models and the screening efficiency.

[0146] In some embodiments, for different target health research topics, a model library corresponding to each target health research topic can be constructed. This model library can be used to store health analysis models that have been selected and match the target health research topic. After the electronic device selects one or more candidate health analysis models corresponding to the target health research topic, it can store the one or more candidate health analysis models into the model library corresponding to the target health research topic; and / or, after the electronic device determines the target health analysis model corresponding to the target health research topic, it can store the target health analysis model into the model library corresponding to the target health research topic.

[0147] For example, Figure 6 This is a flowchart illustrating the decision-making objective health analysis model in another embodiment. Figure 6 As shown, the electronic device can construct target search formulas for each health research topic based on search strategies corresponding to each health research topic. For example, it can construct a target search formula for health research topic 1 based on the search strategy for health research topic 1, and a target search formula for health research topic 2 based on the search strategy for health research topic 2, and so on. Based on the target search formulas for each health research topic, it can filter out target literature corresponding to each health research topic. After extracting key information from each target literature, it performs three rounds of filtering to obtain candidate health analysis models for each health research topic. These candidate health analysis models can be stored in a model library corresponding to each health research topic. Optionally, the electronic device can periodically execute search tasks and update the model library for each health research topic based on the latest retrieved target literature. For example, it can update the model parameters of the candidate health analysis models originally stored in the model library, and add or delete candidate health analysis models from the model library.

[0148] Researchers can select one or more health analysis models for a specific health research topic based on their actual needs. They can also perform external validation on the health analysis models corresponding to each of the one or more health research topics to determine the optimal target health analysis model for each of the one or more health research topics, and then develop and apply the target health analysis model.

[0149] In this embodiment, a model library corresponding to each target health research topic can be constructed, and all health analysis models entering the model library have undergone comprehensive evaluation and screening, ensuring the quality of the health analysis models in the model library. In addition, the acquired target literature can be updated, and the health analysis models in the model library can be dynamically updated according to the updated target literature, ensuring the timeliness and effectiveness of the model library. Furthermore, a standardized model screening and aggregation mechanism is provided, allowing different researchers to select the applicable health analysis models according to their actual needs, thereby improving the applicability and applicability of the solution.

[0150] In some embodiments, such as Figure 7 As shown, before the electronic device validates the model performance of each candidate health analysis model based on the validation dataset, the electronic device may also perform the following steps: Step 702: Obtain the initial dataset.

[0151] Step 704: Preprocess the data in the initial dataset to obtain the validation dataset.

[0152] The initial dataset can be data collected with user consent from a real application environment, or it can be a publicly available database. Electronic devices can preprocess the data in the initial dataset to obtain a validation dataset.

[0153] In some embodiments, the electronic device can analyze the features of datasets where each candidate health analysis model performs well. For example, it can learn the features of datasets where different health analysis models perform well through machine learning models. The electronic device can obtain an initial dataset that matches the features of each candidate health analysis model, thereby enabling intelligent verification of the dataset and improving the verification effect.

[0154] Optionally, the data in the initial dataset may be preprocessed, including but not limited to one or more of the following: deleting data that is irrelevant to the candidate health analysis model; deleting data that does not meet the validation criteria; deleting data with a missing data ratio greater than the ratio threshold; and completing data with missing data.

[0155] For example, data in the initial dataset that is not related to the predictors of the candidate health analysis model can be deleted; data that does not meet the inclusion criteria of the study subjects corresponding to the candidate health analysis model can be deleted; data with a missing proportion of more than 20% can be deleted; or multiple imputation or other completion processing can be performed on the missing data to ensure data integrity.

[0156] By preprocessing the data in the initial dataset to obtain the validation dataset, the accuracy of the first performance information obtained by validating each candidate health analysis model based on the validation dataset can be guaranteed, thereby further improving the accuracy and cross-sectional quality of the target health analysis model.

[0157] In some embodiments, the electronic device may generate a model report based on first performance information corresponding to each candidate health analysis model, the model report being used to indicate the target health analysis model.

[0158] The model report can compare the first performance information corresponding to each candidate health analysis model. For example, the model report can compare the indicator values ​​(obtained based on the validation dataset) of each candidate health analysis model with the target performance indicator, and select the candidate health analysis model with the best indicator value as the target health analysis model.

[0159] By outputting model reports and comparing the primary performance information of each candidate health analysis model within these reports, electronic devices make it easier for researchers to evaluate the performance of each candidate health analysis model and determine whether each model is suitable for application, thus improving the ease of model application.

[0160] In addition to using a single candidate health analysis model as the target health analysis model, electronic devices can also combine multiple candidate health analysis models to obtain the target health analysis model. The target health analysis model can be one of one or more candidate health analysis models obtained through screening; or it can be a combination of multiple candidate health analysis models.

[0161] Optionally, the electronic device can analyze the model performance of each candidate health analysis model according to actual business needs, such as data characteristics in the real application environment, so as to determine the candidate health analysis model that meets the business needs and has excellent performance. Multiple candidate health analysis models can be combined to obtain the target health analysis model, thereby improving the model performance and model quality of the target health analysis model of the application.

[0162] Furthermore, standardized application programming interfaces (APIs) can be developed to correspond to the entire model decision-making scheme, and can be connected with various systems (such as hospital management systems, electronic health record systems, etc.). With the user's permission, user data and prediction results output by the health analysis model can be collected, and the health analysis model in the model library can be iteratively updated using this data to improve the application effect of the health analysis model.

[0163] In this embodiment of the application, by combining theoretical rules with model validation, the model performance and reliability of the target health analysis model are guaranteed, and the screening efficiency of the health analysis model is improved.

[0164] In some embodiments, a model evaluation method is provided, including the following steps: Step 1: Obtain one or more target documents, which include health analysis models; Step 2: Extract key information related to the health analysis model from each target document; Step 3: Based on the model evaluation rules, evaluate the health analysis models in each target document according to the key information corresponding to each target document, and obtain the evaluation results corresponding to each health analysis model.

[0165] In some embodiments, the model evaluation method further includes: determining one or more candidate health analysis models based on the evaluation results corresponding to each health analysis model.

[0166] In some embodiments, step 3 further includes: selecting second target documents containing key information corresponding to one or more rule categories based on the key information corresponding to each target document; evaluating the health analysis model in each second target document based on the evaluation rules corresponding to each rule category and the key information corresponding to each rule category, and obtaining the evaluation results corresponding to each health analysis model.

[0167] It should be noted that the description of the model evaluation method provided in the embodiments of this application can be referred to the description of the model decision method provided in the above embodiments, and will not be repeated here.

[0168] In this embodiment of the application, the health analysis models of each target document are evaluated using model evaluation rules executable by electronic devices, which avoids the subjectivity and randomness of manual selection and ensures the quality of the candidate health analysis models obtained through screening.

[0169] In some embodiments, a model validation method is provided, including the following steps: Step 1: Based on the validation dataset, validate the model performance of one or more candidate health analysis models to obtain the first performance information corresponding to each candidate health analysis model.

[0170] Step 2: Based on the first performance information corresponding to each candidate health analysis model, determine the target health analysis model from one or more candidate health analysis models.

[0171] In some embodiments, prior to step 1, the model validation method further includes: obtaining an initial dataset; preprocessing the data in the initial dataset to obtain a validation dataset; Preprocessing includes one or more of the following: Remove data that is irrelevant to the candidate health analysis model; Delete data that does not meet the verification criteria; Delete data whose missing data ratio exceeds the ratio threshold; Complete any missing data.

[0172] In some embodiments, the model validation method further includes: generating a model report based on the first performance information corresponding to each candidate health analysis model, wherein the model report is used to indicate the target health analysis model.

[0173] It should be noted that the description of the model verification method provided in the embodiments of this application can be referred to the description of the model decision method provided in the above embodiments, and will not be repeated here.

[0174] In this embodiment of the application, the candidate health analysis model is validated using a validation dataset, which can ensure the model performance and reliability of the target health analysis model that is decided upon.

[0175] like Figure 8 As shown, in some embodiments, a model decision-making device 800 is provided, which includes: a literature acquisition module 810, an information extraction module 820, an evaluation module 830, a verification module 840, and a decision-making module 850.

[0176] The document acquisition module 810 is used to acquire one or more target documents, which include health analysis models.

[0177] The information extraction module 820 is used to extract key information related to the health analysis model from various target documents.

[0178] The evaluation module 830 is used to evaluate the health analysis models in each target document based on the model evaluation rules and the key information corresponding to each target document, and to obtain the evaluation results corresponding to each health analysis model.

[0179] The evaluation module 830 is also used to determine one or more candidate health analysis models based on the evaluation results corresponding to each health analysis model.

[0180] The verification module 840 is used to verify the model performance of each candidate health analysis model based on the verification dataset, and obtain the first performance information corresponding to each candidate health analysis model.

[0181] The decision module 850 is used to determine the target health analysis model from one or more candidate health analysis models based on the first performance information corresponding to each candidate health analysis model.

[0182] In some embodiments, the model decision-making device 800 further includes a screening module.

[0183] The filtering module is used to filter out the first target literature that matches the target health research topic based on the key information corresponding to each target literature.

[0184] The evaluation module 830 is also used to evaluate the health analysis models in each first target literature based on the model evaluation rules and the key information corresponding to each first target literature, and to obtain the evaluation results corresponding to each health analysis model.

[0185] In some embodiments, the model evaluation rules include evaluation rules of one or more rule categories; the evaluation module 830 includes a screening unit and an evaluation unit.

[0186] The filtering unit is used to filter out second target documents that contain key information corresponding to one or more rule categories based on the key information corresponding to each target document.

[0187] The evaluation unit is used to evaluate the health analysis models in each second-target literature based on the evaluation rules corresponding to each rule category and the key information corresponding to each rule category, and to obtain the evaluation results corresponding to each health analysis model.

[0188] In some embodiments, the model evaluation rules include evaluation rules from one or more of the following rule categories: The first evaluation rule relevant to the research subjects; Second evaluation rule related to predictors in health analysis models; The third assessment rule related to clinical outcomes; The fourth evaluation rule related to data analysis.

[0189] In some embodiments, key information includes second performance information corresponding to the health analysis model; a candidate health analysis model is a health analysis model that meets all model evaluation rules and whose corresponding second performance information satisfies performance requirements.

[0190] In some embodiments, the document acquisition module 810 is further configured to retrieve one or more target documents from the database based on the target search query.

[0191] In some embodiments, the model decision-making device 800 further includes a storage module.

[0192] The document retrieval module 810 is also used to retrieve one or more target documents related to the target health research topic from the database based on the target retrieval formula corresponding to the target health research topic.

[0193] The storage module is used to store one or more candidate health analysis models into the model library corresponding to the target health research topic; and / or to store the target health analysis model into the model library corresponding to the target health research topic.

[0194] In some embodiments, the document acquisition module 810 is further configured to retrieve one or more target documents published within the current retrieval period from the database based on the target search formula according to the retrieval period.

[0195] In some embodiments, the model decision-making device 800 further includes a preprocessing module.

[0196] The preprocessing module is used to obtain the initial dataset; preprocess the data in the initial dataset to obtain the validation dataset; wherein, the preprocessing includes one or more of the following: Remove data that is irrelevant to the candidate health analysis model; Delete data that does not meet the verification criteria; Delete data whose missing data ratio exceeds the ratio threshold; Complete any missing data.

[0197] In some embodiments, the model decision-making device 800 further includes a generation module.

[0198] The generation module is used to generate model reports based on the first performance information corresponding to each candidate health analysis model. The model reports are used to indicate the target health analysis model.

[0199] In some embodiments, the target health analysis model is one of one or more candidate health analysis models; or, the target health analysis model is a combination of multiple candidate health analysis models.

[0200] In this embodiment of the application, the health analysis models of each target document are evaluated using model evaluation rules executable by electronic devices. This avoids the subjectivity and randomness of manual selection, ensures the quality of the selected candidate health analysis models, and uses a validation dataset to validate the candidate health analysis models. By combining theoretical rules with model validation, the model performance and reliability of the selected target health analysis models are guaranteed, and the screening efficiency of health analysis models is improved.

[0201] Figure 9 This is a structural block diagram of an electronic device in one embodiment. For example... Figure 9As shown, the electronic device 900 may include one or more of the following components: a processor 910 and a memory 920 coupled to the processor 910, wherein the memory 920 may store one or more computer programs, and the one or more computer programs may be configured to, when executed by one or more processors 910, cause the electronic device 900 to perform the methods described in the above embodiments.

[0202] Processor 910 may include one or more processing cores. Processor 910 connects to various parts within the electronic device 900 using various interfaces and lines, and performs various functions and processes data of the electronic device 900 by running or executing instructions, programs, code sets, or instruction sets stored in memory 920, and by calling data stored in memory 920. Optionally, processor 910 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 910 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 910 and may be implemented separately using a communication chip.

[0203] The memory 920 may include random access memory (RAM) or read-only memory (ROM). The memory 920 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the electronic device 900 during use.

[0204] This application discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the methods described in the above embodiments.

[0205] This application discloses a computer program product, which includes a computer program, and when executed by a processor, causes the processor to implement the methods described in the above embodiments.

[0206] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, ROM, etc.

[0207] Any references to memory, storage, databases, or other media used herein may include non-volatile and / or volatile memory. Suitable non-volatile memory may include ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. By way of illustration and not limitation, RAM may take many forms, such as Static RAM (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus DRAM (RDRAM), and Direct Rambus DRAM (DRDRAM).

[0208] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0209] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A model-based decision-making method, characterized in that, Applied to electronic devices, the method includes: Obtain one or more target documents, which include health analysis models; Extract key information related to the health analysis model from each of the target documents; Based on the model evaluation rules, the health analysis models in each of the target documents are evaluated according to the key information corresponding to each target document, and the evaluation results corresponding to each health analysis model are obtained. Based on the evaluation results corresponding to each of the health analysis models, one or more candidate health analysis models are determined. Based on the validation dataset, the model performance of each candidate health analysis model is validated, and the first performance information corresponding to each candidate health analysis model is obtained. Based on the first performance information corresponding to each candidate health analysis model, a target health analysis model is determined from the one or more candidate health analysis models.

2. The method according to claim 1, characterized in that, Before evaluating the health analysis models in each of the target documents based on the model evaluation rules and the key information corresponding to each target document, the method further includes: Based on the key information corresponding to each of the target documents, the first target documents that match the target health research topic are selected. The model evaluation rules, based on the key information corresponding to each of the target documents, evaluate the health analysis models in each of the target documents to obtain the evaluation results corresponding to each health analysis model, including: Based on the model evaluation rules, the health analysis models in each of the first target documents are evaluated according to the key information corresponding to each of the first target documents, and the evaluation results corresponding to each health analysis model are obtained.

3. The method according to claim 1, characterized in that, The model evaluation rules include evaluation rules of one or more rule categories; based on the model evaluation rules, and according to the key information corresponding to each of the target documents, the health analysis models in each of the target documents are evaluated to obtain the evaluation results corresponding to each health analysis model, including: Based on the key information corresponding to each of the target documents, select second target documents that contain the key information corresponding to one or more rule categories respectively; Based on the evaluation rules corresponding to each rule category, and according to the key information corresponding to each second target document and each rule category, the health analysis model in each second target document is evaluated, and the evaluation results corresponding to each health analysis model are obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The model evaluation rules include evaluation rules of one or more of the following rule categories: The first evaluation rule relevant to the research subjects; Second evaluation rule related to predictors in health analysis models; The third assessment rule related to clinical outcomes; The fourth evaluation rule related to data analysis.

5. The method according to any one of claims 1 to 3, characterized in that, The key information includes the second performance information corresponding to the health analysis model; the candidate health analysis model is a health analysis model that meets all model evaluation rules and whose corresponding second performance information meets the performance requirements.

6. The method according to claim 1, characterized in that, The acquisition of one or more target documents includes: Based on the target search query, one or more target documents are retrieved from the database.

7. The method according to claim 5, characterized in that, The method of retrieving one or more target documents from a database based on a target search query includes: Based on the target search formula corresponding to the target health research topic, one or more target documents related to the target health research topic are retrieved from the database; The method further includes: The one or more candidate health analysis models are stored in the model library corresponding to the target health research topic; and / or the target health analysis model is stored in the model library corresponding to the target health research topic.

8. The method according to claim 6, characterized in that, The method of retrieving one or more target documents from a database based on a target search query includes: Based on the search cycle, one or more target documents published within the current search cycle are retrieved from the database using the target search query.

9. The method according to claim 1, characterized in that, Before validating the model performance of each candidate health analysis model based on the validation dataset, the method further includes: Obtain the initial dataset; The data in the initial dataset is preprocessed to obtain the validation dataset; The preprocessing includes one or more of the following: Remove data that is irrelevant to the candidate health analysis model; Delete data that does not meet the verification criteria; Delete data whose missing data ratio exceeds the ratio threshold; Complete any missing data.

10. The method according to claim 1, characterized in that, The method further includes: Based on the first performance information corresponding to each candidate health analysis model, a model report is generated, which is used to indicate the target health analysis model.

11. The method according to claim 1, characterized in that, The target health analysis model is one of the one or more candidate health analysis models; or, the target health analysis model is a combination of multiple candidate health analysis models.

12. A model decision-making device, characterized in that, The device includes: The document acquisition module is used to acquire one or more target documents, including health analysis models. The information extraction module is used to extract key information related to the health analysis model from each of the target documents; The evaluation module is used to evaluate the health analysis models in each of the target documents based on the model evaluation rules and the key information corresponding to each target document, and to obtain the evaluation results corresponding to each health analysis model. The evaluation module is also used to determine one or more candidate health analysis models based on the evaluation results corresponding to each health analysis model. The verification module is used to verify the model performance of each candidate health analysis model based on the verification dataset, and obtain the first performance information corresponding to each candidate health analysis model. The decision module is used to determine the target health analysis model from the one or more candidate health analysis models based on the first performance information corresponding to each candidate health analysis model.

13. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to implement the method as described in any one of claims 1 to 11.

14. A computer program product, characterized in that, The method includes a computer program, and when the computer program is executed by a processor, the processor causes the processor to perform the method as described in any one of claims 1 to 11.

Images