AI-BASED EXPLAINABLE APPLICATION PRE-EVALUATION SYSTEM AND METHOD
Patent Information
- Application Number
- TR202615184
- Authority / Receiving Office
- TR · TR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2026-09-04
- Publication Date
- 2026-09-21
Abstract
Description
AI-BASED EXPLAINABLE APPLICATION PRE-ASSESSMENT SYSTEM AND METHOD Technical Area The invention involves analyzing research and development and innovation applications using artificial intelligence. extracting field-based structured features from application content, etc. the creation of confidence scores and supporting information regarding the subject characteristics, obtained processing of the collected data through a rule-based evaluation mechanism and Strengths, weaknesses, risks, and field-based evaluation of applications. enabling the creation of explainable preliminary assessment outputs that include the results. It relates to an artificial intelligence-based application pre-evaluation system and methodology. The Invention's Infrastructure Project prepared within the scope of research and development and innovation activities. Evaluation of applications; the technical competence of the application, the level of innovation, the relationship between the problem and the proposed solution, its feasibility, the team's competence, commercialization potential, data and resource adequacy, relevant call or support the suitability of the program to the criteria and the risks associated with the application are considered together. It is a multifaceted process that requires taking action. In current practices, this is the case. Evaluation processes largely rely on the review and interpretation of human experts. It is based on and the application documents are reviewed by different experts, The application is evaluated within the framework of the determined criteria, and its strengths and weaknesses are assessed. aspects are determined and the application is made using the evaluation results obtained. An opinion is being formed about it. The proliferation of research and development and innovation activities, along with various support mechanisms... with the creation of programs and the increase in applications to these programs The amount of documents that need to be evaluated together is also increasing. In addition Each support program should have its own specific evaluation criteria. Applications must be prepared in different technical fields and one application must be submitted in multiple formats. 1. The need to examine the evaluation process from the perspective of evaluation dimensions. This makes things more complex. Applications require technical competence, level of innovation, different factors such as commercialization potential, team competence and risk profile separately evaluation requires a significant amount of human resources and time for the evaluation. This can lead to its use. In known practices, the evaluation process is carried out by human experts. to be carried out by utilizing the knowledge and experience of experts while providing this, the evaluation results may vary depending on the personal opinions of the experts and This can lead to it being influenced by different evaluation approaches. Similarly... or applications with similar characteristics will be evaluated differently by different evaluators. the ability to interpret in different ways, consistency among evaluation results This can make it difficult to provide. In addition, the high number of applications... In these cases, experts carefully examine and evaluate each application. the lengthening of their durations and the human resources used in the evaluation processes This can lead to an increase in the need for it. Another approach used in current evaluation processes is to review applications. based on predetermined evaluation criteria and evaluation forms This approach provides a certain standardization, but the application... technical descriptions, problem definitions, and solutions contained within it, all in natural language. their approaches, innovation elements, team information, commercialization statements, and The correlation between risks and assessment criteria is again largely It depends on the evaluator's review. Therefore, it is solely an evaluation. standardization of criteria or scoring forms, application content completely eliminates any differences in interpretation that may arise during the evaluation process. It does not remove it. In known technical terms, project and application processes involve manual review and expert opinion. document review, criteria-based evaluation forms, and evaluation meetings These structures are used to identify the strengths and weaknesses of applications. Isolation of technical risks, measurement of compliance with evaluation criteria, and preliminary The preparation of evaluation reports depends on the assessments of human experts. This is carried out as follows. Therefore, from past applications, past Corporate data derived from 2 evaluation results and expert feedback. accumulated knowledge in subsequent evaluations in a systematic and automatic manner Its use may remain limited. With the development of artificial intelligence technologies, documents prepared in natural language... processing, summarizing, classifying, and evaluating according to specific questions Systems aimed at this have begun to be used, especially large language models. These structures enable the processing of long texts and the semantic aspects within the text. It enables the identification of relationships and the generation of output in natural language. This systems summaries or general information for comprehensive documents such as project applications. It can be used to formulate evaluations. However, in existing systems based on large language models, the user... The text transferred to the system is mostly processed directly by the model, and the model It is created by a summary, comment, suggestion, or general assessment. In this approach, which evaluation domain does the result generated by the model belong to? Which application section, which call criteria, or which information source? It is always possible to clearly demonstrate that it was produced based on This is not the case. Such a structure is particularly unsuitable for decisions such as evaluating an application. traceability of results generated in support applications and This can lead to a lack of explainability. Another problem that arises in well-known large language model-based applications is the model The output can be used directly as an evaluation result. Application Forming a general interpretation based on the text as a whole, different aspects of the application This can make it difficult to examine the evaluation areas separately. However, in research and development and innovation applications, technical feasibility and innovation are important factors. level, fit between problem and solution, team competence, commercialization potential, Data and resource adequacy, compliance with call criteria, and risk level differ from one another. These are areas of evaluation, and each area needs to be evaluated separately. Some known systems automatically process documents according to specific criteria. Scoring, classification, or ranking is carried out. This type of systems in terms of processing electronic documents and certain characteristics A general-purpose document that facilitates comparison between the three. It uses an evaluation approach. Research and development and innovation. specific to the multifaceted nature of applications: technical, innovation, commercial, team, risk, and Evaluation areas such as call compliance should be considered together and each one... Creating structured features related to the evaluation area is the current general This is not adequately addressed in document evaluation systems. Evaluation of patent documents in known technology and specific patents There are also systems for scoring based on characteristics. In these systems... the economic value, technical effect, citation information or portfolio of patents Its position within the context can be assessed. However, These systems for evaluating patent documents are for research purposes. technical competence, team capacity, commercialization of development project applications Different assessments such as potential, risk level, and compliance with call criteria It is not structured for the purpose of analyzing their dimensions together. In known decision support systems related to project selection and project portfolio management, prioritizing projects, allocating resources among projects, and project management. Operations such as portfolio management can be carried out. Systems mostly process structured data, and the content of comprehensive research and development applications prepared in natural language preliminary in a way that can be explained by relating it to the evaluation criteria. It does not offer a framework for evaluation. Knowledge of preparing and managing grant and fund applications. Preparation of application texts in applications, existing institutional content using, developing texts that meet the expectations of funders, appropriate Functions such as locating funds or calls for proposals and monitoring application processes. This can be accomplished. These practices help the applicant prepare their application. supporting the process and reviewing a prepared application using a technical referee approach. extracting field-based features from the application by evaluating them, and listing these features separately a system structure that evaluates and generates a reasoned preliminary assessment result He does not have it. 4. AI-powered applications for literature review: scientific publications Finding, summarizing academic sources, providing answers based on those sources. creation and bringing together different scientific findings This can be achieved. These systems provide access to information in research activities. While facilitating, the call criteria for a research and development application are technical. feasibility, level of innovation, team competence, commercialization potential and risks It is not intended for joint evaluation in terms of these aspects. Natural language processing and semantic document evaluation approaches in known techniques. using semantic similarities between texts to determine the documents Classification and grouping of contents into specific categories is also possible. However, the outputs obtained from these approaches are mostly similar, It remains at the level of classification or content analysis and research and development. a multidimensional evaluation mechanism specific to their applications It is not being converted. Another shortcoming observed in current systems is the reliability of the evaluation results. the level and the information on which the result is based It is the inability to demonstrate a specific application by an artificial intelligence model. Whether it is evaluated positively or negatively in the evaluation area alone This does not demonstrate the reliability of the evaluation result. Evaluation which application statement, which call criteria, or which recalled item is the result of The inability to determine whether the information source is reliable, and the expert's assessment outcome... This makes it difficult for the user to control. In addition, past expert assessments and corporate decisions in existing systems learning their patterns systematically and using them in subsequent assessments There are also limitations in this regard. Different institutions or assessments organizations have different evaluation priorities and criterion weightings for similar applications. or decision-making approaches can be found. A general-purpose artificial intelligence model. spontaneously adapting to the aforementioned institutional evaluation patterns This may not be possible. Therefore, the system's past applications and evaluations... corporate evaluation using results and expert feedback It needs to be adapted to the approach. In order to remedy the aforementioned deficiencies in the structure that is the subject of the invention, the application text Instead of directly converting it into a general assessment result, it undergoes a multi-stage process. An evaluation architecture is used. The application text is first put under evaluation. is being made suitable, information sources related to the application are being recalled, application The recalled information is processed together and field-based within the application. Structured features are extracted. Thus, the artificial intelligence model... The generated output is not used directly as the final evaluation decision; it is used in the next phase. Structured features are obtained that will be used in evaluation processes. One of the key differences of the invention is information retrieval-assisted manufacturing (RAG). low-order adaptation (LoRA) and quantized low-order adaptation approach Adapted to institutional assessment patterns using (QLoRA) methods It is the combined use of the local language model. Recall via an information retrieval structure. texts, evaluation criteria, past submissions, expert feedback, literature, Content relevant to the application is identified from among patent and similar project records, and An evaluation context is being established. Institutional evaluation patterns... The adapted large language model, on the other hand, takes the reference text into the relevant context. by processing the domain-based structured features to be used in the evaluation process. It produces. The invention encompasses model adaptation, low-order adaptation (LoRA), and quantized. This can be achieved using low-order adaptation (QLoRA) methods. Thus, instead of retraining the entire core major language model, training related to the model is performed. Research and development and innovation of the system using adaptation components evaluation tasks specific to applications and institutional evaluation It is possible to adapt them to their patterns. Another distinctive aspect of the invention is that the application content provides a single overall assessment output. Instead, it is divided into domain-based structured features. Technical feasibility, innovation. level, clarity of problem definition, adequacy of solution approach, data and resources competence, team cohesion, commercialization potential, regulatory or integration risk, and Features related to assessment areas such as call compliance are listed separately. This structure allows for different evaluations of the application. The six dimensions can be examined independently and then combined into a common framework. It becomes possible to combine them within the evaluation mechanism. For each structured feature created within the scope of the invention, a value is assigned. Generating confidence scores and supporting information is also an important step compared to known practices. This creates a difference. The confidence score is the relevant score generated by the system. The underlying information indicates the confidence level of the evaluation outcome. Which statement in the application text and which section in the call criteria will be evaluated? or shows which recalled information it is associated with. Thus, only one Instead of generating a score or evaluation result, the rationale behind the result and The information on which it is based can also be reviewed by expert users. The invention directly incorporates structured features derived from an artificial intelligence model. not to be used as the final decision and the features in question are rule-based Processing by the evaluation engine is another technical difference of the system. It consists of a rule-based evaluation engine with structured features. confidence scores and supporting information based on predefined evaluation rules, It operates according to threshold values and weightings; thus, the technical evaluation score, innovation score, team score, commercialization score, risk score, and overall It enables the generation of domain-based results such as preliminary assessment scores. Thanks to this structure, the generative nature of the artificial intelligence model and the predefined... The verifiable nature of the evaluation rules is used in conjunction with artificial intelligence. The intelligence model analyzes the reference content in natural language to identify structured features. while enabling the extraction of these features, the rule-based evaluation engine determines which of these features with weights and which threshold values will be considered in the evaluation This determines whether the model output is directly accepted or rejected. Instead of creating it, the evaluation process is divided into different technical stages. A traceable structure is obtained. The invention also includes a discernible digital peer review report resulting from the evaluation. This enables the creation of a report that identifies the strengths and weaknesses of the application. aspects, weaknesses, critical risks, domain-based scores, confidence scores, It may include supporting information and suggestions for improvement. This report is the final decision. A preliminary assessment output that is not of the 7th quality and is submitted for expert evaluation. It is used as such. Thus, the human being is outside the expert evaluation process. not left unattended, the results generated by the system are reviewed by an expert. This allows for review and correction if necessary. Another advantage of the invention is that approval, rejection, correction, and verification can be carried out by an expert. The process involves transferring comments as feedback to the system. This feedback... By recording the notifications, they can be used in retraining the model adaptation components. in the calibration of the rule-based evaluation engine and system performance It can be used in its development. In this way, the system is not just a fixed one. It does not function as an evaluation mechanism, based on expert feedback. By making use of it, the corporate evaluation approach has become more prevalent over time. It can be developed in a way that allows for adaptation. One of the advantages offered by the invention is that it allows for the prioritization of research and development and innovation applications. Thanks to the ability to automate the review processes, experts can initially... The goal is to reduce the burden of review. Processing of application documents, Identifying areas of assessment, recalling relevant information sources, Extraction of structured features, creation of trust and underlying information, and Since the calculation of field-based scores is performed by the system, the expert... so the user can focus on the later stages of the evaluation process is provided. The invention also applies to the same evaluation areas, the same property structure, and the same rule. the use of a base-based evaluation mechanism on different applications This allows for increased consistency in evaluation by ensuring that applications are processed correctly. the criteria and weightings to be used in the evaluation within the system the ability to define, the approach that may occur among different evaluators It helps to reduce differences. Another advantage offered by the invention is explainability. Each domain-based assessment. By correlating the output with the confidence score and supporting information, the expert can... The user needs to know what information the result generated by the system is based on. It can examine how it was created. This feature is particularly relevant for low confidence scores. 8. Determining the evaluation results and directly applying those results. This allows for the subject to human evaluation. The invention facilitates technical feasibility, data sufficiency, team cohesion, commercialization, and legislation. and different risk areas such as integration are presented separately during the application process. It is possible to review this. Thus, not only a general application score is given. Instead of creating an application, the areas where the application poses a risk should be determined and discussed. The issue involves presenting the risks in a way that is understandable to the expert user. In the preferred application of the invention, the system's local or in-house computing system The fact that it can be run on the same infrastructure provides a significant advantage. This allows... institutional or sensitive projects that may be included in research and development and innovation applications Evaluation of this type of data without the need to transfer it to external systems The operations can be performed. However, the system is external to different applications. It is also possible to integrate with major language models that operate through services. It is possible. The invention also offers an expandable structure. New call types and new features can be added to the system. Evaluation criteria, different areas of expertise, and new field-based features. It can be added; thus, different research and development, innovation, technology development and The system can be adapted to the evaluation requirements of support programs. It is not limited to a specific support program but applies to different application types. It can be structured in a way that can be used in its evaluation. Consequently, inventions are only eligible for research and development and innovation applications. summarizing, rewriting, classifying, or generally using artificial intelligence not subjecting it to interpretation; but pre-processing of the application, relevant information sources recall, language model adapted to corporate evaluation patterns extracting field-based structured features, and those features generating confidence scores and supporting information, and structured outputs. processing in a rule-based assessment engine, domain-based scores and risk determining their levels, creating a explainable digital peer review report, and expert feedback in subsequent model and rule improvement processes by bringing together their use within the same technical structure, the current evaluation The 9 methods are characterized by duration, consistency, explainability, traceability, and institutional knowledge. solutions to the shortcomings regarding its use and early identification of risks It brings. Detailed Description of the Invention This detailed explanation outlines the preliminary evaluation of an AI-based application concerning an invention. It relates to the system and method and is solely aimed at a better understanding of the subject. This is explained in a way that will not create any limiting effects. The invention in question; • application text, call for proposals, evaluation criteria and supplementary documents the data to be entered into the system and cleaned, segmented, normalization and metadata generation processes in the evaluation process The application data retrieval and preprocessing module that enables the preparation of the data. • Call for proposals text, history using information recall-assisted generation (RAG) approach applications, evaluation criteria, literature, patents, and expert feedback the recall of content related to the application and said content Information feedback that enables the creation of an evaluation context using Calling and context creation module, • Low-order adaptation and quantification of corporate evaluation patterns adapted from the application text using low-order adaptation methods domain-based structured features to be used in the evaluation process an adapted local language model that enables its production, • The outputs obtained from the adapted local language model include technical, innovation, commercial, and team aspects. separating them according to assessment areas such as risk and call compliance, and Field-based feature extraction that enables the conversion of structured features. layer, • Confidence score and baseline for each of the generated structured features the creation of information and the relevant supporting documents in the application text, call for proposals trust that enables it to be linked to the criteria and recalled information sources. score and support generation module, • pre-configured features, confidence scores, and underlying information defined evaluation rules, thresholds, and weightings processing using and field-based scores, risk level and prediction Rule-based system that enables the generation of evaluation results. evaluation engine, • Based on the evaluation results, identify the strengths and weaknesses of the application. including aspects, risks, shortcomings, area-based scores, and recommendations. Digital referee enables the creation of a explainable digital referee report. reporting module, • Recording of approvals, rejections, corrections, and comments made by experts. and that the feedback in question be taken into account in the adapted local language model retraining, calibration of the rule-based assessment engine, and human feedback that enables its use in improving system performance The notification and model improvement modules consist of the following parts. The invention involves receiving and evaluating application documents prepared in natural language. making it suitable for the process, evaluating the information sources related to the application. context creation, field-based structured features from the application content extraction, confidence scores and supporting information for the extracted features the creation, rule-based evaluation of the outputs in question, and The evaluation resulted in a verifiable digital peer review report. It is based on the principle of bringing it about. The system described in the invention provides a direct evaluation of application texts in a single stage. instead of transforming it into a final result, it goes through different evaluation stages of the application. This ensures that the application document and evaluation are carried out within this scope. Other information to be used is first prepared for evaluation, then submitted with the application. Information sources related to the topic are identified, creating a context for evaluation, and the application is prepared. through a local language model whose content is adapted to institutional evaluation patterns instead of directly making an acceptance or rejection decision, it is processed and then... Field-based structured features to be used in evaluation stages The generated features are correlated with confidence scores and supporting information. and the preliminary assessment output that can be explained as a result of rule-based assessment. It is being transformed. 11 The first element used in the evaluation process of the system concerning the invention is the application data. It is the data retrieval and preprocessing module. The application data retrieval and preprocessing module; application the text, the call for proposals, the evaluation criteria, and any additional documents related to the application. It ensures that the documents are uploaded to the system. The content of the application uploaded to the system text in order to make it processable in subsequent evaluation stages cleaning, partitioning, normalization, and metadata generation processes This is being done. Long application documents, coherence of meaning within the content. It can be separated into parts in a way that preserves its integrity. For each part created, the source, metadata such as section, date, call type, application type, and subject tag It can be created. Thus, it is prepared in natural language and consists of different sections. The submitted application documents can be used in subsequent evaluation processes. It is arranged in this way. Pre-processing performed by the application data retrieval and pre-processing module. fields that will be used in the evaluation of the application as a result These areas of evaluation can be determined based on the content of the application and technical evaluation, innovation evaluation depending on the evaluation criteria, areas such as team competence, commercialization potential, risk analysis, and call alignment It can include this. This ensures that the application is not only presented as a complete text. Instead of evaluating the application, the different evaluation dimensions of the application should be considered separately. It is ensured that it can be obtained. Another element of the system described in the invention is information retrieval and context creation. It is a module. The information retrieval and context creation module is part of the application. among the information resources that can be used in the evaluation, related to the application. Identifying the content and determining the context of evaluation from that content. This ensures the creation of call texts, past applications, Evaluation criteria, expert feedback, literature, patents and similar projects. The records can be used by associating them with the application. The information retrieval and context creation module supports information retrieval-assisted production. by utilizing this approach, the semantic information related to the application content can be retrieved. It performs the retrieval. The retrieved content is reordered and 12 are filtered, then evaluated using an adapted local language model. It is placed within its context. Thus, during the evaluation process, only the application is considered. Not the information contained in the document, but the call criteria and history related to the application. Assessments and other supporting sources of information are also included in the evaluation context. It can be included. The process is performed by the information retrieval and context creation module. As a result, information related to the application text and the evaluation of the application. They are brought together. This structure specifically ensures that the application complies with the call criteria, relationship with past evaluation records and information related to the application This ensures that the sources can be taken into consideration. The recalled information is directly... It is not used as a final assessment result, but rather in the later stages of the application. It forms the evaluation context to be used in processing. Another element of the system described in the invention is low-order adaptation and quantized low-order adaptation. It is a local language model adapted through sequential adaptation. This local language model, the application text and the created framework were adapted to institutional evaluation patterns. It ensures the processing of the evaluation context. Low in the adaptation of the model. quantized low-order adaptation (LoRA) and / or quantized low-order adaptation (QLoRA) These methods can be used. The primary task of the adapted local language model is to provide a direct, final answer about the application. The goal is not to make an acceptance or rejection decision. The model involves retrieval of information from the application text. and then process the context generated by the context creation module. Field-based structured features to be used in the evaluation stages It produces. Thus, the output of the large language model is directly used as a decision. instead of using them, creating the characteristics to be processed in the evaluation process. is provided. The application's problem definition and solution are presented through an adapted locale model. approach, innovative content, technical feasibility, data and resource adequacy, team cohesion, important in terms of evaluation such as commercialization potential and risks Its characteristics can be determined. The model's institutional evaluation patterns... Thanks to its adaptation, past applications, evaluation results and expert feedback are available. The evaluation approach derived from the 13 notifications within the system It is intended to be usable. The adapted local language model for the preferred application of the invention, local or institutional. It can be run on the internal computing infrastructure. Thus, research and development and without the need to transfer the data included in innovation applications to external systems This allows for the evaluation processes to be carried out. The alternative of the invention... In its applications, the system uses external service-based large language models. It is possible to integrate them. Another element of the system described in the invention is the domain-based feature extraction layer. The domain the feature-based feature inference layer, outputs from the adapted native language model separating and standardizing according to different evaluation areas It ensures that these are transformed into structured features. In this context, the technical aspect... Characteristics relating to assessment areas such as innovation, commercial, team, risk, and call compliance. They are structured separately. Clarity of problem definition and solution through domain-based feature extraction layer adequacy of the approach, level of innovation, technical feasibility, data and resource adequacy, Features such as team cohesion, commercialization potential, and regulatory or integration risks. They can be categorized according to evaluation areas. Thus, different aspects of the application can be examined. separating the evaluation dimensions from each other and the next This allows for separate processing of each stage in the evaluation process. Structured features obtained from the field-based feature extraction layer are modeled. using the natural language output produced by [the source] as the direct final decision to standardize the information to be used in the prevention and evaluation process It enables the transformation. These features then increase confidence levels in the next stage. and in determining their basis and rule-based evaluation processes It is used in its implementation. Another element of the system described in the invention is the confidence score and the underlying production module. The confidence score and backbone generation module is generated by the field-based feature extraction layer. Confidence score and underlying information for each of the structured features created. It enables the creation of 14. The confidence score is the relevant evaluation output of the system. When specifying the level of confidence related to the relationship, the underlying information is the information with which the relevant assessment is based. This shows that they are related. The supporting information consists of statements in the application text, call criteria, and information feedback. with information resources recalled by the call and context creation module It is correlated. In this way, only the outcome value of an evaluation output is correlated. not, nor the source of information used in generating that result. This can be determined. Thus, the evaluation generated by the system. so that the results can be monitored and examined by expert users is provided. A numerical or categorical value can be assigned to each attribute, and the following can be determined: value confidence score along with supporting information in the evaluation process is used. Assessment outcomes with low confidence scores are used by humans. It can be directed to an evaluation and thus generated by artificial intelligence. The results can be used under expert supervision. Another element of the system described in the invention is the rule-based evaluation engine. field-based assessment engine; by field-based feature extraction layer the structured features created, the confidence score and the underlying production module confidence scores generated by predefined underlying information in accordance with evaluation rules, threshold values and weightings It is in operation. Field-based weighting and thresholding via a rule-based evaluation engine. The process involves monitoring, determining risk levels, and calculating scores. As a result of these processes, the technical evaluation score, innovation score, and team score are calculated. evaluation scores for different areas such as commercialization score and risk score These scores can be generated based on a predefined decision logic. In line with this, the overall preliminary assessment score and preliminary It is generated as a result of the evaluation. Adapted local language thanks to the use of a rule-based evaluation engine. The outputs produced by the model are considered as a direct result of the evaluation. This is not done. The features derived by artificial intelligence have been determined. processing in accordance with evaluation rules, weightings and threshold values. by ensuring that the evaluation process is more traceable and standardized. This is ensured. Another element of the system that is the subject of the invention is the digital peer review reporting module. Digital referee reporting module, derived from the rule-based evaluation engine using the results to explain the preliminary assessment report of the application It ensures the creation of a digital referee report. The application is included in the report. Strengths, weaknesses, shortcomings, critical risks, domain-based scores, confidence. Scores, supporting information, and suggestions for improvement may be included. The report generated by the digital referee reporting module will result in final acceptance or rejection. It is not used as a decision-making tool, but rather to support expert assessment. It is presented as a predictable preliminary assessment outcome. Thus, the system As a result of the evaluation processes carried out by human experts not being excluded from the evaluation process and the final decision being made by humans It is left for evaluation. Another element of the system described in the invention is human feedback and model improvement. It is a module. The human feedback and model improvement module is part of the digital peer review report. After being evaluated by the expert user, by the expert recording all approval, rejection, correction, and comment processes performed. This provides the evaluation results generated by the system. The relationship between expert assessments can be observed. Feedback from experts indicates that the adapted local language model has a low ranking. Rule-based evaluation in retraining adaptation (LoRA) components in the calibration of the engine and in improving system performance It can be used. In this way, corrections and final results are obtained by the expert. The evaluations contribute to the development of subsequent evaluation processes of the system. It can be transferred. 16 In the system described in the invention, the eight elements in question follow each other and complement each other. It operates within an evaluation framework that utilizes its outputs. Application Application prepared for evaluation by the data retrieval and preprocessing module. content generated by the information retrieval and context creation module It is linked to the evaluation context; the application text and the context created. It is processed by the adapted local language model; generated by the model. The outputs are structured by the field-based feature extraction layer; Confidence score for features and reliability generated by the reliability and reliability module. Information is being generated; the resulting structured information is used for rule-based evaluation. It is evaluated by the engine and the evaluation results are digitally reviewed by a referee. a preliminary assessment report that can be explained by the reporting module It is being transformed. The expert performed the analysis on the report. The evaluations are recorded via the human feedback and model improvement module. It is taken under control and used in the development of the system. Thanks to this integrated workflow, the application is processed directly by artificial intelligence. by giving it to the model and generating a single-stage and general evaluation result. instead, from the preparation of the application content to the evaluation result from explaining the process to incorporating expert feedback into the system. A phased evaluation process is being carried out. How the System Subject to the Invention Works: The invention concerns an artificial intelligence-based, explainable application pre-evaluation system. the working method, from the submission of research and development and innovation applications to the system Starting with the field-based evaluation of applications, the assessment converting the results into a readable digital peer review report and expert feedback This includes the processes for using notifications in the development of the system. The method in question consists of the following steps: 1. Upload the application document, call for proposals, and evaluation criteria to the system. Applications are being received: evaluated via the application data retrieval and preprocessing module. the subject of the research and development or innovation application, the application document, relevant 17 call for proposals and evaluation criteria are being entered into the system. Thus... Application content and the subject matter of the application to be used in the evaluation process This ensures that the criteria are processed by the system. 2. Pre-processing of the application text: The application text received into the system. It is being cleaned, normalized, and divided into sections. Long The documents are divided into parts while preserving their semantic integrity, and each part... For a piece of paper, this includes source, section, date, call sign, reference type, and subject tag. Metadata is being generated. This allows the application text to be evaluated later. It is being brought into a structure that can be used in its processes. 3. Determining the areas for evaluation regarding the application content: Under pre-processing. fields to be used in the evaluation according to the content of the submitted application These areas are determined through technical evaluation and innovation. assessment, team competency, commercialization potential, risk analysis and call It may include areas of assessment such as compliance. Thus, the application... This allows different dimensions of evaluation to be addressed separately. 4. Retrieving resources related to the application: Information retrieval and context. through the creation module, information recall-assisted manufacturing (RAG) Call criteria related to the application via the infrastructure, past applications, expert. feedback, literature, patents and similar project records are semantically returned. They are called in. This process allows for the evaluation of the application. Relevant information content that can be used is being identified. 5. Transforming recalled content into an evaluation context: Recycle The retrieved content is reordered and filtered, then adapted. It is placed in the context window to be used by the local language model. Thus, the application text is related to the evaluation of the application. context for evaluation so that information sources can be processed together is being created. 6. Transferring the application text and contextual data to the adapted local language model: The application text and the context of the evaluation created are low-order adaptations. (LoRA) and quantized low-order adaptation (QLoRA) with adapted local. It is being transferred to 18 major language models. This involves an adapted local language model. by processing the data together, it will be used in the evaluation of the application. It enables the creation of structured information. 7. Extraction of domain-based structured features: Adaptive native language model field-based structured features from the application content These characteristics are derived from the clarity of the problem definition and the solution. adequacy of the approach, level of innovation, technical feasibility, data and resources competence, team cohesion, commercialization potential, and legislation or integration. It includes assessment features such as risk. This enables the application to function properly. Instead of transforming its content into a single overall assessment result, different Characteristics related to the evaluation areas are obtained separately. 8. Generating value, confidence score, and underlying information for each feature: A numerical or categorical value is assigned to each structured property extracted. Confidence score and supporting information are generated. The confidence score is related to... The underlying information indicates the level of confidence in the assessment outcome. Which expression in the application text corresponds to which of the call criteria in the output in question? It indicates which section or recalled information content is related to it. 9. Transferring structured features to the rule-based evaluation engine: Structured properties created on a field basis and the trust associated with them. scores and supporting information are entered into a rule-based evaluation engine. is transferred. Thus, the structured data obtained by artificial intelligence is transferred. processing of outputs within a predefined evaluation logic is provided. 10. Field-based weighting, threshold value control, risk level determination, and scoring. Performing calculation operations: Rule-based evaluation engine Area-based weightings are applied, and the determined threshold values are used. They are being monitored, risk levels are being identified, and the relevant assessment is being made. Score calculation processes are performed for these areas. 19 11. Creation of area-based scores and overall preliminary assessment score: Rule Technical score, innovation score, team based evaluation processes A score, a commercialization score, and a risk score are generated. The resulting field-based score... Scores are based on a predefined evaluation approach. A general preliminary assessment score is generated using these methods. 12. Creation of the digital referee report: Field-based evaluation results. The application is robustly evaluated through the digital referee reporting module. aspects, weaknesses, shortcomings, critical risks, and suggestions for improvement, A printable digital peer review report including confidence scores and supporting information. is being created. 13. Submission of the digital referee report to the expert user: The generated digital The referee report is presented to the expert user and generated by the system. The preliminary assessment results are reviewed by an expert. The report generated by the system is not used as a final decision. The final decision is left to human judgment. 14. Recording expert feedback: By the expert user digital referee report and evaluation results were used Approval, rejection, correction, or comments are human feedback and model improvement. It is recorded through the module. Thus, the system... feedback between the results generated and expert evaluations The cycle is being created. 15. Utilizing expert feedback to improve the system: Recording it. The expert feedback received will be used for subsequent model adaptation, rule improvement, and It is used in performance monitoring processes. In this context, the aforementioned low-order adaptation (LoRA) components from feedback in updating and calibrating the rule-based evaluation engine by utilizing the system's evaluation performance over time development is ensured. In the preferred application of the invention, the procedural steps involve information retrieval. supported production infrastructure, low-order adaptation and quantized low-order local language adapted to institutional evaluation patterns through adaptation methods the model, domain-based feature extraction, confidence score and backing generation, and rule This is achieved through the combined use of a base-based assessment engine; Thus, applications are only subject to a general artificial intelligence assessment. Instead of being held accountable, the application content is broken down into different areas of evaluation, area-based scores that are linked to the basis of the assessment results created and the results obtained are submitted to human expert evaluation. A transparent preliminary assessment process is provided. 21
Claims
1. The invention is explained through an artificial intelligence-based preliminary application evaluation system. related to; its characteristic is; • application text, call for proposals, evaluation criteria and supplementary documents the data to be entered into the system and cleaned, segmented, normalization and metadata generation processes in the evaluation process The application data retrieval and preprocessing module that enables the preparation of the data. • Call for proposals text, history using information recall-assisted generation (RAG) approach applications, evaluation criteria, literature, patents, and expert feedback the recall of content related to the application and said content Information feedback that enables the creation of an evaluation context using Calling and context creation module, • Low-ranking adaptation to corporate evaluation patterns (LoRA) and by adapting with quantized low-order adaptation (QLoRA) methods from the application text, field-based evaluation process An adapted native language model that enables the generation of structured features. • The outputs obtained from the adapted local language model include technical, innovation, commercial, and team aspects. separating them according to assessment areas such as risk and call compliance, and Field-based feature extraction that enables the conversion of structured features. layer, • Confidence score and baseline for each of the generated structured features the creation of information and the relevant supporting documents in the application text, call for proposals trust that enables it to be linked to the criteria and recalled information sources. score and support generation module, • pre-configured features, confidence scores, and underlying information defined evaluation rules, thresholds, and weightings processing using and field-based scores, risk level and prediction Rule-based system that enables the generation of evaluation results. evaluation engine, • Based on the evaluation results, identify the strengths and weaknesses of the application. including aspects, risks, shortcomings, area-based scores, and recommendations. Digital referee enables the creation of a explainable digital referee report. 35 reporting modules, 22 • Recording of approvals, rejections, corrections, and comments made by experts. and that the feedback in question be taken into account in the adapted local language model retraining, calibration of the rule-based assessment engine, and human feedback that enables its use in improving system performance The notification and model improvement module is characterized by its components.
2. Invention: An artificial intelligence-based, explainable application pre-evaluation system. It relates to the working method and its characteristic is; • application document, call for proposals and evaluation criteria inclusion in the system • Cleaning, normalizing, and dividing the application text into sections. and preprocessing of its metadata, • Identifying areas for evaluation regarding the application content, • Call criteria related to the application, past applications, expert feedback information from notifications, literature, patents and similar project records Recall via recall-assisted manufacturing (RAG) infrastructure, • Reordering and filtering the recalled content establishing the context for evaluation, • the application text and the context of the evaluation created, low ranking adaptation (LoRA) and quantized low-order adaptation (QLoRA) local language adapted to institutional evaluation patterns through methods transferring to the model, • Field-based analysis of application content through an adapted locale model Extraction of structured features, • Value, confidence score, and for each structured feature extracted. Creating the supporting information, • domain-based structured features, confidence scores, and underlying principles transferring the information to a rule-based evaluation engine, • field-based through a rule-based assessment engine Weighting, threshold control, risk level determination, and scoring. performing calculation operations, • technical score, innovation score, team score, commercialization score and risk score Creating a general preliminary assessment score, 23 • the strengths, weaknesses, shortcomings, and critical risks of the application, including improvement suggestions, confidence scores, and supporting information. Creation of a explainable digital peer review report. • Presenting the generated digital referee report to the expert user, • Approval, rejection, correction, and processing performed by expert users. recording comments as feedback, • Recorded expert feedback for model adaptation, rule setting. Used in improvement and performance monitoring processes, It includes the steps of the process. 24