For example, apparatus, data structures, and computer-implemented methods for determining metrics for evaluating the quality of generative, particularly multimodal, base models for optical inspection, identification of sound or technical components, natural language processing, program generation, or data labeling.

Abstract

For example, the present invention relates to apparatus, data structures, and computer-implemented methods for determining metrics for evaluating the quality of generative, particularly multimodal, base models for optical inspection, sound or technical component identification, natural language processing, program generation, or data labeling. [Solution] In this method, a first answer to a first question is determined using a generative base model (202), a second answer to a second question is determined using a generative base model (204), a relationship between the first answer and the second answer is specified (208), and an index is determined depending on the answer and the relationship (210).

Description

Technical Field

[0001] Prior Art The present invention relates to an apparatus, a data structure, and a computer-implemented method for determining an index for evaluating the quality of a generative, particularly multimodal, base model for, for example, optical inspection, sound or technical component identification, natural language processing, program generation, or data labeling.

Background Art

[0002] Generative large-scale, particularly multimodal, language models, i.e., generative base models such as large language models (LLMs), are capable of answering questions regarding multiple different themes, i.e., multiple different fields. Examples of fields are optical inspection, sound or technical component identification, natural language processing, program generation, or data labeling.

Summary of the Invention

Means for Solving the Problems

[0003] Disclosure of the Invention According to a computer-implemented method for determining an index for evaluating the quality of a generative, particularly multimodal, base model for, for example, optical inspection, sound or technical component identification, natural language processing, program generation, or data labeling, it is envisioned that a first answer to a first question is determined using the generative base model, a second answer to a second question is determined using the generative base model, the relationship between the first answer and the second answer is specified, and the index is determined depending on the answer and the relationship. The method provides parameters that make the base model automatically trainable or checkable.

[0004] For example, the degree of confidence that the first and second answers satisfy the relationship is determined as an indicator. The degree of confidence is determined, for example, based on whether different individual relationships are satisfied for each relationship.

[0005] For example, the confidence level is output to the user of the underlying model. This notifies the user of the quality of the underlying model regarding relationships.

[0006] For example, if the confidence level is determined to be below a threshold, the use of the base model for the application area containing the first and / or second questions is blocked. This prevents the base model from being used in areas where it is not trusted.

[0007] For example, if the relationship between the first and second answers is satisfied, a reward is determined as an indicator, and the base model is trained in a reward-dependent manner. The reward enables reward-dependent training.

[0008] For example, the underlying model includes an artificial neural network with weights, the weights being determined in relation to the reward.

[0009] The relationships may preferably be defined in various ways as metamorphic relationships.

[0010] For example, the relationship may be specified that the first answer includes a set of individual answers to the first question, the second answer includes a set of individual answers to the second question, and the first and second answers are sets of individual answers that are each relatively prime, or the relationship may be specified that the first answer is a subset of the second answer.

[0011] For example, the first question might ask about a first program for performing a task in a first domain, and the second question might ask about a second program for performing a task in a second domain, where the first domain encompasses the second domain, and the relationship is such that the first program and the second program solve the task in the second domain with the same result.

[0012] In particular, in the case of optical inspection, a first digital image displaying an object is provided, the first digital image is transformed into a second digital image displaying the object, the first question asks about the position of the object in the first digital image, the second question asks about the position of the object in the second digital image, and the relationship of how the first digital image is transformed into the second digital image is specified.

[0013] In particular, in the case of sound, a first audio sequence having a certain sound and a second audio sequence having a certain sound are provided, the first question asks about the time when the sound in the first audio sequence occurred, and the second question asks about the time when the sound in the second audio sequence occurred, and the relationship between the times is specified.

[0014] In particular, when checking programs automatically generated using a foundational model for computers, for example, the first question might ask about the result of linking a first program with a second program to perform an input-related task, and the second question might ask about the result of a third program to perform an input-related task, specifying that the third program solves the task with the same result as the first program linked with the second program.

[0015] It may be assumed that a third answer to a third question is determined, and that the third answer is expected to contain the first and second answers, or that the third answer is identical to the union of the first and second answers.

[0016] For example, according to an apparatus for determining an index for evaluating the quality of a generative, particularly multimodal, base model for optical inspection, sound or technical component identification, natural language processing, program generation, or data labeling, the apparatus is assumed to include at least one processor and at least one memory, the at least one memory containing instructions, and the method is carried out in the apparatus when the instructions are executed by at least one processor.

[0017] A computer program is assumed to be a computer program that includes computer executable instructions, and in which the method is implemented in the computer when the computer executable instructions are executed by the computer.

[0018] For example, according to a data structure for determining metrics for evaluating the quality of generative, particularly multimodal, base models for optical inspection, sound or technical component identification, natural language processing, program generation, or data labeling, the data structure is assumed to include at least one data field for the generative base model, at least one data field for a second answer to a second question determined using the generative base model, at least one data field for a second answer to a second question determined using the generative base model, at least one data field for the relationship between the first answer and the second answer, and at least one data field for metrics determined depending on the answer and the relationship.

[0019] Further advantageous embodiments can be seen in the following description and drawings.

Brief Description of the Drawings

[0020] [Figure 1] It is a schematic diagram of an apparatus 100 for determining an index for evaluating the quality of a generative, particularly multimodal, base model. [Figure 2] It is a flowchart having steps of a method for determining an index for evaluating the quality of a generative, particularly multimodal, base model. [Figure 3] It is a schematic diagram of a data structure for determining an index for evaluating the quality of a generative, particularly multimodal, base model.

Embodiments for Carrying Out the Invention

[0021] The apparatus 100 includes at least one processor 102 and at least one memory 104. The at least one memory 104 stores instructions executable by the at least one processor 102, and when the instructions are executed by the at least one processor 102 in the apparatus 100, a method for determining an index for evaluating the quality of a generative, particularly multimodal, base model is implemented.

[0022] The apparatus 100 may be assumed to include an interface 106. The interface 106 includes, for example, a human-machine interface or a machine-machine interface.

[0023] The interface 106 is used, for example, to input questions and output answers. The interface 106 may be configured to output hints for the answers.

[0024] The input includes, for example, text or a file. The file includes, for example, text, a digital image, an audio signal, and / or a program, i.e., for example, computer program code. The multimodal base model is configured, for example, to process the input. The output includes, for example, text or a file. The file includes, for example, text, a digital image, an audio signal, and / or a program, i.e., for example, computer program code. The multimodal base model is configured, for example, to output the output.

[0025] The input and output are data, for example, natural language descriptions and programs.

[0026] It may be assumed that at least one memory 104 includes the base model.

[0027] The base model includes, for example, an artificial neural network having weights.

[0028] The generative base model is used, for example, for optical inspection, identification of sounds or technical parts, natural language processing, program generation, and / or data labeling.

[0029] The generative base model is configured, for example, to perform questions regarding fields such as optical inspection, identification of sounds or technical parts, natural language processing, program generation, and / or data labeling.

[0030] The generative multimodal base model is configured to answer questions regarding at least two fields.

[0031] The base model is, for example, a large language model, i.e., a large language model (LLM).

[0032] In this example, the LLM is already trained to answer questions. This means that the LLM is configured to output answers to questions presented to it. The answer to the question is determined using a generative foundational model.

[0033] The underlying model is trained, for example, to consider specified quality features when generating its own output. Training is performed, for example, unsupervised.

[0034] An example of LLM training in the field of program generation is the programming task in "RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (arXiv:2410.02089v1)".

[0035] Methods for determining metrics to evaluate the quality of generative, particularly multimodal, foundational models are based on questions for which expected relationships between multiple answers are specified. These relationships are, for example, metamorphic relationships. These relationships are described, for example, in a formal language.

[0036] This method will be explained using the relationship between the first question, the second question, and the first answer of the generative base model to the first question, and the second answer of the generative base model to the second question as an example. In this example, instead of providing the relationship between the first answer and the second answer, a third question will be provided, and a means of providing the relationship between the first answer, the second answer, and the third answer of the base model to the third question will be explained as an optional choice.

[0037] This method is not limited to relationships between three questions and three answers, or between two or three answers. More than three questions, more than three answers, or two or more relationships between more than three answers may be assumed. Different relationships may be assumed between multiple answers. Different relationships may be assumed between each part of multiple answers. At least one relationship may be assumed between multiple answers, and at least one relationship may be assumed between parts of multiple answers.

[0038] In an example where it is known that the correct answer to the first question includes a first set of individual answers, and the correct answer to the second question includes a set of individual answers that are coprime to the first set, the relationship is specified that, for example, the first answer and the second answer of the base model are sets of individual answers that are coprime to each other.

[0039] In an example where it is known that the correct answer to the first question includes the first set of individual answers, and the correct answer to the second question includes the set of individual answers that encompasses the first set, the relationship is specified that the first answer of the base model is a subset of the set of individual answers in the second answer of the base model.

[0040] It is assumed that, in order to determine a program for performing tasks in the first domain, the first question will ask about a first program for performing tasks in the first domain, and the second question will ask about a second program for performing tasks in the second domain, which encompasses the first domain. For checking purposes, it is assumed that the relationship between the first program and the second program is specified so that they resolve the tasks in the second domain with the same result.

[0041] It can be assumed that the third question is expressed such that the correct answer to the third question includes the first and second answers. This means that the relationship is specified for the third answer that the third answer includes the first and second answers.

[0042] The third question can be assumed to be expressed such that the correct answer to the third question is identical to the union of the first and second answers. This means that the relationship is specified for the third answer that the third answer is identical to the union of the first and second answers.

[0043] When determining the first program, it can be assumed that the first question will be about the result of linking the first program with the second program to perform the input-related task. The second question will be, for example, about the result of a third program to perform the input-related task. For example, it may be specified that the third program solves the task with the same result as the first program linked with the second program.

[0044] When determining the position of an object in a first digital image, it is assumed that the first question asks about the position of the object in the first digital image, and the second question asks about the position of the object in the second digital image. The object is, for example, located at a first position in the first image and at a second position in the second image. The second position is located a certain distance away from the first position in a certain direction. The first answer includes the predicted first position of the object. The second answer includes the predicted second position of the object. For checking purposes, for example, a relationship is specified in which the predicted second position in the second answer to the second question differs from the predicted first position in the first answer to the first question by a certain distance in a certain direction.

[0045] For example, in the case of sound in an audio signal, it can be assumed that a first audio sequence containing a certain sound and a second audio sequence containing a certain sound are provided. The first question may ask about the time when the sound in the first audio sequence occurs, and the second question may ask about the time when the sound in the second audio sequence occurs. For checking purposes, for example, the relationship between these time points may be specified.

[0046] Relationship determination is carried out, for example, depending on the first question. Relationship determination may be carried out through interaction between a computer program configured for this purpose, such as a chatbot, and the user. The computer program may include, for example, a generative foundational model.

[0047] A computer program is configured, for example, to detect a first question from a user.

[0048] Computer programs are configured, for example, to ask the user about relationships in a counter-question. These relationships are, for example, metamorphic relationships.

[0049] A computer program is configured, for example, to determine a second question depending on a relation and a first question. The computer program is also configured, optionally, to determine a third question depending on a relation and a first question.

[0050] The computer program is configured, for example, to present a first and second question to a generative base model. The computer program is also configured to optionally present a third question to the generative base model.

[0051] A computer program is configured, for example, to determine in a check whether the answer to a given question satisfies a given condition.

[0052] A computer program is configured, for example, to output the results of a check to the user. A computer program is configured, for example, to output both the check results and the answer to the user. A computer program is configured, for example, to output the answer to the user if the check is successful, and not to output the answer to the user otherwise. For example, the check is successful if the answer satisfies the relationship.

[0053] If there are multiple relationships, the check will succeed, for example, only if all relationships are satisfied.

[0054] The first exemplary output of the computer program is, "My answer is [...], but I am not certain in this case as my own internal check has failed," where [...] represents at least one of several answers.

[0055] A second exemplary output from the computer program would be, "My answer is [...] and I am fairly confident as my own internal checks have worked fine," where [...] represents at least one of several answers.

[0056] The first exemplary flow of a computer program is illustrated using a question about a list of presidents of a country X. In this example, four presidents of country X are known: President A, President B, President C, and President D. In this example, the two most recent presidents are President A and President D. In this example, the first and second user questions are input into the computer program by the user. For example, the computer program generates the back questions from a template for back questions to the question about the list. The first, second, and third questions to the base model are generated by the computer program, for example, from a template for the question about the list, depending on the first and second questions to the base model. The question about the list is recognized, for example, using a classifier that maps the question to a template. In this example, the template for the question is associated with the relationship expected from the answer to the question to the base model. The back questions and answers are output to the user by the computer program. The computer program questions the base model with each question and outputs the corresponding answer from the base model.

[0057] User question #1: Please list the names of all presidents of country X. First question for the foundational model: Please list the names of all presidents of country X.

[0058] First answer: President A, President B, President C, President D.

[0059] Counter-question: It seems you're looking for a list. Could you give some examples of questions that could help me find a more refined list?

[0060] Second user question: Please list the names of the two most recent presidents of country X. Second question regarding the foundational model: Please name the two most recent presidents of country X.

[0061] Second answer: President A, President D.

[0062] Third question regarding the foundational model: Please list the names of all presidents of country X, excluding the two most recent presidents. Third answer: President B, President C.

[0063] Relationship: {Second answer, Third answer} == First answer. This means that the expected relationship is that the union of the president list in the second answer and the president list in the third answer is equal to the president list in the first answer.

[0064] A second illustrative flow of a computer program is illustrated by a question about a program that checks whether a given graph is a tree. In this example, a graph is considered a tree if it is connected and contains no cycles.

[0065] In this example, user questions are input into the computer program by the user. For example, the computer program generates counter-questions from templates for counter-questions to questions about the program. The counter-questions and answers are output to the user by the computer program. The computer program uses each question to question the underlying model and outputs the corresponding answers from the underlying model. Questions about the program are recognized, for example, using a classifier that maps the questions to templates. In this example, the templates are associated with the relationships expected from the answers to questions about the underlying model.

[0066] User question #1: Please write a checker to determine if a graph is a tree.

[0067] First question regarding the foundational model: Write a checker to determine if a graph is a tree.

[0068] Counter-question: Please list the subtasks that can be obtained by breaking down this task.

[0069] First answer: Program 1 - Checker to determine if a graph is a tree.

[0070] Second user question: 1. Write a checker to determine if graphs are connected. 2. Write a checker to determine if a graph contains cycles.

[0071] Second question regarding the base model: Write a checker to determine if graphs are connected.

[0072] Second answer: Program 2 - Checker to determine if graphs are connected.

[0073] Third question regarding the base model: Write a checker to determine if a graph contains a cycle.

[0074] Third answer: Program 3 - Checker to determine if a graph contains cycles.

[0075] In this example, the relationship relates to the results that each program provides to the test graph.

[0076] Result 1: Apply Program 1 to the test graph. Result 2: Apply Program 2 to the test graph. Result 3: Apply Program 3 to the test graph.

[0077] In this example, Result 1 is true if the graph is a tree, and false otherwise. In this example, Result 2 is true if the graph is connected, and false otherwise. In this example, Result 3 is true if the graph does not contain cycles, and false otherwise.

[0078] Relationship: Result 2 & Result 3 == Result 1. This means that the expected relationship is that Result 1 is true if both Result 2 and Result 3 are true, and false otherwise.

[0079] In one example, the user is given a hint along with their answer. The hint is determined to differ depending, for example, on whether the answer satisfies an expected relationship.

[0080] Example hints for answers that do not satisfy the relationship: "My result is "Program 1". Furthermore, here is the code for the substeps: "Program 2, Program 3". Unfortunately, I cannot be certain, because I tested Program 1 using a test graph. The test showed discrepancies between the results from Program 1 and those from the substeps. Therefore, please exercise caution when using this code in the future.

[0081] Checking whether a question satisfies expected relationships is used to determine metrics for evaluating the quality of generative, and especially multimodal, foundational models.

[0082] By using this method, indicators for evaluating quality can be determined.

[0083] This method includes step 202.

[0084] In step 202, the first answer to the first question is determined using a generative foundational model.

[0085] This method includes step 204.

[0086] In step 204, a generative foundational model is used to determine the second answer to the second question.

[0087] This method optionally includes step 206, which is performed, for example, when the relationship between three answers should be checked.

[0088] In step 206, a generative foundational model is used to determine the third answer to the third question.

[0089] This method includes step 208.

[0090] In step 208, the relationship is specified.

[0091] For example, the relationship between the first answer and the second answer is specified. If a third question is used optionally, the relationship between the first answer, the second answer, and the third answer is specified.

[0092] This method includes step 210.

[0093] In step 210, the indicator is determined based on the response and the relationship.

[0094] It can be assumed that multiple alternative expressions of a question will be presented to the base model. For example, the relationships between the answers provided by the base model to multiple alternative expressions of a question to the base model will be checked individually in each check.

[0095] The indicators are determined, for example, depending on the results of each check.

[0096] For example, the degree of confidence that the first and second answers satisfy the relationship is determined as an indicator.

[0097] If the third question is used at will, it can be assumed that the degree of confidence that the first, second, and third answers satisfy the relationship will be determined as an indicator.

[0098] For example, a value greater than zero represents confidence, while a value of zero represents no confidence.

[0099] For example, when multiple checks are performed, the confidence values ​​from each check are added or multiplied to form a total value, and an increase in the total value represents an increase in confidence.

[0100] For example, if the relationship between the first and second responses is met, the reward is determined as an indicator.

[0101] If the third question is used at the discretion of the participant, it can be assumed that the reward will be determined as an indicator if the relationship is satisfied by the first, second, and third answers.

[0102] For example, if this is not the case, no reward is determined. For example, a value greater than zero represents a reward, and a value of zero represents no reward.

[0103] For example, when multiple checks are performed, the reward values ​​from each check are added or multiplied to form a total value, and an increase in the total value represents an increase in reward.

[0104] This method includes step 212.

[0105] In step 212, it can be assumed that the confidence level will be output, in particular, to the user of the base model.

[0106] In step 212, it may be assumed that if the confidence level is determined to be below a threshold, the use of the base model for the application area containing the first and / or second questions will be blocked.

[0107] In step 212, it can be assumed that the base model is trained in a reward-dependent manner.

[0108] For example, the weight is determined based on the reward.

[0109] Depending on the result of the check, it may be assumed that a computer-controlled or regulated machine, such as a robot system, vehicle, home appliance, machine tool, manufacturing machine, personal assistance system, or access control system, will be driven depending on a first response, a second response, and / or a third response. For example, if the check is determined to be successful, the drive will be performed depending on the first response, the second response, and / or the third response. For example, if not, especially if the check is determined to be unsuccessful, the drive will not be performed depending on the first response, the second response, and / or the third response.

[0110] For example, in the case of a robotic system for grasping an object, the first digital image is an image of the object detected at a first position, and the second digital image is an image of the object detected at a second position. Whether or not the object is grasped at the second position depends on whether the relationship is satisfied such that the position in the answer to the second question differs from the position in the answer to the first question by a predetermined distance in a predetermined direction.

[0111] For example, in the case of a vehicle for autonomous driving, the first digital image is an image in which an object is detected at a first position, and the second digital image is an image in which an object is detected at a second position. For example, whether or not the vehicle runs over the object at the second position depends on whether the relationship is satisfied such that the position in the answer to the second question differs from the position in the answer to the first question by a predetermined distance in a predetermined direction.

[0112] Distance and direction are determined, for example, depending on a predetermined expected trajectory of the object. The trajectory is estimated, for example, depending on multiple detected images.

[0113] In the case of optical inspection, for example, a first digital image displaying an object is provided. The first digital image is transformed into a second digital image. The second digital image displays the object. For example, the relationship of how the first digital image is transformed into the second digital image is specified. This means that it is checked whether the transformation of the first image has changed to the expected display of the object in the second image.

[0114] Images are detected using, for example, cameras, radar sensors, LiDAR sensors, infrared cameras, motion sensors, or ultrasonic sensors.

[0115] Figure 3 schematically shows an exemplary data structure 300 for determining the indicators.

[0116] The data structure 300 includes at least one data field 302 for a generative foundational model, at least one data field 302 for a first answer to a first question determined using the generative foundational model, at least one data field 302 for a second answer to a second question determined using the generative foundational model, at least one data field 302 for the relationship between the first answer and the second answer, and at least one data field 302 for an index determined depending on the answer and the relationship.

Claims

1. For example, in a computer-implemented method for determining metrics for evaluating the quality of generative, particularly multimodal, underlying models for optical inspection, sound or technical component identification, natural language processing, program generation, or data labeling, Using the aforementioned generative base model, the first answer to the first question is determined (202), Using the aforementioned generative base model, the second answer to the second question is determined (204), The relationship between the first answer and the second answer is specified (208), The index is determined depending on the above answer and the above relationship (210) A method characterized by the following:

2. The degree of confidence that the first answer and the second answer satisfy the relationship is determined as an index (210). The method according to claim 1.

3. The confidence level is output to the user of the base model (212). The method according to claim 2.

4. If the confidence level is determined to be below a threshold, the use of the base model for the application area that includes the first question and / or the second question is blocked (212). The method according to claim 2 or 3.

5. If the first response and the second response satisfy the relationship, the reward is determined as an indicator (210), The aforementioned base model is trained in a manner dependent on the aforementioned reward (212), The method according to any one of claims 1 to 4.

6. The aforementioned base model includes an artificial neural network having weights, The aforementioned weight is determined depending on the aforementioned reward (212), The method according to claim 5.

7. The first answer includes a set of individual answers to the first question, The second answer includes a set of individual answers to the second question, The relationship is specified that the first answer and the second answer are sets of individual answers that are each relatively prime (208), or, The relationship is specified that the first answer is a subset of the second answer (208). The method according to any one of claims 1 to 6.

8. The first question was about a first program for carrying out a task in the first field (202), The second question was about a second program for carrying out tasks related to the second area (204), The first field described above encompasses the second field described above. The relationship is specified such that the first program and the second program solve a task relating to the second field with the same result (208). The method according to any one of claims 1 to 7.

9. A first digital image displaying an object is provided. The first digital image is converted by the transformation into a second digital image that displays the object. The first question asked was about the position of the object in the first digital image (202), The second question was about the position of the object in the second digital image (204), The relationship of how the first digital image is converted into the second digital image is specified (208). The method according to any one of claims 1 to 7.

10. A first audio sequence having a certain sound and a second audio sequence having a certain sound are provided. The first question asked was about the timing at which the sound occurred in the first audio sequence (202), The second question was asked about the timing at which the sound occurred in the second audio sequence (204), The relationship between the aforementioned time points is specified (208), The method according to any one of claims 1 to 7.

11. The first question asked about the results of chaining the first program with the second program to perform an input-related task (202), The second question concerns the results of a third program for performing the task with respect to the input (204), The relationship is specified such that the third program solves the task with the same result as the first program being chained with the second program (208). The method according to any one of claims 1 to 7.

12. The third answer to the third question was determined (206), With respect to the third answer, it is expected that the third answer includes the first answer and the second answer (208), or that the third answer is identical to the union of the first answer and the second answer (208), The method according to any one of claims 1 to 7.

13. For example, in an apparatus (100) for determining an index for evaluating the quality of a generative, particularly multimodal, base model for optical inspection, sound or technical component identification, natural language processing, program generation, or data labeling, The aforementioned device is At least one processor (102), At least one memory (104), Includes, The at least one memory (104) includes instructions, When the instruction is executed by the at least one processor (102), the method according to any one of claims 1 to 12 is performed in the apparatus (100), A device (100) characterized by the following.

14. It is a computer program, The computer program includes a computer executable instruction, and when the computer executable instruction is executed by the computer, the method according to any one of claims 1 to 12 is performed in the computer. A computer program characterized by the following features.

15. For example, in a data structure (300) for determining metrics for evaluating the quality of generative, particularly multimodal, base models for optical inspection, sound or technical component identification, natural language processing, program generation, or data labeling, The aforementioned data structure (300) is At least one data field (302) for the generative base model, At least one data field (302) for a first answer to a first question determined using the generative base model, At least one data field (302) for a second answer to a second question determined using the generative base model, A data field (302) for the relationship between the first answer and the second answer, A data field (302) for an index determined depending on the above answer and the above relationship, including, A data structure (300) characterized by the following.

Images