Method for generating and assigning comparative pairs to individual comparative tasks of test participants

By generating and allocating comparison pairs, optimizing the selection and allocation of pairwise comparisons, combined with the evaluation model, the training problem of regression model in the acoustic field is solved, the evaluation accuracy and data collection efficiency are improved, and the effective generation and technology of large-scale regression models are realized. Product optimization.

CN118916800APending Publication Date: 2024-11-08ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410554645.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-08
Filing Date
2024-05-07
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

It is difficult for the prior art to effectively train and apply regression models, especially in the field of acoustics. Model training requires a large amount of labeled data and the labeling process is highly subjective, resulting in extremely poor scalability of data labeling, which in turn limits the popularization and application of models.

Method used

By generating and allocating comparison pairs, using pairwise comparison methods to improve the accuracy of objective evaluation, optimize the selection and allocation of comparison pairs, reduce data collection costs, use computer systems and programs to realize this process, and combine the evaluation model to conduct objective evaluation .

Benefits of technology

It significantly improves the accuracy of objective evaluation of samples or sampling through pairwise comparison, reduces the cost and complexity of data collection, and can effectively generate large-scale regression models, suitable for technical product optimization in acoustics and other fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916800A_ABST
    Figure CN118916800A_ABST
Patent Text Reader

Abstract

A method for generating and assigning comparison pairs to respective comparison tasks of test participants for subjective individual evaluation of paired comparison, the method comprising:-providing (11) a dataset having sensory samples (43),-identifying (12) for each sample (43) potentially desirable comparison samples based on metrics, wherein the metric is based on a correlation between a possible estimate of an evaluation of a test participant and at least one quantitative feature of the sample (43),-assigning (13) a predetermined number of total comparisons to potentially desirable comparison pairs according to the following conditions:-providing each sample (43) the same number of times in the pair comparison, each sample (43) is provided as an answer option A and an answer option B for the same number of times in the paired comparison, generating (14) a comparison task for the test participant by assigning the comparison pairs to the comparison task, taking into account the following conditions: each sample (43) is provided for no more than one number of times in the comparison task; -each sample (43) is provided the same number of times for a predetermined subset of all test participants as a whole.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Due to the continuous development of artificial intelligence (KI) methods, the processing of sensory input parameters by computer-aided models is becoming increasingly common. Common applications in this context involve processing sensory impressions, such as those experienced by humans. This includes, for example, the classification of objects in camera images (e.g. in the context of highly automated driving), the classification of sounds in the context of security systems (recognition of alarm bells, human screams, gunshots, etc.) or the processing of texts (i.e. the classification of words on images).

[0002] Therefore, most applications in this field belong to the field of classification. For example, in the field of acoustics, there are hardly any AI models that can handle regression based on samples such as sounds (i.e., assignment of floating point numbers). One of the main reasons for this is that such models require a lot of input data for model training. In the case of classification, these input data are also relatively easy to obtain, because many different samples can be created from a single sample (the so-called sampling) by slight modifications, but without changing the class of the sample (the so-called label). For example, an image of a cat can be slightly rotated, the contrast changed, mirrored or other filters applied - the subject still shows a cat, the trained label is therefore not changed, and many different samples can be created from a single sample that still provide useful information for the model.

[0003] The same is true in the field of acoustic classification, where slight modifications of frequency or amplitude samples, adding sounds and similar modifications do not ensure, for example, that a warning bell is suddenly no longer a warning bell. Here, too, the labels remain unchanged, and by modifying the sampling, many new samples can be generated to train a data-intensive classification model, which make the model more robust.

[0004] However, in the domain of regression, such modifications change the labels. Imagine that the dominant frequency of a bell changes, in the case of which the model should answer the question: “Is this sound How comfortable are you on a scale of 1 to 10 (Skala)? ". Obviously, in this case, any modification to the samples will also change the labels in unpredictable ways. Therefore, models predicting such regression problems must be trained on samples that have been fully evaluated by humans in advance. In this case, not only the number of labeled samples required to train large AI models, but also the subjectivity of individual answers (hence the need to evaluate each sample by many trial participants to obtain an objective label) leads to the fact that such models are rare due to the poor scalability of data labeling.

[0005] In this context, the challenges of determining an overall rating that is as objective as possible based on subjective individual evaluations have long been known. This is not limited to customer feedback on products, but can also be a complex task when obtaining expert opinions or similarly set questions. In this case, the same questions often arise: How many people should be asked to obtain statistically stable results? How should the survey be designed? For example, should the test participants assign scores to the individual samples according to a defined answer scale, or should other types of surveys be used? In particular in the field of acoustics, the planning, implementation and evaluation of so-called listening tests is an intensively studied field, for which many guidelines already exist. In these cases, the method used is usually to present a scale (e.g., a score from 1 to 6 or a Rohrmann scale with different linguistically expressed answer possibilities) to the test participants, who should classify the presented samples according to these scales. Then, a mean value or other consideration of the answer distribution is performed in order to assign an overall rating to each sample. Alternatively, a pairwise comparison can also be carried out, in which, for example, the test participants should indicate their preference between the sound of two presented products. For example, the overall evaluation of each individual product based on many pairs of comparisons can then be determined by probabilistic methods. However, in order to obtain stable results, each comparison must be evaluated by many participants, and if all existing samples are compared with each other, a large number of comparison pairs will quickly result (for example, in the case of 100 product sounds, 100 2 = 10,000 combinations; if the comparison of identical sounds (A compared to A) is omitted, this number is reduced to 100 2 -100 = 9900; if it is reduced to only one comparison direction (i.e., always only A is compared with B, never B is compared with A), then the number is reduced to (100 2 -100) / 2=4950), this method seems inappropriate in many cases. In addition, when each of the two answer options is selected at least once (i.e., when the probability that A is evaluated as better than B is not 100% or 0%), which again increases the requirement for a large number of test participants, the method based on pairwise comparison can achieve the highest result accuracy.

[0006] Therefore, not only in the case of subjective evaluation based on scales, but also in the case of paired comparisons and other methods, the problem arises of having a sufficient number of test participants answer the questions, at least because the limits of human efficiency (e.g. with regard to attention and memory when evaluating different samples one after another) also represent additional restrictions on the number of test samples and also reduce the precision and thus the overall persuasiveness of the obtained results. Therefore, the training and technical application of large-scale regression models based on such data are rare. Summary of the invention

[0007] A first general aspect of the disclosure relates to a method for generating comparison pairs and assigning them to individual comparison tasks of test participants for the purpose of subjective individual evaluation of pairwise comparisons.

[0008] The method comprises: providing a data set with sensory samples; identifying a potentially promising comparison sample for each sample based on a metric, wherein the metric is based on a correlation between a possible estimate of the evaluation of the test participant and at least one quantitative feature of the sample; allocating a predetermined number of total comparisons to potentially promising comparison pairs according to the following conditions: each sample is provided the same number of times in the pairwise comparisons; each sample is provided the same number of times as answer option A and answer option B in the pairwise comparisons. The method also comprises: generating a comparison task for the test participants by assigning the comparison pairs to the comparison task in such a way that each sample is provided no more than once in the comparison task and each sample is provided the same number of times for a predetermined subgroup of the total number of all test participants, taking into account the following conditions: each sample is provided no more than once in the comparison task and each sample is provided the same number of times for a predetermined subgroup of the total number of all test participants.

[0009] A second general aspect of the present disclosure relates to a method for objectively evaluating a sensory sample, comprising providing a sensory sample and evaluating the sample using an evaluation model.

[0010] A third general aspect of the present disclosure relates to a computer system designed to perform the method according to the first and / or second general aspect (or embodiments thereof).

[0011] A fourth general aspect of the present disclosure relates to a computer program designed to perform the method according to the first and / or second general aspect (or embodiments thereof).

[0012] A fifth general aspect of the present disclosure relates to a computer-readable medium or signal storing and / or containing a computer program according to the fourth general aspect (or an embodiment thereof).

[0013] The techniques of the first, second, third, fourth and fifth general aspects may, in some cases, have one or more of the following advantages.

[0014] The disclosure leads to an increase in the accuracy of the objective evaluation of samples or samplings determined by pairwise comparisons, while at the same time significantly reducing the effort required for their collection, which is reflected, for example, in significantly reduced costs due to the optimized selection of promising comparison pairs.

[0015] The advantages of the present disclosure become apparent in particular when a large number of patterns (e.g., hundreds or thousands) are to be evaluated. This is done, for example, with the aim of subsequently building a model, which can be done, for example, with the goal of subsequently obtaining a model based on the large number of evaluations obtained, for example a model supported by artificial intelligence, for future predictions of the evaluation of new samples.

[0016] Due to the huge outlay for collecting data, it has been usually not possible to create such regression models until now or only at great expense. However, according to the present disclosure, the data can be obtained efficiently and with high quality, and thus these models can be generated, which can then be used, for example, to optimize the operation of technical products.

[0017] In addition (assuming the number of overall evaluations is high enough) the optimized allocation of pairwise comparisons to defined subgroups of the total population of all test participants also makes it possible to analyze the evaluation differences between the individual subgroups (Untergruppe), thereby even making it possible to user-specifically control the technical product or to develop products in a targeted manner for specific user groups based on the user data.

[0018] For example, the term "potentially promising" can be compared to Figure 2 and Figure 3 Here, explicit comparisons, i.e. "sample A is superior to sample B in 100% or 0% of the comparisons performed", therefore lead to a (theoretically) infinite distance between samples A and B. These comparisons therefore do not provide any additional information to the mathematical model for calculating the numerical distances between the samples and are accordingly not considered "potentially promising" comparison pairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart depicting the disclosed technique for generating comparison pairs and assigning them to individual comparison tasks.

[0020] Figure 2 A graph showing the association between the distance of the two BTL evaluations Δθ and the probability p(i>j) according to the present disclosure is shown.

[0021] Figure 3 A diagram for identifying potential comparison partners according to the present disclosure is shown.

[0022] Figure 4 Another chart for identifying potential comparison partners in accordance with the present disclosure is shown.

[0023] Figure 5 A diagram showing the assignment of these comparisons to acoustic pairs i, j and the assignment of these pairwise comparisons to individual comparison tasks k according to the present disclosure is shown.

[0024] Figure 6 Several systems are schematically shown in which the techniques of the present disclosure may be used to generate comparison pairs and assign them to individual comparison tasks. DETAILED DESCRIPTION

[0025] The method according to the present disclosure represents a process for efficiently determining an objective overall evaluation based on a subjective evaluation of paired comparisons, which achieves high result accuracy and can be effectively performed even in the case of a large number of comparison samples. Based on the obtained data, a data-intensive artificial intelligence model can then be trained and installed on the terminal device in order to optimize the control of the terminal device in terms of the comfort of the operating sound, for example, by using microphone signals in an acoustic context at the terminal device. Obtaining the required data is achieved by optimally selecting the following comparison pairs from the population of all possible comparison pairs, which are identified as particularly convincing based on the metrics for estimating the expected overall evaluation.

[0026] Figure 1 A method 10 is shown for generating and assigning comparison pairs to individual comparison tasks of test participants for subjective individual evaluation of pairwise comparisons.

[0027] First, a data set with sensory samples is provided 11. The sensory samples may include stimuli or stimuli perceptible to humans. For example, these are sounds, images, texts, temperatures, smells. This includes everything that can be perceived and evaluated by the human sensory system (Sensorik). The sensory samples are generated by sensors.

[0028] Potentially promising comparison samples are then identified 12 for each sample based on a metric, wherein the metric is based on a correlation between a possible estimate of the evaluation by the test participants and at least one quantitative characteristic of the sample. The metric can roughly estimate the expected evaluation and thus reduce the computational effort. As a metric for identifying promising comparison pairs from the population of all possible comparison pairs, for example, two simple parameters can be used: the samples are either ranked with respect to the maximum perceived aspect of their psychological sensation or with respect to their average amplitude.

[0029] If the predicted BTL distance is too large, i.e. the distance in the objective evaluation space is too large, the evaluation can be aborted, which increases efficiency. It is also possible to fill identified gaps with additional samples instead of aborting, in order to avoid explicit comparisons in subsequent tests. The objective evaluation can be, for example, a numerical evaluation.

[0030] A predetermined number of overall comparisons are assigned 13 to potentially promising comparison pairs according to the following conditions:

[0031] - each sample is presented the same number of times in the pairwise comparisons,

[0032] - Each sample was presented as answer option A and answer option B an equal number of times in these pairwise comparisons.

[0033] Furthermore, a comparison task 14 is generated for the trial participants by assigning comparison pairs to the comparison tasks in the following manner taking into account the following conditions:

[0034] - each sample is presented no more than once in the comparison task,

[0035] - Each sample is provided the same number of times to a predetermined subgroup of the total population of trial participants.

[0036] The predefined subgroups can be, for example, so-called sociocultural groupings and thus allow, for example, men and women to evaluate each sample with equal frequency. The attribution to one or more of the subgroups and / or sociocultural groupings can be checked by asking the test participants predefined questions in advance. For example, appropriate users of the survey can be selectively allowed to conduct individual tests by means of an initial question (e.g. "Do you have a passenger car driver's license?"). Having a driver's license is not necessarily considered a sociocultural attribute.

[0037] In individual cases, minor deviations from the stated specifications for the sample may occur because the previously defined total number of all pairwise comparisons is usually not a multiple of the number of pairwise comparisons in the individual comparison tasks for the test participants. The number of available samples and the number of required sociocultural groups among the test participants can also influence the mathematical feasibility of the stated criteria. Within the scope of the tests carried out to date, these deviations were of the order of magnitude of rounding errors and therefore had little influence on the results.

[0038] Another variant provides that a predetermined number of total comparisons are allocated to potentially promising comparison pairs, taking into account the further condition that a predetermined number of pairwise comparisons with obvious answers (so-called honeypot comparisons) are inserted. By inserting these pairwise comparisons with obvious answers, test participants who performed the test with insufficient attention can be identified and excluded from the evaluation.

[0039] Another variant provides that an objective evaluation is determined for each of the samples involved by evaluating the population of the individual pairwise comparisons performed on a plurality of samples by means of a mathematical model. The mathematical model may be a probabilistic model, for example the Bradley-Terry-Luce model, i.e. the BTL model. For example, the BTL model obtains a list of all completed comparisons, which includes the information on which of the two samples involved "won" the comparison in each case. This therefore results in a probability for each compared pair (e.g. "A compared to B is evaluated 73% in favor of A"). According to the above equation, these probabilities result in the BTL distance.

[0040] Another variant provides that the evaluation of the sample is derived from the probability of the sample preference in all pairwise comparisons in all comparison tasks. Thus, a final overall evaluation can be obtained for the sample, which takes into account all pairwise comparisons performed.

[0041] Another variant provides that, after the evaluation of the sample has been determined by the trial participants, the following is continued:

[0042] - Creation of evaluation models for objective evaluation of sensory samples, including:

[0043] - Define evaluation as output parameter ,

[0044] - annotate the sample with its corresponding rating, which is derived from the probability of preferring the sample across all pairwise comparisons in all comparison tasks, and

[0045] -Optimize the evaluation model by training it with annotated samples.

[0046] Thus, a model is formed based on the expertise of the test participants and the sample, which can then independently evaluate the sample as if it had been evaluated by a series of testers (probanden). The optimization of the model can be, for example, setting or optimizing the weights of a neural network. Deep learning or machine learning methods can also be used. The input parameters of the model can then be one or more quantitative features of the sample.

[0047] Thus, a method for objectively evaluating a sensory sample comprises providing a sensory sample and evaluating the sample using an evaluation model created as described above. Such a model may be, for example, an artificial intelligence model. The input parameters are quantitative features of the sample, such as the spectrum of the sample. For example, the spectrum of a sound may then be provided as input, and the output of the model may be an evaluation having a value of, for example, 4.2.

[0048] This method for objectively evaluating sensory samples can be independent and unrelated to the aforementioned method for generating comparison pairs and assigning them to individual comparison tasks. The two methods can also be combined with each other.

[0049] The described method for generating and assigning comparison pairs to the individual test participants can be considered an important preparatory step for carrying out tests to determine objective evaluations based on subjective individual evaluations, which are absolutely necessary for training data-intensive regression models. For example, in an application within the scope of a listening test, a list of comparison tasks is ultimately generated by the algorithm, each of which contains a defined number of pairwise comparisons (e.g. 100 pairwise comparisons, of which 93 pairs are actual product sound pairs and 7 comparisons are so-called "honeypot" comparisons). In addition, metadata such as the affiliation of the desired socio-cultural grouping of the test participants is stored for each comparison task, so that the comparison task can be offered to users corresponding to the desired socio-cultural profile via an application on a mobile terminal device. This form of assigning the test to a large number of participant circles around the world enables listening tests to be carried out quickly and cost-effectively even with a large number of sound samples (and therefore a high total number of pairwise comparisons).

[0050] A variant provides:

[0051] - Obtain sensory samples from test subjects,

[0052] - generate a review of the sample,

[0053] - manipulating at least one parameter of the test subject affecting the sample, and

[0054] - Repeat these steps until the evaluation reaches the target value.

[0055] By means of a control loop or closed loop, which is realized by manipulating at least one parameter of the test object or the test device, a desired operating state or an operating state that is comfortable for humans of the test object can be set.

[0056] In principle, the described method can always be applied when an objective evaluation of the samples is to be determined by pairwise comparison (particularly in the case of a large number of samples). The data obtained can then be used to form a model, and the resulting model can be used for the development or optimization of a technical application. In order to optimally select particularly promising comparison pairs for data collection, it is applicable in this case that a metric can be used to roughly estimate the expected evaluation.

[0057] The metric can be designed to be arbitrarily complex. The described method has been successfully tested many times in the execution of listening tests. The sounds to be evaluated in this case always come from technical products, such as heat pumps, electric bicycles, drones for delivering packages or dishwashers. These sounds can include the entire range of possible sounds of the product in this case, such as compressor-dominated sounds in the case of heat pumps, fan sounds, start-up sounds, sounds under partial load and full load, etc. This enables the model that is subsequently trained based on the acquired data to predict the human perception of sounds in all relevant operating states.

[0058] Initially, two simple parameters were used as measures for identifying promising comparison pairs from the population of all possible sound pairs: the sound samples were either ranked with respect to their psychoacoustic loudness or with respect to their average sound pressure level. Then, an algorithm for generating these pairs and assigning them to the comparison tasks of the individual test participants ensured that each sound could only be paired with a limited number of adjacent sounds with respect to the selected measure in order to avoid clear sound pairs for which all test participants would give the same answer with a high probability. From the range of possible comparison partners for each sound sample (e.g., 10 adjacent sounds for a smaller loudness / sound pressure level and 10 adjacent sounds for a larger loudness / sound pressure level), the algorithm now formed and assigned these pairs in such a way that the described requirements were met as far as possible.

[0059] In the subsequent listening test, the nonlinear correlation between loudness or sound pressure level and the expected overall evaluation is used as a measure for identifying promising sound pairs, wherein the correlation has been determined based on the results of the listening test already performed. It is also possible to use a preliminary evaluation by a model, for example by means of an existing algorithm for evaluating sounds in view of the expected evaluation thereof by the test participants.

[0060] In order to identify the listening test participants who perform the test with insufficient attention, in addition to the actual comparison pairs of product sounds, obvious sound pairs (so-called "honeypots") that do not meet the criteria set forth for the metric used for the pre-evaluation are also added. For example, in this case, the insertion of a pairwise comparison between the 10 loudest and 10 quietest sounds of all product sounds to be evaluated has been successfully tested. The basic assumption is that users who perform the listening test conscientiously and not just interested in a quick "click" to obtain a financial reward for completing the test will always prefer very quiet products to very loud products. In individual cases where this assumption does not exist at least for the relevant share of the entire user circle (e.g. when evaluating sports engine sounds), specially selected "bad samples" (e.g. the sound of a dentist's drill) are also inserted in these individual cases to avoid ambiguity in the correlation between loudness and evaluation.

[0061] Based on the acquired data, data-intensive models (e.g. neural networks) can then be trained, which predict the pleasantness of the product sound. These models can then be applied to the technical product and fed by the microphone signal, wherein the current pleasantness of the generated product sound is predicted from the microphone signal and carried out during operation: the control of the technical product, such as a heat pump, is optimized in view of this parameter.

[0062] In the aforementioned listening test, a large number of pairwise comparisons were distributed globally to users for evaluation via a crowdsourcing application for mobile terminal devices, whereby a specified composition (Zusammensetzung) of the population of all test participants was taken into account if necessary (e.g. in terms of their gender, age, continental origin, opinions on pre-specified questions (e.g. "Do you like loud engine sounds?") or similar socio-cultural factors). The optimal allocation of the comparison tasks to specific socio-cultural groupings also enables subsequent user-type-specific predictions of the pleasantness of the product sound by means of a model trained on the basis of the acquired data. An electric bicycle or a car can be used as an example here, where the current driver is informed of the product before the start of the journey, for example via an application or a display mounted on the product. By means of targeted questioning when such a user profile is first created, each driver of this example product already knows not only the general socio-cultural parameters (gender, age, etc.) but also the specific user preferences ("Do you prefer a sporty driving experience?"). The current product sound can therefore be predicted by the trained model based on the microphone signals for precisely those user groups who answered the questions asked within the scope of the listening test in a similar way to the current driver and therefore correspond to the same user type.

[0063] The product sound can then be optimized in such a way that it best matches the possible preferences of the current user. The method can also be used within the scope of acoustic product development, for example by preparing the sound of the brake system in a wide variety of operating states for a listening test and training a prediction model based on the data obtained. Such a listening test has already been successfully carried out within the scope of this project. The model can now be used on a smartphone that receives the current acoustic signal in good recording quality via a pluggable USB microphone to predict the human perception of the current brake sound. This allows the brake sound to be assessed immediately and objectively during a test drive.

[0064] Since such an objective evaluation has hitherto only been possible by means of a complex recording of the sounds and a subsequent evaluation within the framework of a listening test, developers have hitherto generally only been able to use the subjective opinions of the test drivers in this regard. The method can also be used outside the field of acoustic evaluation. Thus, for example, it is conceivable to evaluate different product designs. For example, an evaluation of the designs of different power tools can be used as product examples, the appearance of which is to be evaluated with regard to its impact (Durchschlagskraft) in order to identify the most attractive and powerful product design on the basis of a subjective evaluation. In this case, the selectable samples can, for example, be selected with regard to the device parameters. The aspects are ordered and only a limited number of adjacent samples with respect to the device parameters are allowed to be compared pairwise by the algorithm similarly to the scope of the listening test.

[0065] These data can then also be used to train a model that is able to predict the customer's likely opinion of a design immediately during product development. It is also conceivable to apply the method to other areas of visual subjective evaluation, such as evaluating the affective emotions of human facial expressions to obtain data for further applications of machine learning. Here, subjective evaluations of human facial expressions (e.g. "Which of the two people shown looks happier?") can be used as an example. In this case, the identification of promising comparison pairs can be carried out, for example, by pre-sorting all images to be evaluated in advance with respect to the distance between the two corners of the mouth and / or the degree of visibility of the teeth and / or the height (Erhebung) of the corners of the mouth above or below the middle of the mouth. Such data can then be used to train a model that evaluates the current facial expression in real-time applications (e.g. a smartphone camera in "selfie" mode). Such an application is also conceivable within the scope of security systems.

[0066] As already described, a population of many individual pairwise comparisons can be evaluated by a probabilistic model to determine a numerical evaluation for each of the samples involved. A popular model here is the so-called Bradley-Terry-Luce model (BTL model), which determines a so-called BTL evaluation Θi for each sample i involved. The basic assumption is that based on the probability p(i>j) describing the share of experimental participants who prefer sample i to sample j in a pairwise comparison, there exists a distance ΔΘ=Θi-Θj between the BTL evaluations of these two samples according to the following equation:

[0067]

[0068] The relationship between the probability p(i>j) and the distance Δθ between the two BTL evaluations is as follows Figure 2 shown.

[0069] From this representation, it is clear that in the case of perfect agreement among all trial participants (i.e., if 100% or 0% of all trial participants prefer sample i over sample j), the theoretical distance between these two samples on the BTL scale (BTL-Skala) will assume an infinite value. In practice, numerical implementations of such probabilistic methods always yield a finite BTL distance, which, however, may take on very large values. Therefore, such explicit comparisons with p(i>j) = 100% or p(i>j) = 0% come at the expense of the accuracy of the results and also represent an unnecessary cost factor, since these comparisons have been performed without the possibility of providing information gain to the BTL model.

[0070] The object of the present disclosure is in particular to optimize the selection of comparison pairs and the assignment of these comparisons to trial participants, with the aim of subsequent evaluation by probabilistic methods such as the described BTL model. The data required for training large regression models can thereby be efficiently collected, and models based on these data can then be used to optimize technical products. By developing a metric for identifying in an optimized manner those samples that are expected to give rise to similar subjective impressions, and reducing the possible comparison pairs to only this comparison space, the probability for the occurrence of clear comparison pairs with p(i>j)=100% or p(i>j)=0% is significantly reduced during the course of the trial.

[0071] As mentioned above, in the case of a listening test, for example, the loudness of a sample is the most important parameter for the subjective perception of the product sound of almost all products, which can be used to pre-rank the sound samples so that the samples are subsequently only compared with a certain number of neighboring sounds with respect to loudness. Alternatively, however, a more complex association can also be established between the parameters describing the sound and its expected BTL distance (and thus the implicit probability p(i>j)). In principle, the measure can thus be designed to be arbitrarily complex. However, in any case, it must provide an estimate of the subsequent BTL distance based on the known parameters. The determined sound pairs are then assigned to individual tasks of the test participants by the described algorithm.

[0072] The derivation of a measure that provides an estimate of the expected distance in the target variable can be performed as follows. For example, in the case of a listening test, this can be the loudness of the sound samples, or better yet, it can be an equation for estimating the expected BTL distance between the individual samples, which can be calculated based on values ​​such as loudness, sound pressure level or similar values. Pre-trained artificial intelligence models based on previous listening tests can also be used here. For example, it can also use more complex inputs, such as spectral representations of sounds, to pre-evaluate the BTL distance. If the user's data set has too large gaps in terms of the predicted BTL distance between the sound samples, the algorithm can also warn the user at this point (Stelle), because there is an increased risk of a clear comparison answer (p(i>j)=100% or p(i>j)=0%) at these points.

[0073] The identification of potential comparison neighbors for each sound sample based on the derived metric can be performed as follows. Figure 3 The first option, shown in , provides for the use of a fixed number of neighbors for each (sound) sample, which are then available as potential comparison partners. Figure 2 , the sample index i of one sample is plotted along the x-axis, and the sample index j of the other sample is plotted along the y-axis.

[0074] For example, in the case of a listening test, for each sample: the 10 second loudest sounds from the entire data set to be evaluated. The loudest sound sample and the 10 second quietest sound samples are used as potential comparison objects. Figure 3 The case for 100 sound samples is shown in , where the value 1 (black) means that a given comparison pair can be used as a potential pair, where these noises have been pre-sorted according to their BTL evaluation estimated based on loudness.

[0075] according to Figure 4The second option shown in provides for using a variable number of neighbors per (acoustic) sample, the neighbors being identified, for example, by the number of neighbors within a certain distance with respect to this metric. Figure 3 , the sample index i of one sample is plotted along the x-axis, and the sample index j of the other sample is plotted along the y-axis.

[0076] For example, in the case of a listening test, an equation for estimating the BTL distance of all samples can be used to identify for each sample those other sound samples that are within a certain BTL distance of the specified sample. The threshold to be used can be defined by the user of the algorithm. For example, comparisons are only allowed within a BTL distance of ±0.7.

[0077] In order to prevent that, in particular at the two extremes of the data set, the expected BTL evaluation of it by the criterion does not occur: it is limited to only very few or even only a single neighboring sample, a logic can also be implemented, which searches for potential comparison partners within a specified BTL distance threshold, but uses at least a number of neighbors that is still to be specified (according to the logic "take all neighbors within the BTL range of ±0.7, but at least 5 on both sides, if available"). In the extreme case, no neighbor samples appear, for example, no neighbors are within the defined BTL distance (for example + / -0.7). Here, a logic can also be implemented to terminate the algorithm, introduce new samples or, for example, extend the BTL distance. .

[0078] The example shown is Figure 4 , where the value 1 (black) means that the given comparison pair can be used as a potential pair, where the sounds have been pre-sorted according to their estimated BTL ratings.

[0079] All comparison pairs and comparison tasks can be generated as follows. From the set of potential comparison pairs, the pairs that should actually be evaluated in the subjective evaluation are now determined. The number of overall pairs can in this case be determined, for example, by a fixed value pre-given by the user of the algorithm, which describes the number of pairs per sample to be generated by the algorithm. As an example, a case can be mentioned in which the user of the algorithm specifies a value for a listening test of 100 pairs for each sound sample, wherein a total of 800 sound samples are available in the data set.

[0080] The algorithm will now generate for each sample 100 pairwise comparisons with neighbors within the set of identified potential comparison partners for the relevant sample. This results in a total set of 100*800=80,000 pairwise comparisons within the data set. In this terminology, it should be noted that since two samples always appear in the 80,000 pairwise comparisons, a total of 160,000 sounds appear, i.e., on average each sound does not appear 100 times (100x) in the pairwise comparisons, but 200 times (200x).

[0081] The algorithmic assignment of these comparisons to the individual sounds i or to the individual sound pairs i, j is performed in the same way as the assignment of all pairwise comparisons to the individual comparison tasks k for the individual test participants, taking into account the mentioned boundary conditions. This can be illustrated as follows: All test participants receive a task with a certain number of pairwise comparisons (of which a small subset are again so-called honeypot comparisons, which are used to identify negligent test participants). The assignment problem can therefore be understood as a three-dimensional tensor, in which the three dimensions are the occurrence of sound samples i and j (for a total of N samples) and the attribution to a specific comparison task k (in the case of a total of M comparison tasks). This can be illustrated in the very simple case of only 30 sound samples and 5 comparison tasks, for example according to Figure 5 Provide explanation.

[0082] In practice, the number of samples can be significantly higher, hundreds or thousands of samples x. First, it is now necessary to determine the comparisons to be assigned to the population. These depend on the desired accuracy of the final objective evaluation, the number of samples to be evaluated, the number of comparisons for each comparison task, and the number of honeypot comparisons for each comparison task. In the example with 800 samples, where 100 pairwise comparisons with neighbors are to be created for each sample, where a single comparison task for the trial participant should also contain 100 comparisons, of which again 7 are honeypot comparisons, the total number of pairwise comparisons obtained is:

[0083]

[0084] The resulting number of comparison tasks is calculated as follows (rounded up, since no task fragmentation is permitted):

[0085]

[0086] The algorithm now has the task of distributing the set of available total comparisons among the existing empty tensor entries (Tensoreintrag) in such a way that the mentioned boundary conditions are met. In particular, this means:

[0087] - Only an integer number of entries greater than or equal to zero are allowed

[0088] - All comparison tasks k contain a predefined number of product comparisons (e.g. 93 comparisons) in total, to which a predefined number of honeypot comparisons (e.g. 7) are subsequently mixed, which means that the sum of all "real" entries of comparison task k must be 93 in this exemplary case. That is, for N sounds, the number of pairwise comparisons is:

[0089]

[0090] - Compare only within potential comparison partners defined based on the stated metrics (honeypot comparisons are an exception)

[0091] - Each sample appears the same number of times in the comparison, that is, for each sample, the sum of the column sums i and the row sums j over all K comparison tasks is ideally the same size,

[0092] - In the pairwise comparisons, each sample appears the same number of times as answer option A and answer option B, i.e., for each sample, the sum of the column sums i for each sample across all K comparison tasks is ideally equal to the sum of the column sums j across all K comparison tasks.

[0093] - Ideally, for each comparison task, the frequency of each sample does not exceed once, that is, the sum of the column sum i and the row sum j for each of the N sound samples in a single comparison task k is no greater than 1.

[0094] - The samples are assigned to the comparison tasks in such a way that they can be output to the test participants in such a way that each sound sample is ideally evaluated with equal frequency by each of the L number of pre-specified socio-cultural groups. Due to the large number of comparison pairs to be assigned, the pairwise comparisons contained in the individual comparison tasks for the individual test participants are usually always composed in a different way, i.e. there are no identical comparison tasks between different participants. Taking into account the proposed boundary conditions, the solution of the formulated problem can be achieved very efficiently by using a numerical solver in view of the complexity and size of the algorithm.

[0095] Figure 6 Exemplary implementations of software and / or hardware of the present disclosure are schematically shown.

[0096] The system 40 , such as a computer system, is designed to objectively evaluate sensory samples.

[0097] The test object 41 can be a component of the system 40 or can be designed outside the system 40. The test object 41 is, for example, a technical system, such as a heat pump, etc. The sensor 42 detects a parameter, such as temperature or sound, of the test object 41. The sensor 42 then outputs a sample 43, such as a frequency spectrum or an amplitude curve.

[0098] The computing device 44 receives the sample 43 at an input. The computing device 44 may include a processor and / or a working memory and a (non-volatile) memory. The computing device also includes the evaluation model described previously for objectively evaluating the sensory sample 43. The sensor 42 may be an integral part of the computing device 44. The (computer) system 40 may correspond to the computing device 44.

[0099] Processing of the measured values, for example into samples 43 , may be provided in the sensor 42 and / or in the computing device 44 .

[0100] The computing device 44 is designed to apply the evaluation model to the received sample 43 and determine an evaluation of the sample 43 , for example a sensory evaluation.

[0101] This evaluation can now be output, for example, optically on an output device in order to output the evaluation of the sample 43. The output device can be a component of the computing device 44 or be designed externally thereto.

[0102] Likewise, the computing device 44 can manipulate a component or at least one parameter of the test object 41 in the sense of a feedback or control loop, which affects a corresponding quantity or sample 43 of the test object 41. The changed sample 43 is then evaluated again by the computing device 44. This control or optimization continues until a target value and / or an optimal value is reached.

[0103] Also disclosed is a computer program designed to perform the (computer-implemented) method 10 for generating comparison pairs and assigning them to individual comparison tasks and / or the (computer-implemented) method for objectively evaluating sensory samples. The computer program can exist, for example, in an interpretable or compiled form. It can be (even also partially) loaded into the RAM of a computer for execution, for example as a sequence of bits or bytes.

[0104] A computer readable medium or signal storing and / or containing a computer program is also disclosed. The medium may include, for example, one of a RAM, ROM, EPROM, HDD, SDD... on which / in which a signal is stored.

Claims

1. A method for generating comparison pairs and assigning them to individual comparison tasks of trial participants for subjective individual evaluation of pairwise comparisons, the method comprising: - providing (11) a data set with sensory samples (43), - identifying (12) for each sample (43) a potentially promising comparison sample based on a metric, wherein the metric is based on an association between a likely estimate of the evaluations of the trial participants and at least one quantitative characteristic of the sample (43), - the predetermined number of total comparisons is distributed (13) to potentially promising comparison pairs according to the following conditions: - each sample (43) is provided the same number of times in the pairwise comparisons, - each sample (43) is presented an equal number of times as answer option A and answer option B in the pairwise comparisons, - Generate (14) a comparison task for the trial participants by assigning comparison pairs to the comparison task in the following manner taking into account the following conditions: - each sample (43) is presented no more than once in the comparison task, - Each sample (43) is provided the same number of times to a predetermined subgroup of the total population of trial participants.

2. The method of claim 1, wherein the sensory sample (43) comprises a stimulus perceptible to humans.

3. The method according to claim 1 or 2, wherein the subgroups are socio-cultural groupings and / or wherein attribution to one or more of the subgroups is checked by asking the test participants predefined questions.

4. A method according to any one of claims 1 to 3, wherein, taking into account the further condition of inserting a predetermined number of pairwise comparisons with obvious answers, a predetermined number of total comparisons are assigned to potentially promising comparison pairs, and comparison tasks in which one or more pairwise comparisons with obvious answers are answered incorrectly are optionally excluded from further processing.

5. The method according to any one of claims 1 to 4, further comprising: after completing the comparison task: - determining an objective evaluation for each of the samples (43) involved by evaluating the population of the pairwise comparisons performed on each of the plurality of samples by means of a mathematical model.

6. The method according to claim 5, wherein the mathematical model is a probabilistic model, such as a Bradley-Terry-Luce model, ie, a BTL model.

7. The method according to claim 5 or 6, wherein: The evaluation of a sample (43) is derived from the probability of preferring the sample in all pairwise comparisons in all comparison tasks.

8. The method according to any one of claims 5 to 7, after completing determining the evaluation of the sample (43), the method further comprises: - Creation of an evaluation model for objectively evaluating sensory samples (43), including: - defining said evaluation as an output variable, - annotating said sample (43) with its corresponding rating, said rating being derived from the probability of preference of the sample in all pairwise comparisons in all comparison tasks, and - Optimizing the evaluation model by training it with annotated samples.

9. A method for objectively evaluating a sensory sample (43), comprising: - provide sensory samples (43), and - evaluating the sample (43) using an evaluation model created according to claim 8.

10. The method according to claim 9, further comprising: - obtaining a sensory sample (43) from a test subject (41), - generating an evaluation of said sample (43), - manipulating at least one parameter of the test object (41) affecting the sample (43), and - Repeat these steps until the evaluation reaches the target value.

11. A computer system, the computer system being designed to execute the method (10) according to any one of claims 1 to 8 and / or the method (10) according to claim 9 or 10.

12. The computer system according to claim 10, comprising a sensor system for detecting the sample (43) and / or an output device for outputting an evaluation of the sample (43).

13. A computer program, the computer program being designed to carry out the method (10) according to any one of claims 1 to 8 and / or the method (10) according to claim 9 or 10.

14. A computer readable medium or signal storing and / or containing a computer program according to claim 13.