Multi-decision maker collaborative decision-making method, device and application

Abstract

The present invention discloses a method, device and application for collaborative decision-making by multiple decision makers, which relates to the field of human-machine collaborative technology and is used to improve the reliability of collaborative decision-making by multiple decision makers. The present invention determines the adaptive weight of each decision maker in the decision-making task based on the capability vector of each decision maker and the information to be decided. Each decision maker scores each decision option separately and then uses its decision weight to weight it to obtain the final score of each decision option, and then selects the final decision option based on the final score. The capability vector and weight generation model parameters in the present invention are updated and learned end-to-end according to the accuracy of the final decision, thereby improving the reliability of collaborative decision-making.

Description

Technical Field

[0001] The present invention relates to the field of human-machine collaboration technology, and in particular to a multi-decision maker collaborative decision-making method, a multi-decision maker collaborative decision-making device, and applications in image classification and management decision-making. Background Art

[0002] Human-machine collaboration in a narrow sense refers to the collaborative decision-making by humans and artificial intelligence models on the same information to be decided, taking into comprehensive consideration the decision-making results of humans and artificial intelligence models, and leveraging human creative thinking and accumulated experience, as well as the high-speed computing capabilities of artificial intelligence models, to improve the efficiency and quality of decision-making.

[0003] On this basis, the broad concept of human-machine collaboration can be derived, that is, in addition to the collaboration between humans and artificial intelligence models, it also includes scenarios where collaboration is entirely carried out by humans, and scenarios where collaboration is entirely carried out by artificial intelligence models.

[0004] Existing human-machine collaborative decision-making can be broadly divided into two categories. The first involves first selecting a decision-maker, determining whether the decision will be made by a human or an AI model. The selected decision-maker then produces the final decision. This approach completely separates humans from AI models and fails to comprehensively consider the decision-making strengths of both. The second approach involves both humans and AI models first generating alternative decision outcomes, then voting on these alternative outcomes to determine the final decision. Although this approach obtains alternative decision outcomes from both humans and AI models in the interim, the final decision outcome remains a single candidate outcome and fails to integrate the strengths of both. Even when considering fusing the alternative decision outcomes from humans and AI models, it fails to consider the decision-making capabilities of each decision-maker based on the different information at hand. Instead, it assumes that each decision-maker's decision-making capabilities across different expertise and fields are constant. This is clearly inconsistent with objective reality and renders the final fused decision outcome unreliable. Summary of the Invention

[0005] The object of the present invention is to provide a method, device and application for collaborative decision-making by multiple decision-makers to address all or part of the above-mentioned problems, so as to improve the reliability of collaborative decision-making by multiple decision-makers.

[0006] The technical solution adopted in the present invention is as follows:

[0007] A multi-decision maker collaborative decision-making method, comprising:

[0008] Vectorize the information to be decided to obtain a representation sequence;

[0009] Obtaining the capability vector of each decision maker respectively, splicing it with the representation sequence, and inputting it into the weight generation model to obtain the decision weight of each decision maker respectively;

[0010] Obtaining scores of decision makers for each decision option of the information to be decided;

[0011] For each decision option, the score of each decision maker is weighted using the decision weight to calculate the final score of each decision option of the information to be decided.

[0012] To solve the above problems, the present invention also provides a multi-decision-maker collaborative decision-making device, including a processor and a storage medium, wherein the storage medium stores a computer program, and when the processor runs the computer program in the storage medium, it executes the above-mentioned multi-decision-maker collaborative decision-making method.

[0013] The present invention also provides a method for image classification using the above-mentioned multi-decision maker collaborative decision-making method, wherein the information to be decided is the classification of the target image, and the decision options are preset decision options.

[0014] The present invention also provides a method for making management decisions using the above-mentioned multi-decision maker collaborative decision-making method, wherein the information to be decided is to determine decision options for a target decision scenario.

[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0016] The present invention introduces energy vectors of different decision makers in collaborative decision-making. The energy vector indicates the decision-making ability of the decision maker to deal with the decision information, thereby generating a decision weight equivalent to the decision-making ability of the decision maker through the weight generation module, and then weighting the decision results (i.e., scores) based on the decision weights of different decision makers, and further calculating the final score of each decision option. This not only gives play to the advantages of multi-person decision-making, but also fully considers the decision-making ability of each decision maker, making the final decision result more objective and reliable. In addition, the energy vector of the decision maker introduced in the present invention can also be designed as a learnable energy vector. By learning the features of the decision information sample, the energy vector of each decision maker is optimized, and a more complete representation of decision-making ability is learned, which further improves the reliability of the final decision result. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0018] Figure 1 This is a flow chart of the multi-decision maker collaborative decision-making method provided in an embodiment of the present application.

[0019] Figure 2This is a flowchart of dimensional alignment of input and output vectors of the weight generation model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] All features disclosed in this specification, or all steps in the disclosed methods or processes, except mutually exclusive features and / or steps, can be combined in any manner.

[0021] Any feature disclosed in this specification (including any appended claims and abstract), unless otherwise stated, may be replaced by other equivalent or similar features. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0022] In response to the problem in existing human-machine collaborative technology that the decision results of humans and artificial intelligence models are considered independently, or are integrated through a stereotyped fixed fusion method, making the collaborative decision results unreliable, the embodiments of the present application provide a multi-decision-maker collaborative decision-making method, device and application, which optimizes the contribution of different decision-makers to the collaborative decision results by introducing capability vectors indicating the decision-making capabilities of different decision-makers, thereby improving the reliability of collaborative decision-making.

[0023] In some embodiments, as Figure 1 The multi-decision maker collaborative decision-making method includes the following processes:

[0024] S1. Vectorize the information to be decided to obtain a representation sequence.

[0025] The information to be decided describes the decision context and the corresponding decision problem. In different scenarios, the information to be decided takes different forms.

[0026] For example, in problem-solving scenarios, the information to be decided is often textual. For this type of information, we segment the decision context and problem described in the information, convert each segmented word into an embedding vector using word embedding, and then concatenate the embedding vectors to obtain a representation sequence of the information to be decided.

[0027] In some classification decision-making scenarios, such as image recognition, the information to be decided includes not only text but also image information. For this type of information, the image serving as the decision context is encoded, the question text is segmented and embedded, and finally, the image encoding sequence and word embedding vectors are concatenated to obtain a representation sequence of the information to be decided.

[0028] S2. Obtain the capability vector of each decision maker respectively, concatenate it with the representation sequence, and input it into the weight generation model to obtain the decision weight of each decision maker.

[0029] The decision maker's capability vector indicates their decision-making ability. It is a multidimensional, continuous-valued vector used to comprehensively measure the decision maker's capabilities related to the decision-making task. In the embodiments of the present application, regardless of whether the decision maker is a human or an artificial intelligence model, the decision-making ability of the decision maker is measured using a unified dimension, without the need to consider humans and artificial intelligence models separately. The decision maker's energy vector determines the decision weight of their participation in the decision.

[0030] In some embodiments, the capability vectors of each decision maker are all preset capability vectors, or are all learnable capability vectors. The so-called preset energy vector is an energy vector set in advance for each decision maker, and its value does not change. For example, the energy evaluation index is used to construct the capability vector of the decision maker, and the energy vector is constructed by taking the professional knowledge question and answer scores, mathematics test scores, intelligence test scores, etc. as components. As an implementation method that is more favorable to the reliability of the final collaborative result, the use of a learnable energy vector will continuously optimize its value to better match the decision-making ability of the decision maker. For the learnable energy vector, it can be continuously optimized based on a randomly initialized energy vector, or it can be continuously optimized based on a certain set initial value, such as continuously optimized based on the energy vector constructed using the energy evaluation index in the previous article. The learnable energy vector can learn a more complete representation of decision-making ability, thereby obtaining a decision weight that is more matched with the decision-making ability of the decision maker, further improving the reliability of collaborative decision-making.

[0031] As a feasible approach, the above-mentioned learnable ability vector is trained using decision information examples with score labels, with the goal of minimizing the distance between the final score of the decision information example and the corresponding score label.

[0032] The so-called decision information samples are historical decision information samples that have already obtained the final decision results for their decision options. Based on the learning of a sufficient number of decision information samples, the value of the energy vector is continuously updated to make the decision-making ability it reflects more accurately.

[0033] The embodiment of the present application provides a method for learning energy vectors:

[0034] Assuming that there are m decision makers involved in the decision-making, there are m capability vectors, which are represented as {c1,…,c m}.

[0035] Let θ represent the parameters of the weight generation model, x represent the representation sequence of the information to be decided, and f(.) represent the operation of the weight generation model. The decision weight of each decision maker obtained by the weight generation model is expressed as:

[0036] w=(w1,w2,…,w m )=f(c1,…,c m,x,θ),

[0037] Where w1, w2, …, w m They represent the decision weights of the m decision makers respectively.

[0038] In addition, in some preferred embodiments, in order to avoid the weight generation model from confusing the representation sequence and the energy vector, a segmentation embedding vector can be added between the two parts of the embedding vector, usually the embedding vector of the text delimiter.

[0039] Assuming that there are n decision options generated for the decision information sample, the scores of the n decision options by m decision makers constitute an m×n dimensional matrix S∈R m×n The scoring matrix is ​​weighted using the decision weights w of m decision makers to obtain the final score s of n decision options. f ∈R 1×n for:

[0040] s f =wS.

[0041] To facilitate the operation of the training process, Softmax is used to normalize the final score:

[0042]

[0043] represents the final score of the i-th decision alternative.

[0044] The one-hot scoring labels of N decision information samples are expressed as {y (1) ,…,y (i) ,…,y (N)},i∈[1,N], Indicates the kth position of the i-th evaluation label. For N decision information samples, the corresponding score matrix {a (1) ,…,a (i) ,…,a (N)},i∈[1,N],

[0045] The goal is to train the energy vector to minimize and distance.

[0046] In some embodiments, the cross entropy loss is used to measure the distance between the final score of the decision information example and the corresponding score label. The cross entropy loss is expressed as:

[0047]

[0048] When there are enough decision information samples, the optimization of cross entropy loss can update the values ​​of each ability vector.

[0049] The weight generation model in the embodiments of the present application uses a model that maps a vector sequence to a few weight values. A variety of basic network structures can be used, such as encoder structures of Transformers such as BERT, Roberta, and Longformer, decoder models of Transformers such as GPT, Palm, and Transformer-XL, models with both encoding and decoding capabilities such as BART and XLNet, and traditional deep learning or machine learning model architectures with text processing capabilities such as RNN, LSTM, GRU, bi-LSTM, and bi-GRU.

[0050] In addition, when the length of the decision maker's capability vector is inconsistent with the length of the representation sequence of the decision information, it is impossible to directly splice the vectors. Figure 2 As shown, a first linear layer is placed before the weight generation module to unify the dimensions of the capability vectors and representation sequences before concatenation. Furthermore, for the selected basic network structure, the output layer dimensions must be configured so that each decision maker receives a unique decision weight. This can be achieved by modifying the output layer dimensions or placing a second linear layer after the output layer to match the number of decision makers.

[0051] In some embodiments, the parameter θ of the weight generation model can be updated with reference to the energy vector optimization method to further improve the accuracy of decision weight generation. Specifically, the weight generation model is trained using decision information examples with score labels, with the goal of minimizing the distance between the final score of the decision information example and the corresponding score label.

[0052] As a feasible approach, in the embodiment of training the capability vectors described above, the parameter θ of the weight generation model can be updated during the process of optimizing the cross entropy loss and updating the values ​​of the capability vectors.

[0053] In a specific implementation, the value of the ability vector and the parameter θ of the weight generation model can be updated simultaneously or alternately during the process of optimizing the cross entropy loss.

[0054] S3. Obtain scores of each decision maker for each decision option of the decision information.

[0055] In some embodiments, each decision option is a decision option generated by each decision maker based on the information to be decided, or is a preset decision option.

[0056] Preset decision options—predetermined and defined decision options for the information to be decided—are a priori settings, valid for any decision maker and unaffected by their subjective influence. Decision makers only need to rate these decision options individually. For example, in some image classification scenarios, the classification category of the target image is predefined.

[0057] The decision options generated by the decision maker can be generated by the decision maker who participated in the decision or by other decision makers who did not participate in the decision. The difference between the default method and the generated decision options is that they are subject to the subjective influence of the decision maker. This is not limited in the embodiments of the present application.

[0058] In some embodiments, the decision options generated by the decision maker may include the following forms:

[0059] 1) Generated independently by each decision maker. For example, when both humans and AI models are decision makers, the decision options are generated by humans and AI models separately and then aggregated.

[0060] 2) Humans generate examples of some decision options, and then the artificial intelligence model generates a series of other decision options based on the examples.

[0061] 3) The artificial intelligence model generates some decision options, and then humans refer to these decision options and generate more decision options based on their inspiration.

[0062] Each decision maker needs to score each decision option separately to reflect each decision maker's judgment on the degree to which each decision option conforms to the information to be decided.

[0063] S4. For each decision option, weight its score using the decision weight of each decision maker, and calculate the final score of each decision option of the information to be decided.

[0064] For each decision option, after obtaining each decision maker's score for that decision option, each decision maker's decision weight is used to perform a weighted calculation to obtain the final score for that decision option. As a feasible approach, the scores can be weighted by weighting them, either by weighted summation or by weighted averaging. In the weighted averaging calculation, the decision weights of each decision maker are normalized.

[0065] S5. Select the final decision option based on the final score of each decision option.

[0066] After obtaining the final score for each decision option, each decision option can be sorted according to the final score. For single-choice decision scenarios, the decision option with the highest final score can be selected. For multiple-choice decision scenarios, a corresponding number of decision options with the highest final scores can be selected. Of course, based on the final score of each decision option, other rules can also be used to determine the final decision option.

[0067] The present application also provides a multi-decision maker collaborative decision-making device, which includes a processor and a storage medium. The storage medium stores a computer program, and when the processor runs the computer program in the storage medium, it executes the multi-decision maker collaborative decision-making method of the above embodiment.

[0068] The embodiment of the present application also provides a method for image classification using the multi-decision maker collaborative decision-making method of the above embodiment. In this method, the information to be decided is the classification of the target image, that is, it includes the decision background of the target image and the decision problem for the classification. For this scenario, each decision option is a preset decision option, that is, each decision maker can directly score the preset decision option without generating new decision options. The preset decision options can be a determined set or a most likely subset of all possible options given by the decision maker.

[0069] For example, the decision problem may be the activity state of a person in an image (state A, state B, etc.), or which number in the image is 0 to 9.

[0070] In this scenario, AI models can use visual models such as ResNet and Vision-Transformer to determine the scores of different categories based on the probability distribution of different categories in the output layer. Human decision makers may directly identify one or a few categories. In this case, scores can be determined based on the decision maker's confidence in these few categories, while the scores of other categories are set to zero.

[0071] The present application also verifies the effectiveness of the classification method:

[0072] The experiment uses a pre-trained wide resnet-18 as a vectorization model for decision information and an artificial intelligence model for scoring decision options. At the same time, the human decision-making process is simulated as follows: human experts have one or k expertise labels. When the true label of the input decision information sample is in the human expert's expertise label set, the human expert can accurately judge and give a prediction of the true label; when the true label of the input sample is not in the human expert's expertise label set, the human expert has a certain probability p of giving the correct label, and the remaining probability 1-p of giving a randomly selected label. The decision weights corresponding to human experts and artificial intelligence models can be obtained through the weight generation model. When a total of m human experts and artificial intelligence models are used in collaboration, the initial ability vector is selected as an m+1-dimensional 0-1 one-hot vector, where the i-th decision maker corresponds to a 0-1 vector with the i-th position being 1. By end-to-end learning of a post-linear layer, the 0-1 vector corresponding to a specific decision maker can give the ability vector of the corresponding decision maker after linear transformation. The label information in the CIFAR10 dataset is used as the training label. Figure 1 The model shown is trained end-to-end. The classification accuracy of the trained collaborative model on the test set for different values ​​of the number k of human expert capabilities is shown in Table 1. The embodiments of this application compare the method of this application with several baseline models. These baseline models consider selecting a human decision maker or an artificial intelligence model to make the final decision instead of performing a weighted summation of the output probabilities. The results show that the method provided by this application outperforms these baseline models for almost all values ​​of the number k of expertise.

[0073] Table 1 Comparison of the accuracy of this solution and the baseline model on the CIFAR10 dataset with different numbers of expertise

[0074] k 1 2 3 4 5 6 7 8 9 10 This application 95.53 95.85 96.16 97.21 97.66 98.47 98.96 99.25 99.66 100 L.5CE 94.63 95.26 95.53 95.89 96.31 98.08 98.71 99.14 99.42 100 Confidence 90.47 90.56 90.71 91.41 92.52 94.15 95.5 97.35 98.05 100 OracleReject 89.54 89.51 89.48 90.75 90.64 93.25 95.28 96.52 98.16 100 [MPZ18] 90.4 90.4 90.4 90.4 90.4 90.4 90.4 94.48 95.09 100

[0075] Considering different values ​​of the correct probability p for non-expert categories, experiments were conducted on the CIFAR10 dataset. The classification accuracy is shown in Table 2. It can be seen that our method maintains a stable advantage over the baseline model, Confidence. However, the baseline model's performance declines when the overall accuracy of human experts and the model is close. This, to some extent, reflects the shortcomings of using humans or AI models to make decisions compared to using fusion weighted scoring.

[0076] Table 2 Comparison of the accuracy of this scheme and the baseline model on the CIFAR10 dataset under different correct probability values ​​of non-expert categories

[0077] 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 This application 97.66 97.79 97.94 98.09 98.35 98.32 98.6 98.78 98.75 99.14 100 Consistent[6] 96.31 96.58 96.9 97.15 97.13 97.29 95.23 93.88 92.94 95.6 100

[0078] Furthermore, consider that each human expert has only one distinct expertise category, i.e., Expert 1's expertise category is the first category of images labeled 1, Expert 2's expertise category is the second category of images labeled 2, and so on. Since most traditional methods cannot use end-to-end deep learning models to handle multi-decision maker collaboration, experiments compare the accuracy of systems with varying numbers of experts alone and with varying numbers of experts and AI models collaborating. The results are shown in Table 3. The results demonstrate that the proposed method can effectively handle collaborative decision-making in multi-expert systems, and that its accuracy increases almost monotonically with the number of human experts. When the number of human experts is small, the increase in accuracy is more pronounced.

[0079] Table 3 Accuracy of this solution on CIFAR10 dataset with different number of experts (each expert has a different expertise category)

[0080] 0 1 2 3 4 5 6 7 8 9 10 AI+experts 93.5 95.8 96.76 97.21 97.2 97.3 97.2 97.3 97.2 97.2 97.3 experts 0 19 25.7 31.2 35.6 39.3 41.9 45.7 48.9 52.7 56.1

[0081] The present application also provides a method for making management decisions using the multi-decision maker collaborative decision-making method of the above-mentioned embodiment, wherein the information to be decided is a decision option for determining a target decision scenario, typically a text describing the decision background of the target decision scenario and the decision problem for determining the decision option. Each decision option is a decision option generated by each decision maker based on the information to be decided, such as a possible response strategy for a certain dilemma, an expansion strategy that can be adopted, etc. Alternatively, each decision option is a preset decision option, such as deciding whether to provide a loan, whether to hire or fire an employee, whether to enter a certain market, etc.

[0082] As an alternative approach, to address complex management decision-making problems, large generative language models with reasoning capabilities, such as GPT-4, can be used as AI models to participate in decision-making. GPT-4 can also be used to generate decision alternatives.

[0083] Compared with the existing technology, the solution of this application generally has the following advantages:

[0084] a. By learning the decision-making ability vectors of different decision makers, a more accurate decision-making weight distribution can be obtained through adaptive methods under each specific decision-making task. In this application, the distribution of decision weights depends not only on the decision-making ability index of the decision maker, but also on the information to be decided (such as decision background knowledge). For different decision information samples, accurately determining the decision results requires different abilities. For example, red-green color blindness and color weakness have different abilities in recognizing different pictures. There are also certain differences in the accuracy of strategic experts, accountants, and human resources experts in judging different types of management decision problems.

[0085] Using data-driven methods to learn the ability vectors of different decision makers and the mechanism for assigning decision weights based on real-world decision-making information can enable decision-making systems to achieve more accurate results. Existing technologies that directly determine whether a decision is made by a human or a machine based on contextual information suffer from limited accuracy and generalization capabilities.

[0086] b. Use a unified capability vector to represent the decision-related capabilities of different human decision makers and different AI models. The capability vector learned from a large amount of labeled decision information examples is a powerful cross-species measure of human and AI model capabilities. It is reusable and has a wide range of applications.

[0087] c. This application is well-suited for scenarios where multiple humans and multiple AI models make mixed decisions. This means that the decision makers can be all humans, all AI models, or a combination of varying numbers of humans and AI models, with two or more decision makers. Existing technologies consider relatively limited scenarios, including multiple humans, multiple AI models, or one person per machine.

[0088] d. Existing human-machine collaborative decision-making systems mainly focus on discriminative decision-making problems, while the method of this application can be effectively used in generative decision-making scenarios, such as management decisions and unstructured decision-making scenarios where the decision options are a piece of text or a process.

[0089] e. After obtaining the capability vectors of multiple decision makers, this application can optimize the combination of the decision makers' choices.

[0090] The present invention is not limited to the aforementioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A multi-decision maker collaborative decision-making method, characterized in that: include: Vectorizing the information to be decided to obtain a representation sequence, wherein the information to be decided is the classification of the target image; Obtaining the capability vector of each decision maker respectively, splicing it with the representation sequence, and inputting it into the weight generation model to obtain the decision weight of each decision maker respectively; The decision makers are humans and artificial intelligence models; the capability vectors of each decision maker are all learnable capability vectors; the learnable capability vectors are trained using decision information samples with scoring labels, with the goal of minimizing the distance between the final score of the decision information sample and the corresponding scoring label; the weight generation model is trained using decision information samples with scoring labels, with the goal of minimizing the distance between the final score of the decision information sample and the corresponding scoring label; Obtaining scores of decision makers for each decision option of the information to be decided; For each decision option, the score of each decision maker is weighted using the decision weight to calculate the final score of each decision option of the information to be decided.

2. The multi-decision maker collaborative decision-making method according to claim 1, characterized in that: The decision options are decision options generated by the decision makers for the information to be decided, or are preset decision options.

3. The multi-decision maker collaborative decision-making method according to claim 1, wherein: The learnable capability vector is obtained by training with a preset energy vector as an initial value.

4. The multi-decision maker collaborative decision-making method according to claim 1 or 3, characterized in that: The cross entropy loss is used to measure the distance between the final score of the decision information sample and the corresponding score label.

5. A multi-decision maker collaborative decision-making device, comprising a processor and a storage medium, characterized in that: The storage medium stores a computer program, and when the processor runs the computer program in the storage medium, it executes the multi-decision maker collaborative decision-making method as described in any one of claims 1-4.

6. A multi-decision maker collaborative decision-making method, characterized in that: include: Vectorizing the information to be decided to obtain a representation sequence, wherein the information to be decided is determining a decision option for a target decision scenario, and the information to be decided is text information; Obtaining the capability vector of each decision maker respectively, splicing it with the representation sequence, and inputting it into the weight generation model to obtain the decision weight of each decision maker respectively; The decision makers are humans and artificial intelligence models; the capability vectors of each decision maker are all learnable capability vectors; the learnable capability vectors are trained using decision information samples with scoring labels, with the goal of minimizing the distance between the final score of the decision information sample and the corresponding scoring label; the weight generation model is trained using decision information samples with scoring labels, with the goal of minimizing the distance between the final score of the decision information sample and the corresponding scoring label; Obtaining scores of decision makers for each decision option of the information to be decided, respectively; each decision option is a decision option generated by each decision maker for the information to be decided, or is a preset decision option; For each decision option, the score of each decision maker is weighted using the decision weight to calculate the final score of each decision option of the information to be decided.

Images