A human-computer collaborative weight distribution method based on double-layer fuzzy strategy

Abstract

The present invention relates to a method for allocating weights of human-machine collaboration based on a double-layer fuzzy strategy, and belongs to the field of human-machine collaboration technology. A fuzzy interactive system is constructed, and the fuzzy interactive system includes two layers, the first layer includes a conflict situation fuzzy decision maker, a human-machine level fuzzy decision maker, and a motion risk fuzzy decision maker; the second layer is a weight output fuzzy decision maker; human-machine data is obtained, and the human-machine data is fuzzified and converted into a fuzzy subset of a language variable as a state quantity input into the fuzzy interactive system; the fuzzy interactive system outputs a human-machine collaborative control weight allocation coefficient. By adopting a two-layer fuzzy controller, various influencing factors are fully considered, and the whole process of weight output is made interpretable, which is conducive to mutual trust between humans and machines in collaborative tasks. The fuzzy interactive system is trained by reinforcement learning, which reduces the problems caused by empirical design in traditional methods and can effectively solve the problem of human-machine collaborative weight allocation in different scenarios.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human-computer collaboration, and relates to a self-adaptive human-computer collaboration weight distribution strategy, in particular to a human-computer collaboration weight distribution method based on a double-layer fuzzy strategy. The method is applied to human-computer collaboration tasks, and realizes natural human-computer collaboration control by changing the weight distribution of a controlled system in real time. BACKGROUND

[0002] Human-computer collaboration of an intelligent agent is a human-in-the-loop human-computer collaboration control problem. Humans and automatic control systems have different advantages. The intelligent agent has the abilities of fine perception, normative decision-making and precise control, while the human has the ability of decision-making in complex unknown working conditions, and there is strong complementarity. Therefore, through human-computer collaboration control, the weight of humans and intelligent agents in different tasks is reasonably distributed, the hybrid enhancement of human and computer intelligence is realized, the work efficiency is greatly improved, the development of human and computer intelligence is promoted, and the load of humans in the work task is reduced.

[0003] In the working process, the intelligent agent and the human have the perception, decision-making and control authority at the same time, and the control instruction of the control system will directly affect the key problems such as operation safety and comfort, and a dynamic distribution model needs to be established by considering many influencing factors to optimize the flexibility and explainability of the collaborative task. SUMMARY

[0004] The technical problem to be solved by the present application is:

[0005] In order to solve the problem of control weight distribution between humans and intelligent agents in the human-computer collaboration process, the present application provides a human-computer collaboration weight distribution method based on a fuzzy reinforcement learning method, which adaptively adjusts the size of the control weight distribution according to environmental risk perception, human state and human-computer conflict situation.

[0006] In order to solve the above technical problem, the technical scheme adopted by the present application is:

[0007] A human-computer collaboration weight distribution method based on a double-layer fuzzy strategy, characterized in that it comprises:

[0008] A fuzzy interaction system is constructed, which comprises two layers. The first layer comprises a conflict situation fuzzy decision maker, a human-computer level fuzzy decision maker and a motion risk fuzzy decision maker. The second layer is a weight output fuzzy decision maker.

[0009] Human-computer data is acquired, and the human-computer data is fuzzily converted into a fuzzy subset of linguistic variables as state variables input into the fuzzy interaction system.

[0010] The human-machine data includes the position and speed of the agent, the input of the operating personnel, the control signal of the agent, and the position of the environment boundary line; the conflict parameter and the level parameter are obtained by respectively evaluating the human-machine conflict and the human-machine level according to the input of the operating personnel and the control signal of the agent; the risk parameter is obtained by evaluating the risk according to the position and speed of the agent and the position of the environment boundary line;

[0011] The conflict parameter, the level parameter and the risk parameter are respectively input into the conflict situation fuzzy decision maker, the human-machine level fuzzy decision maker and the motion risk fuzzy decision maker; the outputs of the conflict situation fuzzy decision maker, the human-machine level fuzzy decision maker and the motion risk fuzzy decision maker are input into the weight output fuzzy decision maker;

[0012] The weight output fuzzy decision maker outputs the human-machine cooperative control weight distribution coefficient η, and the final control instruction act = ηact h +(1-η)act m , wherein act h represents the control instruction of the human, and act m represents the control instruction of the machine.

[0013] Further technical solutions of the application: the conflict situation fuzzy decision maker, the human-machine level fuzzy decision maker, the motion risk fuzzy decision maker and the weight output fuzzy decision maker are all evolutionary fuzzy neural networks, and the evolutionary fuzzy neural network is composed of N first-order IF-THEN fuzzy rules:

[0014]

[0015] In the formula, R n represents the nth rule, n = 1, 2, … N, and N is the number of fuzzy rules; p i is a rule prior parameter; y i is the output value of the current rule; σ(·) represents an activation function, A n represents a weight value;

[0016] Due to the adoption of the activation function, each set of fuzzy rules in the ENFIS becomes a nonlinear model, and thus the output y can be represented as a weighted sum of the outputs of a set of rules:

[0017]

[0018] Wherein, λ i is a trigger strength, and the expression is:

[0019]

[0020] In addition, is a local density, τ i represents the rule-related kernel function width.

[0021] The further technical solution of the present application further comprises updating parameters of the fuzzy interaction system by using a parameter exploration-based policy gradient algorithm, wherein the parameter updating comprises representing a parameter policy as π θ , that is, a function taking the environment state as input and generating a control action u = π θ (x s ), and the goal of the reinforcement learning algorithm is to find the optimal parameters in the parameter space to maximize the expected return, and the specific process is as follows:

[0022] ρ = (μ, σ 2 ) and v ρ = (μ, σ 2 ) are Gaussian distributions with mean μ and variance σ 2 , and {θ i} i＝1,...,N is a group of N parameters independently sampled from the hyperparameter policy v ρ ; the parameterized policy π θ is run, and M independent events {τ ij} j＝1,...,M are collected; N x M events are used to estimate the target:

[0023]

[0024] wherein R ij is the reward of the jth event of the ith policy;

[0025] In each iteration, the hyperparameter policy is updated by a gradient ascent:

[0026]

[0027] wherein α > 0 is a learning rate, is an estimation of the parameter gradient;

[0028] After the parameter updating process converges, the training of the double-layer evolutionary neural fuzzy inference system is completed.

[0029] A computer system, characterized in that it comprises one or more processors, a computer readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0030] A computer readable storage medium, characterized in that it stores computer executable instructions, which when executed implement the above method.

[0031] A computer program product comprising computer executable instructions for implementing the above method when executed.

[0032] The present application has the following advantages:

[0033] The human-machine collaborative weight distribution method based on a double-layer fuzzy strategy provided by the present application does not need to evaluate the human in advance in the collaborative control process, and establishes a mapping relationship between each item of data and the weight through a fuzzy rule method according to real-time information. A two-layer fuzzy controller is adopted to fully consider various influencing factors and make the whole process of weight output interpretable, which is helpful to human-machine mutual trust in collaborative tasks. In addition, the fuzzy rule library is trained by using reinforcement learning, which reduces the problems caused by experience design in traditional methods and can effectively solve the human-machine collaborative weight distribution problem in different scenarios. Since the fuzzy reinforcement learning method is used to distribute the human-machine collaborative weight, it can realize real-time and flexible and interpretable collaborative work between the human and the agent in different scenarios, and also can fully utilize the respective advantages of the human and the agent to improve work efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0035] Figure 1 A risk field diagram.

[0036] Figure 2 A double-layer fuzzy system structure diagram.

[0037] Figure 3 An evolutionary fuzzy neural network structure diagram.

[0038] Figure 4 A PGPE algorithm parameter optimization and update diagram. DETAILED DESCRIPTION

[0039] The present application will be described in detail below with reference to specific embodiments, which are described herein for the purpose of explanation only and are not intended to limit the present application.

[0040] The application provides a human-machine collaborative weight distribution method combining a fuzzy neural network and a reinforcement learning method, which is capable of adaptively adjusting the size of control weight distribution in view of environmental risk perception, human-machine level and human-machine conflict situation, etc. The method comprises a weight distribution and control loop of a double-layer evolutionary neural fuzzy interaction system and a fuzzy rule parameter update loop of a policy gradient with parameter-based exploration (PGPE) algorithm, and the core lies in structural evolution and parameter update of the fuzzy interaction system. The framework of the algorithm is shown in Figure 1 As an example of human-machine collaborative co-driving, the specific steps include:

[0041] Step 1: A simulation environment based on a Carla simulator in the field of automatic driving is constructed, and various different traffic road scenes are constructed in the simulator.

[0042] A Logitech G923 steering wheel kit is used as an input device of an operator to realize data input of human driving behavior, including steering wheel turning angle, throttle and brake information.

[0043] An automatic driving vehicle is constructed as an agent, which needs to have simple automatic driving capability, and the specific algorithm is not limited, that is, any intelligent driving system, and the output of the agent is the steering wheel turning angle and throttle and brake information according to the road traffic condition.

[0044] A large number of typical traffic scenes are constructed, including but not limited to following scenes, overtaking scenes, crossroads, ramp merging, congested road conditions and the like under all types of road sections.

[0045] A certain amount of random traffic flow is inserted into the traffic environment, and some special dangerous traffic scenes or accident simulation scenes are inserted.

[0046] Step 2: Regularization and normalization of input of fuzzy decision maker

[0047] The steering wheel turning angle, throttle and brake information input by the operator and the intelligent driving system are all in the range of [-1, 1].

[0048] 2.1 Human-machine conflict evaluation

[0049] Evaluation parameter σ of human-machine conflict s1 , which is defined as the conflict of control instruction signals between the operator input and the intelligent driving system, and a linear normalization method is used for processing, and σ s1 is defined as:

[0050]

[0051] In the formula, δ max and δmin are the maximum steering angle and the minimum steering angle of the vehicle, respectively, where the right steering angle is defined as positive and the left steering angle is defined as negative; δ λ is the threshold value for approximation; where:

[0052]

[0053] Where δ ms and δ hs are the steering control signal output by the intelligent driving system and the angle value corresponding to the steering wheel angle input by the operator, respectively. s1 The denominator is the steering angle δ input by the operator hs , so its input is approximated as Reduce the operator's input steering wheel angle within a small range (for example, |δ hs |<δ λ , take δ λ =1°) jitter caused by σ s1 Drastic changes in value.

[0054] The meaning of the conflict parameter can be understood as follows: s1 When it is greater than 0, the steering angle input by the operator and the control signal input by the intelligent driving system are in the same direction, which means that the steering intention of the operator and the intelligent driving system are the same. Figure 1 , in cooperative driving state; when σ′ s1 When it is less than 0, the steering angle input by the operator and the control signal input by the intelligent driving system are in different directions, and the steering signals at this time are in conflict.

[0055] Conflict evaluation parameter σ s2 , σ t is the difference between the control command signal input by the operator and the intelligent driving system, and its expression is defined as:

[0056]

[0057] In the formula and are the normalized values ​​of the steering wheel angles input by the intelligent driving system and the operator, and These are the normalized values ​​of the accelerator and brake input by the intelligent driving system and the operator respectively, and the information range is [-1,1].

[0058] 2.2 Environmental risk assessment

[0059] The definition of the environmental risk parameter r adopts the artificial potential field method, which is mainly used to evaluate the degree of risk of vehicle collision and lane deviation. The schematic diagram is as follows Figure 2 .

[0060] Therefore, the obstacle potential field and the road boundary line potential field are constructed respectively, and are expressed as:

[0061] r=k obs r obs +k Rb r Rb

[0062] In the formula, k obs , k Rb are the weight coefficients of the obstacle potential field and the road boundary potential field respectively, k obs +k Rb =1, which aims to ensure the normalization of the environmental risk parameter, and k obs =0.9, k Rb =0.1 are acceptable; r obs , r Rb are the obstacle potential field value and the road boundary line potential field value respectively.

[0063] And r obs =max(r obs (x,y) i ), that is, the obstacle potential field value takes the value of the obstacle that has the greatest influence on the current point (x, y); the road boundary potential field r Rb is composed of different road lines such as solid lines, dashed lines, and road boundaries:

[0064]

[0065] In the formula, A i represents the intensity value of different road line types, and A represents the distance from the point to the edge line, σ l represents the increase or attenuation coefficient of the potential field.

[0066] The expression of the potential field of the ith obstacle at the point (x, y) is r obs (x,y) i , which is defined as:

[0067]

[0068] Among them,

[0069]

[0070] In the formula, (x i , y i ) represents the global position coordinates of the center point of the obstacle, l i , w i are the length and width of the obstacle respectively (for a general sedan vehicle, l i =5m, wi = 2m), are the potential field shape factors in the lateral and longitudinal directions, respectively, v i is the velocity of the obstacle along its axis, is the heading angle of the vehicle, δ is the steering angle, t s is the critical reaction time, and t s = 2.5 s.

[0071] 2.3 Human-machine level evaluation

[0072] The evaluation parameters a h , a m for the human-machine level are also evaluated using the artificial potential field, which has vector characteristics. Considering that the steering wheel control input of the vehicle, the operator input, and the instructions of the intelligent driving system can all be converted into the forward direction of the vehicle, the cosine similarity is used to evaluate the human-machine level, which is expressed as:

[0073]

[0074] wherein, is the unit vector.

[0075] Step three, structure evolution of the double-layer evolutionary neural fuzzy reasoning system

[0076] As can be seen from Figure 1 , the proposed double-layer evolutionary neural fuzzy interaction system is composed of multiple layered evolutionary fuzzy neural networks. The inputs of the fuzzy interaction system are processed layer by layer until the final output is generated, and the evolutionary fuzzy neural network of the second layer takes the output of the previous layer as its input. Taking any evolutionary fuzzy neural network as an example, the evolution process is introduced, and the structural diagram is as shown in Figure 3 , and the calculation process can be expressed as:

[0077] y = f(x)

[0078] wherein, x = [x1, x2, …, x M ] is an M × 1 dimensional input vector, and y is the output value thereof.

[0079] The evolutionary fuzzy neural network is a neural fuzzy system composed of N first-order IF-THEN fuzzy rules:

[0080]

[0081] wherein, R n represents the nth rule, n = 1, 2, … N, N is the number of fuzzy rules; p i is the rule prior parameter; y iis the output value of the current rule; σ(·) denotes the activation function, A n denotes the weight value;

[0082] Due to the adoption of the activation function, each set of fuzzy rules in the evolutionary fuzzy neural network becomes a nonlinear model, and its output y can be expressed as the weighted sum of the outputs of a set of rules:

[0083]

[0084] where λ i is the firing strength, and the expression is:

[0085]

[0086] In addition, is the local density, τ i denotes the rule-related kernel function width.

[0087] The structure evolution process includes initialization and structure evolution, and needs to learn from the input data.

[0088] In the initialization stage, the input x = [x1, x2, …, x M ] needs to be given, and after the evolutionary fuzzy neural network receives the first input, its global parameters are set as:

[0089] μ←x 1 ; X←||x 1 || 2

[0090] In the formula, μ is the global average value of the input sample, and X is the average value of the Euclidean norm square of all inputs.

[0091] The prior parameters and weight parameters of the first rule R n are initialized as:

[0092]

[0093] In the formula, ε o = [ε o，k ] k＝1:(M+1) is a randomly generated 1 × (M + 1) dimensional vector, and its elements are equal to 0 or 1.

[0094] The initial cluster C1 parameters formed near the prior p1 are initialized as:

[0095] C1:c1←x 1 ; χ1←||x 1 || 2; S1 <- 1

[0096] where c1 is the mean of the C1 related data samples, χ1 is the mean of the square of its 2-norm, and S1 is the number of its members.

[0097] In the structure evolution and output generation stage, the global parameters are updated as:

[0098]

[0099] Check whether the new input data represents a new data pattern that does not exist in the historical input:

[0100]

[0101] then(x t becomes a new prototype)

[0102] else(update parameters)

[0103] where δ o is a threshold value for determining whether the new input data belongs to an existing data pattern, 0 < δ o < 1. In this example, δ o = e -3 According to Chebyshev's rule, the probability that the Euclidean distance between the new input data and the prior is greater than e is less than 33.3%.

[0104] Therefore, when the condition is met, it can be considered that the new data is not within the influence area of the existing prior, that is, it represents a new data pattern, and a new rule needs to be introduced to represent it. The generation and initialization process of the new rule is the same.

[0105] However, if the new input data does not meet the condition, it is used to update the nearest cluster, represented as:

[0106]

[0107] Step four, parameter update of the double-layer evolutionary neural fuzzy reasoning system

[0108] The system parameter update adopts the policy gradient with parameter-based exploration (PGPE) algorithm. A parameter policy is represented as π θ , that is, a function with the environment state as input, which generates a control action u = π θ (x s), the goal of reinforcement learning algorithm is to find the best parameter in the parameter space to maximize the expected return, such as Figure 4 .

[0109] In this example, the parameter strategy is the evolutionary fuzzy neural network, where the parameter θ is the weight A n set, and the control action generated is the final output weight value η.

[0110] Select ρ = (μ, σ 2 ) and ν ρ = (μ, σ 2 ) are Gaussian distributions with mean μ and variance σ 2 , {θ i} i＝1,...,N is a set of N parameters independently sampled from the hyperparameter strategy ν ρ . We run the parameterized strategy π θ , collect M independent events {τ ij} j＝1,...,M . Finally, use N × M events to estimate the target:

[0111]

[0112] In the formula, R ij is the reward of the jth event of the ith strategy.

[0113] In each iteration, the hyperparameter strategy is updated by a gradient ascent:

[0114]

[0115] where α > 0 is the learning rate, is the estimate of the parameter gradient.

[0116] Finally, after the parameter update process converges, the training of the double-layer evolutionary neural fuzzy inference system is completed.

[0117] Step five, human-machine collaborative control instruction generation

[0118] After the learning of the fuzzy interaction system parameters is completed, the human-machine collaborative control weight distribution coefficient η is obtained using the trained double-layer fuzzy inference system, and the final control instruction is act = ηact h + (1-η)act m , where act h represents the control instruction of human, and act m represents the control instruction of machine.

[0119] To sum up, taking human-computer collaborative shared driving control as an example, the human-computer collaborative weight distribution method based on double-layer fuzzy strategy proposed in the application is described in detail.

[0120] The above merely describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements shall be encompassed within the protection scope of the present application.

Claims

1. A human-machine collaboration weight allocation method based on a double-layer fuzzy strategy, characterized in that: include: Constructing a fuzzy interactive system, the fuzzy interactive system includes two layers, the first layer includes a conflict situation fuzzy decision maker, a human-machine level fuzzy decision maker, and a motion risk fuzzy decision maker; The second layer is the weighted output fuzzy decision maker; Obtain human-machine data, fuzzify the human-machine data and convert it into fuzzy subsets of language variables as state quantities to input into the fuzzy interaction system; The human-machine data includes the position and speed of the intelligent body, the operator input, the intelligent body control signal, and the position of the environmental boundary line; the human-machine conflict and the human-machine level are respectively evaluated based on the operator input and the intelligent body control signal to obtain the conflict parameter and the level parameter; the risk is evaluated based on the position and speed of the intelligent body and the position of the environmental boundary line to obtain the risk parameter; Input the conflict parameter, level parameter and risk parameter into the conflict situation fuzzy decision maker, the human-machine level fuzzy decision maker and the motion risk fuzzy decision maker respectively; input the outputs of the conflict situation fuzzy decision maker, the human-machine level fuzzy decision maker and the motion risk fuzzy decision maker into the weight output fuzzy decision maker; The weighted output fuzzy decision maker outputs the human-machine collaborative control weight distribution coefficient η, and the final control instruction act=ηact h +(1-η)act m , among which, act h Indicates human control instructions, act m Indicates the control instructions of the machine.

2. The method for allocating weights for human-machine collaboration based on a double-layer fuzzy strategy according to claim 1, characterized in that: The conflict situation fuzzy decision maker, human-machine level fuzzy decision maker, motion risk fuzzy decision maker and weight output fuzzy decision maker are all evolutionary fuzzy neural networks. The evolutionary fuzzy neural network is composed of N first-order IF-THEN fuzzy rules: R n :IF(x~p i )THEN Where R n represents the nth rule, n = 1, 2, ... N, N is the number of fuzzy rules; p i is the rule prior parameter; y i is the output value of the current rule; σ(·) represents the activation function, A n Indicates the weight value; Due to the use of activation functions, each set of fuzzy rules in ENFIS becomes a nonlinear model, so its output y can be expressed as the weighted sum of the outputs of a set of rules: Among them, λ i is the trigger strength, and the expression is: in addition, is the local density, τ i Indicates the width of the kernel function associated with the rule.

3. The method for allocating weights for human-machine collaboration based on a double-layer fuzzy strategy according to claim 1, characterized in that: It also includes using a policy gradient algorithm based on parameter exploration to update the parameters of the fuzzy interactive system, wherein the parameter update is: a parameter strategy is represented as π θ , which is a function that takes the environment state as input and produces a control action u = π θ (x s ), the goal of the reinforcement learning algorithm is to find the best parameters in the parameter space to maximize the expected return; specifically as follows: Choose ρ=(μ,σ 2 ) and ν ρ =(μ,σ 2 ) is the mean μ and the variance σ 2 Gaussian distribution, {θ i } i＝1,...,N is sampled independently from the hyperparameter strategy ν ρ A set of N parameters; run the parameterized strategy π θ , collect M independent events {τ ij } j＝1,...,M ; Use N×M events to estimate the target: Where R ij is the reward of the jth event of the i-th group strategy; At each iteration, the hyperparameter policy is updated via a gradient ascent: Among them, α>0 is the learning rate, is an estimator of the parameter gradient; Finally, after the parameter updating process converges, the training of the two-layer evolutionary neuro-fuzzy inference system is completed.

4. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

5. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

6. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when the instructions are executed.

Images