Robot multi-task diagnosis skill strategy generation method and device, and storage medium

Abstract

The robot multi-task diagnosis skill strategy generation method, device and storage medium provided by the present disclosure comprise the following steps: a reinforcement learning method in the form of maximum entropy is used to pre-train a constructed diagnosis skill strategy neural network to obtain a maximum entropy diagnosis skill strategy neural network; a divide-and-conquer method is used to divide the same diagnosis task into multiple different categories of subtasks, the maximum entropy diagnosis skill strategy neural network is copied for each subtask, and a reinforcement learning method is used to train each subtask to obtain diagnosis skill strategies for subtasks of different categories of the same task; a strategy distillation method is used to distill the diagnosis skill strategies for subtasks of different categories of the same task to obtain single-task diagnosis skill strategies; and a multi-task skill strategy distillation method is used to distill multiple different single-task diagnosis skill strategies to obtain a multi-task diagnosis skill strategy. The present disclosure can achieve the goal of completing multi-task diagnosis by using one skill strategy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of medical robots, and in particular to a robot multi-task diagnosis skill strategy generation method and device and storage medium. BACKGROUND

[0002] Current medical examination operations such as ultrasound scanning and auscultation are important means indispensable for clinical disease diagnosis. With the development of medical robot technology, some medical robots have been applied to the diagnosis field of ultrasound scanning and auscultation to reduce the workload of doctors. However, the degree of autonomy and intelligence of current robot diagnosis operations is still not high, and a robot can only complete a certain specific examination task, and the cost of robot medical operation skill training is also relatively high. Therefore, how to efficiently improve the intelligent diagnosis operation skill level of the robot and realize the ability of a robot to autonomously complete multiple diagnosis operations has become an important problem to be solved. SUMMARY

[0003] The present disclosure aims to solve one of the above problems.

[0004] To this end, the robot multi-task diagnosis skill strategy generation method provided by the first aspect of the present disclosure can complete multiple different medical diagnosis operations, and comprises:

[0005] A diagnosis skill strategy neural network is constructed, and a reinforcement learning method in the form of maximum entropy is used to pre-train the diagnosis skill strategy neural network to obtain a maximum entropy diagnosis skill strategy neural network;

[0006] For different types of diagnosis tasks, the corresponding single-task diagnosis skill strategy is obtained according to the following steps:

[0007] The same diagnosis task is divided into multiple different categories of sub-tasks by using the divide-and-conquer method, the maximum entropy diagnosis skill strategy neural network is copied for different categories of sub-tasks, and a reinforcement learning method is used for training to obtain diagnosis skill strategy neural networks corresponding to different categories of sub-tasks of the same task, so as to obtain single-task diagnosis skill strategies of different categories of sub-tasks of the same task; and a strategy distillation method is used to distill the single-task diagnosis skill strategies of different categories of sub-tasks of the same task to obtain a single-task diagnosis skill strategy;

[0008] The single-task diagnosis skill strategies of multiple different types are distilled by using a strategy distillation method to obtain a multi-task diagnosis skill strategy.

[0009] The robot multi-task diagnosis skill strategy generation method provided by the first aspect of the present disclosure has the following characteristics and beneficial effects:

[0010] The robot multi-task diagnosis skill strategy generation method provided by the first aspect of the present disclosure can efficiently and conveniently generate a multi-task diagnosis skill strategy to complete various diagnosis tasks. First, a reinforcement learning method in the form of maximum entropy is used for pre-training to increase the strategy entropy and improve the diversity of the strategy action output, thereby facilitating the subsequent learning of multi-task diagnosis skills. Then, the divide-and-conquer idea is used to obtain different skills for different diagnosis operations, and a strategy distillation method is used to distill different operation skill strategies into the same skill neural network to obtain a multi-task diagnosis skill strategy.

[0011] In some embodiments, the parameters of the diagnosis skill strategy neural network are generated in a randomized manner, and a reinforcement learning method in the form of maximum entropy based on formula (1) is used to pre-train the parameters of the diagnosis skill strategy neural network to make the action output by the maximum entropy diagnosis skill strategy neural network diverse. The formula (1) is:

[0012] max∑ t E π [r'(s t ',a t ')]+H(π) (1)

[0013] where s t ',a t ' represent the first state and the first action at time t, respectively, r'(s t ',a t ') represents the first reward value corresponding to the execution of the first action a t ' on the first state s t ', π represents the diagnosis skill strategy, H(π) represents the entropy corresponding to the diagnosis skill strategy π, and E π represents the expected value calculated based on the data generated based on the diagnosis skill strategy π.

[0014] In some embodiments, the divide-and-conquer method is a k-means clustering method, and the same diagnosis task is divided into K categories of sub-tasks based on the same diagnosis task characteristic attribute, and the diagnosis skill strategy of each category of sub-tasks is obtained by training each category of sub-tasks.

[0015] In some embodiments, the single-task diagnosis skill strategy is obtained by distilling the diagnosis skill strategies of different categories of sub-tasks of the same task based on a strategy distillation method based on formula (2):

[0016]

[0017] where, represents the sub-task diagnosis skill strategy KL divergence between single-task examination skill policy strategy π c

[0018] In some embodiments, the examination skill policy neural network comprises four fully connected layers with neuron numbers of 128, 256, 128, and 12 respectively, each of the fully connected layers adopts a ReLU activation function, the input of the examination skill policy neural network is an input vector with a dimension of (C+40), the first C dimensions of the input vector are used to encode the examination task type, the (C+1)th to (C+6)th dimensions of the input vector are six-dimensional pose vectors of the robot end examination executor relative to the detected part of the examination object, the (C+7)th to (C+12)th dimensions of the input vector are six-dimensional velocity values of the robot end examination executor relative to the detected part of the examination object, the (C+13)th to (C+28)th dimensions of the input vector are six joint angle values of the robot end, the (C+29)th to (C+34)th dimensions of the input vector are six joint angular velocity values of the robot end, and the (C+35)th to (C+40)th dimensions of the input vector are six-dimensional force / torque values borne by the robot end examination executor; the output of the examination skill policy neural network is an output vector with a dimension of 12, the first six dimensions of the output vector are six-dimensional poses to which the robot end examination executor should reach, and the seventh to twelfth dimensions of the output vector are six-dimensional force / torque values that the robot end examination executor should apply to the detected part of the examination object.

[0019] In some embodiments, the randomly generated parameters of the examination skill policy neural network include weight values of connections between neurons of adjacent two fully connected layers and corresponding bias term parameters.

[0020] The robot multi-task examination skill policy generation device provided by the second aspect of the present disclosure comprises:

[0021] The maximum entropy examination skill policy neural network acquisition module is configured to construct an examination skill policy neural network, and pre-train the examination skill policy neural network by using a reinforcement learning method in a maximum entropy form to obtain a maximum entropy examination skill policy neural network.

[0022] The single-task examination skill policy generation module is configured to obtain a corresponding single-task examination skill policy by performing the following operations for different types of examination tasks: dividing the same type of examination task into multiple different types of sub-tasks by using a divide-and-conquer method, copying the maximum entropy examination skill policy neural network for different types of sub-tasks respectively and training the copied maximum entropy examination skill policy neural network for different types of sub-tasks respectively by using a reinforcement learning method to obtain a same-task different-type sub-task examination skill policy neural network, and distilling the same-task different-type sub-task examination skill policy by using a policy distillation method to obtain a single-task examination skill policy. ​

[0023] The multi-task diagnosis skill strategy generation module is configured to distill a plurality of different single-task diagnosis skill strategies into a multi-task diagnosis skill strategy by using a strategy distillation method.

[0024] The computer readable storage medium provided by the third aspect of the present disclosure stores computer instructions for causing the computer to execute the robot multi-task diagnosis skill strategy generation method described above. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The flowchart of the robot multi-task diagnosis skill strategy generation method provided by the first aspect of the present disclosure is provided.

[0026] Figure 2 The structural schematic diagram of the electronic device provided by the third aspect of the present disclosure is provided. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0028] On the contrary, the present application covers any alternative, modification, equivalent method and scheme made on the essence and scope of the present application as defined by the claims. Further, in order to make the public better understand the present application, some specific details are described in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0029] Referring to Figure 1 The robot multi-task diagnosis skill strategy generation method provided by the first aspect of the present disclosure comprises:

[0030] The diagnosis skill strategy neural network is constructed, and a reinforcement learning method in the form of maximum entropy is used to pre-train the diagnosis skill strategy neural network to obtain a maximum entropy diagnosis skill strategy neural network.

[0031] For different kinds of diagnosis tasks, the corresponding single-task diagnosis skill strategy is obtained according to the following steps:

[0032] The same kind of diagnosis task is divided into multiple different categories of subtasks by using the divide and conquer method, the maximum entropy diagnosis skill strategy neural network is copied for different categories of subtasks respectively, and reinforcement learning method is used for training respectively to obtain diagnosis skill strategy neural network of different categories of subtasks of the same kind of task, so as to obtain diagnosis skill strategy of different categories of subtasks of the same kind of task; the diagnosis skill strategy of different categories of subtasks of the same kind of task is distilled by using the strategy distillation method to obtain a single task diagnosis skill strategy.

[0033] The single task diagnosis skill strategies of multiple different categories are distilled by using the strategy distillation method to obtain a multi-task diagnosis skill strategy, so as to realize the goal of one skill strategy completing multi-task diagnosis skill.

[0034] In an embodiment, full-scene RGB images and depth images are acquired by using a color depth camera installed in a diagnosis task environment, semantic segmentation of the RGB images is obtained by using an FCN semantic segmentation method, and six-dimensional spatial coordinates p i of the diagnosis object to be diagnosed in the robot coordinate system are obtained by using depth information provided by the depth images and a spatial coordinate transformation method. x , y , z , x , y , z , x , y , z , x , y , z are respectively coordinate components of the diagnosis object to be diagnosed along the x, y, z axes and rotation angles around the x, y, z axes in the robot coordinate system.

[0035] In an embodiment, the diagnosis skill strategy neural network is used to realize three diagnosis tasks of auscultation, low-frequency ultrasound scanning and high-frequency ultrasound scanning, and three corresponding diagnosis task executors (a stethoscope, a low-frequency ultrasound probe and a high-frequency ultrasound probe) are arranged in different directions at the end of the mechanical arm. The diagnosis skill strategy neural network includes four fully connected layers with neuron numbers of 128, 256, 128 and 12 respectively, and each fully connected layer uses a ReLU activation function. The input of the diagnosis skill strategy neural network is a 42-dimensional input vector; wherein the first 2 dimensions of the input vector are used to encode the diagnosis task type, and a feasible scheme is that (0, 0) is used to encode the auscultation task, (0, 1) is used to encode the low-frequency ultrasound scanning task, and (1, 0) is used to encode the high-frequency ultrasound scanning task. The 3rd to 8th dimensions of the input vector are six-dimensional displacement amounts p r of the robot end diagnosis executor relative to the diagnosis object detection part. r , wherein pe -p i , p e is a six-dimensional pose vector of the robot endoscopy executor in the robot coordinate system; the 9th to 14th dimensions of the input vector are six-dimensional velocity values v of the robot endoscopy executor relative to the detection site of the subject r , where v r is obtained by subtracting the six-dimensional pose vector p r of the previous sampling time from the six-dimensional pose vector p r of the current sampling time and then dividing the difference by the time interval between the two samplings; the 15th to 30th dimensions of the input vector are six joint angle values of the robot end; the 31st to 36th dimensions of the input vector are six joint angular velocity values of the robot end; the 37th to 42nd dimensions of the input vector are six-dimensional force / torque values received by the robot endoscopy executor, which are collected by a six-dimensional force / torque sensor arranged on the robot arm. The output of the endoscopy skill policy neural network is a 12-dimensional output vector; wherein the 1st to 6th dimensions of the output vector are six-dimensional poses to which the robot endoscopy executor should reach; the 7th to 12th dimensions of the output vector are six-dimensional force / torque values that the robot endoscopy executor should apply to the detection site of the subject.

[0036] In some embodiments, the parameters of the endoscopy skill policy neural network are generated in a randomized manner, and the randomly generated parameters of the endoscopy skill policy neural network include the weight values and corresponding bias term parameters of the connections between adjacent two layers of neurons in a fully connected neural network with four layers of neurons with numbers of 128, 256, 128 and 12 respectively.

[0037] In some embodiments, a reinforcement learning method in the form of maximum entropy is used to pre-train the endoscopy skill policy neural network, which can increase the policy entropy, so that the actions output by the learned maximum entropy endoscopy skill policy neural network have diversity, thereby facilitating the subsequent learning of multi-task endoscopy skills. The specific steps include:

[0038] S1: construct a first experience pool R' with a data storage size of M' (M' = 1 x 10 5 ), and the data stored in the first experience pool R' is a state transition pair, denoted as a first state transition pair (s t ', a t ', r t ', s t+1 '), wherein s t ' represents a first state at time t, a t ' represents a first action performed at time t, and r t ' represents a first reward value obtained at time t, which is obtained by a first reward function r'(s t ', a tIt is determined that if the diagnosis skill strategy successfully performs the diagnosis task, a first reward value is 1, otherwise the first reward value is -1, s t+1 represents the first state s t represents the first action a t represents the state reached after the first action a

[0039] S2: execute the current diagnosis skill strategy π, collect the current first state transition pair (s t ',a t ',r t ',s t+1 ) data, and store the current first state transition pair data in the first experience pool R';

[0040] S3: collect data from the first experience pool R', update the parameters of the diagnosis skill strategy neural network based on the following formula, and obtain the updated diagnosis skill strategy neural network:

[0041] max∑ t E π [r'(s t ',a t ')]+H(π)

[0042] wherein H(π) represents the entropy corresponding to the current diagnosis skill strategy π, E π represents the expected value calculated based on the data generated based on the current diagnosis skill strategy π;

[0043] Let the current diagnosis skill strategy π be the diagnosis skill strategy generated by the updated diagnosis skill strategy neural network;

[0044] S4: continuously repeat steps S2-S3 until the success rate reaches a set value (95%) or the number of iterations reaches a maximum number of steps (10000 steps), end the pre-training process, and obtain the maximum entropy diagnosis skill strategy neural network.

[0045] In some embodiments, the divide-and-conquer method used is to divide the same diagnosis task into multiple different categories of subtasks according to the characteristics of the diagnosis task, wherein a preferred diagnosis task category division scheme is to divide the same diagnosis task into K categories of subtasks based on the characteristics of the same diagnosis task, which specifically includes the following steps:

[0046] For a certain diagnosis task (for example, high-frequency ultrasound scanning of different parts of the diagnosis object), accurate demonstration diagnosis operation is performed, and the position coordinate sequence of the diagnosis equipment end along the x, y, z axis direction of the robot coordinate system and the Euler angle coordinate sequence Simultaneously record the force sequence applied to the end of the diagnostic device along the x, y, z axis direction of the robot coordinate system in the teaching operation and the torque sequence wherein T is the time sequence length of the teaching operation, represents the sequence {x1, x2, …, xT} from time i=1 to i=T T}, and the other physical quantity sequence has the same meaning. Based on this, the spatial six-dimensional pose change quantity feature ΔP=[Δx max , Δy max , Δz max , Δrx max , Δry max , Δrz max ] is calculated, wherein Meanwhile, the spatial six-dimensional force / torque change quantity feature ΔF={ΔFx max , ΔFy max , ΔFz max , ΔTx max , ΔTy max , ΔTz max} is calculated, wherein Finally, the six-dimensional pose change quantity feature ΔP and the six-dimensional force / torque change quantity feature ΔF and the time T are combined into a feature vector T F as a task feature, wherein the task feature T F is represented as a 13-dimensional vector as follows:

[0047] T F =[T, Δx max , Δy max , Δz max , Δrx max , Δry max , Δrz max , ΔFx max , ΔFy max , ΔFz max , ΔTx max , ΔTy max , ΔTz max ].

[0048] In some embodiments, considering the distribution position and depth of the diagnostic site during ultrasonic examination and taking into account the strategy learning efficiency, K=4 is taken for the low-frequency ultrasonic scanning task, K=4 is taken for the low-frequency ultrasonic scanning task; and K=2 is taken for the auscultation task because only the lungs and heart are usually auscultated in clinical detection.

[0049] In some embodiments, the maximum entropy diagnosis skill policy neural network is copied for each subtask category respectively and learned by reinforcement learning method to obtain diagnosis skill policy for different categories of subtasks wherein k = 1, 2, …, K. The specific learning process includes the following steps:

[0050] S5: A second experience pool R with a data storage size of M (M = 2 x 10 5 ) is constructed, and the data stored in the second experience pool R is a state transition pair, denoted as the second state transition pair (s t , a t , r t , s t+1 ), wherein s t represents the second state at time t, a t represents the second action performed at time t, r t represents the second reward value obtained at time t, which is determined by the second reward function r(s t , a t ). If the diagnosis skill policy successfully performs the diagnosis task, the second reward value is 1, otherwise the second reward value is -1, s t+1 represents the state reached after performing the second action a t in the second state s t ; the second experience pool R is initialized as an empty set, and the parameters of the current subtask diagnosis skill policy are the parameters of the copied maximum entropy diagnosis skill policy neural network;

[0051] S6: The current subtask diagnosis skill policy is executed, and the second state transition pair (s t , a t , r t , s t+1 ) data is collected and stored in the second experience pool R;

[0052] S7: Data is collected from the second experience pool R, and the neural network parameters of the subtask diagnosis skill policy are updated based on the following formula to obtain the updated subtask diagnosis skill policy:

[0053]

[0054] wherein, represents the expected value calculated based on the data generated by the subtask diagnosis skill policy ;

[0055] Let the current subtask diagnosis skill policy a subtask diagnosis skill strategy generated by the updated subtask diagnosis skill policy for the updated subtask;

[0056] S8: repeatedly performing the updated subtask diagnosis skill strategy and calculating the success rate, and when the success rate is greater than 95%, the learning ends, otherwise, steps S6 to S7 are repeatedly performed in turn.

[0057] In some embodiments, a single-task diagnosis skill strategy π is distilled from different types of subtask diagnosis skill strategies for the same task by using a policy distillation method c , specifically including:

[0058] the trained subtask diagnosis skill strategies for K types of subtask categories are distilled to obtain a single-task diagnosis skill strategy π based on the following formula c :

[0059]

[0060] wherein, represents the KL divergence between the subtask diagnosis skill strategy and the single-task diagnosis skill strategy π c .

[0061] In some embodiments, a multi-task learning method is used to obtain different single-task diagnosis skill strategies π c (c = 1, 2, …, C) based on steps S4 and S5 for C (C = 3) different diagnosis tasks (auscultation examination, high-frequency ultrasound scanning, and low-frequency ultrasound scanning).

[0062] In some embodiments, a multi-task diagnosis skill strategy π is distilled based on a plurality of different single-task diagnosis skill strategies π c (c = 1, 2, …, C) using the following formula G :

[0063]

[0064] wherein, D KL (π c ||π G ) represents the KL divergence between the single-task diagnosis skill strategy π c and the multi-task diagnosis skill π G .

[0065] The robot multi-task diagnosis skill strategy generation method provided by the first aspect of the present disclosure ultimately realizes the goal of performing a plurality of continuous diagnosis operation tasks by one diagnosis multi-task skill strategy.

[0066] The robot multi-task diagnosis skill strategy generation apparatus provided by the second aspect embodiment of the present disclosure comprises:

[0067] The maximum entropy diagnosis skill strategy neural network acquisition module is configured to construct a diagnosis skill strategy neural network, and pre-train the diagnosis skill strategy neural network by using a reinforcement learning method in a form of maximum entropy to obtain a maximum entropy diagnosis skill strategy neural network.

[0068] The single-task diagnosis skill strategy generation module is configured to obtain a corresponding single-task diagnosis skill strategy by performing the following operations for different kinds of diagnosis tasks: dividing the same diagnosis task into a plurality of different categories of sub-tasks by using a divide-and-conquer method, copying the maximum entropy diagnosis skill strategy neural network for different categories of sub-tasks respectively, and training the copied maximum entropy diagnosis skill strategy neural network for different categories of sub-tasks respectively by using a reinforcement learning method to obtain diagnosis skill strategy neural networks corresponding to different categories of sub-tasks of the same task, so as to obtain diagnosis skill strategies of different categories of sub-tasks of the same task; and distilling the diagnosis skill strategies of different categories of sub-tasks of the same task by using a strategy distillation method to obtain a single-task diagnosis skill strategy.

[0069] The multi-task diagnosis skill strategy generation module is configured to distill a plurality of different kinds of single-task diagnosis skill strategies by using a strategy distillation method to obtain a multi-task diagnosis skill strategy.

[0070] In order to implement the above-mentioned embodiments, the present disclosure further provides a computer readable storage medium having a computer program stored thereon, which is executed by a processor to perform the above-mentioned embodiments.

[0071] Reference will be made to the following description Figure 2 which shows a structural schematic diagram of an electronic device 100 suitable for implementing the embodiments of the present disclosure. It should be noted that Figure 2 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0072] As shown in Figure 2 , the electronic device 100 can include a processing device (such as a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 102 or loaded from a storage device 108 into a random access memory (RAM) 103. In the RAM 103, various programs and data required for the operation of the electronic device 100 are also stored. The processing device 101, the ROM 102, and the RAM 103 are connected to each other through a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0073] In general, the following devices can be connected to the I / O interface 105: input devices 106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 108 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 109. The communication devices 109 can allow the electronic device 100 to communicate wirelessly or wired with other devices to exchange data. Although Figure 2 The electronic device 100 is shown with various devices, but it is understood that all of the shown devices are not required to be implemented or present. More or less devices can alternatively be implemented or present.

[0074] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the present embodiments include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 109, or installed from the storage devices 108, or installed from the ROM 102. When the computer program is executed by the processing devices 101, the above-mentioned functions defined in the methods of the present disclosure embodiments are performed.

[0075] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the disclosure, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a program for use by or in connection with an instruction execution system, apparatus or device, other than the computer-readable storage medium. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, optical fiber, RF (radio frequency), or any suitable combination of the foregoing.

[0076] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and be not assembled into the electronic device.

[0077] The computer readable medium carries one or more programs when the one or more programs are executed by the electronic device, the electronic device is caused to: construct a diagnosis skill policy neural network, pre-train the diagnosis skill policy neural network by using a reinforcement learning method in a form of maximum entropy to obtain a maximum entropy diagnosis skill policy neural network; obtain a single-task diagnosis skill policy for different types of diagnosis tasks according to the following steps: divide the same diagnosis task into a plurality of different types of sub-tasks by using a divide-and-conquer method, copy the maximum entropy diagnosis skill policy neural network for different types of sub-tasks respectively, and train by using a reinforcement learning method to obtain a diagnosis skill policy neural network for different types of sub-tasks of the same task, so as to obtain a diagnosis skill policy for different types of sub-tasks of the same task; and obtain a single-task diagnosis skill policy by distilling the diagnosis skill policy for different types of sub-tasks of the same task by using a policy distillation method; and obtain a multi-task diagnosis skill policy by distilling a plurality of different types of single-task diagnosis skill policies by using a policy distillation method.

[0078] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, Python, or conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0079] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0080] Moreover, the terms "first", "second", "third", etc. are used herein only to describe different instances, and do not imply or suggest relative importance or a number of indicated technical features. Thus, features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0081] Any process or method descriptions or blocks in flow charts herein, and elsewhere, can be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of the preferred embodiments of the present application in which additional functionality can be added or some functionality can be removed, by, for example, adding one or more steps performing a similar or reciprocal function, by adding one or more steps performing the same functions, by adding one or more steps performing an alternative or parallel function, by adding one or more steps performing one or more additional functions, or by adding one or more steps performing a different function altogether.

[0082] Logic and / or steps represented in flow charts herein, and elsewhere, can be embodied in computer-readable media, for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination thereof. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a product of "tangible" non-transitory computer-readable medium. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electronic connection having one or more wires (electronic devices), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical device), and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer. In this context, a "non-transitory computer readable medium" can be any computer readable medium except transmission media.

[0083] It should be understood that parts of the present application can be realized in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be realized as software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized in hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0084] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiments can be completed by a program instructing the relevant hardware, and the developed program can be stored in a computer readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0085] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0086] The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A method for generating multi-task diagnostic skills strategies for robots, characterized in that, The 15th to 30th dimensions of the input vector are six joint angle values of the robot end; the 31st to 36th dimensions of the input vector are six joint angular velocity values of the robot end; the 37th to 42nd dimensions of the input vector are six force / torque values borne by the robot end examination executor, which are collected by a six-dimensional force / torque sensor arranged on the mechanical arm; the output of the examination skill strategy neural network is a 12-dimensional output vector; wherein the 1st to 6th dimensions of the output vector are six-dimensional postures to which the robot end examination executor should reach; the 7th to 12th dimensions of the output vector are six-dimensional force / torque values that the robot end examination executor should apply to the detection part of the examination object; The diagnostic skill strategy neural network is pre-trained by using a reinforcement learning method in a maximum entropy form to obtain a maximum entropy diagnostic skill strategy neural network; the first two dimensions of an input vector of the diagnostic skill strategy neural network are used for encoding a diagnostic task type, and the third to eighth dimensions are six-dimensional displacement amounts of a robot end diagnostic executor relative to a detection position of a diagnostic object wherein , is a six-dimensional pose vector of the robot end diagnostic executor in a robot coordinate system; the ninth to fourteenth dimensions of the input vector are six-dimensional velocity amounts of the robot end diagnostic executor relative to the detection position of the diagnostic object wherein is obtained by subtracting the six-dimensional pose vector at the sampling time of the previous step from the six-dimensional pose vector at the current sampling time and then dividing the difference by the time interval of the two samplings to obtain the six-dimensional velocity For different kinds of examination tasks, the corresponding single-task examination skill strategy is obtained according to the following steps: The same kind of examination task is divided into multiple different categories of subtasks by using the divide-and-conquer method, the maximum entropy examination skill strategy neural network is replicated for different categories of subtasks, and reinforcement learning method is used for training to obtain the examination skill strategy neural network corresponding to the different categories of subtasks of the same kind of task, so as to obtain the examination skill strategy of the different categories of subtasks of the same kind of task; and the single-task examination skill strategy is obtained by distilling the examination skill strategy of the different categories of subtasks of the same kind of task by using the strategy distillation method. The single-task examination skill strategies of multiple different kinds are distilled by using the strategy distillation method to obtain a multi-task examination skill strategy. The parameters of the examination skill strategy neural network are generated in a randomized manner, and the parameters of the examination skill strategy neural network are pre-trained by using a reinforcement learning method in the form of maximum entropy based on formula (1) to make the actions output by the maximum entropy examination skill strategy neural network diverse, and the formula (1) is:

2. The method of claim 1, wherein the robot multi-task diagnostic skill policy is generated based on a plurality of robot multi-task diagnostic skill policies. The examination skill strategy neural network comprises four fully connected layers with neuron numbers of 128, 256, 128 and 12 respectively, and each of the fully connected layers adopts a ReLU activation function. wherein, respectively represent a first state and a first action at time represents a first reward value corresponding obtained after the first state executes the first action represents a diagnosis skill policy, represents a diagnosis skill policy corresponding entropy, represents a data generated based on the diagnosis skill policy produces an expected value.​​ 3. The method of claim 1, wherein: The divide-and-conquer method is a k-means clustering method, and the same diagnosis task is divided into K categories of subtasks based on the same diagnosis task characteristic attributes, and each category of subtask is trained respectively to obtain the diagnosis skill strategy of different categories of subtasks of the same diagnosis task , .

4. The method of claim 3, wherein, The single task diagnosis skill strategy is obtained by distilling the diagnosis skill strategy of the subtasks of the same task and different categories using a distillation method based on formula (2) : wherein, represents the KL divergence between the subtask diagnosis skill policy and the single task diagnosis skill policy .

5. The method of claim 4, wherein, distilling a plurality of different kinds of the single-task diagnostic skill strategies using a strategy distillation method based on equation (3) distilling a multi-task diagnostic skill strategy : wherein, represents the KL divergence between the single-task diagnostic skill strategy and the multi-task diagnostic skill strategy , and C represents the number of diagnostic task categories.

6. The robotic multi-task diagnostic check skill policy generation method of claim 1, wherein, The randomly generated parameters of the examination skill strategy neural network include weight values and corresponding bias term parameters of connections between neurons of adjacent two fully connected layers.

7. The robotic multi-task diagnostic skill policy generation method of claim 6, wherein, The maximum entropy examination skill strategy neural network acquisition module is configured to construct an examination skill strategy neural network, and pre-train the examination skill strategy neural network by using a reinforcement learning method in the form of maximum entropy to obtain a maximum entropy examination skill strategy neural network; 8. A robotic multi-task diagnostic skill policy generation apparatus, comprising: The single-task examination skill strategy generation module is configured to obtain a corresponding single-task examination skill strategy for different kinds of examination tasks according to the following operations: divide the same kind of examination task into multiple different categories of subtasks by using the divide-and-conquer method, replicate the maximum entropy examination skill strategy neural network for different categories of subtasks, and train the maximum entropy examination skill strategy neural network by using a reinforcement learning method to obtain an examination skill strategy neural network corresponding to the different categories of subtasks of the same kind of task, so as to obtain the examination skill strategy of the different categories of subtasks of the same kind of task; and obtain a single-task examination skill strategy by distilling the examination skill strategy of the different categories of subtasks of the same kind of task by using the strategy distillation method. ​ ​ The multi-task diagnosis skill strategy generation module is configured to distill a plurality of different single-task diagnosis skill strategies into a multi-task diagnosis skill strategy by using a strategy distillation method.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the computer to perform the robot multi-task diagnosis skill strategy generation method in any one of claims 1-7.

Images