Data processing method, device, electronic device and storage medium

Abstract

The present disclosure proposes a data processing method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology, and in particular to the field of deep learning technology. The scheme is: obtain the state data of the environment in which the target object is located, input the state data into the policy network of the reinforcement model, sample multiple actions corresponding to the state data from the action set, input the multiple actions and the state data into the guidance network of the reinforcement model, output the target matching degree between each action and the state data, and determine the target action of the target object from the multiple actions obtained by sampling according to the target matching degree of each action. By sampling multiple actions from the action set output by the policy network, the efficiency of subsequent processing is improved, and the state data and multiple actions are combined and input into the guidance network, and the target matching degree corresponding to each action is calculated. The target matching degree indicates a stronger correlation between each action and the state data, thereby improving the accuracy of determining the target action.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning technology, and specifically to data processing methods, devices, electronic devices and storage media. Background Art

[0002] In recent years, reinforcement learning has been applied to many fields, such as games, robots, and recommendation systems. However, training a reinforcement learning model is time-consuming because during the training process, the reinforcement learning model needs to interact with the environment a lot to determine matching actions, and these costs are high. At the same time, simple exploration strategies slow down the learning speed of the model and may even cause the model to take actions that are harmful to the environment. Therefore, how to improve the accuracy of action determination is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present disclosure provides a data processing method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided a data processing method, comprising:

[0005] Get the status data of the target object's environment;

[0006] Inputting the state data into a policy network of a reinforcement model to sample a plurality of actions corresponding to the state data from an action set;

[0007] Inputting the plurality of actions and the state data into a guidance network of the reinforcement model to output a target matching degree between each of the actions and the state data;

[0008] According to the target matching degree between each action and the state data, a target action of the target object is determined from the multiple actions obtained by sampling.

[0009] According to another aspect of the present disclosure, there is provided a data processing device, comprising:

[0010] An acquisition module is used to obtain the status data of the environment where the target object is located;

[0011] A first determination module is used to input the state data into a policy network of a reinforcement model to sample a plurality of actions corresponding to the state data from an action set;

[0012] A second determination module is used to input the plurality of actions and the state data into a guidance network of the reinforcement model to output a target matching degree between each of the actions and the state data;

[0013] The third determination module is used to determine the target action of the target object from the multiple actions obtained by sampling according to the target matching degree between each action and the state data.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method described in the aforementioned aspect.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the steps of the method described in the aforementioned aspect.

[0017] According to another aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which implement the steps of the method described in the aforementioned aspect when executed by a processor.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of another data processing method provided by an embodiment of the present disclosure;

[0022] Figure 3 A structural diagram of a reinforced network provided in an embodiment of the present disclosure;

[0023] Figure 4 A flowchart of another data processing method provided by an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of an action distribution provided by an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure;

[0026] Figure 7 is a block diagram of an example electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] The data processing method, device, electronic device, and storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0029] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure.

[0030] The executor of the data processing method of the embodiment of the present disclosure may be a data processing device of the embodiment of the present disclosure. The data processing device may be arranged in an electronic device. The electronic device may be a robot, a mobile phone or other wearable device, which is not limited in the present embodiment.

[0031] like Figure 1 As shown, the method comprises the following steps:

[0032] Step 101, obtaining status data of the environment where the target object is located.

[0033] In the disclosed embodiments, the state data may indicate a state of the environment in which the target object is located. For example, the target object is a robot, the robot is used to grasp the target object, and the robot's mechanical arm is in a raised state; or the robot is in a target following state, or the robot is in a state of receiving questions in a question-and-answer scenario, etc. The target object may also be a vehicle in an intelligent driving scenario, etc. The business scenario is different and the target object is different, which is not limited in this embodiment. Among them, the state data may be presented in the form of image data, text data, voice data, or video data, etc., which is not limited in this embodiment.

[0034] Among them, in the technical solution of the present disclosure, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0035] Step 102: input the state data into the policy network of the reinforcement model to sample multiple actions corresponding to the state data from the action set.

[0036] The reinforcement model includes a policy network and a guidance network. As an implementation method, the reinforcement model can be a soft actor-critic (SAC) model, in which the actor is the policy network and the critic is the guidance network.

[0037] In the disclosed embodiment, the state data is input into the policy network of the reinforcement model, and an action set consisting of multiple actions corresponding to the state data output by the policy network can be obtained, and multiple actions corresponding to the state data can be sampled from the action set. As an implementation method, a set number of actions are randomly sampled from the action set to obtain a set number of actions. As a second implementation method, sampling can be performed from the action set based on set conditions to obtain multiple actions corresponding to the state data. The set conditions can be an association relationship between the action and the state data, or a matching relationship, or an environmental state, such as a sudden change in the environment.

[0038] Step 103, multiple actions and state data are input into the guidance network of the reinforcement model to output the target matching degree between each action and state data.

[0039] In the disclosed embodiment, multiple actions and state data are converted into corresponding action vectors and state vectors, the action vectors and state vectors are fused to obtain a fusion vector, and the fusion vector is input into the guidance network of the reinforcement model to output the target matching degree between each action and the state data, wherein the target matching degree indicates the possible row of selecting the corresponding action in the next step based on the state data of the current environment of the target object, and the higher the value of the target matching degree, the higher the possibility of the corresponding action being selected.

[0040] In the disclosed embodiment, the training of the policy network depends on the guidance network, so the cumulative expected value corresponding to the action selected by the guidance network is greater than or equal to the cumulative expected value corresponding to the action selected by the policy network, that is, the accuracy of the matching degree between each action and state data output by the guidance network is higher than the matching degree between each action and state output by the policy network. Therefore, the state data of the environment in which the target object is located and multiple actions are combined and input into the guidance network, the target matching degree corresponding to each action is calculated, and the target action is selected based on the target matching degree corresponding to each action, which can improve the accuracy of determining the target action.

[0041] Step 104 , determining a target action of the target object from the sampled multiple actions according to the target matching degree between each action and the state data.

[0042] In the disclosed embodiment, according to the target matching degree between each action and the state data, the action whose matching degree meets the set threshold is used as the target action of the target object. For example, in a question-and-answer scenario, the state data is the current question and the action is the corresponding answer, then the answer with the highest target matching degree is selected as the target answer.

[0043] In the data processing method of the disclosed embodiment, the state data of the environment in which the target object is located is obtained, and the state data is input into the policy network of the reinforcement model to sample multiple actions corresponding to the state data from the action set, and the multiple actions and the state data are input into the guidance network of the reinforcement model to output the target matching degree between each action and the state data, and the target action of the target object is determined from the multiple actions obtained by sampling according to the target matching degree between each action and the state data, and the efficiency of subsequent processing is improved by sampling multiple actions from the action set output by the policy network, and the state data and the multiple actions are combined and input into the guidance network, and the target matching degree corresponding to each action is calculated, and the target matching degree indicates a stronger correlation between each action and the state data, so that the target action is selected based on the target matching degree corresponding to each action, thereby improving the accuracy of determining the target action.

[0044] Based on the above embodiments, the present disclosure provides another data processing method. Figure 2 A flowchart of another data processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0045] Step 201, obtaining status data of the environment where the target object is located.

[0046] Step 202: input the state data into the policy network of the reinforcement model to sample multiple actions corresponding to the state data from the action set.

[0047] Among them, step 201 and step 202 can refer to the explanation in the above-mentioned embodiment, and the principles are the same, so they will not be repeated in this embodiment.

[0048] Step 203: Input the plurality of actions and state data into at least one sub-network in the guidance network respectively, so as to obtain the candidate matching degree corresponding to each action output by each sub-network.

[0049] The guide network includes at least one sub-network, for example, 2 sub-networks, or 3 sub-networks, which is not limited in this embodiment.

[0050] In the disclosed embodiment, multiple actions and state data are respectively input into at least one sub-network included in the guiding network to obtain the candidate matching degree of each action output by each sub-network. Figure 3A schematic diagram of a structure of a reinforced network provided by an embodiment of the present disclosure, such as Figure 3 As shown, a guide network includes two sub-networks as an example. For ease of distinction, they are called the first sub-network and the second sub-network. Multiple action data and state data are input into the first sub-network and the second sub-network, respectively, to obtain the candidate matching degree of each action output by the first sub-network and the candidate matching degree of each action output by the second sub-network.

[0051] Step 204: Determine the target matching degree corresponding to each action according to the candidate matching degree corresponding to each action output by each sub-network.

[0052] For example, two sub-networks are still used as an example for explanation.

[0053] As an implementation method, for each action, the candidate matching degree of the action output by the first sub-network and the candidate matching degree of the action output by the second sub-network are averaged, and the average value is used as the target matching degree corresponding to the action, thereby achieving balanced processing of the target matching degree of the final output and improving the accuracy of the target matching degree.

[0054] As another implementation method, for each action, determine the smaller value between the candidate matching degree of the action output by the first sub-network and the candidate matching degree of the action output by the second sub-network, and use the candidate matching degree corresponding to the smaller value as the candidate matching degree corresponding to the action.

[0055] In the disclosed embodiment, a guidance network is configured to include a plurality of sub-networks, and a target matching degree corresponding to each action is determined based on the candidate matching degrees of each action output by each sub-network. For example, a smaller candidate matching degree is selected as the candidate matching degree of each action, thereby preventing the matching degree output by a single guidance network from being over-estimated, and setting a plurality of sub-networks improves the reliability of determining the target matching degree.

[0056] Step 205 , determining a target action of the target object from the sampled multiple actions according to the target matching degree between each action and the state data.

[0057] For details, please refer to the explanation in the signing embodiment. The principle is the same and will not be repeated in this embodiment.

[0058] In the data determination method of the disclosed embodiment, a guidance network is set to include multiple sub-networks, and the target matching degree corresponding to each action is determined based on the candidate matching degree of each action output by each sub-network. For example, a smaller candidate matching degree is selected as the candidate matching degree of each action, thereby preventing the value of the matching degree output by a single guidance network from being over-estimated, and setting multiple sub-networks improves the reliability of the target matching degree determination.

[0059] Based on the above embodiments, the present disclosure provides another data processing method. Figure 4 A flowchart of another data processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the method comprises the following steps:

[0060] Step 401, obtaining status data of the environment where the target object is located.

[0061] Among them, step 401 can refer to the explanation in the above-mentioned embodiment, and the principle is the same, so it will not be repeated in this embodiment.

[0062] Step 402: input the state data into the policy network, and output the initial action distribution corresponding to the state data.

[0063] The initial action distribution is used to indicate the initial matching degree between each action and the state data in the action set. The initial matching degree corresponding to each action indicates the probability of the state data of the target object's environment selecting the corresponding action in the next step, or the expectation value of selecting the action.

[0064] Step 403: based on the initial matching degree between each action in the initial action distribution and the state data, a plurality of actions corresponding to the state data are sampled from the action set.

[0065] In the disclosed embodiment, according to the initial matching degree, multiple actions corresponding to the state data are sampled from the action set, and sampling based on probability, that is, the initial matching degree, is implemented, and the randomness of sampling is implemented. The higher the initial matching degree, the higher the possibility of the action being sampled. In addition, a set number of actions are sampled, where the value of the set number is adjusted according to the scene requirements. For example, the set number of samples is usually 200. If the sampling is based on the initial matching degree, if the sampling number is too small, the actions with higher matching degrees may not be sampled, and if the sampling number is too large, the difficulty of data processing will increase and the efficiency will be low. Therefore, sampling a set number of actions from the initial distribution can ensure the uniformity of the distribution of the sampled actions, improve the reliability of the sampled multiple actions, and improve the reliability of the subsequent determination of the target action.

[0066] For the initial state distribution, as an example, Figure 5 A schematic diagram of an action distribution provided by an embodiment of the present disclosure, such as Figure 5 As shown in the figure, P is the initial action distribution, which is a Gaussian distribution, indicating the initial matching degree between each action and state data in the action set. The white circular points A on the horizontal axis indicate the sampling points sampled from the action set, and each sampling point corresponds to an action.

[0067] Step 404, multiple actions and state data are input into the guidance network of the reinforcement model to output the target matching degree between each action and state data.

[0068] Among them, step 404 can refer to the explanation in the above-mentioned embodiment, the principle is the same, and it will not be repeated in this embodiment.

[0069] Step 405: Generate a target action distribution corresponding to the state data according to the target matching degree corresponding to each action.

[0070] The target action distribution is used to indicate the matching degree between each action in the sampled action and the state data.

[0071] In one implementation of the disclosed embodiment, the target matching degree corresponding to each action is normalized to obtain the matching degree corresponding to each action after normalization, and the target action distribution corresponding to the state data is generated according to the matching degree corresponding to each action after normalization, that is, the target action distribution indicates each sampled action and the normalized matching degree between each action and the state data. By normalizing the target matching degree corresponding to each action, the target matching degree corresponding to each action is mapped to the same data interval, such as a value between 0 and 1, so as to facilitate subsequent data processing or data comparison, thereby improving the efficiency of data processing.

[0072] Among them, as an implementation method of normalization processing, the target matching degree corresponding to each sampled action is normalized by a normalization function to obtain the matching degree corresponding to each action after normalization, wherein the normalization function is, for example, a softmax function. The matching degree corresponding to each action obtained by normalization processing with the normalization function can generate a smoother target action distribution, that is, a more uniform distribution of matching degrees corresponding to each action, which can improve the stability of subsequent selection of the target action corresponding to the target object, thereby improving the accuracy of determining the target action.

[0073] As another implementation method, normalization processing can be performed according to the maximum and minimum values ​​of the target matching degrees corresponding to each action to obtain the normalized matching degrees corresponding to each action, and based on the normalized matching degrees, a target action distribution is generated, and a target action that interacts with the environmental state of the target object is selected from the target action distribution.

[0074] like Figure 5As shown, according to the target matching degree corresponding to each action, the target action distribution corresponding to the state data is generated, that is, P', wherein the selection probability corresponding to each action sampled in the initial action distribution P and the target action distribution P', that is, the matching degree is different, that is to say, the initial matching degree and the normalized matching degree between each sampled action and the state data are different, wherein the guidance network outputs the target matching degree between each action and the state data, and the target matching degree is higher than the initial matching degree between each action and the state output by the strategy network, so that the target action is selected based on the matching degree corresponding to each action, which can improve the accuracy of the target action determination.

[0075] Step 406 , determining a target action of the target object from the sampled multiple actions according to the matching degree between each action in the target action distribution and the state data.

[0076] Among them, step 406 can refer to the explanation in the above-mentioned embodiment, and the principle is the same, which will not be repeated here.

[0077] In the data processing method of the disclosed embodiment, the state data of the environment in which the target object is located is obtained, and the state data is input into the policy network of the reinforcement model to sample multiple actions corresponding to the state data from the action set, and the multiple actions and the state data are input into the guidance network of the reinforcement model to output the target matching degree between each action and the state data, and the target action of the target object is determined from the multiple actions obtained by sampling according to the target matching degree between each action and the state data, and the efficiency of subsequent processing is improved by sampling multiple actions from the action set output by the policy network, and the state data and the multiple actions are combined and input into the guidance network, and the target matching degree corresponding to each action is calculated, and the target matching degree indicates a stronger correlation between each action and the state data, so that the target action is selected based on the target matching degree corresponding to each action, thereby improving the accuracy of determining the target action.

[0078] In order to implement the above embodiment, this embodiment provides a data processing device.

[0079] Figure 6 A structural diagram of a data processing device provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the device comprises:

[0080] The acquisition module 61 is used to acquire the state data of the environment where the target object is located.

[0081] The first determination module 62 is used to input the state data into the policy network of the reinforcement model to sample a plurality of actions corresponding to the state data from the action set.

[0082] The second determination module 63 is used to input the multiple actions and the state data into the guidance network of the reinforcement model to output the target matching degree between each of the actions and the state data.

[0083] The third determination module 64 is used to determine the target action of the target object from the multiple actions obtained by sampling according to the target matching degree between the actions and the state data.

[0084] Further, in an implementation of the embodiment of the present disclosure, the second determining module 63 is specifically configured to:

[0085] Inputting the plurality of actions and the state data into at least one sub-network in the guidance network to obtain candidate matching degrees corresponding to the actions output by each sub-network;

[0086] According to the candidate matching degrees corresponding to the actions output by the sub-networks, the target matching degrees corresponding to the actions are determined.

[0087] In an implementation of the embodiment of the present disclosure, the first determining module 62 is specifically configured to:

[0088] Input the state data into the policy network, and output an initial action distribution corresponding to the state data; wherein the initial action distribution is used to indicate the initial matching degree between each action in the action set and the state data;

[0089] According to the initial matching degree between each action in the initial action distribution and the state data, a plurality of actions corresponding to the state data are sampled from the action set.

[0090] In an implementation of the embodiment of the present disclosure, the third determining module 64 is specifically configured to:

[0091] Generate a target action distribution corresponding to the state data according to the target matching degree corresponding to each of the actions, wherein the target action distribution is used to indicate the matching degree between each of the actions obtained by the sampling and the state data;

[0092] According to the matching degree between each of the actions in the target action distribution and the state data, a target action of the target object is determined from the multiple actions obtained by sampling.

[0093] In an implementation of the embodiment of the present disclosure, the third determining module 64 is further configured to:

[0094] Normalizing the target matching degree corresponding to each of the actions to obtain the matching degree corresponding to each of the actions after normalization;

[0095] According to the matching degree corresponding to each of the normalized actions, a target action distribution corresponding to the state data is generated.

[0096] In an implementation of the embodiment of the present disclosure, the third determining module 64 is further configured to:

[0097] The matching degrees corresponding to the actions are normalized using a normalization function to obtain the matching degrees corresponding to the actions after normalization.

[0098] It should be understood that the explanations in the aforementioned method embodiment are also applicable to the device in this embodiment, and the principles are the same, which will not be repeated in this embodiment.

[0099] In the data processing device of the embodiment of the present disclosure, the state data of the environment in which the target object is located is obtained, and the state data is input into the policy network of the reinforcement model to sample multiple actions corresponding to the state data from the action set, and the multiple actions and the state data are input into the guidance network of the reinforcement model to output the target matching degree between each action and the state data, and the target action of the target object is determined from the multiple actions obtained by sampling according to the target matching degree between each action and the state data, and the efficiency of subsequent processing is improved by sampling multiple actions from the action set output by the policy network, and the state data and the multiple actions are combined and input into the guidance network, and the target matching degree corresponding to each action is calculated, and the target matching degree indicates a stronger correlation between each action and the state data, so that the target action is selected based on the target matching degree corresponding to each action, thereby improving the accuracy of determining the target action.

[0100] In order to implement the above embodiment, the present disclosure also provides an electronic device, including:

[0101] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method described in the aforementioned method embodiment.

[0102] In order to implement the above embodiments, the embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the steps of the method described in the above method embodiments.

[0103] In order to implement the above embodiments, the embodiments of the present disclosure further provide a computer program product, including computer instructions, which implement the steps of the method described in the above method embodiments when executed by a processor.

[0104] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0105] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0106] Figure 7 1 is a block diagram of an example electronic device of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0107] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 702 or a computer program loaded from a storage unit 708 to a RAM (Random Access Memory) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An I / O (Input / Output) interface 705 is also connected to the bus 704.

[0108] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as a data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the data processing method in any other appropriate manner (eg, by means of firmware).

[0110] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0112] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or an LCD (Liquid Crystal Display)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0115] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0116] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Acquire state data of the environment in which the target object is located, wherein the state data includes image data, text data, voice data, or video data; Inputting the state data into a policy network of a reinforcement model to sample a plurality of actions corresponding to the state data from an action set; Inputting the plurality of actions and the state data into at least one sub-network in the guidance network of the reinforcement model to obtain candidate matching degrees corresponding to the actions output by each sub-network; Determining a target matching degree between each of the actions and the state data according to the candidate matching degrees corresponding to each of the actions output by each of the sub-networks; According to the target matching degree between each action and the state data, a target action of the target object is determined from the multiple actions obtained by sampling.

2. The method of claim 1, wherein: The step of inputting the state data into a policy network of a reinforcement model to sample a plurality of actions corresponding to the state data from an action set includes: Input the state data into the policy network, and output an initial action distribution corresponding to the state data; wherein the initial action distribution is used to indicate the initial matching degree between each action in the action set and the state data; According to the initial matching degree between each action in the initial action distribution and the state data, a plurality of actions corresponding to the state data are sampled from the action set.

3. The method according to any one of claims 1 to 2, wherein: The step of determining the target action of the target object from the plurality of actions obtained by sampling according to the target matching degree between each action and the state data includes: Generate a target action distribution corresponding to the state data according to the target matching degree corresponding to each of the actions, wherein the target action distribution is used to indicate the matching degree between each of the actions obtained by the sampling and the state data; According to the matching degree between each of the actions in the target action distribution and the state data, a target action of the target object is determined from the multiple actions obtained by sampling.

4. The method of claim 3, wherein: Generating a target action distribution corresponding to the state data according to the target matching degree corresponding to each of the actions includes: Normalizing the target matching degree corresponding to each of the actions to obtain the matching degree corresponding to each of the actions after normalization; According to the matching degree corresponding to each of the normalized actions, a target action distribution corresponding to the state data is generated.

5. The method of claim 4, wherein: The step of normalizing the target matching degrees corresponding to the actions to obtain the normalized matching degrees corresponding to the actions includes: The matching degrees corresponding to the actions are normalized using a normalization function to obtain the matching degrees corresponding to the actions after normalization.

6. A data processing device, comprising: An acquisition module, used to acquire state data of the environment in which the target object is located, wherein the state data includes image data, text data, voice data, or video data; A first determination module is used to input the state data into a policy network of a reinforcement model to sample a plurality of actions corresponding to the state data from an action set; A second determination module is used to input the plurality of actions and the state data into at least one sub-network in the guidance network of the reinforcement model to obtain a candidate matching degree corresponding to each of the actions output by each of the sub-networks; Determining a target matching degree between each of the actions and the state data according to the candidate matching degrees corresponding to each of the actions output by each of the sub-networks; The third determination module is used to determine the target action of the target object from the multiple actions obtained by sampling according to the target matching degree between each action and the state data.

7. The device according to claim 6, wherein: The first determining module is specifically configured to: Input the state data into the policy network, and output an initial action distribution corresponding to the state data; wherein the initial action distribution is used to indicate the initial matching degree between each action in the action set and the state data; According to the initial matching degree between each action in the initial action distribution and the state data, a plurality of actions corresponding to the state data are sampled from the action set.

8. The device according to any one of claims 6 to 7, wherein: The third determination module is specifically used to: Generate a target action distribution corresponding to the state data according to the target matching degree corresponding to each of the actions, wherein the target action distribution is used to indicate the matching degree between each of the actions obtained by the sampling and the state data; According to the matching degree between each of the actions in the target action distribution and the state data, a target action of the target object is determined from the multiple actions obtained by sampling.

9. The device of claim 8, wherein: The third determination module is further specifically used for: Normalizing the target matching degree corresponding to each of the actions to obtain the matching degree corresponding to each of the actions after normalization; According to the matching degree corresponding to each of the normalized actions, a target action distribution corresponding to the state data is generated.

10. The device of claim 9, wherein: The third determination module is further specifically used for: The matching degrees corresponding to the actions are normalized using a normalization function to obtain the matching degrees corresponding to the actions after normalization.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make the computer execute the steps of the method according to any one of claims 1-5.

13. A computer program product comprising computer instructions, which when executed by a processor implement the steps of the method according to any one of claims 1 to 5.

Images