Reinforcement learning-based neural network segmentation methods, devices, equipment, and media

Abstract

This invention discloses a neural network segmentation method, apparatus, device, and medium based on reinforcement learning. The method includes: acquiring the neural network to be segmented and all terminal devices on which it depends; determining the state space and behavior space associated with reinforcement learning; constructing a reward function based on the latency states of all terminal devices during operation; acquiring a training sample set from the historical learning environment and importing it into a memory pool; iteratively training the evaluation network and the target network using the memory pool until a preset loss function is minimized; then outputting the optimal segmentation scheme obtained by updating the target network after training; and segmenting the neural network to be segmented into several sub-networks according to the optimal segmentation scheme and correspondingly assigning them to all the terminal devices for operation. This invention effectively improves the execution efficiency of the neural network to be segmented by considering the practical factors that may exist during the operation of all terminal devices and introducing reinforcement learning to train and obtain the optimal segmentation scheme.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a neural network segmentation method, apparatus, device, and medium based on reinforcement learning. Background Technology

[0002] To enable neural networks to complete data processing tasks more efficiently, existing researchers have proposed two approaches: one is to increase hardware resources, that is, to install more CPUs (Central Processing Units) and GPUs (Graphics Processing Units) on a terminal device to provide computing resources, but this also reduces the portability of the terminal device; the other is to publish the computing tasks to the cloud, which integrates a large number of computing resources for processing, but the computing cost in the cloud is high and the risk of privacy leakage is greater.

[0003] To address this, some technicians have proposed dividing the neural network and distributing it across multiple terminal devices. However, since these terminal devices are scattered and their transmission delays are significantly affected by the hardware status of the devices and the network environment they are in, existing neural network segmentation methods do not take these factors into account. They rely solely on pre-provided historical data for analysis, which may result in a suboptimal segmentation solution. Consequently, the efficiency of the segmented neural network across multiple terminal devices may not be effectively improved. Summary of the Invention

[0004] This invention provides a neural network segmentation method, apparatus, device, and medium based on reinforcement learning to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

[0005] Firstly, a neural network segmentation method based on reinforcement learning is provided, the method comprising:

[0006] Acquire the neural network to be segmented and all the terminal devices on which it depends for operation, and determine the state space and behavior space associated with reinforcement learning;

[0007] A reward function is constructed based on the latency status of all the terminal devices operating in coordination.

[0008] The training sample set is obtained from the historical learning environment and imported into the constructed memory pool. Each training sample in the training sample set includes the current state space, behavior space, and reward score, as well as the state space at the next time step.

[0009] The constructed evaluation network and target network are iteratively trained using the memory pool until the preset loss function reaches its minimum. Then, the optimal segmentation scheme obtained by updating the target network after training is output.

[0010] The neural network to be segmented is divided into several sub-networks according to the optimal segmentation scheme, and then the several sub-networks are assigned to all the terminal devices for operation.

[0011] Furthermore, the determined state space includes the expected number of segmentation points of the neural network to be segmented, the terminal device currently in operation, the network environment of the terminal device, the network layer allocated to the terminal device, the amount of input data of the network layer, and the amount of output data of the network layer.

[0012] Furthermore, the determined behavior space includes the state of the split point setting between any two adjacent network layers of the neural network to be segmented.

[0013] Furthermore, the expression for the reward function is:

[0014] R=αN+βT+εD

[0015] Where R is the reward score, N is the average network latency of all terminal devices, T is the average execution time of all terminal devices, D is the average transmission latency of all terminal devices, α is the learning rate, β is the learning decay rate, and ε is the exploration rate.

[0016] Furthermore, the preset loss function is used to characterize the mean square error between the score value of the target network's single training output and the score value of the evaluation network's single training output, wherein any score value includes a reward score.

[0017] Furthermore, the step of iteratively training the constructed evaluation network and target network using the memory pool until the preset loss function reaches its minimum includes:

[0018] The weight parameters of the evaluation network, the weight parameters of the target network, and the Q-table are initialized. The Q-table is used to store the optimal segmentation scheme and its corresponding score during the training process.

[0019] During each training process, a portion of training samples are randomly selected from the memory pool. The evaluation network is trained using the current state space and behavior space of the selected training samples. The target network is trained using the state space of the next time step of the selected training samples. The Q-table is then updated based on the optimal segmentation scheme output by the target network and its corresponding score. If the preset loss function is not determined to be at its minimum, the weight parameters of the evaluation network are updated using the backpropagation algorithm, and the next training iteration is performed. Alternatively, if the preset loss function is determined to be at its minimum, the training ends and the updated Q-table is output.

[0020] Furthermore, during the iterative training of the evaluation network and the target network, when it is detected that the current training time is an integer multiple of a preset time threshold, the weight parameters of the target network are directly updated to the current weight parameters of the evaluation network.

[0021] Secondly, a neural network segmentation device based on reinforcement learning is provided, the device comprising:

[0022] The determination module is used to acquire the neural network to be segmented and all the terminal devices on which it depends for operation, and at the same time determine the state space and behavior space associated with reinforcement learning;

[0023] The construction module is used to construct a reward function based on the latency status of all the terminal devices during operation.

[0024] The acquisition module is used to acquire a training sample set from the historical learning environment and import it into the constructed memory pool. Each training sample in the training sample set includes the current state space, behavior space, and reward score, as well as the state space at the next time step.

[0025] The output module is used to iteratively train the constructed evaluation network and target network using the memory pool until the preset loss function reaches its minimum, and then output the optimal segmentation scheme obtained by updating the target network after the training is completed.

[0026] The allocation module is used to divide the neural network to be segmented into several sub-networks according to the optimal segmentation scheme, and then allocate the several sub-networks to all the terminal devices for operation.

[0027] Thirdly, a computer device is provided, comprising:

[0028] At least one memory;

[0029] At least one processor;

[0030] The memory stores a computer program, and the processor executes the computer program as described in the first aspect: a neural network segmentation method based on reinforcement learning.

[0031] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the reinforcement learning-based neural network segmentation method as described in the first aspect.

[0032] This invention offers at least the following advantages: By introducing reinforcement learning to train and obtain the optimal segmentation scheme for the neural network to be segmented, and considering the realistic factors that may exist when the neural network is distributed across all terminal devices during the entire training process—namely, the current network environment, data processing status, and data transmission status of all terminal devices—the reliability of the optimal segmentation scheme can be improved, thereby effectively increasing the operating efficiency of the neural network distributed across all terminal devices. Furthermore, each terminal device only needs to perform the corresponding processing on its assigned sub-network, without interfering with the operation of other terminal devices, thus minimizing the risk of privacy leakage. Attached Figure Description

[0033] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0034] Figure 1 This is a flowchart illustrating a neural network segmentation method based on reinforcement learning in an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of the composition of a neural network segmentation device based on reinforcement learning in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the hardware structure of the computer device in an embodiment of this disclosure. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0038] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0039] First, some of the terms used in this invention are explained as follows:

[0040] Reinforcement learning is a goal-oriented learning method that aims to learn how to make decisions through various rewards or punishments obtained in the process of interacting with the environment. The simulation of the reinforcement learning decision-making process includes six elements, as follows: The first element is the Agent (algorithm), which interacts with the environment through actions to execute actions instead of making decisions for humans; the second element is the Environment, which is the world in which the Agent interacts, taking the Agent's current state and actions as input, and the reward and the next state as output; the third element is the State, which is the state the Agent is in at a specific point in time, which can be a specific... The first element is the location or configuration of the agent, and the State can be the current state fed back by the environment or the future state; the second element is the Action, which is the set of all actions that the agent can take, and the agent will choose from the list of possible actions; the third element is the Reward, which is the indicator of the success or failure of the actions taken by the agent, and the reward can be immediate or delayed, but the actions taken by the agent must be effectively evaluated; the fourth element is the Policy, which is the strategy by which the agent decides its actions, and it will be adjusted according to different states to obtain the highest possible reward.

[0041] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a neural network segmentation method based on reinforcement learning provided in an embodiment of the present invention. The method includes the following:

[0042] Step S110: Obtain the neural network to be segmented and all terminal devices on which it depends for operation, and at the same time determine the state space and behavior space associated with reinforcement learning;

[0043] Step S120: Construct a reward function based on the latency status of all terminal devices during operation;

[0044] Step S130: Obtain the training sample set from the historical learning environment and import it into the constructed memory pool. Each training sample in the training sample set includes the state space at the current time (denoted as S).t The current action space (denoted as A) t The reward score at the current moment (denoted as R) t and the state space at the next time step (denoted as S) t+1 );

[0045] Step S140: Use the memory pool to iteratively train the constructed evaluation network and target network until the preset loss function reaches its minimum, and then output the optimal segmentation scheme obtained by updating the target network after training.

[0046] Step S150: Divide the neural network to be segmented into several sub-networks according to the optimal segmentation scheme, and then assign the several sub-networks to all the terminal devices for operation.

[0047] In this embodiment of the invention, all the terminal devices mentioned in step S110 above include servers, gateways, etc., with different computing power.

[0048] The state space mentioned in step S110 above actually includes the expected number of segmentation points of the neural network to be segmented, the terminal devices currently in operation among all terminal devices, the network layers allocated to the terminal devices from the neural network to be segmented, the input data volume and output data volume of the network layers, and the network environment in which the terminal devices are located. It should be noted that when the terminal devices are allocated multiple network layers from the neural network to be segmented, the state space should record the multiple network layers and the input data volume and output data volume of each of the network layers; when there are multiple terminal devices currently in operation among all terminal devices, the state space should also record the multiple terminal devices and the network environment in which each of the terminal devices is located.

[0049] The behavior space mentioned in step S110 above actually includes the state of the splitting point setting between any two adjacent network layers of the neural network to be segmented. It can be seen that the behavior space is also related to the number of network layers contained in the neural network to be segmented. The behavior space is illustrated by the following example: Suppose that when the behavior space is A = (0,1,0), it means that the neural network to be segmented has a total of four network layers. There is no splitting point set between the first and second network layers, there is a splitting point set between the second and third network layers, and there is no splitting point set between the third and fourth network layers.

[0050] In this embodiment of the invention, the specific implementation process of step S120 includes the following:

[0051] Step S121: After dividing the neural network to be segmented into several sub-networks, the several sub-networks are assigned to all the terminal devices for operation.

[0052] Step S122: When any sub-network receives a large amount of burst data, but the terminal device where that sub-network is located is in a network congestion state, it may cause the terminal device to shut down directly, thereby affecting the operating efficiency of other terminal devices in their respective assigned sub-networks. To avoid the above situation, it is necessary to roughly assess the network congestion of all terminal devices before the neural network to be segmented officially runs. At this time, the average network latency of all terminal devices is calculated as follows:

[0053]

[0054] Step S123: The better the performance of any terminal device, the shorter the time required for the sub-network allocated to that terminal device to run on that terminal device. The total time required for the sub-networks to run on all terminal devices is minimized, which is a reference basis for judging the rationality of the segmentation of the neural network to be segmented. At this time, the average execution time of all terminal devices is calculated as follows:

[0055]

[0056] Step S124: Since there is a sequential connection between the several sub-networks, the transmission order between all terminal devices can be defined, and a transmission process will occur between every two terminal devices. Each transmission process will generate a certain data transmission delay. At this time, the average transmission delay of all terminal devices is calculated as follows:

[0057]

[0058] Step S125: Determine the specific expression of the reward function as: R = αN + βT + εD;

[0059] Where num is the total number of all terminal devices, t i Let s be the current network latency of the i-th terminal device. i For the total data generated by the i-th terminal device, FT i ST is the end time when the i-th terminal device processes its assigned subnetwork. i Let l be the start time when the i-th terminal device begins processing its assigned sub-network. m,i v represents the amount of data to be transmitted by the i-th terminal device. m,i Let l be the data processing rate of the i-th terminal device. c,i v is the channel length that the i-th terminal device relies on.c,i Let R be the propagation speed of the electromagnetic wave emitted by the i-th terminal device in the channel, R be the reward score, α be the learning rate, β be the learning decay rate, and ε be the exploration rate.

[0060] In this embodiment of the invention, the above step S130 is implemented as follows: several training samples are obtained from the historical learning environment through random sampling to form a training sample set, and the training sample set is stored in the constructed memory pool, and the number of training samples that the memory pool can hold is set to 2000. It should be noted that by constructing an additional memory pool, it is possible to avoid directly obtaining a continuous training sample from the historical learning environment in the subsequent iterative training process, and further avoid the final training result from converging to an incorrect peak interval because the correlation of the continuous training samples may be too close.

[0061] In this embodiment of the invention, the specific implementation process of step S140 includes the following:

[0062] Step S141: Construct the evaluation network and target network associated with reinforcement learning, wherein the evaluation network and the target network are both convolutional neural networks with the same structure;

[0063] Step S142: Construct a preset loss function associated with reinforcement learning. The specific expression of the preset loss function is: Loss = E[(Q_target - Q_evaluation)] 2 ], where Q_target is the score value obtained by the target network through a single training, Q_evaluation is the score value obtained by the evaluation network through a single training, and the score value of either of the above includes the reward score, and E is the solution symbol for the mean square error;

[0064] Step S143: Set the maximum number of iterations to 500, and initialize the Q-table associated with reinforcement learning, the weight parameters of the target network and the weight parameters of the evaluation network respectively. The Q-table is mainly used to store the optimal segmentation scheme and its corresponding score value in the entire iterative training process.

[0065] Step S144: During the j-th training process, a portion of training samples are randomly obtained from the memory pool. Since each training sample in the memory pool includes the state space at the current time (denoted as S...), ... t The current action space (denoted as A) t ) and the state space of the next time step (denoted as S) t+1 ), and all S contained in the aforementioned training samples t Data and all A tThe data is imported into the evaluation network for training to obtain the j-th score (denoted as the P1 score), and all S values ​​contained in the partial training samples are also included. t+1 The data is imported into the target network for training to obtain the j-th score (denoted as P2 score) and its corresponding optimal segmentation scheme. The training output of the target network is then used to update the current Q table.

[0066] Step S145: Based on the P1 score and P2 score obtained during the j-th training process, determine whether the preset loss function can be minimized; if not, update only the weight parameters of the evaluation network using the existing backpropagation algorithm, assign j+1 to j, and return to execute the above step S144; if yes, end the iterative training and directly execute the following step S146.

[0067] Step S146: Output the Q-table updated after the j-th training. At this time, the Q-table actually stores the final optimal segmentation scheme.

[0068] It should be noted that step S144 above is executed starting from j=1; in step S145 above, before assigning j+1 to j, it is actually necessary to first determine whether j is less than or equal to the maximum number of iterations. If so, the assignment continues; otherwise, the iterative training ends directly. However, since in most cases the training of the convolutional neural network has already caused the preset loss function to converge before reaching the maximum number of iterations, this embodiment of the invention does not focus on describing this number determination step.

[0069] In step S141 above, the network structure of the evaluation network is described as follows: When the number of network layers of the neural network to be segmented is K, the evaluation network includes an input layer, a second fully connected layer, a third fully connected layer and an output layer connected in sequence, wherein the size of the input layer is K-1, the size of the second fully connected layer is (K-1) / 2, the size of the third fully connected layer is (K-1) / 8, and the size of the output layer is 1.

[0070] In step S144 above, updating the current Q-table using the training output of the target network specifically involves: the current Q-table internally stores the optimal segmentation scheme (denoted as the first scheme) and its corresponding score value (denoted as the first score value) obtained after the (j-1)th training; the target network obtains the optimal segmentation scheme (denoted as the second scheme) and its corresponding score value (denoted as the second score value, also referred to as the P2 score value mentioned in step S144 above); it is determined whether the second score value is greater than the first score value; if so, it means that the second scheme is better than the first scheme, and the first scheme stored in the current Q-table is directly replaced by the second scheme, and the first score value stored in the current Q-table is directly replaced by the second score value, thereby obtaining the updated current Q-table; if not, it means that the first scheme is better than the second scheme, and no update is needed for the current Q-table.

[0071] In this embodiment of the invention, step S145 only mentions the implementation of the task of updating the weight parameters of the evaluation network. The target network also needs to update its weight parameters throughout the entire iterative training process. Specifically, timing is started from the beginning of the entire iterative training process, and when the current training time is detected to be an integer multiple of a preset time threshold, the current weight parameters of the evaluation network are directly obtained, and then the target network updates them to its own weight parameters at a certain rate. This process does not require recording the number of training sessions that the target network has completed.

[0072] It should be noted that after performing the above step S150, the present invention will start timing when all terminal devices start running the neural network to be segmented, and when it is detected that the current running time has reached the preset running time threshold, it will re-enter the iterative training stage, that is, re-execute the above step S140, so as to dynamically obtain a new optimal segmentation scheme output, thereby maximizing the overall running efficiency of the neural network to be segmented on all terminal devices.

[0073] In this embodiment of the invention, reinforcement learning is introduced to train and obtain the optimal segmentation scheme for the neural network to be segmented. Throughout the training process, realistic factors that may arise when the neural network is distributed across all terminal devices—namely, the current network environment, data processing status, and data transmission status of each terminal device—are considered. This improves the reliability of the optimal segmentation scheme, thereby effectively increasing the operational efficiency of the neural network distributed across all terminal devices. Furthermore, each terminal device only needs to perform the corresponding processing on its assigned sub-network, without interfering with the operation of other terminal devices, thus minimizing the risk of privacy breaches.

[0074] Please refer to Figure 2 , Figure 2 This is a schematic diagram of a neural network segmentation device based on reinforcement learning provided in an embodiment of the present invention. The device includes:

[0075] The determination module 210 is used to acquire the neural network to be segmented and all terminal devices, wherein the neural network to be segmented depends on the cooperative operation of all terminal devices, and at the same time determines the behavior space and state space associated with reinforcement learning.

[0076] Construction module 220 is used to construct a reward function by utilizing the latency state that occurs when all the terminal devices are running in coordination;

[0077] The acquisition module 230 is used to acquire a training sample set from the historical learning environment and store the training sample set into a pre-built memory pool. Each training sample in the training sample set includes the state space of the next time step and the behavior space, state space and reward score of the current time step.

[0078] Output module 240 is used to perform iterative training on the constructed target network and evaluation network using the memory pool until the preset loss function is minimized, and finally output the optimal segmentation scheme obtained by the target network after the training is completed.

[0079] The allocation module 250 is used to segment the neural network to be segmented using the optimal segmentation scheme to obtain several sub-networks, and then allocate the several sub-networks to all the terminal devices for operation.

[0080] The content of the above method embodiments is applicable to the device embodiments. The functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are the same as those of the above method embodiments, so they will not be repeated here.

[0081] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the reinforcement learning-based neural network segmentation method described in the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium on which a device (e.g., a computer, mobile phone, etc.) stores or transmits information in a readable form, and can be a read-only memory, a disk, or an optical disk, etc.

[0082] also, Figure 3 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. The computer device includes components such as a processor 320, a memory 330, an input unit 340, and a display unit 350. Those skilled in the art will understand that... Figure 3 The illustrated device structure is not intended to limit all devices and may include more or fewer components than shown, or combine certain components. The memory 330 can be used to store the computer program 310 and various functional modules. The processor 320 runs the computer program 310 stored in the memory 330, thereby performing various functional applications and data processing of the device. The memory can be internal memory or external memory, or include both internal and external memory. Internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. External memory may include hard disks, floppy disks, ZIP disks, USB flash drives, magnetic tapes, etc. The memory 330 disclosed in the embodiments of this invention includes, but is not limited to, these types of memory. The memory 330 disclosed in the embodiments of this invention is only an example and not a limitation.

[0083] Input unit 340 is used to receive signal input and user-input keywords. Input unit 340 may include a touch panel and other input devices. The touch panel can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel) and drive the corresponding connection device according to a pre-set program; other input devices may include, but are not limited to, one or more of physical keyboards, function keys (such as play control buttons, power buttons, etc.), trackballs, mice, joysticks, etc. Display unit 350 can be used to display user-input information or information provided to the user, as well as various menus of the terminal device. Display unit 350 may be in the form of a liquid crystal display, organic light-emitting diode, etc. Processor 320 is the control center of the terminal device, connecting various parts of the entire device through various interfaces and lines, performing various functions and processing data by running or executing software programs and / or modules stored in memory 320, and calling data stored in memory.

[0084] As one embodiment, the computer device includes at least one processor 320, at least one memory 330, and at least one computer program 310, wherein the at least one computer program 310 is stored in the at least one memory 330 and configured to be executed by the at least one processor 320, and the at least one computer program 310 is configured to perform the reinforcement learning-based neural network segmentation method in the above embodiment.

[0085] Although the description of this application has been quite detailed and particularly focused on several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment. Rather, it should be considered as effectively covering the intended scope of this application by referring to the appended claims and taking into account the prior art, which provides for a broad possible interpretation of these claims. Furthermore, the foregoing description of this application with respect to embodiments foreseeable by the inventors is intended to provide a useful description, and non-substantial modifications to this application that have not yet been foreseen may still represent equivalent modifications.

Claims

1. A neural network segmentation method based on reinforcement learning, characterized in that, The method includes: The process involves acquiring the neural network to be segmented and all the terminal devices on which it depends, while simultaneously determining the state space and behavior space associated with reinforcement learning. The state space includes the desired number of segmentation points for the neural network to be segmented, the terminal devices currently in operation, the network environment of the terminal devices, the network layers allocated to the terminal devices, the amount of input data for each network layer, and the amount of output data for each network layer. The behavior space includes the segmentation point setting state between any two adjacent network layers of the neural network to be segmented. Based on the latency status of all the terminal devices operating in coordination, a reward function is constructed, and the expression of the reward function is: In the formula, As a reward point, The average network latency for all the terminal devices. The average execution time for all the terminal devices. The average transmission delay of all the terminal devices. For learning rate, To learn the decay rate, For exploration rate; The training sample set is obtained from the historical learning environment and imported into the constructed memory pool. Each training sample in the training sample set includes the current state space, behavior space, and reward score, as well as the state space at the next time step. The constructed evaluation network and target network are iteratively trained using the memory pool until the preset loss function reaches its minimum. Then, the optimal segmentation scheme obtained by updating the target network after training is output. The neural network to be segmented is divided into several sub-networks according to the optimal segmentation scheme, and then the several sub-networks are assigned to all the terminal devices for operation.

2. The neural network segmentation method based on reinforcement learning according to claim 1, characterized in that, The preset loss function is used to characterize the mean square error between the score value of the target network's single training output and the score value of the evaluation network's single training output, wherein any score value includes a reward score.

3. The neural network segmentation method based on reinforcement learning according to claim 2, characterized in that, The step of iteratively training the constructed evaluation network and target network using the memory pool until the preset loss function reaches its minimum includes: The weight parameters of the evaluation network, the weight parameters of the target network, and the Q-table are initialized. The Q-table is used to store the optimal segmentation scheme and its corresponding score during the training process. During each training process, a portion of training samples are randomly selected from the memory pool. The evaluation network is trained using the current state space and behavior space of the selected training samples. The target network is trained using the state space of the next time step of the selected training samples. The Q-table is then updated based on the optimal segmentation scheme output by the target network and its corresponding score. If the preset loss function is not determined to be at its minimum, the weight parameters of the evaluation network are updated using the backpropagation algorithm, and the next training iteration is performed. Alternatively, if the preset loss function is determined to be at its minimum, the training ends and the updated Q-table is output.

4. The neural network segmentation method based on reinforcement learning according to claim 3, characterized in that, During the iterative training of the evaluation network and the target network, when it is detected that the current training time is an integer multiple of a preset time threshold, the weight parameters of the target network are directly updated to the current weight parameters of the evaluation network.

5. A neural network segmentation device based on reinforcement learning, characterized in that, The device includes: A determination module is used to acquire the neural network to be segmented and all terminal devices on which it depends for operation, and to determine the state space and behavior space associated with reinforcement learning; wherein, the state space includes the expected number of segmentation points of the neural network to be segmented, the terminal devices currently in operation, the network environment of the terminal devices, the network layers allocated to the terminal devices, the amount of input data of the network layers, and the amount of output data of the network layers; the behavior space includes the segmentation point setting state between any two adjacent network layers of the neural network to be segmented. The construction module is used to construct a reward function based on the latency status of all the terminal devices during operation. The expression of the reward function is: In the formula, As a reward point, The average network latency for all the terminal devices. The average execution time for all the terminal devices. The average transmission delay of all the terminal devices. For learning rate, To learn the decay rate, For exploration rate; The acquisition module is used to acquire a training sample set from the historical learning environment and import it into the constructed memory pool. Each training sample in the training sample set includes the current state space, behavior space, and reward score, as well as the state space at the next time step. The output module is used to iteratively train the constructed evaluation network and target network using the memory pool until the preset loss function reaches its minimum, and then output the optimal segmentation scheme obtained by updating the target network after the training is completed. The allocation module is used to divide the neural network to be segmented into several sub-networks according to the optimal segmentation scheme, and then allocate the several sub-networks to all the terminal devices for operation.

6. A computer device, characterized in that, include: At least one memory; At least one processor; The memory stores a computer program, and the processor executes the computer program to implement the neural network segmentation method based on reinforcement learning as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the neural network segmentation method based on reinforcement learning as described in any one of claims 1 to 4.

Images