A method and device for remote upgrading of an internet of things terminal based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's internet of things terminal remote upgrading method and device, it is related to internet of things technical field, the method includes by gateway sends upgrade request to internet of things terminal, to notify internet of things terminal to carry out remote upgrading, the upgrade request includes the address of update server, the latest deadline of remote upgrading;The flash area of internet of things terminal is divided, and flash is divided into application software program area and upgrade boot program area;Internet of things terminal receives upgrade request and switches interrupt vector table from application software program to upgrade boot program, carries out distributed autonomous upgrading decision based on reinforcement learning algorithm;Based on FTP protocol, communication between internet of things terminal and update server, to download the file required for updating, and carry out the update of Q table, and carry out the update of application software firmware.The application can guarantee the reliability of remote update upgrading.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and specifically to a method and apparatus for remotely upgrading IoT terminals based on reinforcement learning. Background Technology

[0002] With the development of IoT technology, the number of IoT sensing devices, such as sensors, has exploded, generating massive amounts of real-time data. In sensor networks, terminal devices need to report data to a gateway for management. As practical needs change and security issues arise, the firmware of IoT terminal devices often needs to be updated remotely. During a remote upgrade, the IoT terminal needs to obtain the new firmware and load it into memory to complete the update. However, updating all terminal devices simultaneously often exceeds the gateway's processing capacity. When the number of update requests the gateway can handle exceeds its capacity, it can easily lead to response timeouts, update failures, and other problems, resulting in a waste of time and energy.

[0003] Currently, conventional remote upgrade solutions for IoT devices control the number of devices being upgraded simultaneously by assigning them numbers and sending update requests in batches through a gateway. While this method is simple to manage, it suffers from low automation, performs poorly in complex environments, lacks real-time performance, cannot respond to changes in the network environment, and fails to fully utilize gateway resources. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a method and device for remote upgrading of Internet of Things terminals based on reinforcement learning, which can ensure the reliability of remote updates and upgrades.

[0005] To achieve the above objectives, this invention provides a method for remotely upgrading IoT terminals based on reinforcement learning, specifically including the following steps: The upgrade request is sent to the IoT terminal through the gateway to notify the IoT terminal to perform a remote upgrade. The upgrade request includes the address of the update server and the latest deadline for the remote upgrade. The Flash memory area of ​​the IoT terminal is divided into an application software program area and an upgrade boot program area. After receiving an upgrade request, the IoT terminal switches the interrupt vector table from the application software program to the upgrade boot program, and makes distributed autonomous upgrade decisions based on reinforcement learning algorithms. Based on the FTP protocol, IoT terminals communicate with the update server to download the necessary update files, update the Q table, and update the application software firmware.

[0006] Based on the above technical solution, the IoT terminal runs an application software program in normal operation. When it receives an upgrade request sent by the gateway, it switches the interrupt vector table from the application software program to the upgrade boot program.

[0007] Based on the above technical solution, the specific steps of making distributed autonomous upgrade decisions based on reinforcement learning algorithms include: The number of upgrades of the IoT terminal is determined. If the IoT terminal is being upgraded for the first time, the IoT terminal will randomly initialize the time slot number as the initial state and initialize the Q table. Otherwise, the time slot number used in the last upgrade will be used as the state and the Q table will be reused. The action is selected according to the ε-greedy policy. It is randomly selected with a probability of ε, and the action with the largest Q value in the current state is selected by looking up the Q table with a probability of 1-ε. The time slot number is modified according to the selected action to obtain the new state. The start time of this remote upgrade is calculated based on the latest deadline for remote upgrade in the upgrade request sent by the gateway and the time slot number. The remote upgrade will begin after the waiting time has elapsed.

[0008] Based on the above technical solutions, During Q-table initialization, values ​​are assigned to the Q-table according to the needs of the application scenario. Furthermore, the Q-value of the action of decreasing the time slot number can be increased in each time slot state of the Q-table, so as to complete the upgrade in a shorter time slot.

[0009] Based on the above technical solution, the actions include decreasing the time slot number, maintaining the time slot number, and increasing the time slot number.

[0010] Based on the above technical solution, the communication between the IoT terminal and the update server based on the FTP protocol to download the required update files includes the following specific steps: The IoT terminal communicates with the update server via the FTP protocol. The IoT terminal downloads the application software firmware description file and parses the file content. The application software firmware description file includes the firmware version number, firmware file size, author and modification date. The IoT terminal compares the original firmware version number with the firmware version number obtained by parsing the application software firmware description file to determine whether a new application software firmware needs to be downloaded. If so, the IoT terminal downloads the new application software firmware, verifies its integrity, and then switches to the application software program; If not, switch directly to the application software program.

[0011] Based on the above technical solution, the Q-table is updated, wherein the update formula used for updating the Q-table is:

[0012] in, Indicates the state The following actions are adopted Q value, Indicates the current state. Indicates the action of selection. Indicates the learning rate. Represents the reward function, Indicates the discount factor. Indicates that the action has been completed. The next state after that, Indicates the state The action to choose from, Indicates the state The following actions are adopted. The obtained Q value.

[0013] Based on the above technical solution, before updating the application software firmware, the following is also included: Determine if the application software firmware has been downloaded successfully: If so, then update the application software firmware; If not, switch from the upgrade bootloader to the application software program, report the update failure to the gateway, and end the update process.

[0014] Based on the above technical solution, the specific steps for updating the application software firmware include: Save the interrupt vector table of the upgrade bootloader, set the upgrade flag, disable interrupts, and clear the application software program code area; The downloaded application software firmware is read and parsed in 512-byte blocks. The application software firmware includes address identifiers and machine code in ASCII format. The upgrade bootloader extracts the address identifier and stores it in the address variable, and converts the machine code in ASCII format into hexadecimal machine code and stores it in the buffer area; The application software firmware is continuously read until the entire file is read. When the end of the file (EOF) is reached, the data in the buffer is written to the Flash memory pointed to by the address variable. The upgrade flag is changed to "completed". The data in the additional interrupt vector table buffer is written in blocks to the interrupt vector table of the main controller and overwritten. Interrupts are enabled. The IoT terminal restarts the main controller, switches to the application software program, and reports the new firmware version number to the gateway. In the process of parsing the application software firmware, when the cache storage space is full, the cache is processed as follows: if the address variable points to the interrupt vector table, the data in the cache is written to an additional interrupt vector table cache according to the offset; otherwise, the contents of the cache are written to the Flash pointed to by the address variable in blocks, and the address variable is modified to point to the address of the next machine code write.

[0015] This invention provides a remote upgrade device for IoT terminals based on reinforcement learning, comprising: The sending module is used to send an upgrade request to the IoT terminal through the gateway to notify the IoT terminal to perform a remote upgrade. The upgrade request includes the address of the update server and the latest deadline for the remote upgrade. The partitioning module is used to partition the Flash area of ​​the IoT terminal, dividing the Flash into an application software program area and an upgrade boot program area. The decision module is used to drive the IoT terminal to switch the interrupt vector table from the application software program to the upgrade boot program after receiving the upgrade request, and to make distributed autonomous upgrade decisions based on reinforcement learning algorithms. The update module is used to drive communication between IoT terminals and the update server based on the FTP protocol, so as to download the files required for the update, update the Q table, and update the application software firmware.

[0016] Compared with the prior art, the advantages of the present invention are as follows: (1) For IoT devices, a distributed autonomous update decision is made using reinforcement learning in IoT devices with limited resources, thereby realizing stable, reliable and intelligent remote upgrades of IoT devices; (2) Using reinforcement learning algorithms for distributed autonomous decision-making, terminals do not communicate with each other and the communication load between terminals and gateways is reduced, avoiding excessive competition for resources between terminals, which can reduce the time required for updates and reduce energy waste; (3) The remote update mechanism involved in this invention uses the memory of the Internet of Things networking module, which can realize remote updates on the main controller device with limited storage resources. The reliability of remote updates is guaranteed by the verification mechanism of software firmware and description file. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for remotely upgrading an IoT terminal based on reinforcement learning, as described in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the overall process framework of the reinforcement learning-based remote upgrade method for IoT terminals according to the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0020] See Figure 1 As shown in the figure, the present invention provides a method for remotely upgrading an IoT terminal based on reinforcement learning, which specifically includes the following steps: S1: Send an upgrade request to the IoT terminal through the gateway to notify the IoT terminal to perform a remote upgrade. The upgrade request includes the address of the update server and the latest deadline for the remote upgrade. S2: Divide the Flash area of ​​the IoT terminal into an application software program area and an upgrade boot program area; S3: After receiving the upgrade request, the IoT terminal switches the interrupt vector table from the application software program to the upgrade boot program, and makes distributed autonomous upgrade decisions based on reinforcement learning algorithms. In this invention, the IoT terminal runs an application software program in its daily operation. When it receives an upgrade request sent by the gateway, it switches the interrupt vector table from the application software program to the upgrade boot program, thereby starting to make distributed autonomous upgrade decisions.

[0021] In this invention, distributed autonomous upgrade decision-making based on reinforcement learning algorithms includes the following specific steps: S301: Determine the number of upgrades for the IoT terminal. If the IoT terminal is being upgraded for the first time, the IoT terminal will randomly initialize the time slot number as the initial state and initialize the Q table. Otherwise, the time slot number used in the last upgrade will be used as the state and the Q table will be reused. The reinforcement learning algorithm of this invention is the QLearning reinforcement learning algorithm, which is an algorithm for recording behavior values ​​(Q).

[0022] During Q-table initialization, values ​​are assigned to the Q-table according to the needs of the application scenario. Furthermore, the Q-value of the action of decreasing the time slot number can be increased in each time slot state of the Q-table, so as to complete the upgrade in a shorter time slot. This enables the terminal to have the trend of starting remote upgrades earlier.

[0023] S302: Select an action according to the ε-greedy policy. Randomly select an action with a probability of ε. Select the action with the largest Q value in the current state by looking up the Q table with a probability of 1-ε. Modify the slot number according to the selected action to obtain the new state. The actions include decreasing the time slot number, maintaining the time slot number, and increasing the time slot number. That is, depending on the selected action... The new state is obtained by modifying the time slot number. .

[0024] S303: Calculate the start time of this remote upgrade based on the latest deadline for remote upgrade in the upgrade request sent by the gateway and the time slot number, and start the remote upgrade after the waiting time is reached.

[0025] S4: Based on the FTP (File Transfer Protocol) protocol, IoT terminals communicate with the update server to download the necessary update files, update the Q table, and update the application software firmware.

[0026] In this invention, based on the FTP protocol, the IoT terminal communicates with the update server to download the necessary update files. Specific steps include: S401: The IoT terminal communicates with the update server via the FTP protocol. The IoT terminal downloads the application software firmware description file and parses the file content. The application software firmware description file includes the firmware version number, firmware file size, author, and modification date. That is, the application software firmware description file includes necessary information such as firmware version number and firmware file size, and may also include non-essential information such as author and modification date.

[0027] S402: The IoT terminal compares the original firmware version number with the firmware version number obtained by parsing the application software firmware description file to determine whether a new application software firmware needs to be downloaded. If so, the IoT terminal downloads the new application software firmware, verifies its integrity, and then switches to the application software program; If not, switch directly to the application software program.

[0028] The IoT terminal downloads application software firmware using the IoT networking module and stores it in the network module's memory. After the download is complete, it performs integrity verification based on the relevant information in the description file. If the verification fails, it tries to download again until the preset number of fault tolerances is reached.

[0029] In this invention, the Q-table is updated, and the update formula used for updating the Q-table is as follows:

[0030] in, Indicates the state The following actions are adopted. Q value, Indicates the current state. Indicates the action of selection. Indicates the learning rate. This represents the reward function, which is determined by the download status. If the download is successful, the reward function should be positive; otherwise, it can be set to different negative values ​​depending on the number of retries, indicating the intensity of competition in the current state. The discount factor can be set according to the actual application scenario. In this embodiment of the invention, it is set as follows: It is 0.5. It is 0.5. Indicates that the action has been completed. The next state after that, Indicates the state The action to choose from, Indicates the state The following actions are adopted. The obtained Q value.

[0031] In this invention, before updating the application software firmware, the following steps are also included: Determine if the application software firmware has been downloaded successfully: If so, then update the application software firmware; If not, the update process switches from the upgrade bootloader to the application software program, reports the update failure to the gateway, and terminates the update. The main controller used in this embodiment of the invention is the MSP430F5438A chip.

[0032] In this invention, the specific steps for updating application software firmware include: S411: Save the interrupt vector table of the upgrade bootloader, set the upgrade flag, disable interrupts, and clear the application software program code area; S412: Read and parse 512 bytes of data from the downloaded application software firmware in sequence. The application software firmware includes address identifiers and machine code in ASCII format. S413: The upgrade bootloader extracts the address identifier and stores it in the address variable, and converts the machine code in ASCII form into hexadecimal machine code and stores it in the buffer. S414: Continuously reads the application software firmware until all reading is complete. When the end of the file (EOF) is reached, the data in the buffer is written to the Flash pointed to by the address variable, the upgrade flag is modified to complete, the data in the additional interrupt vector table buffer is written in blocks to the interrupt vector table of the main controller and overwritten, and interrupts are enabled. S415: The IoT terminal restarts the main controller, switches to the application software program, and reports the new firmware version number to the gateway; In the process of parsing the application software firmware, when the cache storage space is full, the cache is processed as follows: if the address variable points to the interrupt vector table, the data in the cache is written to an additional interrupt vector table cache according to the offset; otherwise, the contents of the cache are written to the Flash pointed to by the address variable in blocks, and the address variable is modified to point to the address of the next machine code write.

[0033] The overall flowchart of the reinforcement learning-based remote upgrade method for IoT terminals in this invention is as follows: Figure 2 As shown.

[0034] The remote upgrade method for IoT terminals of the present invention mainly includes three parts: a gateway, an IoT terminal, and an update server.

[0035] The gateway is used to initiate remote upgrade requests, communicates directly with IoT terminals running application software programs, and provides IoT terminals with information such as the address of the update server and the latest deadline for remote upgrades. IoT terminals make distributed autonomous upgrade decisions using reinforcement learning algorithms, avoiding excessive competition among multiple terminals for the limited resources of the update server. After making a decision using the reinforcement learning algorithm, the terminal waits for the update time and then communicates with the update server. Throughout the process, the terminals do not communicate with each other; they only learn about the environment through reinforcement learning algorithms to coordinate the relationship between multiple terminals and the update server. The IoT terminals download the necessary update files from the update server via the FTP protocol. The update server provides FTP service for IoT terminals, allowing them to download application software firmware and application software firmware description files.

[0036] In this invention, the IoT terminal makes distributed autonomous upgrade decisions using a reinforcement learning algorithm. The QLearning reinforcement learning algorithm divides the time into multiple time slots based on the latest deadline for remote upgrades provided by the gateway. Randomly initialized time slot numbers are used as the model's states, and decreasing, maintaining, or increasing the time slot number are used as the model's actions. The model is iteratively trained to obtain a suitable action strategy. The terminal selects an action using the reinforcement learning algorithm and updates the state (time slot number). It observes the performance of the remote upgrade under the new state and modifies the expected cumulative reward obtained by taking a specific action in the corresponding state in the Q-table.

[0037] During the initialization phase, values ​​can be flexibly assigned to the Q table according to actual needs. For example, if it is desired to reduce the waiting time of IoT terminals, the Q value of the action to reduce the time slot number can be increased in each state during the initialization of the Q table.

[0038] During the iteration phase, actions are selected according to the ε-greedy algorithm. Actions are randomly selected with a probability of ε, and the action with the highest expected cumulative reward in the current state is selected with a probability of 1-ε.

[0039] In this invention, multiple IoT terminals do not need to communicate with each other, and there is no management relationship between the IoT terminals and the update server; the update server only needs to provide FTP service. After receiving an update request from the gateway, the IoT terminal switches from the application software program to the upgrade bootloader to perform the update. The terminal obtains the application firmware description file from the update server and compares it with the locally stored application software firmware information to determine if an update is needed. If an update is required, the application firmware is downloaded. After downloading, integrity verification can be performed using information in the application software firmware description file, such as file size and MD5 checksum. After integrity verification, the firmware is moved.

[0040] In this invention, the update status of the remote upgrade of the observed terminal in the current time slot is used as the input to the reward function. Based on the principle of reinforcement learning, the agent learns a suitable action strategy through interaction with the environment. The IoT terminal observes the update status of the remote upgrade, including whether the upgrade was successful, the number of attempts, etc., to infer the competition situation of other terminals and the resource availability of the update server, and finally obtains an action strategy and time slot number suitable for the terminal.

[0041] In this invention, the IoT terminal operates normally within the application software program. Upon receiving an update request from the gateway, it switches to the upgrade bootloader. The upgrade bootloader is responsible for distributed decision-making and remote upgrades, including downloading, parsing, and loading the application software firmware. The downloaded firmware is stored in the storage area of ​​the IoT networking module. During firmware loading, the upgrade bootloader reads 512 bytes of firmware at a time, parses it, and gradually overwrites the application software program. After all bytes are read, the upgrade bootloader switches back to the application software program, which then reports the version number to the gateway.

[0042] The present invention provides a remote upgrade method for IoT terminals based on reinforcement learning, in which the IoT terminal autonomously decides when to update and is able to learn from the environment.

[0043] The IoT terminal main controller contains two programs: an upgrade bootloader and an application software program. Both programs reside in Flash memory. In normal operation, the main controller runs the application software program. Only upon receiving an upgrade request from the gateway does it switch to the upgrade bootloader for remote upgrades. After switching to the upgrade bootloader, it first obtains the ideal upgrade time slot number based on a reinforcement learning algorithm, then waits for the specific time slot to initiate the upgrade. It then communicates with the update server via FTP through the IoT networking module to download the new version of the application firmware and application firmware description file. After verifying its integrity, it loads the new firmware into Flash memory, overwriting the old application. Based on environmental feedback, it updates the reinforcement learning model, switches to the new application software program, and the new application software reports the new version number to the gateway, completing the update process.

[0044] In this invention, after the terminal runs the upgrade bootloader for the first time, it randomly generates an initial timeslot number and stores it in Flash memory. Subsequently, the upgrade bootloader maintains this number using a reinforcement learning algorithm.

[0045] In this invention, the reinforcement learning algorithm used is a Q-learning-based algorithm. This algorithm uses a Q-table to store the Q-value corresponding to each state and action. The Q-value represents the expected cumulative reward obtained by taking a certain action in a given state. During the iterative process of Q-learning, the IoT terminal communicates with the update server, obtains the reward signal and new state based on the update, and updates the Q-values ​​of the corresponding states and actions in the Q-table accordingly. The update formula for the Q-table is: .

[0046] In this invention, a time slot number is selected as the state, and actions include decreasing the time slot number, retaining the time slot number, and increasing the time slot number. During each update, the terminal employs an ε-greedy strategy to select an action; that is, it selects a random action with a probability of ε and consults the Q-table with a probability of 1-ε to select the action with the highest Q value to make a decision. After making a decision, the terminal immediately modifies the time slot number according to the selected action and waits to observe the update situation. Based on the feedback from the update situation, it updates according to a formula. As the communication Q-table between the IoT terminal and the update server is continuously updated iteratively, the terminal can gradually learn the optimal action strategy and adjust to obtain a stable time slot number.

[0047] In this invention, the IoT terminal and the update server communicate via FTP, and the downloaded data is stored in the memory of the IoT networking module. The update server stores two files: application software firmware and an application software firmware description file. The application software firmware description file contains basic information such as firmware size and version number, and can be used to verify the integrity of the downloaded application software firmware. The terminal first downloads the application software firmware description file and determines whether its local application firmware version number is the latest version. If an update is confirmed, the terminal then downloads the application software firmware. After downloading, the information in the application software firmware and the application software firmware description file is compared for consistency. Once the integrity verification is successful, the terminal reads the memory in the IoT networking module and writes the application software firmware code into Flash memory. After completion, it updates the local application firmware version number, and finally updates the interrupt vector table and restarts to achieve the switch from the upgrade bootloader to the application software program.

[0048] This invention employs QLearning reinforcement learning, using update information as feedback to continuously adjust time slot numbers. This avoids excessive competition among multiple terminals for limited update server resources, enabling intelligent distributed autonomous decision-making and stable, reliable remote upgrades. This invention provides a reinforcement learning-based remote upgrade method for IoT terminals, addressing the technical problem of providing an easy-to-implement, efficient, and reliable remote upgrade mechanism. This invention is achieved through the following technical solutions: distributed autonomous decision-making based on QLearning reinforcement learning and a remote update mechanism between the terminal and the update server.

[0049] The algorithm flow for QLearning's autonomous update decision includes the following steps: (1) Randomly initialize the time slot number as the initial state and initialize the Q table; (2) According to the ε-greedy strategy, a random action is selected with a probability of ε, and the action with the highest Q value is selected by querying the Q table with a probability of 1-ε. The state and time slot number are updated according to the selected action. (3) After the waiting time slot arrives, perform a remote update, observe the result of the remote upgrade update as the input of the reward function, and update the Q table using the following formula: .

[0050] (4) If the update is successful, the decision algorithm ends and the application software program is switched; otherwise, the process jumps to step (2).

[0051] In one possible implementation, the present invention also provides a non-transitory computer-readable storage medium located in a PLC (Programmable Logic Controller) controller. The storage medium stores a computer program that, when executed by a processor, implements the steps of the following reinforcement learning-based remote upgrade method for IoT terminals: The upgrade request is sent to the IoT terminal through the gateway to notify the IoT terminal to perform a remote upgrade. The upgrade request includes the address of the update server and the latest deadline for the remote upgrade. The Flash memory area of ​​the IoT terminal is divided into an application software program area and an upgrade boot program area. After receiving an upgrade request, the IoT terminal switches the interrupt vector table from the application software program to the upgrade boot program, and makes distributed autonomous upgrade decisions based on reinforcement learning algorithms. Based on the FTP protocol, IoT terminals communicate with the update server to download the necessary update files, update the Q table, and update the application software firmware.

[0052] Storage media may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0053] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0054] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0055] This invention provides a remote upgrade device for an Internet of Things (IoT) terminal based on reinforcement learning, comprising a sending module, a partitioning module, a decision-making module, and an update module.

[0056] The sending module sends upgrade requests to the IoT terminal via the gateway to notify the IoT terminal to perform a remote upgrade. The upgrade request includes the address of the update server and the latest deadline for the remote upgrade. The partitioning module partitions the Flash area of ​​the IoT terminal into an application software program area and an upgrade boot program area. The decision-making module drives the IoT terminal to switch the interrupt vector table from the application software program to the upgrade boot program after receiving the upgrade request, and performs distributed autonomous upgrade decision-making based on reinforcement learning algorithm. The update module drives the IoT terminal to communicate with the update server based on the FTP protocol to download the required update files, update the Q table, and update the application software firmware.

[0057] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. A method for remotely upgrading IoT terminals based on reinforcement learning, characterized in that, Specifically, the following steps are included: The upgrade request is sent to the IoT terminal through the gateway to notify the IoT terminal to perform a remote upgrade. The upgrade request includes the address of the update server and the latest deadline for the remote upgrade. The Flash memory area of ​​the IoT terminal is divided into an application software program area and an upgrade boot program area. After receiving an upgrade request, the IoT terminal switches the interrupt vector table from the application software program to the upgrade boot program, and makes distributed autonomous upgrade decisions based on reinforcement learning algorithms. Based on the FTP protocol, the IoT terminal communicates with the update server to download the necessary update files, update the Q table, and update the application software firmware. The specific steps of making distributed autonomous upgrade decisions based on reinforcement learning algorithms include: The number of upgrades of the IoT terminal is determined. If the IoT terminal is being upgraded for the first time, the IoT terminal will randomly initialize the time slot number as the initial state and initialize the Q table. Otherwise, the time slot number used in the last upgrade will be used as the state and the Q table will be reused. The action is selected according to the ε-greedy policy. It is randomly selected with a probability of ε, and the action with the largest Q value in the current state is selected by looking up the Q table with a probability of 1-ε. The time slot number is modified according to the selected action to obtain the new state. The start time of this remote upgrade is calculated based on the latest deadline for remote upgrade in the upgrade request sent by the gateway and the time slot number. The remote upgrade will begin after the waiting time has elapsed. The specific steps for updating the application software firmware include: Save the interrupt vector table of the upgrade bootloader, set the upgrade flag, disable interrupts, and clear the application software program code area; The downloaded application software firmware is read and parsed in 512-byte blocks. The application software firmware includes address identifiers and machine code in ASCII format. The upgrade bootloader extracts the address identifier and stores it in the address variable, and converts the machine code in ASCII format into hexadecimal machine code and stores it in the buffer. The application software firmware is continuously read until the entire file is read. When the end of the file (EOF) is reached, the data in the buffer is written to the Flash memory pointed to by the address variable. The upgrade flag is changed to "completed". The data in the additional interrupt vector table buffer is written in blocks to the interrupt vector table of the main controller and overwritten. Interrupts are enabled. The IoT terminal restarts the main controller, switches to the application software program, and reports the new firmware version number to the gateway. In the process of parsing the application software firmware, when the cache storage space is full, the cache is processed in the following way: if the address variable points to the interrupt vector table, the data in the cache is written to an additional interrupt vector table cache according to the offset; otherwise, the contents of the cache are written to the Flash pointed to by the address variable in blocks, and the address variable is modified to point to the address of the next machine code write.

2. The method for remotely upgrading an IoT terminal based on reinforcement learning as described in claim 1, characterized in that: The IoT terminal runs application software in normal operation. When it receives an upgrade request from the gateway, it switches the interrupt vector table from the application software to the upgrade boot program.

3. The method for remotely upgrading an IoT terminal based on reinforcement learning as described in claim 1, characterized in that: During Q-table initialization, values ​​are assigned to the Q-table according to the needs of the application scenario. Furthermore, the Q-value of the action of decreasing the time slot number can be increased in each time slot state of the Q-table, so as to complete the upgrade in a shorter time slot.

4. The method for remotely upgrading an IoT terminal based on reinforcement learning as described in claim 1, characterized in that: The actions include decreasing the time slot number, maintaining the time slot number, and increasing the time slot number.

5. The method for remotely upgrading an IoT terminal based on reinforcement learning as described in claim 1, characterized in that, The communication between the IoT terminal and the update server, based on the FTP protocol, to download the necessary update files includes the following steps: The IoT terminal communicates with the update server via the FTP protocol. The IoT terminal downloads the application software firmware description file and parses the file content. The application software firmware description file includes the firmware version number, firmware file size, author and modification date. The IoT terminal compares the original firmware version number with the firmware version number obtained by parsing the application software firmware description file to determine whether a new application software firmware needs to be downloaded. If so, the IoT terminal downloads the new application software firmware, verifies its integrity, and then switches to the application software program; If not, switch directly to the application software program.

6. The method for remotely upgrading an IoT terminal based on reinforcement learning as described in claim 5, characterized in that, The update of the Q-table is performed using the following formula: in, Indicates the state The following actions are adopted. Q value, Indicates the current state. Indicates the action of selection. Indicates the learning rate. Represents the reward function, Indicates the discount factor. Indicates that the action has been completed. The next state after that, Indicates the state The action to choose from, Indicates the state The following actions are adopted. The obtained Q value.

7. The method for remotely upgrading an IoT terminal based on reinforcement learning as described in claim 6, characterized in that, Before updating the application software firmware, the following steps are also included: Determine if the application software firmware has been downloaded successfully: If so, then update the application software firmware; If not, switch from the upgrade bootloader to the application software program, report the update failure to the gateway, and end the update process.

8. A remote upgrade device for IoT terminals based on reinforcement learning, characterized in that, include: The sending module is used to send an upgrade request to the IoT terminal through the gateway to notify the IoT terminal to perform a remote upgrade. The upgrade request includes the address of the update server and the latest deadline for the remote upgrade. The partitioning module is used to partition the Flash area of ​​the IoT terminal, dividing the Flash into an application software program area and an upgrade boot program area. The decision module is used to drive the IoT terminal to switch the interrupt vector table from the application software program to the upgrade boot program after receiving the upgrade request, and to make distributed autonomous upgrade decisions based on reinforcement learning algorithms. The update module is used to drive communication between IoT terminals and the update server based on the FTP protocol, so as to download the files required for the update, update the Q table, and update the application software firmware. The specific steps of making distributed autonomous upgrade decisions based on reinforcement learning algorithms include: The number of upgrades of the IoT terminal is determined. If the IoT terminal is being upgraded for the first time, the IoT terminal will randomly initialize the time slot number as the initial state and initialize the Q table. Otherwise, the time slot number used in the last upgrade will be used as the state and the Q table will be reused. The action is selected according to the ε-greedy policy. It is randomly selected with a probability of ε, and the action with the largest Q value in the current state is selected by looking up the Q table with a probability of 1-ε. The time slot number is modified according to the selected action to obtain the new state. The start time of this remote upgrade is calculated based on the latest deadline for remote upgrade in the upgrade request sent by the gateway and the time slot number. The remote upgrade will begin after the waiting time has elapsed. The specific steps for updating the application software firmware include: Save the interrupt vector table of the upgrade bootloader, set the upgrade flag, disable interrupts, and clear the application software program code area; The downloaded application software firmware is read and parsed in 512-byte blocks. The application software firmware includes address identifiers and machine code in ASCII format. The upgrade bootloader extracts the address identifier and stores it in the address variable, and converts the machine code in ASCII format into hexadecimal machine code and stores it in the buffer. The application software firmware is continuously read until the entire file is read. When the end of the file (EOF) is reached, the data in the buffer is written to the Flash memory pointed to by the address variable. The upgrade flag is changed to "completed". The data in the additional interrupt vector table buffer is written in blocks to the interrupt vector table of the main controller and overwritten. Interrupts are enabled. The IoT terminal restarts the main controller, switches to the application software program, and reports the new firmware version number to the gateway. In the process of parsing the application software firmware, when the cache storage space is full, the cache is processed in the following way: if the address variable points to the interrupt vector table, the data in the cache is written to an additional interrupt vector table cache according to the offset; otherwise, the contents of the cache are written to the Flash pointed to by the address variable in blocks, and the address variable is modified to point to the address of the next machine code write.

Images