Bus load monitoring method
By collecting and analyzing the robot's operating environment information, using an intelligent model to predict future dynamics, optimizing data transmission, the problem of bus load overload is solved and the stable operation of the robot system is ensured.
Patent Information
- Application Number
- CN202411176580.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-03
Smart Images

Figure CN119065291A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent device bus data transmission, and more specifically, to a bus load monitoring method. Background Art
[0002] Robot control usually requires some form of bus to enable communication and coordination between different components. A bus is a hardware communication line that connects multiple electronic devices and allows data to be transferred between them. In robot control systems, the bus plays a very critical role, enabling the robot to efficiently coordinate the actions of various components and integrate data from various sensors and actuators;
[0003] Online learning usually requires the agent to interact frequently with the environment or other agents. This means that more data needs to be transmitted between the agent and its learning environment, which increases the communication traffic on the network or bus. In the prior art, a fixed frequency is generally used to start training and update the parameters of the agent. However, the robot will encounter different working states during operation. When the movement or working frequency is high, a peak of data transmission will occur. If the peak coincides with the agent training, the superimposed data transmission volume may exceed the bus load, causing delays or transmission interruptions. Summary of the invention
[0004] The invention provides a bus load monitoring method to solve the technical problems in related technologies.
[0005] The present invention provides a bus load monitoring method, comprising the following steps:
[0006] Step 100, collecting the environment information of the robot operation, the environment information includes bus load information;
[0007] Step 200: The currently collected environmental information S t Input it into the trained agent model, output the Q values corresponding to all first actions in the first action space, and use the first action selection strategy to select a first action from all first actions to decode and execute;
[0008] Step 300, define time t as the current time, and the first action taken at time t is A t , execute the first action A t The environmental information S collected at time t+1 t+1 , execute the first action A t Rewards received R t+1 , combined into a high-dimensional vector exp and stored in the experience replay pool, exp = {A t ,S t ,S t+1 ,R t+1},S t represents the environmental information collected at time t;
[0009] Step 400, input the environmental information from time tn to time t into the trained decision model, output the time when the training starts and the number of experiences collected during the training, and when the time when the training starts is reached, collect the corresponding number of experiences from the experience replay pool to train the intelligent model.
[0010] Furthermore, the robot is an armed robot used in combat or confrontation environments, and the environmental information includes:
[0011] combat terrain data;
[0012] Enemy information: enemy location, enemy equipment, and appearance parameters;
[0013] Robot information: the azimuth angles of each joint and the position of the robot.
[0014] Furthermore, the bus load information is the following information obtained by sampling in a unit time:
[0015] Bandwidth utilization = actual transmission rate / theoretical maximum bandwidth;
[0016] Access time rate = (arbitration time + transmission time) / access time reference value;
[0017] The access time reference value may be a mean value of the access time.
[0018] Waiting time rate = time from request to bus acquisition / reference value of time from request to bus acquisition;
[0019] The reference value of the time from request to obtaining the bus may be the average value of the time from request to obtaining the bus;
[0020] Jitter rate = coefficient of variation of access time.
[0021] Furthermore, the resource utilization calculation formula of the server is:
[0022] B natural =W1*B width +W2*B acc +W3*B wait +W4*B shake
[0023] B natural is the resource utilization of the server, B width is the bandwidth utilization, B acc is the access time rate, B wait is the waiting time rate, B shake is the jitter rate, W CPUis the weight of CPU utilization, W Memory is the weight of memory utilization, W Disk is the weight of disk I / O utilization, W Network is the weight of network bandwidth utilization.
[0024] Furthermore, before the environmental information is input into DQN, it needs to be generated as input features through feature engineering.
[0025] Furthermore, a first action in the first action space is expressed as:
[0026] {joint1,joint2,…joint n ,posit n};
[0027] Among them, joint1, joint2 and joint n They represent the azimuth angles of the joints of the 1st, 2nd, and nth robots after movement, respectively;
[0028] posit n Indicates the position of the robot after moving.
[0029] Furthermore, R t+1 The calculation formula is:
[0030]
[0031] Among them, i1 represents the number of times the robot attacks the enemy after executing the first action, i2 represents the number of times the robot is attacked after executing the first action, i3 represents the proportion of the robot's own damage after executing the first action, a, b, and c represent the first, second, and third tendency coefficients respectively. The sum of the three tendency coefficients is 1, and the default values are 0.3, 0.3, and 0.4 respectively.
[0032] Furthermore, the agent model includes a Q network and a targetQ network, and the Q network is the same as the targetQ network;
[0033] The calculation formula of the agent model is as follows:
[0034] Z=σ(W z S t +b z )
[0035] O q =σ(W q *Z+b q )
[0036] Where Z represents the latent vector, W z and W q Both represent trainable weight parameters, b z and bq Both represent trainable bias parameters, O q represents the first output vector, the value of the i-th component of the first output vector represents the Q value of the i-th first action, and σ represents the Sigmoid function.
[0037] Furthermore, the calculation formula of the decision model is as follows:
[0038] H (0) =x (1)
[0039] u (L) =σ(W u X (L) +W u H (L-1) +b u )
[0040] r (L) =σ(W r X (L) +W r H (L-1) +b r )
[0041] C (L) =tanh(W c X (L) +W c r (L) ⊙H (L-1) +b c )
[0042] H (L) =(1-u (L) )⊙C (L) +u (L) ⊙H (L-1)
[0043] OUT two =σ(W two *H (L) +b two )
[0044] Among them, x (1) and X (L) Respectively represent the Lth environmental information of the input, n≥L≥1, W u , W r , W c and W two Both represent trainable weight parameters, b u 、b r 、b c and b two All represent trainable bias parameters, ⊙ represents dot product, u (L) represents the Lth first intermediate feature, r(L) represents the Lth second intermediate feature, C (L) represents the Lth third intermediate feature, H (L-1) and H (L) Respectively represent the L-1th and Lth output features, OUT two represents the second output vector, the value of the i-th component of the second output vector represents the Q value of the i-th second action, tanh represents the tanh function, and σ represents the Sigmoid function.
[0045] A storage medium stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, one or more steps of the aforementioned bus load monitoring method can be executed.
[0046] The beneficial effects of the present invention are as follows: the present invention predicts the future dynamics of the robot by observing the environment in which the robot operates, rationally plans the future online training parameters of the intelligent body that controls the robot, reduces the maximum bus load, and ensures the normal operation of the robot. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 is a flow chart of a bus load monitoring method of the present invention;
[0048] Figure 2 It is a control block diagram of a bus load monitoring method of the present invention. DETAILED DESCRIPTION
[0049] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present specification. Various examples may omit, replace, or add various processes or components as needed. In addition, the features described in some examples may also be combined in other examples.
[0050] At least one embodiment of the present invention discloses a bus load monitoring method, such as Figure 1 As shown, the following steps are included:
[0051] Step 100, collecting the environment information of the robot operation, the environment information includes bus load information;
[0052] In one embodiment of the present invention, the robot is an armed robot used in combat or confrontation environments, and the environmental information includes:
[0053] combat terrain data;
[0054] Enemy information: enemy location, enemy equipment, and appearance parameters;
[0055] Robot information: the azimuth angles of each joint and the position of the robot;
[0056] The bus load information is the following information sampled per unit time:
[0057] Bandwidth utilization = actual transmission rate / theoretical maximum bandwidth;
[0058] Access time rate = (arbitration time + transmission time) / access time reference value;
[0059] The access time reference value may be a mean value of the access time.
[0060] Waiting time rate = time from request to bus acquisition / reference value of time from request to bus acquisition;
[0061] The reference value of the time from request to obtaining the bus may be the average value of the time from request to obtaining the bus;
[0062] Jitter rate = coefficient of variation of access time;
[0063] The specific weights can be determined based on actual demand analysis. In one embodiment of the present invention, a set of default values is provided:
[0064] W1=0.4, W2=0.2, W3=0.3, W4=0.1
[0065] The formula for calculating server resource utilization is:
[0066] B natural =W1*B width +W2*B acc +W3*B wait +W4*B shake
[0067] B natural is the resource utilization of the server, B width is the bandwidth utilization, B acc is the access time rate, B wait is the waiting time rate, B shake is the jitter rate, W CPU is the weight of CPU utilization, W Memory is the weight of memory utilization, W Disk is the weight of disk I / O utilization, W Network is the weight of network bandwidth utilization.
[0068] Step 200: The currently collected environmental information S tInput it into the trained agent model, output the Q values corresponding to all first actions in the first action space, and use the first action selection strategy to select a first action from all first actions to decode and execute;
[0069] In one embodiment of the present invention, the environment information needs to be generated as input features through feature engineering before being input into DQN;
[0070] Specifically, text information is encoded to generate text features;
[0071] Generate image features by convolution or filtering image information;
[0072] Concatenate text features and image features to obtain input features.
[0073] In one embodiment of the present invention, a first action in the first action space is expressed as:
[0074] {joint1,joint2,…joint n ,posit n};
[0075] Among them, joint1, joint2 and joint n They represent the azimuth angles of the joints of the 1st, 2nd, and nth robots after movement, respectively;
[0076] posit n Indicates the position of the robot after moving.
[0077] The movement and action of the robot can be controlled according to the first action. The azimuth angle after the movement of the robot's joints is the result after the robot's action, and the robot's action is controlled by feedback through the result.
[0078] In one embodiment of the present invention, the first action selection strategy includes:
[0079] Input to the agent model, which predicts the Q-values of all first actions;
[0080] Choose the first action according to the ∈-greedy strategy:
[0081] Set the initial value of ∈ to 0.05;
[0082] Generate a random number r∈[0,1], if r<∈, randomly select a first action, otherwise, select the first action with the largest Q value.
[0083] Step 300, define time t as the current time, and the first action taken at time t is A t , execute the first action A tThe environmental information S collected at time t+1 t+1 , execute the first action A t Rewards received R t+1 , combined into a high-dimensional vector exp and stored in the experience replay pool, exp = {A t ,S t ,S t+1 ,R t+1},S t represents the environmental information collected at time t;
[0084] R t+1 The calculation formula is:
[0085]
[0086] Where i1 represents the number of times the robot attacks the enemy after executing the first action, i2 represents the number of times the robot is attacked after executing the first action, i3 represents the proportion of the robot's own damage after executing the first action, a, b, c represent the first, second, and third tendency coefficients respectively, the sum of the three tendency coefficients is 1, and the default values are 0.3, 0.3, and 0.4 respectively;
[0087] Step 400, input the environmental information from time tn to time t into the trained decision model, output the time when the training starts and the number of experiences collected during the training, and when the time when the training starts is reached, collect the corresponding number of experiences from the experience replay pool to train the intelligent model.
[0088] As a further optimization solution of the present invention, the agent model includes a Q network and a targetQ network, and the Q network is the same as the targetQ network;
[0089] In one embodiment of the present invention, the calculation formula of the agent model is as follows:
[0090] Z=σ(W z S t +b z )
[0091] O q =σ(W q *Z+b q
[0092] Where Z represents the latent vector, W z and W q Both represent trainable weight parameters, b z and b q Both represent trainable bias parameters, O q represents the first output vector, the value of the i-th component of the first output vector represents the Q value of the i-th first action, and σ represents the Sigmoid function.
[0093] In one embodiment of the present invention, the agent model adopts a recurrent neural network.
[0094] In one embodiment of the present invention, the step of training the agent model includes:
[0095] Step 101, randomly obtain an experience from the experience pool;
[0096]
[0097] Indicates S t+1 The maximum Q value output by the targetQ network;
[0098] according to and Find the loss and update the Q network;
[0099]
[0100] Indicates state S t and the first action A t The output Q value of the lower Q network;
[0101] γ is the discount factor, which is a value between 0 and 1, used to balance the weights of immediate rewards and future rewards. The default value is 0.6;
[0102] Step 102, every fixed number of network updates, update the targetQ network so that its parameters are the same as those of the current Q network;
[0103] The default value for the fixed network update interval is 10.
[0104] Step 103, until the number of collected experiences equals the number of experiences collected in this training, the step is terminated.
[0105] In one embodiment of the present invention, the calculation formula of the decision model is as follows:
[0106] H (0) =x (1)
[0107] u (L) =σ(W u X (L) +W u H (L-1) +b u )
[0108] r (L) =σ(W r X (L) +W r H(L-1) +b r )
[0109] C (L) =tanh(W c X (L) +W c r (L) ⊙H (L-1) +b c )
[0110] H (L) =(1-u (L) )⊙C (L) +u (L) ⊙H (L-1)
[0111] OUT two =σ(W two *H (L) +b two )
[0112] Among them, x (1) and X (L) They represent the Lth environmental information input (the environmental information at time tn is the first environmental information), n≥L≥1, W u , W r , W c and W two Both represent trainable weight parameters, b u 、b r 、b c and b two All represent trainable bias parameters, ⊙ represents dot product, u (L) represents the Lth first intermediate feature, r (L) represents the Lth second intermediate feature, C (L) represents the Lth third intermediate feature, H (L-1) and H (L) Respectively represent the L-1th and Lth output features, OUT two represents the second output vector, the value of the i-th component of the second output vector represents the Q value of the i-th second action, tanh represents the tanh function, and σ represents the Sigmoid function.
[0113] In one embodiment of the present invention, the second action belongs to the second action space. The method for generating the second action in the second action space is as follows:
[0114] The range from 0% to 100% is discretized into multiple point values, which are then combined with an optional number of empirical values collected for training.
[0115] For example, a second action is represented as: {4%, 12};
[0116] Among them, 4% means that the ratio of the time difference between the time when the training starts and time t to the time difference from time t to time t+1 is 4%; 12 represents the number of experiences collected during the training.
[0117] It should be noted that 100% means that training is not started.
[0118] In one embodiment of the present invention, the decision model is trained using a Q-learning method, and the state of Q-learning is defined as a set of environmental information from time tn to time t;
[0119] The reward function for Q learning is:
[0120]
[0121] Among them, U t+1 is the reward obtained by taking the second action selected from time t to t+1, It is the resource utilization of the server sampled and calculated at time t+1.
[0122] It should be noted that the training of the decision model is performed during the intervals of robot operation, so the bus will not be occupied when the robot is running.
[0123] In one embodiment of the present invention, an application for a firefighting scenario is provided. The firefighting bus is an unconventional communication method. Currently, there are no related tools to support monitoring, which makes it difficult to obtain real-time data for monitoring the communication quality and communication process, and is even more difficult for on-site debugging personnel. By monitoring the bus data, it is not only easy to find problems, but also a major guarantee for the testing process during the development stage.
[0124] The existing technology uses software simulation to output log confirmation through J-Link, but the disadvantage is that it takes up program storage space and cannot be implemented by on-site debugging personnel.
[0125] like Figure 2 As shown, based on the above method, the present invention provides a bus load monitoring method, comprising:
[0126] The PD deception unit obtains 20V power input through the USB interface to power the power module, then charges the lithium battery through the dual-cell lithium battery charging chip, and the battery protection chip is used to protect against overcharging. Finally, the boost chip outputs 9V to the outside to power the serial port screen and bus transceiver module.
[0127] In the bus transceiver unit, the bus equivalent waveform is obtained through the decoding circuit, and the data is transmitted to the main control chip of the control unit for processing through the isolation chip. The data is then sorted to confirm whether a reply to the bus command is required, and the UART is used to interact with the serial port screen to realize display and configuration download, thereby simulating the detector, debugging and testing.
[0128] The serial port screen performs human-computer interaction, selects functions in the interface, and displays the corresponding data processing results.
[0129] The advantage is that for bus-type products, development and debugging at the installed site can be easily carried out.
[0130] In summary, the bus data can be monitored in real time and displayed through the serial port screen. The detector can also be simulated by selecting the device type and device status for development and on-site debugging. This patent intends to protect products related to dual-bus monitoring and dual-bus detector simulation.
[0131] The present invention provides a storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, one or more steps in the aforementioned bus load monitoring method can be executed.
[0132] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation mode. The above-mentioned specific implementation mode is merely illustrative and not restrictive. Under the guidance of this embodiment, ordinary technicians in this field can also make more forms of equivalent embodiments, all of which are within the protection of this embodiment.
Claims
1. A bus load monitoring method, characterized in that: The following steps are involved: Step 100, collecting the environment information of the robot operation, the environment information includes bus load information; Step 200: The currently collected environmental information S t Input it into the trained agent model, output the Q values corresponding to all first actions in the first action space, and use the first action selection strategy to select a first action from all first actions to decode and execute; Step 300, define time t as the current time, and the first action taken at time t is A t , execute the first action A t The environmental information S collected at time t+1 t+1 , execute the first action A t Rewards received R t+1 , combined into a high-dimensional vector exp and stored in the experience replay pool, exp = {A t ,S t ,S t+1 ,R t+1 },S t represents the environmental information collected at time t; Step 400, input the environmental information from time tn to time t into the trained decision model, output the time when the training starts and the number of experiences collected during the training, and when the time when the training starts is reached, collect the corresponding number of experiences from the experience replay pool to train the intelligent model.
2. A bus load monitoring method according to claim 1, characterized in that: The robot is an armed robot used in combat or confrontation environments. The environmental information includes: combat terrain data; Enemy information: enemy location, enemy equipment, and appearance parameters; Robot information: the azimuth angles of each joint and the position of the robot.
3. A bus load monitoring method according to claim 1, characterized in that: The bus load information is the following information sampled per unit time: Bandwidth utilization = actual transmission rate / theoretical maximum bandwidth; Access time rate = (arbitration time + transmission time) / access time reference value; The access time reference value may be a mean value of the access time; Waiting time rate = time from request to bus acquisition / reference value of time from request to bus acquisition; The reference value of the time from request to obtaining the bus may be the average value of the time from request to obtaining the bus; Jitter rate = coefficient of variation of access time.
4. A bus load monitoring method according to claim 1, characterized in that: The formula for calculating server resource utilization is: B natural =W1*B width +W2*B acc +W3*B wait +W4*B shake B natural is the resource utilization of the server, B width is the bandwidth utilization, B acc is the access time rate, B wait is the waiting time rate, B shake is the jitter rate, W CPU is the weight of CPU utilization, W Memory is the weight of memory utilization, W Disk is the weight of disk I / O utilization, W Network is the weight of network bandwidth utilization.
5. A bus load monitoring method according to claim 1, characterized in that: Before the environmental information is input into DQN, it needs to be generated as input features through feature engineering.
6. A bus load monitoring method according to claim 1, characterized in that: A first action in the first action space is expressed as: {joint1,joint2,…joint n ,posit n }; Among them, joint1, joint2 and joint n They represent the azimuth angles of the joints of the 1st, 2nd, and nth robots after movement, respectively; posit n Indicates the position of the robot after moving.
7. A bus load monitoring method according to claim 1, characterized in that: R t+1 The calculation formula is: Among them, i1 represents the number of times the robot attacks the enemy after executing the first action, i2 represents the number of times the robot is attacked after executing the first action, i3 represents the proportion of the robot's own damage after executing the first action, a, b, and c represent the first, second, and third tendency coefficients respectively. The sum of the three tendency coefficients is 1, and the default values are 0.3, 0.3, and 0.4 respectively.
8. A bus load monitoring method according to claim 1, characterized in that: The agent model includes a Q network and a targetQ network, where the Q network is the same as the targetQ network; The calculation formula of the agent model is as follows: Z=σ(W z S t +b z ) ABOUT q =σ(W q *Z+b q Where Z represents the latent vector, W z and W q Both represent trainable weight parameters, b z and b q Both represent trainable bias parameters, O q represents the first output vector, the value of the i-th component of the first output vector represents the Q value of the i-th first action, and σ represents the Sigmoid function.
9. A bus load monitoring method according to claim 1, characterized in that: The calculation formula of the decision model is as follows: H (0) =x (1) u (L) =σ(W u X (L) +W u H (L-1) +b u ) r (L) =σ(W r X (L) +W r H (L-1) +b r ) C (L) =tanh(W c X (L) +W c r (L) ⊙H (L-1) +b c ) H (L) =(1-u (L) )⊙C (L) +u (L) ⊙H (L-1) OUT two =σ(W two *H (L) +b two ) Among them, x (1) and X (L) Respectively represent the Lth environmental information of the input, n≥L≥1, W u , W r , W c and W two Both represent trainable weight parameters, b u 、b r 、b c and b two All represent trainable bias parameters, ⊙ represents dot product, u (L) represents the Lth first intermediate feature, r (L) represents the Lth second intermediate feature, C (L) represents the Lth third intermediate feature, H (L-1) and H (L) Respectively represent the L-1th and Lth output features, OUT two represents the second output vector, the value of the i-th component of the second output vector represents the Q value of the i-th second action, tanh represents the tanh function, and σ represents the Sigmoid function.
10. A storage medium, characterized in that: Non-transitory computer-readable instructions are stored therein, and when the non-transitory computer-readable instructions are executed by a computer, one or more steps in a bus load monitoring method as claimed in any one of claims 1 to 9 can be executed.
