Path planning method, device, equipment and storage medium for underwater unmanned vehicle

Abstract

The present application relates to the field of path planning technology, and provides a path planning method, apparatus, device, and storage medium for an underwater unmanned vehicle. The method includes: obtaining a node access sequence for the underwater unmanned vehicle; using the first acquisition node as the target acquisition node to determine the current state of the underwater unmanned vehicle; executing the current action corresponding to the current state to obtain the next state and the amount of remaining node data corresponding to the next state; calculating a dynamic compound reward value; determining whether the amount of remaining node data is 0; if the amount of remaining node data is not 0 and is within a preset communication range, using the dynamic compound reward value to update the expected return value corresponding to a preset action-value table to obtain a target action-value table; and obtaining a data acquisition path for the underwater unmanned vehicle based on the target action-value table. In this way, the throughput and efficiency of underwater unmanned vehicle data acquisition can be improved.

Description

Technical Field

[0001] The present application relates to the field of path planning technology, and in particular to a path planning method, device, equipment and storage medium for an underwater unmanned vehicle. Background Art

[0002] With the development of ocean science and advancements in artificial intelligence, autonomous underwater vehicles (AUVs) are moving towards self-learning and self-adaptation. Currently, most UUVs used for deepwater exploration are underactuated, typically consisting of only a stern thruster, with steering and pitching achieved through vector propulsion or rudders.

[0003] Path planning is a core issue in the field of underactuated underwater vehicles (UUVs). It is also a crucial prerequisite for autonomous decision-making and is essential throughout the entire underwater operation process. Commonly used path planning algorithms include artificial potential field methods, ant colony algorithms, and genetic algorithms. With the rapid development of machine learning, reinforcement learning algorithms have become a popular solution for path planning.

[0004] However, existing autonomous underwater vehicles have relatively low throughput and efficiency in the data collection process. Summary of the Invention

[0005] The embodiments of the present application provide a path planning method, apparatus, computer equipment, and storage medium for an underwater unmanned vehicle, aiming to solve the problem of relatively low throughput and efficiency of existing autonomous underwater vehicles during data collection.

[0006] In a first aspect, an embodiment of the present application provides a path planning method for an underwater unmanned vehicle, the method comprising:

[0007] Obtaining a node access sequence of the underwater unmanned vehicle, wherein the node access sequence includes N acquisition nodes;

[0008] Taking the first collection node as the target collection node, determining the current state of the underwater unmanned vehicle at the target collection node;

[0009] Executing a current action corresponding to the current state to obtain a next state of the underwater unmanned vehicle at the target collection node and a remaining amount of node data corresponding to the next state;

[0010] Updating the preset initial node remaining data volume using the node remaining data volume corresponding to the next state;

[0011] Calculating a dynamic compound reward value for the current action, wherein the dynamic compound reward value includes valid information of a data packet;

[0012] Determine whether the remaining data amount of the updated node is 0;

[0013] If the amount of remaining data of the node is not 0 and is within the preset communication range, the dynamic compound reward value is used to update the corresponding expected return value in the preset action-value table to obtain a target action-value table;

[0014] Based on the target action-value table, a data collection path of the underwater unmanned vehicle is obtained.

[0015] In some possible implementations, determining whether the amount of remaining data of the updated node is 0 further includes:

[0016] If the amount of remaining data of the updated node is 0, the next collection node of the target collection node is set as the new target collection node, and the process returns to the step of determining the current state of the underwater unmanned vehicle at the target collection node.

[0017] In some possible implementations, calculating the dynamic compound reward value of the current action includes:

[0018] Obtaining a first distance reward between the underwater unmanned vehicle and the target collection node, a direction angle reward between the underwater unmanned vehicle and the target collection node, a second distance reward between the underwater unmanned vehicle and the collection node next to the target collection node, an effective reward for the proportion of valid data in the data packets received by the underwater unmanned vehicle, and a final reward;

[0019] A dynamic composite reward value for the current action is obtained based on the first distance reward, the direction angle reward, the second distance reward, the effective reward, and the final reward.

[0020] In some possible implementations, the effective reward for the proportion of valid data in the data packets received by the underwater unmanned vehicle is calculated using the following formula:

[0021] ;

[0022] in, For effective rewards, Indicates the target collection node that has been collected by the underwater unmanned vehicle The sum of the data loads, r represents the number of data packets collected by the underwater unmanned vehicle, j is the index of the received data packet, Collect nodes for the target The total amount of data, is the weight coefficient, C is the effective data ratio of the data packet;

[0023] in, , k is the effective data length, n is the total length of the data packet.

[0024] In some possible implementations, the expected return value is updated using the following formula:

[0025] ;

[0026] in, Indicates the current state Take action of value, is the learning rate, Indicates the next state The estimated maximum reward under For the next state Actions taken, is the discount factor, Represents a dynamic compound reward value.

[0027] In some possible implementations, determining the current state of the underwater unmanned vehicle at the target collection node includes:

[0028] Obtaining the absolute position of the underwater unmanned vehicle relative to the origin of the geodetic coordinate system, the pitch angle and yaw angle of the underwater unmanned vehicle, and the signal-to-noise ratio of the underwater unmanned vehicle at the current position coordinates and the surrounding adjacent position coordinates;

[0029] Obtaining a state space of the underwater unmanned vehicle based on the absolute position, the pitch angle, the yaw angle, and the signal-to-noise ratio;

[0030] The current state of the underwater unmanned vehicle at the target collection node is selected from the state space.

[0031] In some possible implementations, the amount of remaining node data corresponding to the next state is obtained in the following manner:

[0032] Obtaining a payload of the next state acquisition data packet;

[0033] Based on the effective load and the preset initial node remaining data volume, the node remaining data volume corresponding to the next state is obtained.

[0034] In a second aspect, an embodiment of the present application further provides a path planning device for an underwater unmanned vehicle, which includes a unit for executing the above method.

[0035] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0037] The embodiments of the present application provide a path planning method, apparatus, device, and storage medium for an underwater unmanned vehicle. The embodiments of the present application introduce a dynamic compound reward value. Since the dynamic compound reward value contains valid information of a data packet and can dynamically update the corresponding expected reward value in a preset action-value table, the expected reward value in the preset action-value table can be continuously optimized as the environment changes and the task progresses, rather than being static. In this way, the underwater unmanned vehicle can collect data under conditions of good communication quality, thereby reducing the probability of data packet loss, reducing the number of retransmissions, and improving the throughput of data collection. In addition, the underwater unmanned vehicle can select an optimal data collection path from the target action-value table obtained after the dynamic update, thereby improving data collection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0041] Figure 1 A flowchart of a first embodiment of a path planning method for an underwater unmanned vehicle provided in an embodiment of the present application.

[0042] Figure 2 A schematic diagram of RS encoding provided in an embodiment of the present application.

[0043] Figure 3 A schematic diagram of the structure of a data packet provided in an embodiment of the present application.

[0044] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application.

[0045] Description of the accompanying drawings:

[0046] Processor 111 , communication interface 112 , memory 113 , and communication bus 114 . DETAILED DESCRIPTION

[0047] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0049] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0050] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0051] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0052] As used in this specification and the appended claims, the term “if” can be interpreted as “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [described condition or event] is detected” can be interpreted as meaning “upon determination” or “in response to determining” or “upon detection of [described condition or event]” or “in response to detecting [described condition or event],” depending on the context.

[0053] In order to solve the technical problem in the prior art that the throughput and efficiency of autonomous underwater vehicles in the data collection process are relatively low, the present application provides a path planning method, device, equipment and storage medium for underwater unmanned vehicles, which can enable underwater unmanned vehicles to achieve dynamic collection, thereby improving the throughput and efficiency of underwater unmanned vehicle data collection.

[0054] See Figure 1 , Figure 1 A schematic flow chart of a first embodiment of a path planning method for an underwater unmanned vehicle provided in an embodiment of the present application, the method comprising:

[0055] Step 110: Obtain the node access sequence of the underwater unmanned vehicle.

[0056] The node access sequence includes N collection nodes.

[0057] Step 120: Taking the first collection node as the target collection node, determining the current state of the underwater unmanned vehicle at the target collection node.

[0058] Step 130: Execute the current action corresponding to the current state to obtain the next state of the underwater unmanned vehicle at the target collection node and the amount of remaining node data corresponding to the next state.

[0059] Step 140: Update the preset initial node remaining data volume using the node remaining data volume corresponding to the next state.

[0060] Step 150: Calculate the dynamic compound reward value of the current action.

[0061] The dynamic compound reward value includes valid information of the data packet.

[0062] Step 160: Determine whether the remaining data amount of the updated node is 0.

[0063] Step 170: If the amount of remaining data of the node is not 0 and is within the preset communication range, the dynamic compound reward value is used to update the corresponding expected return value in the preset action-value table to obtain a target action-value table.

[0064] The preset action-value table is a two-dimensional array, and each cell in the preset action-value table is Indicates that the status Take action The expected return is usually initialized to zero or other small random values, i.e. .

[0065] Step 180: Based on the target action-value table, obtain the data collection path of the underwater unmanned vehicle.

[0066] In this way, this embodiment introduces a dynamic compound reward value. Since the dynamic compound reward value contains valid information of the data packet and can dynamically update the corresponding expected reward value in the preset action-value table, the expected reward value in the preset action-value table can be continuously optimized as the environment changes and the task progresses, rather than being static. In this way, the underwater unmanned vehicle can collect data under conditions of good communication quality, thereby reducing the probability of data packet loss, reducing the number of retransmissions, and improving the data collection throughput. In addition, the underwater unmanned vehicle can select an optimal data collection path from the target action-value table obtained after dynamic update, thereby improving data collection efficiency.

[0067] In some possible implementations, determining whether the amount of remaining data of the updated node is 0 further includes:

[0068] If the amount of remaining data of the updated node is 0, the next collection node of the target collection node is set as the new target collection node, and the process returns to the step of determining the current state of the underwater unmanned vehicle at the target collection node.

[0069] In some possible implementations, step 140, namely calculating the dynamic compound reward value of the current action, includes:

[0070] Step 141: Obtain a first distance reward between the underwater unmanned vehicle and the target collection node, a direction angle reward between the underwater unmanned vehicle and the target collection node, a second distance reward between the underwater unmanned vehicle and the collection node next to the target collection node, an effective reward for the proportion of valid data in the data packet received by the underwater unmanned vehicle, and a final reward.

[0071] Step 142: Based on the first distance reward, the direction angle reward, the second distance reward, the effective reward and the final reward, a dynamic composite reward value of the current action is obtained.

[0072] For steps 141 and 142, the dynamic compound reward value can be calculated using the following formula:

[0073] .

[0074] in, Represents the dynamic compound reward value, Represents the underwater unmanned vehicle and the target collection node (i.e., the i-th collection node) The first distance reward, Indicates the underwater unmanned vehicle and the target collection node The direction angle reward, Indicates the underwater unmanned vehicle and the next node The second distance reward, Indicates the effective reward for the proportion of valid data received by the underwater unmanned vehicle. Indicates the final reward.

[0075] In some possible implementations, It can be calculated using the following formula:

[0076] .

[0077] in, Indicates the position of the underwater unmanned vehicle and the target acquisition node at time slot t The Euclidean distance of Indicates that at time slot t+1, the position of the underwater unmanned vehicle is equal to the next node The Euclidean distance of is the weight coefficient.

[0078] if If it is positive, it means that the underwater unmanned vehicle is far away from the target collection node. Closer, first distance reward The bigger, on the contrary, The smaller the value, the higher the first distance reward. The smaller.

[0079] In some possible implementations, It can be calculated using the following formula:

[0080] .

[0081] in, express and The angle between is the weight coefficient, The smaller it is, the closer the path length chosen by the underwater unmanned vehicle is to the straight-line distance. The direction angle reward The bigger.

[0082] In some possible implementations, It can be calculated using the following formula:

[0083] .

[0084] in, Indicates the position of the underwater unmanned vehicle at time slot t and the next node The Euclidean distance of Indicates that at time slot t+1, the position of the underwater unmanned vehicle is equal to the next node The Euclidean distance of is the weight coefficient.

[0085] like If it is positive, it means that the underwater unmanned vehicle is close to the next node Closer, second distance reward The bigger.

[0086] In some possible implementations, the underwater unmanned vehicle receives an effective reward for a valid data percentage of the data packet. The following formula can be used for calculation:

[0087] .

[0088] in, For effective rewards, Indicates the target collection node that has been collected by the underwater unmanned vehicle The sum of the data loads, r represents the number of data packets collected by the underwater unmanned vehicle, j is the index of the received data packet, Collect nodes for the target The total amount of data, is the weight coefficient, C is the effective data ratio of the data packet;

[0089] in, , k is the effective data length, n is the total length of the data packet.

[0090] when , that is, when the sum of the currently collected data load is less than 2 / 3 of the total node data, the larger the effective data proportion C, the more effective rewards The bigger; when When the effective data ratio C is larger, the effective reward The smaller.

[0091] Because the underwater acoustic channel is a key factor affecting data communication performance, its unique propagation characteristics have a significant impact on data transmission. In the three-dimensional space where underwater unmanned vehicles perform data acquisition tasks, the bit error rate and throughput of the underwater unmanned vehicles receiving data packets are usually used as indicators to judge the efficiency of the acquisition task. The bit error rate and packet loss rate are mainly determined by the signal-to-noise ratio (SNR) of the received signal, which in turn depends on the underwater acoustic channel environment. For an underwater acoustic signal with a frequency of f, according to the sonar equation, the SNR of this signal at the receiving end can be expressed as:

[0092] .

[0093] in, is the emission sound source level, is the propagation loss, is the ocean ambient noise, is the frequency bandwidth of the underwater acoustic signal, is the directivity index, usually set to 0dB.

[0094] The bit error rate (BER) refers to the ratio of the number of erroneous bits received during a communication system to the total number of bits transmitted. There is a close relationship between the bit error rate and the signal-to-noise ratio (SNR). According to the theoretical formula, the relationship between the bit error rate (BER) and the signal-to-noise ratio (SNR) can be expressed as:

[0095] .

[0096] in, Represents the Gaussian error function. This formula shows that the bit error rate decreases as the signal-to-noise ratio increases. That is, the higher the signal-to-noise ratio, the lower the bit error rate, and the higher the system throughput.

[0097] Throughput refers to the amount of data passing through the network per unit time, expressed as:

[0098] .

[0099] in, is the throughput, is the total length of data collected by the underwater unmanned vehicle, The total duration of the collection task.

[0100] To address the above problems, this application adopts the Reed-Solomon (RS) channel coding method, referred to as the RS channel coding method. RS channel coding is a powerful forward error correction (FEC) coding technology that plays a key role in communication systems, especially in combating channel noise, interference and data loss, and is good at handling sudden errors and high-noise environments.

[0101] Specifically, if Figure 2 As shown, the form of RS coding is usually recorded as RS(n,k), where n represents the total number of symbols after coding (including redundancy), which is generally a fixed value; k represents the number of symbols of valid data; represents the number of redundant symbols added; t represents the maximum number of symbol errors that can be detected and corrected.

[0102] In RS coding, the codeword length is in bytes, and each symbol contains 8 bits. Therefore, the symbol error rate (SER) is introduced to measure the reliability of data transmission at the symbol level. The general formula for bit error rate (BER) and symbol error rate (SER) is:

[0103] .

[0104] The derivation formula of symbol error rate (SER) and signal-to-noise ratio (SNR) is:

[0105] .

[0106] In this application, the total length of the data packet is a fixed value, that is, n is a fixed value. To adapt to different channel environments, when the number of symbol errors during transmission does not exceed t, RS decoding can successfully recover the original data. Therefore, to ensure successful decoding, the following conditions must be met:

[0107] .

[0108] .

[0109] Therefore, the effective data length of the RS-encoded data packet can be dynamically set to:

[0110] .

[0111] Among them, k changes dynamically according to the channel quality, so dynamic encoding can be performed by controlling the value of the effective data k. ( The minimum amount of data that can be transmitted is set to 0, which means that the current communication channel conditions are too poor and do not meet the effective transmission requirements. In order to reduce the energy waste caused by invalid communication, this application can set the minimum amount of data that can be transmitted to , that is, k< When , the sensor node does not transmit data until the channel quality is restored.

[0112] Due to the limitations of underwater acoustic transmission, such as limited bandwidth, slow underwater sound speed, and large transmission attenuation, as well as the complex ocean environment, underwater acoustic channels have the characteristics of low propagation rate, large delay, and high bit error rate. Based on this, this application adopts the RS channel coding method to send data packets in the form of RS encoding, which can effectively correct symbol errors and improve the reliability of data transmission.

[0113] In some possible implementations, the data packet length of the present application can be assumed to be 256 bytes, OFDM modulation is used to enhance anti-multipath capability, and the symbol duration is ≥20ms (to adapt to the long delay of the underwater acoustic channel).

[0114] For example, the data packet structure can be as follows Figure 3 shown.

[0115] like Figure 3 As shown, the valid data k includes a preamble, a packet header, a data payload d, and a packet tail.

[0116] Among them, the value of data load d is , B is byte.

[0117] In some possible implementations, the amount of remaining node data corresponding to the next state may be calculated based on the data load d (ie, the effective load), as described below:

[0118] 1) Obtaining the payload of the next state acquisition data packet;

[0119] 2) Based on the payload and the preset initial node remaining data volume, obtain the node remaining data volume corresponding to the next state.

[0120] The remaining data volume of the node can be calculated using the following formula:

[0121] .

[0122] The next state after update is To explain, for The remaining data volume of the node after the update, for The amount of remaining data of the node before the next update, where The remaining data volume of the preset initial node , or After multiple updates, d can be the value in state Collect the payload of the data packet.

[0123] In some possible implementations, the expected return value is updated using the following formula:

[0124] .

[0125] in, Indicates the current state Take action of value, is the learning rate, Indicates the next state The estimated maximum reward under For the next state Actions taken, is the discount factor, Represents a dynamic compound reward value.

[0126] If the underwater unmanned vehicle does not receive the target acquisition node The data packet sent is is 0.

[0127] In some possible implementations, if the underwater unmanned vehicle reaches the end range, then R5=5; if the underwater unmanned vehicle does not reach the end range, then is 0.

[0128] In some possible implementations, determining the current state of the underwater unmanned vehicle at the target collection node includes:

[0129] Step 121: Obtain the absolute position of the underwater unmanned vehicle relative to the origin of the earth coordinate system, the pitch angle and yaw angle of the underwater unmanned vehicle, and the signal-to-noise ratio of the underwater unmanned vehicle at the current position coordinates and the surrounding adjacent position coordinates.

[0130] Step 122: Obtain the state space of the underwater unmanned vehicle based on the absolute position, the pitch angle, the yaw angle and the signal-to-noise ratio.

[0131] In some possible implementations, the state space S of the underwater unmanned vehicle includes the position coordinates and the rotation angle posture of the underwater unmanned vehicle relative to the earth coordinate system, which can be specifically expressed as follows:

[0132] .

[0133] in, , represents the absolute position of the underwater unmanned vehicle relative to the origin of the earth coordinate system, , It is expressed as the pitch angle of the underwater unmanned vehicle (tilt motion with the y-axis as the center), represents the yaw angle of the underwater unmanned vehicle (yaw motion around the z axis), Indicates the signal-to-noise ratio of the current position coordinates of the underwater unmanned vehicle.

[0134] The underwater unmanned vehicle receives and sends data through the underwater transceiver. Since the underwater transceiver and the underwater unmanned vehicle are fixedly connected and the distance between them is relatively short, the underwater transceiver coordinate system in this embodiment approximately coincides with the underwater unmanned vehicle body coordinate system.

[0135] Step 123: Select the current state of the underwater unmanned vehicle at the target collection node from the state space of the underwater unmanned vehicle.

[0136] In some possible implementations, selecting and executing a current action corresponding to the current state and using the action as a target action includes:

[0137] Step 131: Obtain a first action space selected by the underwater unmanned vehicle on the yaw angle and a second action space selected by the underwater unmanned vehicle on the pitch angle.

[0138] Step 132: Based on the first action space and the second action space, obtain the target action space of the underwater unmanned vehicle.

[0139] Step 133: Select a current action corresponding to the current state from the target action space.

[0140] Since the turning attitude of underwater unmanned vehicles is usually changed by vector propulsion or rudder, they can only achieve forward, left and right turns, and upward and downward pitches on the horizontal plane. Based on this, the present invention selects the target action space of the underwater unmanned vehicle as .

[0141] in, , represents the action space that the underwater unmanned vehicle can select on the yaw angle, that is, the first action space, It is a 90° left turn. It is a 45° left turn. Indicates going straight. It is a 45° right turn. It is a 90° right turn. represents the action space that the underwater unmanned vehicle can select in terms of pitch angle, i.e., the second action space, 45° upward pitch, Indicates going straight. It is 45° downward.

[0142] Based on the above embodiments, the path planning method for an underwater unmanned vehicle provided in this application mainly includes the following steps:

[0143] S10: Initialize the Q table.

[0144] The Q table is usually initialized to zero or other small random values, that is, .

[0145] S11: Initialize environmental parameters.

[0146] Among them, the environmental parameters include the collection area, the underwater unmanned vehicle posture, the number of sensor nodes N, the location Sn, the total amount of data D, the sensor node access sequence V, and the amount of remaining data at the node.

[0147] For example, let the remaining data of the preset initial node be It is D.

[0148] S12: Initialize the index i=0 of the node access sequence V.

[0149] in, , represents the target acquisition node of the AUV, and the position of the target acquisition node is .

[0150] Each collection node has the functions of monitoring, sending and receiving environmental data. The total amount of data for each collection node is different. The collection node broadcasts a data packet at the beginning of each time slot, and the size of the data packet sent each time is the same.

[0151] For example, you can use Indicates the total amount of data of the target collection node, Indicates the remaining data volume of the target collection node.

[0152] S13: Based on the current status of the AUV Select Action .

[0153] S14: AUV gets the next state , and update the remaining data of the node .

[0154] For S13 and S14, the main thing is to select and execute actions in each time slot. The underwater unmanned vehicle needs to select and execute actions according to the current state. Select an action For example, the underwater unmanned vehicle performs the selected action ,observe Dynamic compound reward value obtained and the new state , update status The remaining data volume of the node .

[0155] S15: Calculate the dynamic compound reward value of the current action .

[0156] In the Q-learning algorithm, the dynamic compound reward value Used to measure action Is this correct? Based on the influence of channel quality and its own maneuverability during the navigation of underwater unmanned vehicles, this application introduces a dynamic compound reward value .

[0157] Among them, the dynamic compound reward value The calculation of can refer to the above embodiments, and this application will not go into details here.

[0158] S16: Determine the target acquisition node The remaining data Whether it is not 0.

[0159] If yes, execute S17; if no, execute S27.

[0160] S17: Determine whether the underwater unmanned vehicle is within the communication range.

[0161] If yes, execute S18; if no, execute S20.

[0162] S18: Update the Q table, let .

[0163] S19: Determine whether i is equal to N.

[0164] If yes, execute S20; if no, return to S14.

[0165] S20: Determine whether the Q table converges.

[0166] If yes, execute S21, if no, return to S12.

[0167] S21: According to the converged Q table, simulate and obtain the position coordinates of the underwater unmanned vehicle, and generate the optimal operation path of the underwater unmanned vehicle.

[0168] In some embodiments, after data collection is completed at all acquisition nodes, a non-uniform rational B-spline (NURBS) curve can be introduced to smooth the data acquisition path output by Q-learning, so that the output data acquisition path is more consistent with the navigation maneuverability of the AUV.

[0169] Corresponding to the above-mentioned path planning method for an underwater unmanned vehicle, the present application further provides a path planning device for an underwater unmanned vehicle. The path planning device for an underwater unmanned vehicle includes a unit for executing the above-mentioned path planning method for an underwater unmanned vehicle, and the path planning device for an underwater unmanned vehicle can be configured in a terminal such as a desktop computer, a tablet computer, or a laptop computer.

[0170] like Figure 4 As shown, an embodiment of the present application provides a computer device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0171] Memory 113, for storing computer programs;

[0172] In one embodiment of the present application, the processor 111 is configured to execute a program stored in the memory 113 to implement the path planning method for an underwater unmanned vehicle provided by any of the aforementioned method embodiments, including:

[0173] Obtaining a node access sequence of the underwater unmanned vehicle, wherein the node access sequence includes N acquisition nodes;

[0174] Taking the first collection node as the target collection node, determining the current state of the underwater unmanned vehicle at the target collection node;

[0175] Executing a current action corresponding to the current state to obtain a next state of the underwater unmanned vehicle at the target collection node and a remaining amount of node data corresponding to the next state;

[0176] Updating the preset initial node remaining data volume using the node remaining data volume corresponding to the next state;

[0177] Calculating a dynamic compound reward value for the current action, wherein the dynamic compound reward value includes valid information of a data packet;

[0178] Determine whether the remaining data amount of the updated node is 0;

[0179] If the amount of remaining data of the node is not 0 and is within the preset communication range, the dynamic compound reward value is used to update the corresponding expected return value in the preset action-value table to obtain a target action-value table;

[0180] Based on the target action-value table, a data collection path of the underwater unmanned vehicle is obtained.

[0181] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0182] Therefore, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the path planning method for an underwater unmanned vehicle provided in any of the aforementioned method embodiments are implemented, including:

[0183] Obtaining a node access sequence of the underwater unmanned vehicle, wherein the node access sequence includes N acquisition nodes;

[0184] Taking the first collection node as the target collection node, determining the current state of the underwater unmanned vehicle at the target collection node;

[0185] Executing a current action corresponding to the current state to obtain a next state of the underwater unmanned vehicle at the target collection node and a remaining amount of node data corresponding to the next state;

[0186] Updating the preset initial node remaining data volume using the node remaining data volume corresponding to the next state;

[0187] Calculating a dynamic compound reward value for the current action, wherein the dynamic compound reward value includes valid information of a data packet;

[0188] Determine whether the remaining data amount of the updated node is 0;

[0189] If the amount of remaining data of the node is not 0 and is within the preset communication range, the dynamic compound reward value is used to update the corresponding expected return value in the preset action-value table to obtain a target action-value table;

[0190] Based on the target action-value table, a data collection path of the underwater unmanned vehicle is obtained.

[0191] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other physical storage media capable of storing program code. The computer-readable storage medium may be either non-volatile or volatile.

[0192] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0193] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented.

[0194] The steps in the method of the embodiment of the present application can be adjusted in order, combined, and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided, and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0195] If this integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application.

[0196] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0197] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, as long as these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0198] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A path planning method for an underwater unmanned vehicle, characterized in that: The method comprises: Obtaining a node access sequence of the underwater unmanned vehicle, wherein the node access sequence includes N acquisition nodes; Taking the first collection node as the target collection node, determining the current state of the underwater unmanned vehicle at the target collection node; Executing a current action corresponding to the current state to obtain a next state of the underwater unmanned vehicle at the target collection node and a remaining amount of node data corresponding to the next state; Updating the preset initial node remaining data volume using the node remaining data volume corresponding to the next state; Calculating a dynamic compound reward value for the current action, wherein the dynamic compound reward value includes valid information of a data packet; Determine whether the remaining data amount of the updated node is 0; If the amount of remaining data of the node is not 0 and is within the preset communication range, the dynamic compound reward value is used to update the corresponding expected return value in the preset action-value table to obtain a target action-value table; Based on the target action-value table, a data collection path of the underwater unmanned vehicle is obtained.

2. The method according to claim 1, characterized in that The determining whether the remaining data amount of the updated node is 0 further includes: If the amount of remaining data of the updated node is 0, the next collection node of the target collection node is set as the new target collection node, and the process returns to the step of determining the current state of the underwater unmanned vehicle at the target collection node.

3. The method according to claim 1, characterized in that Calculating the dynamic compound reward value of the current action, including: Obtaining a first distance reward between the underwater unmanned vehicle and the target collection node, a direction angle reward between the underwater unmanned vehicle and the target collection node, a second distance reward between the underwater unmanned vehicle and the collection node next to the target collection node, an effective reward for the proportion of valid data in the data packets received by the underwater unmanned vehicle, and a final reward; A dynamic composite reward value for the current action is obtained based on the first distance reward, the direction angle reward, the second distance reward, the effective reward, and the final reward.

4. The method according to claim 3, characterized in that The effective reward for the proportion of valid data received by the underwater unmanned vehicle is calculated using the following formula: ； in, For effective rewards, Indicates the target collection node that has been collected by the underwater unmanned vehicle The sum of the data loads, r represents the number of data packets collected by the underwater unmanned vehicle, j is the index of the received data packet, Collect nodes for the target The total amount of data, is the weight coefficient, C is the effective data ratio of the data packet; in, , k is the effective data length, n is the total length of the data packet.

5. The method according to claim 1, characterized in that The expected return value is updated using the following formula: ； in, Indicates the current state Take action of value, is the learning rate, Indicates the next state The estimated maximum reward under For the next state Actions taken, is the discount factor, Represents a dynamic compound reward value.

6. The method according to claim 1, characterized in that Determining the current state of the underwater unmanned vehicle at the target collection node includes: Obtaining the absolute position of the underwater unmanned vehicle relative to the origin of the geodetic coordinate system, the pitch angle and yaw angle of the underwater unmanned vehicle, and the signal-to-noise ratio of the underwater unmanned vehicle at the current position coordinates and the surrounding adjacent position coordinates; Obtaining a state space of the underwater unmanned vehicle based on the absolute position, the pitch angle, the yaw angle, and the signal-to-noise ratio; The current state of the underwater unmanned vehicle at the target collection node is selected from the state space.

7. The method according to claim 1, characterized in that The amount of remaining data of the node corresponding to the next state is obtained in the following way: Obtaining a payload of the next state acquisition data packet; Based on the effective load and the preset initial node remaining data volume, the node remaining data volume corresponding to the next state is obtained.

8. A path planning device for an underwater unmanned vehicle, characterized in that: The method comprises a unit for executing the method according to any one of claims 1 to 7.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the computer program can implement the method according to any one of claims 1 to 7.

Images