Object trapping method and device
By processing the environmental observation information of the submarine unmanned submarine and making strategic network model decisions, the problem of low success rate of object enclosure in unknown areas of the seabed is solved, and efficient object detection and roundup is achieved.
Patent Information
- Application Number
- CN202210920715.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-02
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-08-02
AI Technical Summary
In unknown areas of the seabed, traditional methods are difficult to achieve high success rate roundup of objects because the position or trajectory of objects cannot be determined.
By obtaining the environmental observation information of the target unmanned submarine at the previous moment, performing operation trend processing, updating the environmental observation information at the current moment, and using the policy network model to make decisions based on environmental data to ensure that the number of unmanned submarines around the objects to be surrounded reaches the Complete the roundup task when preset.
It improves the success rate of seabed object enclosure, realizes efficient object detection and roundup in unknown environments, and ensures distributed decision-making and efficient completion of object enclosure tasks.
Smart Images

Figure CN115422211B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an object capturing method and device. Background Art
[0002] With the widespread application of unmanned underwater vehicles, multiple autonomous underwater vehicles (Multi-AUVs) have become popular in complex submarine tasks.
[0003] Taking the task of capturing objects on the seabed as an example, in related technologies, the method of capturing objects usually requires first determining the position of the object, or predicting in advance the movement trajectory of the object when it enters the seabed, and then capturing the object based on its position or movement trajectory.
[0004] However, it is impossible to determine the location or movement trajectory of an object in an unknown area on the seabed, which makes it difficult to achieve a high success rate in capturing the object. Summary of the invention
[0005] Based on this, it is necessary to provide a method and device for capturing objects in response to the above-mentioned technical problems, which can perform global environmental detection of unknown areas on the seabed and improve the success rate of capturing objects.
[0006] In a first aspect, the present application provides an object capture method, the method comprising:
[0007] Acquire the first environmental observation information of the target unmanned underwater vehicle at the moment before the current moment; the target unmanned underwater vehicle is any unmanned underwater vehicle in the object capture mission;
[0008] Performing operation trend processing on the first environmental observation information to obtain operation data of the target unmanned underwater vehicle at the current moment, and updating the second environmental observation information of the target unmanned underwater vehicle at the current moment with the operation data at the current moment;
[0009] If there is an object to be captured in the second environmental observation information, and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than a preset value, it is determined that the object capture mission is completed.
[0010] In one embodiment, the first environmental observation information is subjected to operation trend processing to obtain the operation data of the target unmanned underwater vehicle at the current moment, including:
[0011] The first environmental observation information is input into a preset strategy network model to obtain the operating data of the target unmanned underwater vehicle at the current moment; each training of the strategy network model during the training process is based on the quantified results of the accuracy of the sample environmental observation information of the sample unmanned underwater vehicle during the previous training.
[0012] In one embodiment, the process of constructing the policy network model includes:
[0013] Obtaining sample environment observation information of the sample unmanned underwater vehicle; the sample environment observation information includes sample environment observation information of multiple states;
[0014] Input the sample environmental observation information into the initial strategy grid model to obtain the test operation data of the sample unmanned underwater vehicle;
[0015] Determine the model training loss based on the test run data;
[0016] The quantified result of the accuracy of the sample environment observation information is obtained, and the initial policy grid model is trained according to the model training loss and the quantified result of the accuracy of the sample environment observation information, until the initial policy grid model training is completed, and the policy network model is obtained.
[0017] In one embodiment, the environment to which the sample unmanned underwater vehicle belongs includes multiple areas; the sample environment observation information includes sample state information, surrounding state information, object state information and map information; obtaining the sample environment observation information of the sample unmanned underwater vehicle includes:
[0018] Obtaining sample status information from the sample unmanned underwater vehicle;
[0019] Acquire surrounding state information of other unmanned underwater vehicles within a preset communication range of the sample unmanned underwater vehicle and state information of objects within a preset detection range of the sample unmanned underwater vehicle;
[0020] According to the preset communication range, map information of the sample unmanned underwater vehicle is obtained.
[0021] In one of the embodiments, the quantified result of the accuracy of the sample environment observation information includes quantified information of the accuracy of the sample environment observation information and reward information;
[0022] Get quantitative results of the accuracy of sample environmental observation information, including:
[0023] Obtaining the information age of the sample unmanned underwater vehicle in each area of the environment; the information age is the time interval between the sample unmanned underwater vehicle and the last detection of each area;
[0024] According to the sample environment observation information and information age of the sample unmanned underwater vehicle, the global observation information of the sample unmanned underwater vehicle is obtained;
[0025] Performing action evaluation operations on the global observation information to determine quantitative information about the accuracy of the sample environment observation information;
[0026] Based on the global observation information, determine the reward information for the accuracy of the sample environment observation information.
[0027] In one embodiment, obtaining the information age of the sample unmanned underwater vehicle in each area of the environment includes:
[0028] For any area, if the sample unmanned submersible meets the first preset condition and does not meet the second preset condition, the information age of the sample unmanned submersible in the area at the current moment is determined according to the information age of the sample unmanned submersible at the previous moment and the information age of other unmanned submersibles in the environment to which the sample unmanned submersible belongs at the current moment;
[0029] If the sample unmanned underwater vehicle meets the second preset condition, it is determined that the information age of the unmanned underwater vehicle in the area is 0;
[0030] If the sample unmanned underwater vehicle does not meet the first preset condition and does not meet the second preset condition, the information age of the sample unmanned underwater vehicle in the area at the current moment is determined according to the information age of the unmanned underwater vehicle at the previous moment and the detection time interval of the sample unmanned underwater vehicle;
[0031] Among them, the first preset condition is that there are other unmanned submersibles within the communication range of the sample unmanned submersible; the second preset condition is that the area is within the detection range of the sample unmanned submersible.
[0032] In one embodiment, the reward information includes a collision reward, a distance keeping reward, a tracking time reward, a balance task reward, and an environment exploration reward; the global observation information also includes the number of unmanned underwater vehicles that detect objects within a preset detection range of the sample unmanned underwater vehicle;
[0033] Based on the global observation information, the reward information for determining the accuracy of the sample environment observation information includes:
[0034] Obtaining a first distance value between the sample unmanned underwater vehicle and other unmanned underwater vehicles in the environment to which it belongs, and determining a collision reward value of the sample environment observation information according to the first distance value;
[0035] Obtaining a second distance value between the sample unmanned underwater vehicle and an object within the detection range, and determining a distance maintenance reward value for the sample environment observation information according to the second distance value;
[0036] Obtain the tracking time of the sample unmanned underwater vehicle for the object within the detection range, and determine the tracking time reward value of the sample environmental observation information according to the tracking time;
[0037] Determine the reward value of the balancing task of the sample environment observation information according to the number of unmanned underwater vehicles that detect objects within the preset detection range of the sample unmanned underwater vehicle;
[0038] Obtain the difference between the information age of the sample unmanned underwater vehicle at the current moment and the information age at the previous moment, and determine the environmental exploration reward value of the sample environmental observation information; the previous moment is the moment before the current moment;
[0039] The collision reward value, distance keeping reward value, tracking time reward value, balance task reward value and environment exploration reward value are weightedly calculated to determine the reward information of the accuracy of the sample environment observation information.
[0040] In one of the embodiments, the quantified result of the accuracy of the sample environment observation information includes quantified information of the accuracy of the sample environment observation information and reward information;
[0041] Based on the quantified results of the model training loss and the accuracy of the sample environment observation information, the initial policy grid model is trained, including:
[0042] Determine the parameter value of the advantage function based on the quantitative information and the reward information;
[0043] Determine the label value of the quantitative information according to the parameter value of the advantage function and the quantitative information;
[0044] Normalize the label values to obtain the standard information corresponding to the quantitative information;
[0045] The initial policy grid model is trained based on the model training loss, quantization information, and standard information corresponding to the quantization information.
[0046] In one embodiment, the test operation data includes acceleration and angular velocity; the sample state information of the sample unmanned underwater vehicle includes a first position, a first velocity and a first angle; and the method further includes:
[0047] Determining a second angle of the sample unmanned underwater vehicle based on the first angle and the angular velocity;
[0048] Determining a second speed of the sample unmanned underwater vehicle based on the first speed and the acceleration;
[0049] Determining a second position of the sample unmanned underwater vehicle based on the first position, the first speed, and the second speed;
[0050] The second angle, the second speed and the second position are the sample status information of the sample unmanned underwater vehicle at the next moment.
[0051] In a second aspect, the present application also provides an object capturing device, the device comprising:
[0052] An acquisition module, used for acquiring first environmental observation information of a target unmanned underwater vehicle at a moment before the current moment; the target unmanned underwater vehicle is any unmanned underwater vehicle in an object capture mission;
[0053] An updating module is used to perform operation trend processing on the first environmental observation information, obtain the operation data of the target unmanned underwater vehicle at the current moment, and update the second environmental observation information of the target unmanned underwater vehicle at the current moment with the operation data at the current moment;
[0054] The judgment module is used to determine that the object capture task is completed if there is an object to be captured in the second environmental observation information and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than a preset value.
[0055] In one embodiment, the update module includes:
[0056] An input unit is used to input the first environmental observation information into a preset strategy network model to obtain the operating data of the target unmanned underwater vehicle at the current moment; each training of the strategy network model during the training process is based on the quantified results of the accuracy of the sample environmental observation information of the sample unmanned underwater vehicle during the previous training.
[0057] In one embodiment, the device further comprises:
[0058] The sample acquisition module is used to obtain sample environment observation information of the sample unmanned underwater vehicle; the sample environment observation information includes sample environment observation information of multiple states;
[0059] An input module is used to input sample environmental observation information into the initial strategy grid model to obtain the test operation data of the sample unmanned underwater vehicle;
[0060] A first determination module is used to determine the model training loss according to the test operation data;
[0061] The training module is used to obtain the quantitative results of the accuracy of the sample environment observation information, and train the initial policy grid model according to the model training loss and the quantitative results of the accuracy of the sample environment observation information until the initial policy grid model training is completed to obtain the policy network model.
[0062] In one embodiment, the sample acquisition module includes:
[0063] A first acquisition unit is used to acquire sample status information of the sample unmanned underwater vehicle;
[0064] A second acquisition unit is used to acquire surrounding state information of other unmanned underwater vehicles within a preset communication range of the sample unmanned underwater vehicle and state information of objects within a preset detection range of the sample unmanned underwater vehicle;
[0065] The third acquisition unit is used to acquire the map information of the sample unmanned underwater vehicle according to the preset communication range.
[0066] In one embodiment, the training module includes:
[0067] A fourth acquisition unit is used to acquire the information age of the sample unmanned underwater vehicle in each area of the environment; the information age is the time interval between the sample unmanned underwater vehicle and the last detection of each area;
[0068] An obtaining unit, used for obtaining global observation information of the sample unmanned underwater vehicle according to the sample environment observation information and information age of the sample unmanned underwater vehicle;
[0069] A first determination unit, configured to perform an action evaluation operation on the global observation information to determine quantitative information of the accuracy of the sample environment observation information;
[0070] The second determination unit is used to determine the reward information of the accuracy of the sample environment observation information according to the global observation information.
[0071] In one embodiment, the fourth acquisition unit includes:
[0072] The first judgment subunit is used to determine the information age of the sample unmanned submersible in the area at the current moment according to the information age of the sample unmanned submersible at the previous moment and the information age of other unmanned submersibles in the environment to which the sample unmanned submersible belongs at the current moment for any area if the sample unmanned submersible meets the first preset condition and does not meet the second preset condition;
[0073] A second judgment subunit is used to determine that the information age of the unmanned underwater vehicle in the area is 0 if the sample unmanned underwater vehicle meets the second preset condition;
[0074] The third judgment subunit is used to determine the information age of the sample unmanned underwater vehicle in the area at the current moment according to the information age of the unmanned underwater vehicle at the previous moment and the detection time interval of the sample unmanned underwater vehicle if the sample unmanned underwater vehicle does not meet the first preset condition and does not meet the second preset condition;
[0075] Among them, the first preset condition is that there are other unmanned submersibles within the communication range of the sample unmanned submersible; the second preset condition is that the area is within the detection range of the sample unmanned submersible.
[0076] In one embodiment, the second determining unit includes:
[0077] A first determination subunit is used to obtain a first distance value between the sample unmanned underwater vehicle and other unmanned underwater vehicles in the environment to which it belongs, and determine a collision reward value of the sample environment observation information according to the first distance value;
[0078] A second determination subunit is used to obtain a second distance value between the sample unmanned underwater vehicle and the object within the detection range, and determine a distance maintenance reward value of the sample environment observation information according to the second distance value;
[0079] The third determination subunit is used to obtain the tracking time of the sample unmanned underwater vehicle for the object within the detection range, and determine the tracking time reward value of the sample environment observation information according to the tracking time;
[0080] A fourth determination subunit is used to determine a reward value for a balancing task of the sample environment observation information according to the number of unmanned underwater vehicles that have detected objects within a preset detection range of the sample unmanned underwater vehicle;
[0081] The fifth determination subunit is used to obtain the difference between the information age of the sample unmanned underwater vehicle at the current moment and the information age at the previous moment, and determine the environment exploration reward value of the sample environment observation information; the previous moment is the moment before the current moment;
[0082] The sixth determination subunit is used to perform weighted calculation on the collision reward value, the distance keeping reward value, the tracking time reward value, the balance task reward value and the environment exploration reward value to determine the reward information of the accuracy of the sample environment observation information.
[0083] In one embodiment, the training module includes:
[0084] a third determining unit, configured to determine a parameter value of the advantage function according to the quantization information and the reward information;
[0085] a fourth determining unit, configured to determine a label value of the quantization information according to a parameter value of the advantage function and the quantization information;
[0086] A fifth determining unit, configured to perform normalization calculation on the label value to obtain standard information corresponding to the quantitative information;
[0087] The training unit is used to train the initial policy grid model according to the model training loss, the quantization information and the standard information corresponding to the quantization information.
[0088] In one embodiment, the device further comprises:
[0089] A second determination module is used to determine a second angle of the sample unmanned underwater vehicle according to the first angle and the angular velocity;
[0090] A third determination module is used to determine a second speed of the sample unmanned underwater vehicle according to the first speed and the acceleration;
[0091] The fourth determination module is used to determine the second position of the sample unmanned underwater vehicle according to the first position, the first speed and the second speed; the second angle, the second speed and the second position are the sample state information of the sample unmanned underwater vehicle at the next moment.
[0092] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods provided in the embodiment of the first aspect are implemented.
[0093] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods provided in the embodiment of the first aspect above are implemented.
[0094] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods provided in the embodiments of the first aspect above.
[0095] The embodiment of the present application provides an object capture method and device, which obtains the first environmental observation information of the target unmanned submersible at the previous moment before the current moment, and then performs operation trend processing on the first environmental observation information to obtain the operation data of the target unmanned submersible at the current moment, and updates the second environmental observation information of the target unmanned submersible at the current moment with the operation data at the current moment. If there is an object to be captured in the second environmental observation information, and the number of unmanned submersibles within a preset range around the object to be captured is greater than a preset value, it is determined that the object capture task is completed. In this method, by performing operation trend analysis on the first environmental observation information of the target unmanned submersible at the previous moment, the new operation data of the target unmanned submersible at the current moment is obtained, and the second environmental observation information at the current moment is generated with the new operation data, and the target unmanned submersible is any unmanned submersible in the object capture task, so that each unmanned submersible can make the next movement decision according to the observation result at the previous moment, thereby realizing the distributed decision-making of its own task and ensuring a high success rate of object capture. BRIEF DESCRIPTION OF THE DRAWINGS
[0096] Figure 1 A diagram showing an application environment of an object capturing method in an embodiment;
[0097] Figure 2 A schematic diagram of a flow chart of an object capturing method in one embodiment;
[0098] Figure 3 A schematic diagram of direction coordinates of an object capturing method in one embodiment;
[0099] Figure 4 is a flow chart of an object capturing method in another embodiment;
[0100] Figure 5 is a flow chart of an object capturing method in another embodiment;
[0101] Figure 6 is a flow chart of an object capturing method in another embodiment;
[0102] Figure 7 is a flow chart of an object capturing method in another embodiment;
[0103] Figure 8 is a flow chart of an object capturing method in another embodiment;
[0104] Fig. 9 is a flow chart of an object capturing method in another embodiment;
[0105] Fig.10 is a flow chart of an object capturing method in another embodiment;
[0106] Fig.11 A schematic diagram of an application scenario of an object capture method in an embodiment;
[0107] Fig.12 A schematic diagram of the structure of an object capturing method in one embodiment;
[0108] Fig.13 is a flow chart of an object capturing method in another embodiment;
[0109] Fig.14 is a structural block diagram of an object capturing device in one embodiment;
[0110] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0111] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0112] The object capture method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the target unmanned underwater vehicle 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the target unmanned underwater vehicle 102 is any unmanned underwater vehicle when performing an object capture mission. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.
[0113] Multiple unmanned underwater vehicles are widely used in various underwater tasks such as marine resource development, marine environment monitoring and object capture. Among them, object capture is a cluster of unmanned underwater vehicles (AUVs) that obtains information about possible intruding objects based on information within its own communication range and communication between clusters, so that a sufficient number of AUVs can approach a certain range of the object and eliminate the target.
[0114] However, traditional capture algorithms mostly know the location of the intruding object, or predict the movement trajectory of the intruding object in advance, but this cannot be applied to actual unknown environments. At the same time, due to the dynamic escape of the intruding object in the environment, the fixed-position algorithm is not adaptive.
[0115] Environmental detection is also an important application of AUV clusters, but most existing methods are for searching fixed intrusive objects. There is a lack of methods to guide clusters to conduct regional periodic detection, and there is no standard for measuring the degree of environmental detection. For AUVs to perform multiple tasks, a task scheduling method should be designed to achieve a balance of mixed tasks in the balanced system and the efficient completion of each task. However, traditional AUV scheduling algorithms are mostly applied to the allocation of fixed multi-tasks, which cannot adapt to the dynamic changes and on-demand triggering of underwater task chains, and naturally there is no allocation method suitable for comprehensive scenarios of environmental search and intrusive object capture. In addition, due to the harsh underwater communication environment and limited equipment, the communication range and detection range of AUVs are limited, which also makes it impossible for AUVs to obtain global information about the environment. Therefore, the design based on centralized methods cannot solve actual underwater tasks.
[0116] Therefore, how unmanned underwater vehicles can achieve periodic detection of the environment in an open and unknown environment and capture intrusive targets with a high success rate, thereby maintaining the security of the controlled area and grasping the dynamic information of the environment, is a problem of great research value.
[0117] Based on this, the embodiments of the present application provide a method and device for capturing an object, which can perform global environmental detection of unknown areas on the seabed and improve the success rate of capturing objects.
[0118] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments.
[0119] In one embodiment, an object capture method is provided for application in Figure 1Taking the application environment in as an example, this embodiment involves performing operation trend processing on the first environmental observation information of the target unmanned underwater vehicle at the previous moment, obtaining the operation data of the target unmanned underwater vehicle at the current moment, and updating the second environmental observation information of the target unmanned underwater vehicle at the current moment with the operation data at the current moment. If there is an object to be captured in the second environmental observation information, and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than a preset value, the specific process of determining the completion of the object capture task is as follows: Figure 2 As shown, this embodiment includes the following steps:
[0120] S201, obtaining first environmental observation information of a target unmanned underwater vehicle at a moment before the current moment; the target unmanned underwater vehicle is any unmanned underwater vehicle in an object capture mission.
[0121] The object is any intrusion target into the environment of the target unmanned underwater vehicle, including people, objects, etc. If there is an intrusion object entering the environment, it is necessary to perform an object capture mission on the intrusion object.
[0122] When performing an object capture mission, multiple unmanned underwater vehicles are required to capture the object, and the environment in which the multiple unmanned underwater vehicles are located is used for detection to achieve the capture of the object. Therefore, when the unmanned underwater vehicle is used to capture the object, the operation trend of the unmanned underwater vehicle needs to be determined so that the unmanned underwater vehicle can perform a global detection of the environment through the operation trend.
[0123] First, for any unmanned submersible in the environment of the object capture mission, before determining the operating trend of the unmanned submersible, it is necessary to obtain the first environmental observation information of the unmanned submersible at the previous moment before the current moment; wherein the first environmental observation information is the information observed by the target unmanned submersible at the previous moment before the current moment, including the status information of the target unmanned submersible itself, the status information of other unmanned submersibles within the communication range of the target unmanned submersible, the status information of objects within the detection range of the target unmanned submersible, and the map information detected by the target unmanned submersible; optionally, the previous moment of the current moment is the moment before the current moment.
[0124] Optionally, the state information includes position, speed and angle, wherein the angle is the angle between the moving direction of the unmanned underwater vehicle and the positive direction of the x-axis of the preset rectangular coordinate system, such as Figure 3 shown.
[0125] Furthermore, the status information of the object within the detection range of the target unmanned underwater vehicle also includes the number of unmanned underwater vehicles that have detected the object and are observed by the target unmanned underwater vehicle within the detection range.
[0126] S202, performing operation trend processing on the first environmental observation information to obtain the operation data of the target unmanned underwater vehicle at the current moment, and updating the second environmental observation information of the target unmanned underwater vehicle at the current moment with the operation data at the current moment.
[0127] Based on the first environmental observation information of the target unmanned submersible obtained as described above, since the first environmental observation information is the environmental observation information of the target unmanned submersible at the previous moment, the first environmental observation information is processed for operation trend, so that the operation data of the target unmanned submersible at the current moment can be determined.
[0128] Specifically, the method for obtaining the operating data of the target unmanned submersible at the current moment can be to input the first environmental observation information of the target unmanned submersible at the previous moment into a preset neural network model to directly obtain the operating data of the target unmanned submersible at the current moment. Optionally, the operating data can be information such as the speed and position of the target unmanned submersible.
[0129] Then, based on the current operating data obtained above, the second environmental observation information of the target unmanned submersible is updated. The second environmental observation information is the information observed by the target unmanned submersible at the current position, including the status information of the target unmanned submersible itself, the status information of other unmanned submersibles within the communication range of the target unmanned submersible, the status information of objects within the detection range of the target unmanned submersible, and the map information of the environment detected by the target unmanned submersible.
[0130] Among them, the information in the second environment observation information is the same as the information in the first environment observation information, which will not be repeated here.
[0131] For example, if the operating data includes the position of the target unmanned submersible, after determining the operating data of the target unmanned submersible at the current moment, the status information of other unmanned submersibles within the communication range of the target unmanned submersible and the status information of objects detected within the detection range, as well as the map information of the environment detected by the target unmanned submersible will be determined according to the position of the target unmanned submersible.
[0132] Optionally, if the operating data includes the angular velocity and acceleration of the target unmanned submersible, and the status information includes the position, speed and angle of the target submersible, the speed, position and angle of the target unmanned submersible at the current moment are determined based on the speed, position, angle, acceleration and angular velocity of the target unmanned submersible at the previous moment, and the angular velocity and acceleration at the current moment; then, based on the position of the target unmanned submersible, the status information of other unmanned submersibles within the communication range of the target unmanned submersible at the current moment, the status information of objects detected within the detection range and the map information of the environment detected by the target unmanned submersible are determined.
[0133] S203: If there is an object to be captured in the second environmental observation information, and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than a preset value, it is determined that the object capture task is completed.
[0134] Because the second observation information includes the status information of the object detected by the target unmanned submersible within the detection range of its location, and the status information of the object also includes the number of unmanned submersibles that have detected the object observed by the target unmanned submersible within the detection range, for example, if object A is within the detection range of the target unmanned submersible, the target unmanned submersible can detect the number of unmanned submersibles that have detected object A.
[0135] If there is an object to be captured in the second environmental observation information, and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than a preset value, it is determined that the object has been captured successfully.
[0136] Specifically, the preset range is the detection range. If there is an unmanned submersible within the preset range around the object to be captured, it means that the unmanned submersible can detect the object. Therefore, if there is object status information in the second environment observation information of the target unmanned submersible, it means that there is an object to be captured in the second environment information, and if the number of unmanned submersibles of the object to be captured detected in the second environment observation information is greater than the preset value, it is determined that the object capture task is completed.
[0137] It should be noted that the number of unmanned underwater vehicles within a preset range around the object to be captured is the number of unmanned underwater vehicles detected around the object to be captured in the second environmental observation information.
[0138] Optionally, the object capture task includes capturing an object and successfully destroying the object.
[0139] In one embodiment, if there is no object to be captured in the second environmental observation information, or there is an object to be captured in the second environmental observation information, but the number of unmanned underwater vehicles within a preset range around the object to be captured is not greater than a preset value, the second environmental observation information continues to be processed for operation trends to obtain the third environmental observation information of the target unmanned underwater vehicle at the next moment. If there is an object to be captured in the third environmental observation information, and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than a preset value, it is determined that the object capture task is completed; otherwise, the third environmental observation information continues to be processed for operation trends until the object capture task is completed.
[0140] The above-mentioned object capture method obtains the first environmental observation information of the target unmanned submersible at the previous moment, and then processes the operation trend of the first environmental observation information to obtain the operation data of the target unmanned submersible at the current moment, and updates the second environmental observation information of the target unmanned submersible at the current moment with the operation data at the current moment. If there is an object to be captured in the second environmental observation information, and the number of unmanned submersibles within a preset range around the object to be captured is greater than a preset value, it is determined that the object capture task is completed. In this method, by performing an operation trend analysis on the first environmental observation information of the target unmanned submersible at the previous moment, the new operation data of the target unmanned submersible at the current moment is obtained, and the second environmental observation information at the current moment is generated with the new operation data, and the target unmanned submersible is any unmanned submersible in the object capture task, so that each unmanned submersible can make the next movement decision according to the observation result at the previous moment, thereby realizing the distributed decision-making of its own task and ensuring a high success rate for object capture.
[0141] In one embodiment, the operation trend of the first environmental observation information is processed to obtain the operation data of the target unmanned underwater vehicle at the current moment, including: inputting the first environmental observation information into a preset strategy network model to obtain the operation data of the target unmanned underwater vehicle at the current moment; each training of the strategy network model during the training process is carried out based on the quantified result of the accuracy of the sample environmental observation information of the sample unmanned underwater vehicle during the previous training.
[0142] Before carrying out the object capture mission, the sample environmental observation information of the sample unmanned underwater vehicle is trained to obtain a policy network model; each training of the policy network model is carried out based on the quantified results of the accuracy of the sample observation information of the sample unmanned underwater vehicle during the previous training.
[0143] Specifically, the accuracy of the sample environmental observation information of the sample unmanned underwater vehicle is quantified to obtain the quantified result of the accuracy of the sample unmanned underwater vehicle, and then the policy network model is updated and trained according to the quantified result of the accuracy of the sample unmanned underwater vehicle to obtain a policy network model with higher accuracy.
[0144] The method of determining the operating data at the current moment based on the observation information at the previous moment is to input the first environmental observation information of the target unmanned underwater vehicle at the previous moment into a preset strategy network model, and then output the operating data of the target unmanned underwater vehicle at the current moment.
[0145] In the above embodiment, the operation data of the target unmanned underwater vehicle is obtained according to the preset strategy network model. The construction process of the strategy network model is given below through an embodiment. In one embodiment, Figure 4 As shown in Figure 1, the construction process of the policy network model includes the following steps:
[0146] S401, obtaining sample environment observation information of a sample unmanned underwater vehicle; the sample environment observation information includes sample environment observation information of various states.
[0147] Before constructing the policy network model, it is necessary to obtain sample environmental observation information of the sample unmanned underwater vehicle in advance, wherein the sample environmental observation information includes sample environmental observation information of various states of the sample unmanned underwater vehicle.
[0148] Optionally, the sample environment observation information of multiple states may be one or more combinations of observation information of sample unmanned submersibles at different positions, observation information of sample unmanned submersibles at different speeds, observation information of different states and numbers of surrounding unmanned submersibles, etc.
[0149] S402, inputting sample environmental observation information into the initial strategy grid model to obtain test operation data of the sample unmanned underwater vehicle.
[0150] The sample observation information of the sample unmanned underwater vehicle obtained above is input into the initial strategy network model to obtain the test operation data of the unmanned underwater vehicle.
[0151] Optionally, the initial policy network model may be an Actor network model, which is also called a policy network. The Actor network model can be used to select a suitable policy for the unmanned underwater vehicle to determine the next operation of the sample unmanned underwater vehicle.
[0152] As shown in formula (1), formula (1) is a formula based on the initial strategy network model.
[0153] P(t)=π θ (a(t)|O(t)) (1)
[0154] Among them, O(t) represents the sample environmental observation information, a(t) represents the test operation data of the sample unmanned underwater vehicle, and P(t) can represent the probability distribution of the test operation data.
[0155] Optionally, the initial strategy model determines the test operation data a(t) to P(t) according to the probability of each strategy, and the initial strategy model outputs the test operation data and also outputs the probability corresponding to the test operation data.
[0156] S403: Determine the model training loss based on the test operation data.
[0157] The model training loss of the initial policy model is determined according to the test operation data obtained above. Specifically, the model training loss can be determined by the loss function of the initial policy network model. As shown in formula (2), formula (3) and formula (4), formula (2) is the loss function of the initial policy network model.
[0158]
[0159]
[0160]
[0161] Among them, L(θ) represents the model training loss, represents the variables in the loss function, H is the entropy of the strategy, π θ (a i (t)|O i (t)) represents the probability value corresponding to the test run data, represents the probability value corresponding to the running data at the previous moment, κ is a hyperparameter that controls the entropy coefficient, N a is the number of strategies in the initial strategy model, and ε is a constant.
[0162] S404, obtaining the quantitative results of the accuracy of the sample environment observation information, and training the initial policy grid model according to the model training loss function and the quantitative results of the accuracy of the sample environment observation information, until the initial policy grid model training is completed, and the policy network model is obtained.
[0163] Based on the above-mentioned sample environmental observation information and the test operation data of the sample unmanned underwater vehicle, the quantitative result of the accuracy of the sample environmental observation information is determined. Specifically, the sample environmental observation information and the test operation data of the sample unmanned underwater vehicle are used as inputs of a preset neural network model, and the preset neural network model is trained to output the quantitative result of the accuracy of the sample environmental observation information.
[0164] Among them, the quantitative results of the accuracy of the sample environment observation information can indicate the quality of the sample environment observation information. The better the sample environment observation information is, the better the ability of the sample unmanned submersible to detect the environment and objects is.
[0165] Based on the quantified results of the model training loss of the above-mentioned initial measurement network model and the accuracy of the sample environment observation information, the comprehensive loss of the initial measurement network model can be obtained, so as to update the parameters in the initial measurement network model, and then continue to train the initial policy network model according to the updated parameters until the training of the initial policy network model is completed to obtain the policy network model.
[0166] Optionally, the condition for completing the training of the initial policy network model may be that the number of training times of the initial policy network model reaches a preset number of iterations; or, the quantified results of the model training loss of the initial policy network model and the accuracy of the sample environment observation information both meet preset convergence conditions; optionally, the convergence condition may be a threshold condition.
[0167] The object capture method obtains sample environmental observation information of a sample unmanned underwater vehicle, the sample environmental observation information includes sample environmental observation information of multiple states, inputs the sample environmental observation information into the initial policy grid model, obtains the test operation data of the sample unmanned underwater vehicle, determines the model training loss based on the test operation data, obtains the quantitative result of the accuracy of the sample environmental observation information, and trains the initial policy grid model based on the model training loss function and the quantitative result of the accuracy of the sample environmental observation information, until the initial policy grid model training is completed, and obtains the policy network model. In this method, the initial policy network model is trained by the sample environmental observation information of the sample unmanned underwater vehicle, and the initial policy network model is further trained and updated by the quantitative result of the accuracy of the sample environmental observation information, thereby improving the accuracy of the policy network model.
[0168] In one embodiment, Figure 5 As shown, the environment to which the sample unmanned underwater vehicle belongs includes multiple areas; the sample environment observation information includes sample state information, surrounding state information, object state information and map information; obtaining the sample environment observation information of the sample unmanned underwater vehicle includes the following steps:
[0169] S501, obtaining sample status information of the sample unmanned underwater vehicle.
[0170] The sample state information of the sample unmanned underwater vehicle includes the speed state, position and angle of the sample unmanned underwater vehicle, etc. The sample state information of the sample unmanned underwater vehicle can be expressed by formula (5).
[0171] S i (t) = {x i (t),y i (t),v i,x (t),v i,y (t),θ i (t)} (5)
[0172] Among them, S i (t) is the sample status information of sample unmanned underwater vehicle i, x i (t) represents the x position component of the sample unmanned underwater vehicle i, y i (t) represents the y position component of the sample unmanned underwater vehicle i, v i,x (t) represents the x-velocity component of the sample unmanned underwater vehicle i, v i,y (t) represents the y velocity component of the sample unmanned underwater vehicle i, θ i (t) represents the angle of sample unmanned underwater vehicle i, and t represents time.
[0173] It should be noted that the x position component, y position component, x velocity component, y velocity component and angle are all in Figure 3 The direction of the rectangular coordinate system set in the embodiment is determined, and the x position component, y position component, x velocity component, y velocity component and angle in this application are all based on Figure 3 The direction of the rectangular coordinate system set in is determined by , and will not be repeated below.
[0174] S502, obtaining surrounding status information of other unmanned underwater vehicles within the preset communication range of the sample unmanned underwater vehicle and status information of objects within the preset detection range of the sample unmanned underwater vehicle.
[0175] The sample unmanned underwater vehicle communicates with other unmanned underwater vehicles within the reliable communication range of the sample unmanned underwater vehicle to obtain the state information of other unmanned underwater vehicles, and uses the state information as the surrounding state information; the surrounding state information can be expressed in the form of formula (6).
[0176] Sψi(t)={{x j (t),y j (t),v j,x (t),v j,y (t),θ j (t)}|j∈ψ i} (6)
[0177] ψ i ={j|d i,j ≤r c ,j∈Λ,j≠i} (7)
[0178] Among them, S ψi (t) represents the surrounding state information of other unmanned underwater vehicles within the preset communication range of the sample unmanned underwater vehicle i, ψ i is the reliable communication radius r of the sample unmanned underwater vehicle i c The set of other unmanned underwater vehicles in the same region, j is the reliable communication radius r of the sample unmanned underwater vehicle i c Other unmanned underwater vehicles in the cluster, Λ={1,...,i,...,N} is the cluster of AUVs, x j (t) represents the x position component of other unmanned underwater vehicles, y j (t) represents the y position component of other unmanned underwater vehicles, v j,x (t) represents the x-velocity component of other unmanned underwater vehicles, v j,y (t) represents the y velocity component of other unmanned underwater vehicles, θ j (t) represents the angle of other unmanned underwater vehicles.
[0179] The sample unmanned underwater vehicle can detect the object state information of the object within the reliable detection range. The object state information can be expressed in the form of formula (8).
[0180] Si,T (t) = {x T (t),y T (t),v T,x (t),v T,y (t),N T (t)} (8)
[0181] Among them, S i,T (t) indicates that the sample unmanned underwater vehicle i is within the reliable detection range ( <r d ) The object state information of the detected object T, x T (t) represents the x position component of object T, y T (t) represents the y position component of object T, v T,x (t) represents the x-velocity component of object T, v T,y (t) represents the y velocity component of object T, N T (t) represents the number of UUVs that have detected object T.
[0182] S503: Acquire map information of the sample unmanned underwater vehicle according to the preset communication range.
[0183] The sample unmanned underwater vehicle can be based on the preset communication range. If the sample unmanned underwater vehicle is r c The communication range of the sample unmanned underwater vehicle can be centered on the sample unmanned underwater vehicle and 0.3r c The sample unmanned submersible stores a global 2D map information. When obtaining the map information, the sample unmanned submersible is taken as the center and the radius is 0.3r c Obtained for the radius.
[0184] Based on the sample state information of the sample unmanned underwater vehicle obtained above, the surrounding state information of other unmanned underwater vehicles within the preset communication range of the sample unmanned underwater vehicle, the state information and map information of objects within the preset detection range of the sample unmanned underwater vehicle, the sample environment observation information of the sample unmanned underwater vehicle can be obtained. The form of the sample environment observation information can be expressed by formula (9).
[0185] O i (t) = {S i (t),S ψi (t),S i,T (t),G i (t)} (9)
[0186] Among them, O i (t) represents the sample environment observation information of the sample unmanned underwater vehicle at time t, G i (t) represents the global 2D map information stored by the sample unmanned underwater vehicle.
[0187] The object capture method described above obtains sample state information of the sample unmanned submersible; obtains surrounding state information of other unmanned submersibles within the preset communication range of the sample unmanned submersible and state information of objects within the preset detection range of the sample unmanned submersible; and obtains map information of the sample unmanned submersible according to the preset detection range. In this method, sample environmental observation information of the sample unmanned submersible is obtained, and the policy grid model can be trained with the sample environmental observation information, thereby improving the accuracy of the policy network model.
[0188] In one embodiment, Figure 6 As shown, the quantified result of the accuracy of the sample environment observation information includes the quantified information of the accuracy of the sample environment observation information and the reward information; obtaining the quantified result of the accuracy of the sample environment observation information includes the following steps:
[0189] S601, obtaining the information age of the sample unmanned underwater vehicle in each area of the environment; the information age is the time interval from the last time the sample unmanned underwater vehicle detected each area.
[0190] The information age of the sample unmanned underwater vehicle in each area of the environment is obtained; the information age is the time interval from the last time the unmanned underwater vehicle detected each area.
[0191] The environment described by the sample submersible is projected into a two-dimensional grid structure, and the environment can be divided into multiple rectangular areas with the grid resolution as the side length. The set of rectangular areas is Θ = {1, ..., k, ..., N Θ}, the information age is used to record the time since each grid area was last detected by an unmanned underwater vehicle. First, we initialize the information age of each area k to the maximum value that can be recorded The information age of the area detected by the unmanned underwater vehicle is set to zero, otherwise it increases with time, up to
[0192]
[0193] Among them, A k (t) represents the information age of region k at time t, represents the maximum value of the information age, Δt represents the detection time interval of the unmanned underwater vehicle, T k Indicates the time when region k was last detected.
[0194] Since unmanned submersibles are distributed and cannot know the actual information of all ranges in their environment, each unmanned submersible maintains its own two-dimensional grid table of the same size as the actual environment, in which its own understanding of the information age of each area is marked, and then updated according to the unmanned submersible's own activities. It is a true global approximation. In order to adapt to this distributed structure, the unmanned submersible can communicate with other unmanned submersibles within the communication range when interacting with the environment, thereby updating its own age storage. Specifically, the minimum information age about area k among the surrounding unmanned submersibles is updated as its own corresponding storage result. The method for updating the storage information age of area k of the sample unmanned submersible i is taken as an example to illustrate.
[0195] In one embodiment, Figure 7 As shown, in one embodiment, obtaining the information age of the sample unmanned underwater vehicle in each area of the environment includes the following steps:
[0196] S701, for any area, if the sample unmanned submersible meets the first preset condition and does not meet the second preset condition, the information age of the sample unmanned submersible in the area at the current moment is determined based on the information age of the sample unmanned submersible at the previous moment and the information age of other unmanned submersibles in the environment to which the sample unmanned submersible belongs at the current moment.
[0197] The first preset condition is that there are other unmanned submersibles within the communication range of the sample unmanned submersible; the second preset condition is that the area is within the detection range of the sample unmanned submersible.
[0198] For any area, when there are other unmanned submersibles within the communication range of the sample unmanned submersible and the area is not within the detection range of the sample unmanned submersible, the information age of the sample unmanned submersible in the area at the current moment is determined based on the information age of the sample unmanned submersible at the previous moment and the information age of other unmanned submersibles in the environment to which the unmanned submersible belongs at the current moment.
[0199] Specifically, the information age of the sample unmanned underwater vehicle at the previous moment and the time interval between the current moment and the previous moment are calculated. The information age of the sample unmanned underwater vehicle in the area at the current moment is the sum of the information age of the sample unmanned underwater vehicle at the previous moment and the detection time interval between the current moment and the previous moment, and the minimum value of the information age of other unmanned underwater vehicles in the environment to which the sample unmanned underwater vehicle belongs at the current moment. It can be expressed by formula (11).
[0200]
[0201] Among them, A i,k (t) represents the information age of the sample unmanned underwater vehicle i in the area k at the current time t, Ai,k (t-1) represents the information age of the sample unmanned underwater vehicle i in the area k at the previous moment of the current moment t, Δt represents the time interval between the current moment and the previous moment, and A j,k (t) represents the information age of other unmanned underwater vehicles j in area k at the current moment, Indicates that area k is not within the detection range of sample unmanned underwater vehicle i, ψ i ≠Empty set means that there are other unmanned underwater vehicles within the reliable communication range of the sample unmanned underwater vehicle.
[0202] S702: If the sample unmanned underwater vehicle meets the second preset condition, it is determined that the information age of the unmanned underwater vehicle in the area is 0.
[0203] When area k is within the reliable detection range of the sample unmanned underwater vehicle, the information age of the sample unmanned underwater vehicle in area k is 0, as expressed in formula (12).
[0204] A i,k (t)=0 k∈Ξ i (12)
[0205] Among them, A i,k (t) represents the information age of the sample unmanned underwater vehicle i in the region k at the current time t, k∈Ξ i Indicates that area k is within the reliable detection range of sample unmanned underwater vehicle i.
[0206] S703, if the sample unmanned submersible does not meet the first preset condition and does not meet the second preset condition, determine the information age of the sample unmanned submersible in the area at the current moment based on the information age of the unmanned submersible at the previous moment and the detection time interval of the sample unmanned submersible.
[0207] For any area, if there are no other unmanned underwater vehicles within the communication range of the sample unmanned underwater vehicle, and the area is not within the detection range of the sample unmanned underwater vehicle, the information age of the sample unmanned underwater vehicle in the area at the current moment is determined according to the sum of the information age of the unmanned underwater vehicle at the previous moment and the detection time interval between the current moment and the previous moment of the sample unmanned underwater vehicle, as shown in formula (13).
[0208]
[0209] Among them, A i,k (t) represents the information age of the sample unmanned underwater vehicle i in the area k at the current time t, A i,k (t-1) represents the information age of the sample unmanned underwater vehicle i in the area k at the previous moment of the current moment t, Δt represents the detection time interval between the current moment and the previous moment, Indicates the maximum value of the information age, It means that area k is not within the detection range of the sample unmanned underwater vehicle i, and ψ = empty set means that there are no other unmanned underwater vehicles within the reliable communication range of the sample unmanned underwater vehicle.
[0210] S602, obtaining global observation information of the sample unmanned underwater vehicle based on the sample environment observation information and information age of the sample unmanned underwater vehicle.
[0211] The sample environment observation information and information age of the sample unmanned submersible are determined as the global observation information of the sample unmanned submersible, that is, the global observation information of the sample unmanned submersible includes the sample environment observation information and information age of the sample unmanned submersible.
[0212] S603, performing an action evaluation operation on the global observation information to determine quantitative information of the accuracy of the sample environment observation information.
[0213] Based on the global observation information obtained above, an action evaluation operation is performed on the global observation information. In one embodiment, an action evaluation operation is performed on the global observation information to determine the quantitative information of the accuracy of the sample environment observation information, including: inputting the global observation information into an initial evaluation network model to obtain the quantitative information of the accuracy of the sample environment observation information.
[0214] The quantitative information of the accuracy of the sample environment observation information represents the value function corresponding to the sample environment observation information.
[0215] Specifically, the global observation information is input into the initial evaluation network model, and the quantitative information of the accuracy of the sample environment observation information is directly output, as shown in formula (14).
[0216] V(t)=V φ (z(t)) (14)
[0217] Among them, V(t) represents the quantitative information of the accuracy of the sample environment observation information, and z(t) represents the global observation information.
[0218] Optionally, the initial evaluation network model may be a critic grid.
[0219] S604: Determine reward information for the accuracy of the sample environment observation information based on the global observation information.
[0220] Based on the global observation information, the global observation information is rewarded, including quantitatively rewarding the global observation information if the global observation information meets the preset reward conditions, and obtaining reward information for the accuracy of the sample environment observation information, thereby realizing the mapping from state to reward.
[0221] In one embodiment, Figure 8As shown, the reward information includes collision reward, distance keeping reward, tracking time reward, balance task reward and environment exploration reward; the global observation information also includes the number of unmanned underwater vehicles that detect objects within the preset detection range of the sample unmanned underwater vehicle; based on the global observation information, the reward information for the accuracy of the sample environment observation information is determined, including the following steps:
[0222] S801, obtaining a first distance value between the sample unmanned underwater vehicle and other unmanned underwater vehicles in its environment, and determining a collision reward value of the sample environment observation information according to the first distance value.
[0223] The collision reward means ensuring that no collision occurs within the unmanned underwater vehicle cluster and that it remains outside the minimum safe distance, and punishing distances that do not meet safety requirements.
[0224]
[0225] Among them, r i (1) (t) represents the collision reward value of UUV i at time t, the clip(x,a,b) function clips x to [a,b], N represents the number of UUV clusters, j represents other UUVs in the UUV cluster, d i,j represents the distance between unmanned underwater vehicle i and unmanned underwater vehicle j, r s Indicates the minimum safe distance between unmanned underwater vehicles.
[0226] Therefore, according to formula (15), the distance between the sample unmanned underwater vehicle and other unmanned underwater vehicles in its environment, that is, the first distance value, is obtained, and then the collision reward value of the sample environment observation information of the sample unmanned underwater vehicle is determined according to each first distance value substituted into formula (15).
[0227] Optionally, a method for obtaining a first distance value between the sample unmanned submersible and other unmanned submersibles in its environment may be that the global observation information of the sample unmanned submersible includes sample status information of the sample submersible and surrounding status information of other unmanned submersibles, and therefore, the first distance value is determined based on the position of the sample unmanned submersible in the sample status information and the positions of other unmanned submersibles in the surrounding status information.
[0228] It should be noted that the distance between the sample unmanned submersible and the unmanned submersible that is not within the detection range of the sample unmanned submersible is greater than the communication radius, that is, the distance between the sample unmanned submersible and the unmanned submersible that is not within the detection range of the sample unmanned submersible is definitely greater than the minimum safety distance.
[0229] S802, obtaining a second distance value between the sample unmanned underwater vehicle and an object within its detection range, and determining a distance maintenance reward value for the sample environment observation information according to the second distance value.
[0230] The distance keeping reward means getting the UUV closer to the object and keeping the object within a closer detection distance while not exceeding the minimum safety distance.
[0231]
[0232] Among them, r i (2) (t) represents the distance maintenance reward value of the unmanned underwater vehicle i at time t, Δd is the boundary buffer distance, and d i,T (t) represents the distance between the unmanned underwater vehicle and the object, r d For reliable detection distance, r s Indicates the minimum safe distance.
[0233] Therefore, the second distance value between the sample unmanned underwater vehicle and the object within the detection range is obtained, and then the second distance value is substituted into formula (16) to calculate the distance keeping reward value of the sample environment observation information.
[0234] Optionally, a method for obtaining a second distance value between the sample unmanned submersible and an object within the detection range may be that the global observation information of the sample unmanned submersible includes sample status information of the sample submersible and object status information of the detected object, and the second distance value between the sample unmanned submersible and the object within the detection range is determined based on the position of the unmanned submersible in the global observation information and the position of the object in the object status information.
[0235] S803, obtaining the tracking time of the sample unmanned underwater vehicle for the object within the detection range, and determining the tracking time reward value of the sample environmental observation information according to the tracking time.
[0236] The tracking time reward refers to the time the unmanned underwater vehicle tracks the target, that is, the time to maintain the detection distance, so that the unmanned underwater vehicle can wait until other unmanned underwater vehicles approach and finally capture the target.
[0237]
[0238] Among them, r i (3) (t) represents the tracking time reward value of the unmanned underwater vehicle i at time t, Δt represents the time interval between two adjacent detections, T d,i is the timestamp when the unmanned underwater vehicle i starts to detect the object, and N is the number of unmanned underwater vehicles.
[0239] Specifically, the historical observation information of the sample unmanned underwater vehicle is stored in the database, so the time when the unmanned underwater vehicle i starts to detect the object can be obtained, and then the tracking time of the sample unmanned underwater vehicle for the object within the detection range can be determined. Then, the tracking time is substituted into formula (17) to obtain the tracking time reward value of the sample environmental observation information.
[0240] S804, determining a reward value for the balancing task of the sample environment observation information according to the number of unmanned underwater vehicles that have detected objects within a preset detection range of the sample unmanned underwater vehicle.
[0241] The reward value of the balanced mission means controlling the number of unmanned underwater vehicles participating in the roundup to avoid too many unmanned underwater vehicles participating in the roundup, which will lead to a decrease in environmental exploration performance. In the early stage, more unmanned underwater vehicles are encouraged to participate in the roundup. When the number exceeds the set number, no reward will be given.
[0242]
[0243] Among them, r i (4) (t) represents the balance task reward value of the unmanned underwater vehicle i at time t, N T (t) represents the number of objects involved in the roundup, Indicates setting the capture quantity threshold.
[0244] Optionally, the object state information of the global observation information includes the number of unmanned underwater vehicles that capture objects. Therefore, the number of unmanned underwater vehicles that capture objects within the preset detection range of the sample unmanned underwater vehicle is directly obtained from the global observation information, and the number of unmanned underwater vehicles that capture objects within the preset detection range of the sample unmanned underwater vehicle is substituted into formula (18) to obtain the balance task reward value of the sample environmental observation information.
[0245] S805, obtain the difference between the information age of the sample unmanned underwater vehicle at the current moment and the information age at the previous moment, and determine the environmental exploration reward value of the sample environmental observation information; the previous moment is the moment before the current moment.
[0246] Environmental exploration rewards represent the innovative use of information age to drive the unmanned underwater vehicle cluster to conduct periodic exploration of high-risk environments. At the same time, in order to prevent the unmanned underwater vehicle from circling in place, rewards are given based on the difference in information age in the map information stored in itself, driving the unmanned underwater vehicle to conduct efficient exploration.
[0247]
[0248] Among them, r i (5) (t) represents the environmental exploration reward value of the unmanned underwater vehicle i at time t, N Θrepresents the number of regions in the environment, k represents any region in the environment, Indicates the maximum value of information age, A k,i (t-1) represents the information age of UUV i in region k at time t-1, A k,i (t) represents the information age of UUV i in region k at time t.
[0249] Optionally, the information age of the sample unmanned underwater vehicle at the previous moment can be obtained from the historical database, and the information age of the sample unmanned underwater vehicle at the current moment can be obtained from the global observation information. The difference between the information age of the sample unmanned underwater vehicle at the current moment and the information age of the previous moment is calculated, and then substituted into formula (19) to obtain the environmental exploration reward value of the sample environmental observation information.
[0250] S806, performing weighted calculation on the collision reward value, the distance keeping reward value, the tracking time reward value, the balance task reward value and the environment exploration reward value to determine the reward information of the accuracy of the sample environment observation information.
[0251] Based on the collision reward value, distance keeping reward value, tracking time reward value, balance task reward value and environment exploration reward value obtained above, they are weighted and calculated as shown in formula (20).
[0252]
[0253] Among them, λ (l) ∈(0,1) represents the weight corresponding to each reward value, R(z i (t),a i (t)) is the reward information of the accuracy of the sample environment observation information.
[0254] The object capture method obtains the information age of the sample unmanned underwater vehicle in each area of the environment to which it belongs, and the information age is the time interval between the sample unmanned underwater vehicle and the last detection of each area. According to the sample environment observation information and information age of the sample unmanned underwater vehicle, the global observation information of the sample unmanned underwater vehicle is obtained, and the action evaluation operation is performed on the global observation information to determine the quantitative information of the accuracy of the sample environment observation information. According to the global observation information, the reward information of the accuracy of the sample environment observation information is determined. In this method, the quantitative information of the accuracy of the sample environment observation information is calculated, and the initial policy network model is further updated, thereby improving the accuracy of the obtained policy network model.
[0255] In one embodiment, Fig. 9 As shown, the quantified result of the accuracy of the sample environment observation information includes the quantified information of the accuracy of the sample environment observation information and the reward information;
[0256] According to the quantitative results of the model training loss function and the accuracy of the sample environment observation information, the initial policy grid model is trained, including:
[0257] S901, determining a parameter value of an advantage function according to the quantization information and the reward information.
[0258] Based on the test operation data obtained by the above initial strategy network model, the sample unmanned underwater vehicle changes the environmental state according to the test operation data, obtains new sample state information, and determines the new sample environmental observation information O(t+1) and global observation information z(t+1) based on the new sample state information, and adds them to the state group queue Γ+=[z(t),O(t),a(t),R(t),z(t+1),O(t+1)].
[0259] Among them, according to the new global observation information z(t+1), the quantitative information of the accuracy of the new sample environment observation information is obtained.
[0260] When the environment interaction is completed, the GAE method can be used to determine the parameter value of the advantage function based on the quantitative information and reward information, as shown in Equations (21) and (22).
[0261] ζ(t)=R(z(t),a(t))+γV φ (z(t+1))-V φ (z(t)) (21)
[0262]
[0263] in, is the parameter value of the advantage function, τ is the attenuation factor of the GAE method, γ is the discount factor, R(z(t), a(t)) is the reward information, V φ (z(t) is the quantitative information.
[0264] S902, determining a label value of the quantization information according to the parameter value of the advantage function and the quantization information.
[0265] The method for determining the label value of the quantitative information can be used as shown in formula (23).
[0266]
[0267] in, is the label value of the quantitative information, is the parameter value of the advantage function, V φ (z(t+1) is the quantitative information, and the quantitative information is the quantitative information corresponding to the test running data.
[0268] S903, performing normalization calculation on the label value to obtain standard information corresponding to the quantitative information.
[0269] The PopArt method can be used to normalize the label value. First, the first-order moment and the second-order moment of the label value are updated using the exponential moving average method.
[0270]
[0271]
[0272]
[0273]
[0274] in, represents the first-order moment of the label value, represents the second-order moment of the label value, is the standard information corresponding to the quantitative information, T is the number of iterations, N is the number of unmanned underwater vehicles, μ, δ and v are constants, is the tag value.
[0275] S904, training the initial strategy grid model according to the model training loss, the quantization information, and the standard information corresponding to the quantization information.
[0276] First, the loss corresponding to the quantization result of the accuracy of the sample environment observation information is determined based on the quantization information and the standard information corresponding to the quantization information, as shown in formula (28).
[0277]
[0278] Among them, L(φ) represents the loss corresponding to the quantified result of the accuracy of the sample environment observation information, V φ (z i (t)) represents the quantitative information of the sample environment observation information, Represents the standard information corresponding to the quantitative information of the sample environment observation information, T represents the number of iterations, and N represents the number of unmanned underwater vehicles.
[0279] Then, the total comprehensive loss is determined based on the loss corresponding to the quantified result of the model training loss and the accuracy of the sample environment observation information, as shown in formula (29).
[0280] L(θ,φ)=L(φ)+αL(θ) (29)
[0281] Among them, L(θ,φ) is the total comprehensive loss, L(φ) represents the loss corresponding to the quantified result of the accuracy of the sample environment observation information, L(θ) represents the model training loss, and α is a constant.
[0282] The object capture method described above determines the parameter value of the advantage function according to the quantitative information and the reward information; determines the label value of the quantitative information according to the parameter value of the advantage function and the quantitative information, performs normalization calculation on the label value, obtains the standard information corresponding to the quantitative information, and trains the initial policy grid model according to the model training loss function, the quantitative information, and the standard information corresponding to the quantitative information. This method trains the initial policy grid model through the model training loss function, the quantitative information, and the standard information corresponding to the quantitative information, thereby ensuring the accuracy of the policy network model.
[0283] In one embodiment, Fig.10 As shown, the test operation data includes acceleration and angular velocity; the sample state information of the sample unmanned underwater vehicle includes a first position, a first velocity and a first angle; the embodiment includes the following steps:
[0284] S1001, determining a second angle of the sample unmanned underwater vehicle based on the first angle and angular velocity.
[0285] The unmanned submersible is a uniformly accelerated translational and rotational motion. Therefore, the unmanned submersible can be controlled using acceleration and angular velocity. Therefore, the behavior space of the test operation data is:
[0286] a i (t) = {a i (t),ω i (t)} (30)
[0287] Among them, a i (t) is the test operation data of the sample unmanned underwater vehicle i at time t, a i (t) is the acceleration of the sample unmanned underwater vehicle i at time t, ω i (t) is the angular velocity of the sample unmanned underwater vehicle i at time t.
[0288] Optionally, due to the limitation of the equipment performance of the unmanned underwater vehicle, the magnitude of acceleration and the range of angular velocity are limited:
[0289] a i (t)∈[0,a max ] (31)
[0290] ω i (t)∈[ω min ,ω max ] (32)
[0291] Among them, a max is the maximum angular velocity, ω min and ω max are the minimum and maximum angular velocity respectively.
[0292] Among them, the angle θ between the sample unmanned underwater vehicle and the x-axis is i (t)∈[-π,π], so the update formula of the second angle of the sample unmanned underwater vehicle is:
[0293]
[0294] Among them, θ i (t) represents the second angle of the sample UUV, θ i (t-1) represents the first angle of the sample unmanned underwater vehicle, ω i (t-1) represents the angular velocity of the sample unmanned underwater vehicle at the last moment, ω i (t) is the angular velocity at the current moment, i.e., the angular velocity in the test run data, Δt is the time interval between the current moment and the previous moment, and K is the adjustment θ i (t)∈[-π,π].
[0295] S1002, determining a second speed of the sample unmanned underwater vehicle based on the first speed and acceleration.
[0296] The velocity of the sample unmanned underwater vehicle can be divided into an x velocity component and a y velocity component, and the relationship between the velocity and the x velocity component and the y velocity component is:
[0297]
[0298] Among them, v i (t) represents the speed of the unmanned underwater vehicle i at time t, v i,x (t) represents the x velocity component of the unmanned underwater vehicle i at time t, v i,y (t) represents the y velocity component of the unmanned underwater vehicle i at time t, v max is the maximum value of speed.
[0299] According to the acceleration of the sample unmanned underwater vehicle, the speed can be updated:
[0300]
[0301]
[0302]
[0303] in, represents the second speed of the sample unmanned underwater vehicle, v i,x (t) represents the x velocity component of the second velocity of the sample unmanned underwater vehicle, v i,y (t) represents the y velocity component of the second velocity of the sample unmanned underwater vehicle, v i,x(t-1) represents the first speed of the sample unmanned underwater vehicle, a i (t) represents the acceleration of the sample unmanned underwater vehicle, θ represents the angle of the sample unmanned underwater vehicle, v i,x (t-1) represents the x velocity component of the first velocity of the sample unmanned underwater vehicle, v i,y (t-1) represents the y velocity component of the first velocity of the sample unmanned underwater vehicle.
[0304] In one embodiment, the second speed of the sample unmanned submersible can be determined based on the first speed of the sample unmanned submersible, the acceleration and angle of the sample unmanned submersible, and then the x speed component of the second speed can be obtained based on the second speed of the sample unmanned submersible and the x speed component of the first speed, and the y speed component of the second speed can be obtained based on the second speed of the sample unmanned submersible and the y speed component of the first speed.
[0305] S1003, determining a second position of the sample unmanned underwater vehicle based on the first position, the first speed, and the second speed.
[0306] The second angle, the second speed and the second position are the sample state information of the sample unmanned underwater vehicle at the next moment. The first position includes an x position component and a y position component, and the second position includes an x position component and a y position component.
[0307] Based on the above-mentioned update of the speed, the position of the sample unmanned underwater vehicle can be updated; wherein, according to the x speed component and the y speed component of the second speed in the above-mentioned embodiment, the x position component and the y position component of the second position of the sample unmanned underwater vehicle are updated. The update can be performed by formula (38) and formula (39).
[0308]
[0309]
[0310] Among them, x i (t) represents the x velocity component of the second velocity of the sample unmanned underwater vehicle, y i (t) represents the y component of the second velocity of the unmanned underwater vehicle, x i (t-1) represents the x position component of the first position in the sample UUV, v i,x (t-1) represents the x velocity component of the first velocity of the sample unmanned underwater vehicle, v i.x (t) represents the x velocity component of the second velocity of the sample unmanned underwater vehicle, y i (t-1) The y position component of the first position in the sample UUV, v i,y (t-1) represents the y velocity component of the first velocity of the sample unmanned underwater vehicle, v i.y(t) represents the y velocity component of the second velocity of the sample unmanned underwater vehicle.
[0311] Specifically, the x-position component of the second position of the sample unmanned submersible is determined based on the x-position component of the first position, the x-velocity component of the first velocity, and the x-velocity component of the second velocity; the y-position component of the second position of the sample unmanned submersible is determined based on the y-position component of the first position, the y-velocity component of the first velocity, and the y-velocity component of the second velocity.
[0312] The object capture method described above determines the second angle of the sample unmanned submersible according to the first angle and angular velocity, determines the second speed of the sample unmanned submersible according to the first speed and acceleration, and determines the second position of the sample unmanned submersible according to the first position, the first speed and the second speed. The second angle, the second speed and the second position are the sample state information of the sample unmanned submersible at the next moment. In this method, the sample state information of the sample unmanned submersible can be updated through the obtained test operation data to update the sample environment observation information, further ensuring the accuracy of the sample update and improving the accuracy of the policy network model training.
[0313] In one embodiment, when at least N unmanned underwater vehicles detect an object at the same time, it is considered that the object has been successfully captured and eliminated, and the area within the detection range of the unmanned underwater vehicles is considered to be detected. The risk of the remaining undetected areas increases with time, and the unmanned underwater vehicle cluster is required to perform periodic visits to obtain regional information to ensure the safety of the controlled area.
[0314] Deep reinforcement learning (DRL) is a combination of deep learning and reinforcement learning. On the one hand, it can use the powerful feature representation ability of deep learning to extract the state of the environment, and on the other hand, it can use the adaptability of reinforcement learning to the environment to enable the AUV to adapt to changes in the underwater task chain. The biggest feature of reinforcement learning is learning in interaction, and the core of deep learning is deep neural network. Neural network has been proven to fit any function, thus having extremely powerful feature representation ability. Multi Agent Proximal Policy Optimization (MAPPO) is an effective method in deep reinforcement learning applied to multi-agent multi-task field.
[0315] In one embodiment, the embodiment is based on the framework of MAPPO and is improved to adapt to distributed environments. At the same time, a reward function is designed so that the AUV cluster can efficiently implement task allocation, and the Age of Information (AoI) is introduced into the reward function for the first time. In this application, the age of information is used to represent the freshness of regional information, that is, the time since the last detection of the region, so as to quantitatively describe the degree to which the region needs to be detected (risk level), thereby guiding the cluster to periodically explore the environment.
[0316] In one embodiment, before performing actual multi-unmanned underwater vehicle mission planning for target capture and area search, the AUV cluster is first simulated to cruise in a set area, and each AUV is simulated to train the action network and decision network (Actor network and Critic network) under the MAPPO framework, such as Fig.11 and Fig.12 As shown, Fig.11 is a schematic diagram of an application scenario of this embodiment, Fig.12 The structure diagram of the distributed MAPPO applied to AUV, where: Fig.12 The process of multiple AUVs interacting with the environment is included in this figure. It should be noted that the interaction process of other AUVs is the same as that of AUV1. Fig.12 It is no longer drawn specifically.
[0317] After the training starts, first of all, the parameters θ of the strategy π in the Actor network and the parameters φ of the value function V in the Critic network are adjusted. Under the total training step, the storage data set Ω = {} is set to store the collected data. At the same time, B parallel simulation experiments are set to store data collection. In each parallel experiment, Γ = [] is set to store the interaction state, and the interaction step t = 1~T is set to use the AUV to interact with the environment. For each AUV at each t, there are local observations and approximate global information. Based on the existing Actor network and Critic network, the action strategy probability distribution and the corresponding value function can be obtained to complete the update of each AUV. Based on Γ, The stored results of Ω are updated. When B parallel experiments are completed, the parameters θ and φ are updated based on Ω using the Adam method.
[0318] After the training is completed, the real-time unmanned underwater vehicle can carry out task planning for target capture and environmental exploration. During an underwater multi-task test, the AUV makes corresponding actions based on the observation status information according to its own Actor network. After that, its own status information, the status of other AUVs within the communication range, the detected target status, the environmental map information stored by itself, the information age and other data (approximate global information) are sent to its own Critic neural network to update the parameters of the two network neural networks. After multiple rounds of training, the network of each AUV can make the next movement decision in real time based on the current observation results, thereby realizing distributed decision-making on its own tasks and efficiently completing the capture of targets and the detection of the environment.
[0319] When AUVs perform multiple tasks underwater, the new data obtained can be used to dynamically update the neural network of each AUV, making the overall capabilities of the AUV cluster more in line with the needs of the scene. i (t) and Critic network C i (t), based on obtaining a new set of approximate global information, the new data is used for online updating of AUVi's stored map information and network parameters. After each AUV is updated, the updated model can further improve the AUV cluster's success rate in capturing the target, reduce the time taken for capturing, improve the efficiency of AUV cluster environment exploration, and achieve more efficient dynamic task planning.
[0320] In one embodiment, Fig.13 As shown, this embodiment includes the following steps:
[0321] S1301, randomly initialize the local observation information of the sample unmanned underwater vehicle at the current moment;
[0322] S1302, inputting the local observation information of the sample unmanned underwater vehicle at the current moment into the corresponding initial strategy network model, and outputting the action information of the sample unmanned underwater vehicle at the next moment;
[0323] Among them, the action is acceleration and angular velocity.
[0324] S1303, obtaining the local observation information of the sample unmanned underwater vehicle at the next moment according to the action information of each underwater vehicle at the next moment;
[0325] Among them, the local observation information includes the status information of the sample unmanned submersible itself, the status information of other unmanned submersibles within the reliable communication range of the sample unmanned submersible, the status information of objects within the reliable detection range of the sample unmanned submersible and map information, the status information includes position, speed and angle, and the status information of the object also includes the number of unmanned submersibles that have observed the object within the detection range of the sample unmanned submersible; optionally, the angle at the next moment is updated according to the angular velocity at the current moment and the next moment; the speed at the next moment is determined according to the x and y velocity components at the current moment and the acceleration at the next moment; the x velocity component and y velocity component of the speed at the next moment are determined according to the x velocity component, y velocity component and angle at the current moment, and the speed and acceleration at the next moment; the x position component at the next moment is determined according to the x position component at the current moment and the x velocity components at the current moment and the next moment; the y position component at the next moment is determined according to the y position component at the current moment and the y velocity components at the current moment and the next moment; the local observation information of the sample unmanned submersible is acquired according to the position.
[0326] S1304, determining the approximate global state information at the next moment based on the local observation information of the sample unmanned underwater vehicle at the next moment, where the approximate global state information includes the information age of the sample unmanned underwater vehicle in each area of the environment to which it belongs.
[0327] S1305, input the approximate global state information into the corresponding Critic evaluation network to obtain the value corresponding to the local observation information at the next moment.
[0328] S1306, determining reward information according to the local observation information of the sample unmanned underwater vehicle at the current moment, and determining the parameter value of the normalized value function according to the reward information, the value of the local observation information at the current moment, and the value corresponding to the local observation information at the next moment;
[0329] Specifically, according to the approximate global state information of the sample unmanned underwater vehicle at the current moment, the parameter value of the reward function of the sample unmanned underwater vehicle at the current moment is determined; wherein the reward function includes: collision reward, distance keeping reward, tracking time reward, balance task reward and environment exploration reward; information age is introduced into the environment exploration reward;
[0330] Determine the estimated value of the advantage function based on the parameter value of the reward function at the current moment, the value of the local observation information at the current moment, and the value corresponding to the local observation information at the next moment; determine the label value of the value function based on the estimated value and the corresponding value of the advantage function; calculate the first-order moment and second-order moment of the value function label value; calculate the parameter value of the normalized value function based on the first-order moment and second-order moment of the value function label value.
[0331] S1307, calculate the parameter value of the loss function based on the action information at the current moment and the next moment, the value of the local observation information at the current moment and the parameter value of the normalized value function.
[0332] S1308, according to the parameter value of the loss function, update the parameters in the policy network and the evaluation network until the policy network and the evaluation network meet the iteration condition, and obtain the policy network model.
[0333] S1309, substituting the local observation information of the target unmanned underwater vehicle into the corresponding strategy network model to obtain the action information of the unmanned underwater vehicle at the next moment, namely, acceleration and angular velocity;
[0334] S1310, determining the local observation information of each unmanned underwater vehicle at the next moment according to the action of each unmanned underwater vehicle at the next moment.
[0335] S1311, when at least a preset number of unmanned underwater vehicles detect an object at the same time, it is considered that the object has been captured successfully and the object is destroyed.
[0336] The specific limitations of the object capturing method provided in this embodiment can be found in the above-mentioned step limitations of each embodiment of the object capturing method, which will not be repeated here.
[0337] It should be understood that, although each step in the attached flow chart in the above-described embodiment is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the attached figure in the above-described embodiment can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.
[0338] In one embodiment, Fig.14 As shown, the embodiment of the present application further provides an object capturing device 1400, the device 1400 includes: an acquisition module 1401, an update module 1402 and a judgment module 1403, wherein:
[0339] The acquisition module 1401 is used to acquire the first environmental observation information of the target unmanned underwater vehicle at the last moment from the current moment; the target unmanned underwater vehicle is any unmanned underwater vehicle in the object capture mission;
[0340] The updating module 1402 is used to perform operation trend processing on the first environmental observation information, obtain the operation data of the target unmanned underwater vehicle at the current moment, and update the second environmental observation information of the target unmanned underwater vehicle at the current moment with the operation data at the current moment;
[0341] The judgment module 1403 is used to determine that the object capture task is completed if there is an object to be captured in the second environmental observation information and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than a preset value.
[0342] In one embodiment, the update module 1402 includes:
[0343] An input unit is used to input the first environmental observation information into a preset strategy network model to obtain the operating data of the target unmanned underwater vehicle at the current moment; each training of the strategy network model during the training process is based on the quantified results of the accuracy of the sample environmental observation information of the sample unmanned underwater vehicle during the previous training.
[0344] In one embodiment, the apparatus 1400 further includes:
[0345] The sample acquisition module is used to obtain sample environment observation information of the sample unmanned underwater vehicle; the sample environment observation information includes sample environment observation information of multiple states;
[0346] An input module is used to input sample environmental observation information into the initial strategy grid model to obtain the test operation data of the sample unmanned underwater vehicle;
[0347] A first determination module is used to determine the model training loss according to the test operation data;
[0348] The training module is used to obtain the quantitative results of the accuracy of the sample environment observation information, and train the initial policy grid model according to the model training loss and the quantitative results of the accuracy of the sample environment observation information until the initial policy grid model training is completed to obtain the policy network model.
[0349] In one embodiment, the sample acquisition module includes:
[0350] A first acquisition unit is used to acquire sample status information of the sample unmanned underwater vehicle;
[0351] A second acquisition unit is used to acquire surrounding state information of other unmanned underwater vehicles within a preset communication range of the sample unmanned underwater vehicle and state information of objects within a preset detection range of the sample unmanned underwater vehicle;
[0352] The third acquisition unit is used to acquire the map information of the sample unmanned underwater vehicle according to the preset communication range.
[0353] In one embodiment, the training module includes:
[0354] A fourth acquisition unit is used to acquire the information age of the sample unmanned underwater vehicle in each area of the environment; the information age is the time interval between the sample unmanned underwater vehicle and the last detection of each area;
[0355] An obtaining unit, used for obtaining global observation information of the sample unmanned underwater vehicle according to the sample environment observation information and information age of the sample unmanned underwater vehicle;
[0356] A first determination unit, configured to perform an action evaluation operation on the global observation information to determine quantitative information of the accuracy of the sample environment observation information;
[0357] The second determination unit is used to determine the reward information of the accuracy of the sample environment observation information according to the global observation information.
[0358] In one embodiment, the fourth acquiring unit includes:
[0359] The first judgment subunit is used to determine the information age of the sample unmanned submersible in the area at the current moment according to the information age of the sample unmanned submersible at the previous moment and the information age of other unmanned submersibles in the environment to which the sample unmanned submersible belongs at the current moment for any area if the sample unmanned submersible meets the first preset condition and does not meet the second preset condition;
[0360] A second judgment subunit is used to determine that the information age of the unmanned underwater vehicle in the area is 0 if the sample unmanned underwater vehicle meets the second preset condition;
[0361] The third judgment subunit is used to determine the information age of the sample unmanned underwater vehicle in the area at the current moment according to the information age of the unmanned underwater vehicle at the previous moment and the detection time interval of the sample unmanned underwater vehicle if the sample unmanned underwater vehicle does not meet the first preset condition and does not meet the second preset condition;
[0362] Among them, the first preset condition is that there are other unmanned submersibles within the communication range of the sample unmanned submersible; the second preset condition is that the area is within the detection range of the sample unmanned submersible.
[0363] In one embodiment, the second determining unit includes:
[0364] A first determination subunit is used to obtain a first distance value between the sample unmanned underwater vehicle and other unmanned underwater vehicles in the environment to which it belongs, and determine a collision reward value of the sample environment observation information according to the first distance value;
[0365] A second determination subunit is used to obtain a second distance value between the sample unmanned underwater vehicle and the object within the detection range, and determine a distance maintenance reward value of the sample environment observation information according to the second distance value;
[0366] The third determination subunit is used to obtain the tracking time of the sample unmanned underwater vehicle for the object within the detection range, and determine the tracking time reward value of the sample environment observation information according to the tracking time;
[0367] A fourth determination subunit is used to determine a reward value for a balancing task of the sample environment observation information according to the number of unmanned underwater vehicles that have detected objects within a preset detection range of the sample unmanned underwater vehicle;
[0368] The fifth determination subunit is used to obtain the difference between the information age of the sample unmanned underwater vehicle at the current moment and the information age at the previous moment, and determine the environment exploration reward value of the sample environment observation information; the previous moment is the moment before the current moment;
[0369] The sixth determination subunit is used to perform weighted calculation on the collision reward value, the distance keeping reward value, the tracking time reward value, the balance task reward value and the environment exploration reward value to determine the reward information of the accuracy of the sample environment observation information.
[0370] In one embodiment, the training module includes:
[0371] a third determining unit, configured to determine a parameter value of the advantage function according to the quantization information and the reward information;
[0372] a fourth determining unit, configured to determine a label value of the quantization information according to a parameter value of the advantage function and the quantization information;
[0373] A fifth determining unit, configured to perform normalization calculation on the label value to obtain standard information corresponding to the quantitative information;
[0374] The training unit is used to train the initial policy grid model according to the model training loss, the quantization information and the standard information corresponding to the quantization information.
[0375] In one embodiment, the apparatus 1400 further includes:
[0376] A second determination module is used to determine a second angle of the sample unmanned underwater vehicle according to the first angle and the angular velocity;
[0377] A third determination module is used to determine a second speed of the sample unmanned underwater vehicle according to the first speed and the acceleration;
[0378] The fourth determination module is used to determine the second position of the sample unmanned underwater vehicle according to the first position, the first speed and the second speed; the second angle, the second speed and the second position are the sample state information of the sample unmanned underwater vehicle at the next moment.
[0379] For the specific definition of the object capture device, please refer to the definition of each step in the object capture method above, which will not be repeated here. Each module in the above-mentioned object capture device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the target device in the form of hardware, or can be stored in the memory of the target device in the form of software, so that the target device can call and execute the operations corresponding to the above modules.
[0380] In one embodiment, a computer device is provided, such as Fig.15 As shown, the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an object capture method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0381] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0382] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0383] The implementation principles and technical effects of each step implemented by the processor in this embodiment are similar to those of the above-mentioned object capture method and will not be repeated here.
[0384] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0385] The implementation principles and technical effects of the steps implemented when the computer program in this embodiment is executed by the processor are similar to the principles of the above-mentioned object capture method and will not be repeated here.
[0386] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0387] The implementation principles and technical effects of the steps implemented when the computer program in this embodiment is executed by the processor are similar to the principles of the above-mentioned object capture method and will not be repeated here.
[0388] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0389] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0390] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0391] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for capturing an object, It is characterized in that The method comprises: Acquire first environmental observation information of a target unmanned underwater vehicle at a moment before the current moment; the target unmanned underwater vehicle is any unmanned underwater vehicle in an object capture mission; The first environmental observation information is input into a preset strategy network model to obtain the operation data of the target unmanned underwater vehicle at the current moment, and the second environmental observation information of the target unmanned underwater vehicle at the current moment is updated with the operation data at the current moment; each training of the strategy network model in the training process is carried out based on the quantified result of the accuracy of the sample environmental observation information of the sample unmanned underwater vehicle in the previous training; If there is an object to be captured in the second environmental observation information, and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than a preset value, it is determined that the object capture task is completed; The construction process of the policy network model includes: Obtaining sample environmental observation information of the sample unmanned underwater vehicle; the sample environmental observation information includes sample environmental observation information of multiple states; inputting the sample environmental observation information into the initial strategy network model to obtain test operation data of the sample unmanned underwater vehicle; determining the model training loss according to the test operation data; For any area, if the sample unmanned submersible meets the first preset condition and does not meet the second preset condition, the information age of the sample unmanned submersible in the area at the current moment is determined according to the information age of the sample unmanned submersible at the previous moment and the information age of other unmanned submersibles in the environment to which the sample unmanned submersible belongs at the current moment; if the sample unmanned submersible meets the second preset condition, the information age of the unmanned submersible in the area is determined to be 0; if the sample unmanned submersible does not meet the first preset condition and does not meet the second preset condition, the information age of the sample unmanned submersible at the previous moment and the detection time interval of the sample unmanned submersible are determined. The information age of the submersible in the area at the current moment; wherein the first preset condition is that there are other unmanned submersibles within the communication range of the sample unmanned submersible; the second preset condition is that the area is within the detection range of the sample unmanned submersible; the information age is the time interval between the sample unmanned submersible and the last detection of each area; the global observation information of the sample unmanned submersible is obtained based on the sample environment observation information of the sample unmanned submersible and the information age; the global observation information includes the number of unmanned submersibles that have detected objects within the preset detection range of the sample unmanned submersible; an action evaluation operation is performed on the global observation information to determine the quantitative information of the accuracy of the sample environment observation information; Obtain a first distance value between the sample unmanned underwater vehicle and other unmanned underwater vehicles in the environment to which it belongs, and determine a collision reward value for the sample environment observation information according to the first distance value; obtain a second distance value between the sample unmanned underwater vehicle and an object within its detection range, and determine a distance maintenance reward value for the sample environment observation information according to the second distance value; obtain the tracking time of the sample unmanned underwater vehicle for the object within the detection range, and determine a tracking time reward value for the sample environment observation information according to the tracking time; determine a balance task reward value for the sample environment observation information according to the number of unmanned underwater vehicles that detect objects within a preset detection range of the sample unmanned underwater vehicle; obtain a difference between the information age of the sample unmanned underwater vehicle at the current moment and the information age at the previous moment, and determine an environment exploration reward value for the sample environment observation information; the previous moment is the moment before the current moment; perform weighted calculation on the collision reward value, the distance maintenance reward value, the tracking time reward value, the balance task reward value, and the environment exploration reward value to determine the reward information for the accuracy of the sample environment observation information; According to the quantified results of the model training loss and the accuracy of the sample environment observation information, the initial policy network model is trained until the training of the initial policy network model is completed to obtain the policy network model; the quantified results of the accuracy of the sample environment observation information include the quantified information of the accuracy of the sample environment observation information and the reward information; the reward information includes the collision reward, the distance keeping reward, the tracking time reward, the balance task reward and the environment exploration reward.
2. The method according to claim 1, It is characterized in that The environment to which the sample unmanned underwater vehicle belongs includes multiple areas; the sample environment observation information includes sample state information, surrounding state information, object state information and map information; the acquisition of the sample environment observation information of the sample unmanned underwater vehicle includes: Obtaining sample status information of the sample unmanned underwater vehicle; Acquire surrounding state information of other unmanned underwater vehicles within a preset communication range of the sample unmanned underwater vehicle and state information of objects within a preset detection range of the sample unmanned underwater vehicle; According to the preset communication range, map information of the sample unmanned underwater vehicle is obtained.
3. The method according to claim 1, It is characterized in that The initial strategy network model is trained according to the quantified results of the model training loss and the accuracy of the sample environment observation information, including: Determining a parameter value of an advantage function according to the quantization information and the reward information; Determining a label value of the quantization information according to a parameter value of the advantage function and the quantization information; Performing normalization calculation on the label value to obtain standard information corresponding to the quantitative information; The initial strategy network model is trained according to the model training loss, the quantization information and the standard information corresponding to the quantization information.
4. The method according to any one of claims 1 to 3, It is characterized in that The method further comprises: If there is no object to be captured in the second environmental observation information, or if there is an object to be captured in the second environmental observation information, but the number of unmanned underwater vehicles within a preset range around the object to be captured is not greater than the preset value, then continue to perform operation trend processing on the second environmental observation information to obtain the third environmental observation information of the target unmanned underwater vehicle at the next moment; If there is an object to be captured in the third environmental observation information, and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than the preset value, it is determined that the object capture task is completed; otherwise, the third environmental observation information continues to be processed for operating trends until the object capture task is completed.
5. The method according to any one of claims 1 to 3, It is characterized in that The condition for completing the training of the initial strategy network model is that the training times of the initial strategy network model reaches a preset number of iterations.
6. The method according to any one of claims 1 to 3, It is characterized in that The condition for completing the training of the initial policy network model is that the model training loss of the initial policy network model and the quantified results of the accuracy of the sample environment observation information both meet the preset threshold conditions.
7. The method according to any one of claims 1 to 3, It is characterized in that The sample state information of the sample unmanned underwater vehicle includes the speed state, position and angle of the sample unmanned underwater vehicle.
8. The method according to any one of claims 1 to 3, It is characterized in that The performing of the action evaluation operation on the global observation information to determine the quantitative information of the accuracy of the sample environment observation information includes: The global observation information is input into the initial evaluation network model to obtain quantitative information on the accuracy of the sample environment observation information.
9. The method according to any one of claims 1 to 3, It is characterized in that The test operation data includes acceleration and angular velocity; the sample state information of the sample unmanned underwater vehicle includes a first position, a first velocity and a first angle; the method further includes: Determining a second angle of the sample unmanned underwater vehicle according to the first angle and the angular velocity; Determining a second speed of the sample unmanned underwater vehicle based on the first speed and the acceleration; Determining a second position of the sample unmanned underwater vehicle based on the first position, the first speed, and the second speed; The second angle, the second speed and the second position are sample status information of the sample unmanned underwater vehicle at the next moment.
10. An object trapping device, It is characterized in that The device comprises: An acquisition module, used for acquiring first environmental observation information of a target unmanned underwater vehicle at a moment before the current moment; the target unmanned underwater vehicle is any unmanned underwater vehicle in an object capture mission; An updating module is used to input the first environmental observation information into a preset strategy network model, obtain the operating data of the target unmanned underwater vehicle at the current moment, and update the second environmental observation information of the target unmanned underwater vehicle at the current moment with the operating data at the current moment; each training of the strategy network model in the training process is carried out based on the quantified results of the accuracy of the sample environmental observation information of the sample unmanned underwater vehicle in the previous training; A judgment module, configured to determine that the object capture task is completed if there is an object to be captured in the second environmental observation information and the number of unmanned underwater vehicles within a preset range around the object to be captured is greater than a preset value; The construction process of the policy network model includes: Obtaining sample environmental observation information of the sample unmanned underwater vehicle; the sample environmental observation information includes sample environmental observation information of multiple states; inputting the sample environmental observation information into the initial strategy network model to obtain test operation data of the sample unmanned underwater vehicle; determining the model training loss according to the test operation data; For any area, if the sample unmanned submersible meets the first preset condition and does not meet the second preset condition, the information age of the sample unmanned submersible in the area at the current moment is determined according to the information age of the sample unmanned submersible at the previous moment and the information age of other unmanned submersibles in the environment to which the sample unmanned submersible belongs at the current moment; if the sample unmanned submersible meets the second preset condition, the information age of the unmanned submersible in the area is determined to be 0; if the sample unmanned submersible does not meet the first preset condition and does not meet the second preset condition, the information age of the sample unmanned submersible at the previous moment and the detection time interval of the sample unmanned submersible are determined. The information age of the submersible in the area at the current moment; wherein the first preset condition is that there are other unmanned submersibles within the communication range of the sample unmanned submersible; the second preset condition is that the area is within the detection range of the sample unmanned submersible; the information age is the time interval between the sample unmanned submersible and the last detection of each area; the global observation information of the sample unmanned submersible is obtained based on the sample environment observation information of the sample unmanned submersible and the information age; the global observation information includes the number of unmanned submersibles that have detected objects within the preset detection range of the sample unmanned submersible; an action evaluation operation is performed on the global observation information to determine the quantitative information of the accuracy of the sample environment observation information; Obtain a first distance value between the sample unmanned underwater vehicle and other unmanned underwater vehicles in the environment to which it belongs, and determine a collision reward value for the sample environment observation information according to the first distance value; obtain a second distance value between the sample unmanned underwater vehicle and an object within its detection range, and determine a distance maintenance reward value for the sample environment observation information according to the second distance value; obtain the tracking time of the sample unmanned underwater vehicle for the object within the detection range, and determine a tracking time reward value for the sample environment observation information according to the tracking time; determine a balance task reward value for the sample environment observation information according to the number of unmanned underwater vehicles that detect objects within a preset detection range of the sample unmanned underwater vehicle; obtain a difference between the information age of the sample unmanned underwater vehicle at the current moment and the information age at the previous moment, and determine an environment exploration reward value for the sample environment observation information; the previous moment is the moment before the current moment; perform weighted calculation on the collision reward value, the distance maintenance reward value, the tracking time reward value, the balance task reward value, and the environment exploration reward value to determine the reward information for the accuracy of the sample environment observation information; According to the quantified results of the model training loss and the accuracy of the sample environment observation information, the initial policy network model is trained until the training of the initial policy network model is completed to obtain the policy network model; the quantified results of the accuracy of the sample environment observation information include the quantified information of the accuracy of the sample environment observation information and the reward information; the reward information includes the collision reward, the distance keeping reward, the tracking time reward, the balance task reward and the environment exploration reward.
Citation Information
Patent Citations
Multi-robot cooperative multi-target sampling method for open scene
CN113189983A
Underwater target tracking method and device and computer equipment
CN114384530A
