A task scheduling method and device for sea area perception task

Abstract

The application provides a task scheduling method and device for a sea area perception task. The task scheduling method provided by the application comprises the following steps: for a pre-built sea area perception system, a corresponding task scheduling model, a communication transmission model and a calculation processing model are constructed; a task scheduling problem of the sea area perception system is constructed according to the models; the task scheduling problem takes the calculation unloading, the edge node selection, the relay node selection, the container resource level and the container survival time level decision as variables, takes the maximization of the task completion number as an optimization target, and is subjected to the decision variable, the task time delay, the signal-to-noise ratio and the transmission resource constraint; the task scheduling problem is modeled as a partially observable Markov decision process, the preset deep reinforcement learning intelligent agent is trained by using offline simulation data, and a trained strategy model is obtained; and the strategy model is deployed on a perception device, so that the perception device calls the strategy model for distributed online inference according to a local observation state, and obtains a task scheduling scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of marine communication network and edge computing technology, and in particular to a task scheduling method and device for marine sensing tasks. BACKGROUND

[0002] The complex characteristics of marine environment bring three challenges: first, the original monitoring data quality is low, such as water turbidity, salinity change leading to blurred optical image, temperature gradient and multipath effect leading to acoustic signal propagation path distortion; second, the marine sensing nodes such as buoys and unmanned boats are limited by volume and power consumption, and can only support basic data collection, or cannot handle a large number of computing-intensive sensing tasks; third, the coverage range of near-shore base stations in the sea area is limited, and satellite communication has high delay of seconds and high cost, which is difficult to meet the needs of regular real-time task processing and decision-making.

[0003] Although the server edge computing architecture can alleviate the above problems to some extent, there are problems such as mismatch between resource rental mode and business flexibility demand, and complex resource operation and management. The serverless computing architecture can effectively solve the problems in the server computing architecture by decoupling business logic, paying for business volume, and transferring operation and maintenance transactions to the cloud platform. However, the inherent cold start delay and coupled resource configuration mechanism in serverless computing will reduce the quality of service and cause soft budget constraint problems, and at the same time, the dynamic spatio-temporal load distribution and resource heterogeneity and fragmentation in the edge network will exacerbate the above problems.

[0004] The current research methods for task scheduling in marine sensing edge computing network have significant limitations: first, existing solutions focus on a single technical link and do not consider the collaborative optimization of network-computing architecture; second, the calculation complexity of traditional convex optimization and game theory single-step solution algorithm is high and the convergence speed is slow, which is difficult to respond to the real-time requirements of marine sensing decision-making; finally, the decision space of centralized algorithm increases exponentially with the number of sensing nodes, which has the problems of difficulty in training and scalability bottleneck. SUMMARY

[0005] Therefore, the present application provides a task scheduling method and device for marine sensing tasks to improve the efficiency of task scheduling and resource management in marine sensing network, and further improve the quality of user service completion.

[0006] Specifically, the present application is realized by the following technical solutions:

[0007] The first aspect of the present application provides a task scheduling method for marine sensing tasks, the method comprising:

[0008] For the pre-built sea area perception system, a task scheduling model, a communication transmission model and a computing processing model corresponding to the sea area perception system are constructed; wherein the task scheduling model is used to describe the computing offloading decision, the edge node selection decision, the relay node selection strategy, the container resource level decision and the container survival time level decision in the task scheduling process; the communication transmission model is used to describe the link transmission characteristics of the sea area perception system; the computing processing model is used to describe the relationship between the total execution delay of the task and the computing offloading decision, the edge node selection decision, the local computing delay and the transmission delay of the task;

[0009] According to the task request model, the communication transmission model and the computing processing model, a task scheduling problem corresponding to the sea area perception system is constructed;

[0010] The task scheduling problem is modeled as a partially observable Markov decision process, and a preset deep reinforcement learning agent is trained using offline simulation data to obtain a trained strategy model; the strategy model is deployed in each perception device, so that each perception device calls the strategy model for distributed online inference according to the local observation state to obtain a task scheduling scheme.

[0011] The second aspect of the application provides a task scheduling device for sea area perception tasks, the device comprising a construction module, a modeling module, a training module and an application module; wherein,

[0012] The construction module is configured to construct a task scheduling model, a communication transmission model and a computing processing model corresponding to the sea area perception system for a pre-built sea area perception system; wherein the task scheduling model is used to describe the computing offloading decision, the edge node selection decision, the relay node selection strategy, the container resource level decision and the container survival time level decision in the task scheduling process; the communication transmission model is used to describe the link transmission characteristics of the sea area perception system; the computing processing model is used to describe the relationship between the total execution delay of the task and the computing offloading decision, the edge node selection decision, the local computing delay and the transmission delay of the task;

[0013] The modeling module is configured to construct a task scheduling problem corresponding to the sea area perception system according to the task request model, the communication transmission model and the computing processing model;

[0014] The training module is configured to model the task scheduling problem as a partially observable Markov decision process, and train a preset deep reinforcement learning agent using offline simulation data to obtain a trained strategy model;

[0015] The application module is configured to deploy the policy model in each sensing device, so that each sensing device invokes the policy model for distributed online inference according to a local observation state, and obtains a task scheduling scheme.

[0016] The task scheduling method and device for sea area perception tasks provided in the present application effectively optimize the task scheduling and resource management process by constructing a task scheduling model, a communication transmission model, and a computing processing model of the sea area perception system. The method models the task scheduling problem as a partially observable Markov decision process and uses deep reinforcement learning to train the agent offline, enabling the system to make adaptive task scheduling decisions in dynamic and complex sea area environments. By maximizing the number of completed tasks and optimizing under multiple constraints, the method can significantly improve task scheduling efficiency, reduce computational delay, optimize resource utilization, and thus improve the overall performance of the sea area perception network. Furthermore, by deploying the trained policy model in each sensing device, the agent can perform distributed online inference based on local observations, making the task scheduling decisions more flexible and real-time, and thus improving the quality of user service completion and the response speed of the system. This method improves the robustness, efficiency, and scalability of the system, meeting the demand for efficient and reliable task scheduling in sea area perception tasks. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of the task scheduling method for sea area perception tasks according to Embodiment One of the present application is shown.

[0018] Figure 2 A schematic diagram of a sea area perception system according to an exemplary embodiment of the present application is shown.

[0019] Figure 3 A flowchart of the task scheduling method for sea area perception tasks according to Embodiment Two of the present application is shown.

[0020] Figure 4 A flowchart of the task scheduling method for sea area perception tasks according to Embodiment Three of the present application is shown.

[0021] Figure 5 A structural schematic diagram of the task scheduling device for sea area perception tasks according to Embodiment One of the present application is shown. DETAILED DESCRIPTION

[0022] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application.

[0023] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or", as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0024] It is to be understood that the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It is to be further understood that the terms "approximately", "substantially", and "about" are used herein to represent an acceptable quality, value, quantity, fact, or like attribute of a result that is close enough to the actual one within a range defined by technical or commercial standards or that would be within a range permissible under law, regulation, or policy. Precise

[0025] The specific embodiments are given as follows to introduce the technical scheme of the application in detail.

[0026] Figure 1 The flow chart of the task scheduling method for the sea area perception task provided by the present application is shown in Figure 1. Please refer to Figure 1 The task scheduling method for the sea area perception task provided by the present application includes the following steps.

[0027] S101, for the pre-built sea area perception system, a task scheduling model, a communication transmission model and a calculation processing model corresponding to the sea area perception system are constructed.

[0028] Figure 2 The schematic diagram of the sea area perception system shown in an exemplary embodiment of the present application is shown in Figure 2. Please refer to Figure 2 The pre-built sea area perception system includes a plurality of buoys, a plurality of small unmanned boats, at least one large unmanned boat mother ship, at least one offshore operation platform, one high-altitude communication base station, at least one unmanned aerial vehicle and a plurality of shore-based macro base stations.

[0029] In specific implementation, for example, in an embodiment, one embodiment can deploy a buoy, denoted as ; a small unmanned boat, denoted as ; a large unmanned boat mother ship; an offshore operation platform (such as an offshore oil platform); a high-altitude communication base station; an unmanned aerial vehicle; a shore-based macro base station, denoted as . The buoys, small unmanned boats, large unmanned boat mother ships, offshore operation platforms, high-altitude communication base stations, unmanned aerial vehicles and shore-based macro base stations are all perception devices, and the set of perception devices is denoted as .

[0030] It should be noted that the high-altitude communication base station is deployed in the stratosphere with a vertical height of about 20 km, and its coverage range reaches hundreds of kilometers. In order to improve the coverage efficiency of the high-altitude communication base station, the coverage range of the high-altitude communication base station is divided into multiple cells by using beamforming technology, and the coverage range of each cell is 15 km, and other sensing devices are within the coverage range of a cell of the high-altitude communication base station.

[0031] Further, the sensing devices are divided into first-type sensing devices and second-type sensing devices, the first-type sensing devices only have local computing capability, and the second-type sensing devices have both local computing capability and the capability of assisting other nodes to process tasks. In the embodiment, the buoy and the unmanned aerial vehicle belong to the first-type sensing devices, and the small unmanned ship, the large unmanned ship mother ship, the offshore operation platform, the high-altitude communication base station and the shore-based macro base station belong to the second-type sensing devices.

[0032] Each first-type sensing device has a transmission waiting queue, an edge computing waiting queue, an edge cold start waiting queue, an edge computing queue and a hot state container queue. The first-type sensing device can only process the task generated by itself in a serial mode. The first-type sensing device has a local computing waiting queue and a local transmission waiting queue.

[0033] In a possible implementation, when the task scheduling model corresponding to the sea area sensing system is constructed, the following steps are included:

[0034] (1) A computing offloading decision model corresponding to the sea area sensing system is constructed; wherein the computing offloading decision model is used to decide whether the current task is offloaded to an edge node.

[0035] Specifically, the edge node is a sensing device close to a data source, for example, close to a buoy or an unmanned ship, and has certain computing capability and storage capability, and can quickly respond to computing demand. The computing offloading decision model is used to decide whether the current task on the device is transferred to other edge nodes for processing.

[0036] Specifically, after the sensing device collects sensing data, the sensing task to be processed is generated in a single time slot. For the generated sensing task, a suitable computing node needs to be selected, for example, local computing or edge computing. Local computing refers to computing and processing by a local sensing device, and edge computing refers to offloading the task to other sensing devices for processing.

[0037] It can be understood that by calculating the computing offloading decision model, it can be decided whether the current task is offloaded to the edge node, and the problem that the local perception device cannot efficiently process the task due to limited computing capability can be avoided. For simple tasks, the local node can directly process, reducing data transmission overhead; for complex tasks, offloading to the edge node can make full use of the stronger computing capability of the edge node to complete quickly, improve the overall task processing efficiency, and guarantee the task real-time performance.

[0038] In a specific implementation, the computing offloading decision model is denoted as , , indicates that the perception task of the nth perception device in the current time slot is offloaded to the edge node, otherwise ; when , a suitable edge node needs to be selected for the perception task, and the computing offloading decision vector of all perception devices is denoted as . The perception device is , and the total perception device set is denoted as . In this example, the time domain is discretized into a time slot set, the length of a single time slot is , and the discrete time slot set is denoted as .

[0039] (2) constructing an edge node selection decision model corresponding to the sea area perception system; wherein the edge node selection decision model is used to decide the specific edge node to which the current task is offloaded when the current task is offloaded to the edge node.

[0040] Specifically, the edge node selection decision model is used to select a specific edge node from a plurality of edge nodes after determining that the current task needs to be offloaded, so as to process the current task through the edge node.

[0041] It can be understood that the edge node selection decision model can ensure that the current task is offloaded to a specific edge node. In a specific implementation, by comprehensively considering the current load of the edge node, the remaining amount of computing resources, and the distance between the edge node and the task source, a specific edge node is selected to reduce the delay of the current task in the transmission and processing process, and improve the efficiency and reliability of task processing.

[0042] In a specific implementation, the edge node selection decision model is denoted as , , indicates that the perception task of the nth perception device in the current time slot is offloaded to the edge node , otherwise , indicates that the perception task of the nth perception device in the current time slot is not offloaded to the edge node . The edge node decision vector of all nodes is denoted as . It should be noted that the total edge computing node set is denoted as .

[0043] (3) Construct a relay node selection strategy model corresponding to the marine perception system; wherein, the relay node selection strategy model is used to describe whether the current task needs a relay node and the specific relay node needed.

[0044] Referring to the preceding description, for the sensing task of the current time slot of the nth sensing device, if it is determined that the sensing task needs to be offloaded to an edge node, and the specific edge node is determined, a decision can be made on whether to select a relay node based on the length of the transmission path, and after determining that a relay node needs to be selected, the specific relay node is determined.

[0045] Specifically, relay nodes are nodes used to forward data during data transmission. When direct communication between the task source and the target edge node is poor or impossible due to distance, obstacles, or other reasons, relay nodes act as intermediaries, facilitating smooth data transmission. The goal of relay node selection strategies is to ensure that tasks outside the communication coverage area can be transmitted through appropriate relay nodes, guaranteeing efficient and stable transmission to the target edge node.

[0046] Understandably, the relay node selection strategy model is used to determine whether a task requires a relay node and, if so, which specific relay node. This can solve communication problems caused by distance, terrain, and other factors in maritime areas. It ensures that task data can be transmitted stably and efficiently to the target edge node, avoiding data loss or transmission interruption, and guaranteeing the continuity and timeliness of task processing. It is especially suitable for scenarios where sensing devices are widely distributed in vast maritime areas.

[0047] In specific implementation, when When it is determined that the sensing task of the nth sensing device in the current time slot needs to be offloaded to the edge node, and it is determined that the offloading will be to the edge node e, if the nth sensing device in the current time slot needs to be offloaded to the edge node e, then the nth sensing device in the current time slot needs to be offloaded to the edge node e. The sensing device is not at the edge node. Within the communication coverage area, a suitable relay node needs to be selected for this sensing task. Referring to the example above, there are two relay nodes: a high-altitude communication base station and a drone. In this example, the relay node selection decision is denoted as... That is, the value of the relay node selection decision can be 0, 1, or 2. This indicates that no relay node is needed; data is transmitted directly to the edge node. . A value of 1 indicates that the relay node for this sensing task transmission is a high-altitude communication base station. A value of 2 indicates that the relay node for this sensing task transmission is a drone. Furthermore, the relay node decision vector for all sensing devices is denoted as... .

[0048] (4) constructing a container resource level model corresponding to the sea area perception system; wherein the container resource level model is used to describe the resource level of each container in each edge node; the resource level of each container is determined based on the storage resource and the computing resource of the edge node where the container is located, and different combinations of storage resources and computing resources correspond to different resource levels.

[0049] Specifically, the container is a standardized unit for carrying task processing in the edge node, which independently runs in the edge node. The container resource configuration decision is to allocate an appropriate resource level container for the perception task on the edge node after the perception task is determined to be processed by a certain edge node.

[0050] It should be noted that in the present embodiment, the resource level of each container is a level divided according to the storage resource and the computing resource of the edge node where the container is located, and different resource levels correspond to different task processing capabilities. Different combinations of storage resources and computing resources correspond to different resource levels. According to the storage resource and the computing resource of the edge node, a plurality of storage resource and computing resource combinations can be divided, and each combination corresponds to a resource level.

[0051] In specific implementation, the present embodiment adopts a decoupled edge container resource configuration scheme, and a plurality of types of resource levels are preset to cope with differentiated task processing requirements. The resource level of the container is denoted as The container resource configuration decision is denoted as When , that is, at least one of the values of the offloading decision and the edge node selection decision is 0, the edge container allocation is not triggered in the case of no need for relaying or no need for offloading ; when , that is, the values of the offloading decision and the edge node selection decision are both 1, the edge container allocation is triggered in the case of need for relaying and need for offloading . represents a container resource level corresponding to a combination of storage resource and computing resource. The specific storage resource and computing resource are denoted as and .

[0052] It can be understood that according to different requirements of storage resource and computing resource, the capacity of the edge node is divided into multiple levels. First, the computing resource of the edge device is divided into a plurality of virtual machines, and each virtual machine is an independent computing unit. Each virtual machine is further divided into a plurality of containers, and each container is responsible for processing an independent task. The resource level of the container is directly related to the storage resource and the computing resource allocated to it. The configuration and allocation of container resources ensure that the task can run efficiently on the edge node, while optimizing the utilization rate of resources.

[0053] (5) constructing a container survival time level model corresponding to the sea area perception system; wherein the container survival time level model is used to describe the survival time level of each container in each edge node.

[0054] Specifically, the container survival time level is a level divided according to the length of time that the container remains in a running state in the edge node. Different container survival time levels correspond to different time ranges from starting to destroying the container. The main role of the container survival time level model is to reasonably plan the life cycle of the container resource, so that the edge node can efficiently use the computing resource.

[0055] As can be understood, when a container is created and started for the first time, the start is a cold start, and because of the lack of previous state, the start process needs a certain time to initialize. When the container is already in an active state, the start is a hot start, and new tasks can quickly enter the container for processing. The delay of hot start is relatively low because the container is already running and does not need to be reinitialized.

[0056] In specific implementation, considering the container reservation mechanism, therefore, multiple types of container survival time levels are preset, denoted as The increment of the container survival time is denoted as Therefore, the set of container survival time values corresponding to all levels is The container survival time level model is denoted as The container survival time decision vector of all nodes is denoted as When a perception task is successfully processed in an edge node and , the container corresponding to the task will be added to the idle container queue of the edge node. When the idle containers in the idle container queue reach the survival time threshold, the occupied storage resource will be released and the container will be destroyed.

[0057] As can be understood from the foregoing description, the running logic of the computing service is decoupled into multiple independent functions, each of which can be requested by a user individually, and then a dynamic scheduling and resource configuration service is provided for each function request. In view of the independent feature of each function in the service, the method considers a computing service composed of one function, and therefore the task scheduling can be represented as function scheduling.

[0058] After the perception device collects the perception data, a perception task to be processed is generated in a single time slot. For the generated task request, a suitable computing node needs to be selected, which can be divided into local computing and edge computing. Local computing refers to computing processing by a local server, while edge computing refers to offloading the task to other node servers for processing. If the transmission path of offloading exceeds one hop, a relay node decision also needs to be made. If edge computing is selected for the task request, edge container resource configuration and container reservation decisions also need to be made. The total set of decision variables is composed of the function offloading decision, edge computing node selection decision, air relay node selection, edge container resource configuration and container reservation decision;

[0059] When a function request arrives at the target edge server, the server deploys a containerized function instance inside a virtual machine to execute the request. To effectively manage these function instances and optimize resource usage efficiency, this module implements two different instantiation modes: cold start and hot start.

[0060] Cold start: The cold start process involves a complete deployment process, including pulling container images from the image repository, configuring the runtime environment, resolving dependencies, and initializing the processing function. This process usually takes hundreds of milliseconds to tens of seconds, depending on the complexity of the function. Since multiple steps are involved, cold start can introduce significant delays, especially during peak loads or resource shortages.

[0061] Hot start: Hot start utilizes pre-initialized container instances, which are maintained in a "ready pool". When a new request is received, the system can directly reuse these existing instances, thereby skipping the initialization phase. By preserving the execution context and network state of previous calls, hot start can greatly reduce the startup time, which can be negligible, providing near-instant response capabilities.

[0062] Considering the performance difference between cold start and hot start, a complex scheduling algorithm must be designed to optimize resource utilization, reduce unnecessary cold start times, and improve overall system response speed and service quality.

[0063] Specifically, after introducing the task scheduling model, the communication transmission model is introduced as follows:

[0064] It can be understood that the communication transmission model is a model for describing the transmission characteristics of the data transmission link between nodes in the sea perception system.

[0065] Specifically, the embodiment mainly includes two types of transmission links, the first is the link of the sea surface, such as the link of the buoy to the unmanned ship; the second is the link from the air to the sea surface, such as the link of the unmanned aerial vehicle to the unmanned ship. The communication transmission models of the two types of links are as follows:

[0066] Specifically, since the sea surface has strong reflection effect on the wireless link, the embodiment adopts a non-visual link model in the first type of transmission link, and the channel gain model is as follows:

[0067] ;

[0068] Specifically, wherein, and respectively represent the antenna transmission gain of the sensing device and the antenna reception gain of the sensing device ; represents the reference path loss at the reference transmission distance of 1 m when the carrier frequency is ; represents the small-scale fading coefficient of the NLOS link, and the Rayleigh fading model is considered here; represents the transmission distance from the sensing device to the sensing device at time ; represents the path loss exponent.

[0069] Further, since the transmission environment quality from the air to the sea surface is good, the embodiment adopts a visual link model in the second type of transmission link, and the channel gain model is as follows:

[0070] ;

[0071] wherein, and respectively represent the antenna transmission gain of the sensing device and the antenna reception gain of the sensing device ; represents the speed of light; represents the transmission distance from the sensing device to the sensing device at time ; represents the carrier frequency; represents the small-scale fading coefficient of the LOS link, and the Rician fading model is considered here.

[0072] Specifically, based on the second type of channel gain model, the transmission rate of the sensing task can be obtained as follows:

[0073] ;

[0074] wherein, denotes the time at which the sensing device receives the data from the transmission bandwidth of the sensing device ; denotes the transmission power of the sensing device ; denotes the Gaussian noise power spectral density.

[0075] Specifically, after introducing the communication transmission model of the sea area sensing system, the following introduces the calculation processing model. Specifically, in a possible implementation manner, the process of constructing the calculation model includes:

[0076] Step 1, determining the local waiting time delay calculation model of the task in local calculation and the edge execution time delay calculation model of the task in edge node processing.

[0077] Specifically, the local calculation time delay calculation model is used to determine the time for which the task waits for processing in local calculation. When there are multiple tasks in the local calculation queue at the same time, the newly arrived task needs to be queued and wait, and the waiting time delay is related to the current task queue length, the remaining processing time of the task being processed, and the local calculation capability. It can provide a key basis for calculation offloading decision, and if the local waiting time delay is too long, combined with other factors, the task can be decided to be offloaded to the edge node.

[0078] It can be understood that the edge execution time delay calculation model is used to calculate the time for which the task is processed from the beginning to the end on the edge node, which is mainly related to the calculation resources of the edge node, the calculation amount of the task itself, and the current load condition of the edge node. It can help to evaluate the processing efficiency of the task in the edge node, and is an important reference index when selecting the edge node, so as to ensure that the selected edge node can complete the task processing within the expected time.

[0079] Specifically, the node to the node at time The input data amount of the task to be processed in the transmission waiting queue is denoted as , and the updating strategy is as follows:

[0080] ;

[0081] wherein, denotes whether the node receives the task data from the node ; denotes the amount of task data transmitted by the node to the node ; denotes the amount of task data transmitted by the node to the node Is the transmission link interrupted? Represents a node Pass to node The amount of task data; and These represent the data volume and latency threshold of the first task in the transmission waiting queue, respectively.

[0082] Task The local waiting delay in local computing can be expressed as:

[0083] ;

[0084] in, Indicates task The node The local computation waits for the queue to pop; Indicates time The set of all previous moments; operators express .

[0085] Specifically, based on the definition of local wait latency mentioned above, the task can be given... of :

[0086] ;

[0087] Among them, operators This represents the vertex function, which returns the smallest integer greater than or equal to the given real number. It is expressed as the length of a single time slot; and These represent the computational resource requirements and latency threshold of the first task in the local computation waiting queue, respectively. This indicates the computational resource capacity of the buoy.

[0088] Furthermore, the task can be obtained. The local computation latency is as follows:

[0089] ;

[0090] Specifically, nodes At any moment The computational requirements of the pending tasks in the local computation waiting queue are denoted as . Its update strategy is as follows:

[0091] ;

[0092] in, Represents a node Are there any newly generated tasks? and denote the computing resource requirement and the latency threshold of the head task in the local computing waiting queue, respectively; denotes the computing resource capacity of the virtual machine.

[0093] Specifically, to construct the edge execution latency calculation model, the task is first obtained. The starting time of the computation in the edge computing node is defined as follows:

[0094]

[0095] wherein, denotes the generation time of ; t denotes the current time; denotes the hot state container of the function type whether in the hot state container queue of the virtual machine ; I(•) is an indication function for judging whether a certain condition is true or not; denotes the cold start latency; denotes the edge computing queue length of the virtual machine ; denotes the maximum queue length; denotes the available storage resource of the node ; denotes the available computing resource of the virtual machine . Based on the above definitions of , the time when the task

[0096] is popped out by the virtual machine is obtained as follows:

[0097] .

[0098] wherein, denotes the waiting latency of the task in the edge computing node .

[0099] Further, the number of tasks to be processed in the edge computing waiting queue of the node at time is denoted as , and the updating strategy is as follows:

[0100] .

[0101] Specifically, the computing resource requirement of the virtual machine in the edge computing queue at time is denoted as​​ According to the change of the edge computing task queue, the computing resource requirement of the virtual machine is updated, and the update strategy is as follows:

[0102] ;

[0103] Further, if the task is completed in the virtual machine , the total edge execution delay is:

[0104] ;

[0105] Step 2, according to the local waiting time calculation model and the edge execution time calculation model, determine the total execution time calculation model of the task.

[0106] Specifically, the total execution time calculation processing model of the task is used to describe the relationship between the total execution time of the task and the computing offloading decision, the edge node selection decision, the local computing time delay, and the transmission time delay of the task. The total execution time of the task is used to calculate the total time from the generation to the final completion of the task according to the local waiting time calculation model and the edge execution time calculation model, combined with the task scheduling decision. It can intuitively reflect the overall efficiency of task processing, is the core index for measuring the pros and cons of task scheduling strategy, can provide quantitative basis for optimizing task scheduling decision, and ensures that the task can meet the real-time requirement.

[0107] In specific implementation, combined with the above description, combined with the local computing time delay and the edge execution time delay mentioned in step 1, the total execution time of the task is expressed as follows:

[0108] ;

[0109] Specifically, wherein, represents the relay node selection decision; the resource capacity level set of the edge container is denoted as , and the edge container resource configuration decision is denoted as ; the container survival time level set is denoted as , and the edge container survival time decision is denoted as ; the discrete time slot set is denoted as .

[0110] Further, this formula considers the binary selection of the task execution mode, 1-x n (t) represents that the task takes the local computing path, and x n (t) represents that the task takes the edge computing path. Under the two paths, the total time delay of the task before execution and the actual execution is calculated, and finally x n (t) selects one path from the total time delay of the two modes and outputs.

[0111] S102, according to the task scheduling model, the communication transmission model and the computing processing model, the task scheduling problem corresponding to the sea area perception system is constructed.

[0112] Combined with the task scheduling model, the communication transmission model and the computing processing model, the task scheduling problem needs to be realized by optimizing the decision variables. Specific decision variables include computing offloading decision, edge node selection decision, relay node selection decision, container resource configuration decision and container survival time reservation decision. Through these decisions, the sea area perception system can select whether to need a suitable edge node, decide whether to need a relay node to transmit data, and allocate sufficient computing resources and storage resources according to the requirements of the task.

[0113] The goal of the task scheduling problem is to maximize the number of completed tasks while meeting the corresponding constraint conditions. Through the constraint conditions, it is guaranteed that the task delay will not exceed the preset threshold, and at the same time, the resource requirements of each task can be met. In this process, the decision variable constraints of the task, the channel quality constraints, the task delay threshold constraints, the queue length constraints and the resource constraints of the edge node must be considered at the same time to ensure the accuracy and efficiency of the scheduling.

[0114] In specific implementation, since the task generation has randomness, and the processing of each task will span multiple time slots, the following task scheduling problem is constructed to maximize the number of completed tasks per unit time:

[0115] ;

[0116] Further, after the task scheduling problem is clarified, constraint conditions need to be set, including decision variable constraints, channel quality constraints, task delay threshold constraints, queue length constraints and edge node resource constraints.

[0117] Specifically, the decision variable constraint represents the value constraint of each decision variable and the value constraint between the decision variables with dependent relationship; the task delay constraint represents that the execution delay of each task needs to be within the task delay threshold; the signal-to-noise ratio constraint represents that the establishment of a wireless link between two nodes needs to meet the minimum signal-to-noise ratio threshold, which can be represented as ; the queue length constraint represents that the number of tasks in each queue cannot exceed the maximum queue length, which can be represented as , wherein represents the queue length, in addition, there are multiple independent virtual machines in each edge server, represents the virtual machine set in the edge computing node ; the resource constraint of the edge node represents that the computing and storage resources required by the tasks carried in each edge node cannot exceed the maximum computing and storage resource capacity of this node, which can be represented as and .

[0118] It can be understood that, according to the above constraints and optimization objectives, the mathematical model of the task scheduling problem is obtained as follows:

[0119] ;

[0120] It can be understood that, constraint condition C1 represents whether the task is to be offloaded to the edge node; constraint condition C2 represents whether the task needs a relay node edge node; constraint condition C3 represents that the edge container allocation is not triggered; constraint condition C4 represents that the container survival time decision of task n on the edge node e must belong to the preset container survival time level set; constraint condition C5 represents that the selection of offloading to a specific edge node is a prerequisite for offloading; constraint condition C6 represents that the decision of whether to select offloading and the selected node changes, and the resource allocation and the container state also change accordingly; constraint condition C7 represents that the actual execution delay of task n will not exceed the task delay threshold; constraint condition C8 represents that the communication link quality cannot be too poor; and constraint condition C9 represents that the number of tasks in the queue cannot exceed the node processing capacity.

[0121] S103, model the task scheduling problem as a partially observable Markov decision process, and train a preset deep reinforcement learning intelligent agent using offline simulation data to obtain a trained strategy model.

[0122] In this embodiment, the long-term task and container scheduling problem in the fusion networking need to be solved, and a collaborative decision needs to be made in a continuous time slot. Since the decision of each perception device only depends on the system state information in its time slot, this feature indicates that the system state transition process satisfies the Markov property, and the subsequent state only depends on the current condition. Therefore, the above task scheduling problem can be reconstructed as a partially observable Markov decision process, in which each perception device operates as a heterogeneous intelligent agent with local environmental observation capability. The system collectively achieves the optimization objective in the above task scheduling problem through collaborative decision-making of all perception devices. The partially observable Markov decision process can be represented as: .

[0123] Specifically, the partially observable Markov decision process includes the local observation state of each perception device in the perception system, the global state of the perception system, the action space, and the reward function. The action space is constructed based on the decision variable;

[0124] The reward function is constructed based on the optimization target; reward items of the reward function include completing a task within a specified time delay threshold, a number of cold starts of the edge computing node, a hot start of the edge computing node, the edge computing node being in local computing, a number of function requests in a computing queue of the edge computing node; and penalty items of the reward function include not completing a task within a specified time delay threshold, a memory requirement of a task exceeding an available memory capacity of the edge node, a communication requirement of the task exceeding a communication capacity of the edge node, a signal-to-noise ratio between the task and the edge node being lower than a preset threshold, and a computing waiting queue of the edge node exceeding a preset queue length.

[0125] Referring to the foregoing description, the perception system includes a first type of perception device and a second type of perception device, the first type of perception device only has local computing capability, and the second type of perception device has both local computing capability and capability of assisting other nodes to process tasks. The buoy and the unmanned aerial vehicle belong to the first type of perception device; and the small unmanned ship, the large unmanned ship mother ship, the offshore operation platform, the high-altitude communication base station, and the shore-based macro base station belong to the second type of perception device.

[0126] Specifically, the local observation state of the first type of perception device includes: a node number, a type number of a newly-arrived function request, a channel gain vector between the node and other reachable nodes, a type number vector of all function requests in a transmission waiting queue of the node, a task data volume vector of a head function request in the transmission waiting queue of the node, an action number selected by the node at a previous time, a computing resource demand vector of all function requests in a local computing waiting queue of the node, a function type number of a head function request in the local computing waiting queue of the node, and a to-be-processed computing resource demand.

[0127] Similarly, the local observation state of the second type of perception device includes: a node number, a type number of a newly-arrived function request, a channel gain vector between the node and other reachable nodes, a type number vector of all function requests in a transmission waiting queue of the node, a task data volume vector of a head function request in the transmission waiting queue of the node, an action number selected by the node at a previous time, a remaining storage resource of the node, a remaining computing resource vector of all virtual machines in the node, a type number vector of all function requests in an edge computing waiting queue of the node, a container survival time vector of all function requests in the edge computing waiting queue of the node, a type number vector of all function requests in an edge cold start waiting queue of the node, a type number vector of all function requests in an edge computing queue of the node, a type number vector of all function requests in an edge hot state container queue of the node, a computing resource demand vector of all function requests in the edge computing queue of the node, and a container survival time vector of all function requests in the edge computing queue of the node.

[0128] Specifically, different types of sensing devices have different local observation states, the first type of sensing device and the second type of sensing device have some common local observation variables, denoted as , which are as follows:

[0129] ;

[0130] wherein the elements in the vector respectively represent: node number, type number of newly arrived function request, channel gain vector between the node and other reachable nodes, type number vector of all function requests in the transmission waiting queue of the node, task data volume vector of the head function request in the transmission waiting queue of the node, action number selected by the node at the last time.

[0131] Further, for the first type of sensing device, the specific local observation vector is denoted as , which is as follows:

[0132] ;

[0133] wherein the elements in the vector respectively represent: computing resource demand vector of all function requests in the local computing waiting queue of the node, function type number and to-be-processed computing resource demand of the head function request in the local computing waiting queue of the node.

[0134] For the second type of sensing device, the specific local observation vector is denoted as , which is as follows:

[0135] ;

[0136] wherein the elements in the vector respectively represent: remaining storage resource of the node, remaining computing resource vector of all virtual machines in the node, type number vector of all function requests in the edge computing waiting queue of the node, container survival time vector of all function requests in the edge computing waiting queue of the node, type number vector of all function requests in the edge cold start waiting queue of the node, type number vector of all function requests in the edge computing queue of the node, type number vector of all function requests in the edge hot state container queue of the node, computing resource demand vector of all function requests in the edge computing queue of the node, container survival time vector of all function requests in the edge computing queue of the node.

[0137] In summary, the local observation state vector of the first type of sensing device is , and the local observation state vector of the second type of sensing device is .

[0138] Further, the global state of the perception system comprises part of the local observation state of the first type of perception device and part of the local observation state of the second type of perception device.

[0139] In particular, the global state plays a key role in enhancing the environment cognition ability of the agent. The traditional method usually constructs the global state by simply concatenating all the local observations of the agents, but this approach has two significant defects: first, it may miss system-level information beyond the observation range of a single agent, such as the queue state and the remaining resource information of the macro base station in this example; second, the high-dimensional state produced by direct concatenation will cause the curse of dimensionality. To solve these defects, in this embodiment, a feature extraction strategy is adopted to select key features from the local observation of each agent, thereby constructing a concise and comprehensive global state representation.

[0140] For example, in an embodiment, the part of the state information extracted from the local observation state of the first type of perception device is denoted as , specifically:

[0141] ;

[0142] wherein the elements in the vector respectively represent: node number, type number of newly arrived function request, channel gain vector between the node and other reachable nodes, type number vector of all function requests in the transmission waiting queue of the node, task data volume vector of the head function request in the transmission waiting queue of the node, action number selected by the node at the last time; computing resource demand vector of all function requests in the local computing waiting queue of the node, function type number and to-be-processed computing resource demand of the head function request in the local computing waiting queue of the node.

[0143] Further, the part of the state information extracted from the local observation of the second type of perception device is denoted as , specifically:

[0144] ;

[0145] Wherein, the elements in the vector respectively represent: node number, type number of newly arrived function request, channel gain vector between the node and other reachable nodes, type number vector of all function requests in the transmission waiting queue of the node, task data volume vector of the head function request in the transmission waiting queue of the node, action number selected by the node at the last time. The remaining storage resource of the node, the remaining computing resource vector of all virtual machines in the node, type number vector of all function requests in the edge computing waiting queue of the node, container survival time vector of all function requests in the edge computing waiting queue of the node, type number vector of all function requests in the edge cold start waiting queue of the node, type number vector of all function requests in the edge computing queue of the node, type number vector of all function requests in the edge hot state container queue of the node, computing resource demand vector of all function requests in the edge computing queue of the node, container survival time vector of all function requests in the edge computing queue of the node.

[0146] It can be understood that the global state of the perception system is:

[0147] ;

[0148] The action space can be represented as:

[0149] ;

[0150] Further, in the embodiment, the specific reward function is represented as:

[0151] ;

[0152] Wherein, and respectively represent the number of cold start and hot start of the edge node in the time ; and respectively represent the number of function requests in the local computing and edge computing queue of the edge node at time .

[0153] Referring to the foregoing description, the optimization objective in the task scheduling problem established in the foregoing is to maximize the number of task completions, that is, to maximize In the embodiment, the , , and are added in the reward function, which can provide more indirect reward signals for the agent, can solve the reward sparsity problem, and improve the convergence performance.

[0154] It should be noted that by introducing cold start, hot start, the number of function requests in the ongoing local computing and edge computing queue and other factors, more rich and diverse reward signals can be provided. These reward signals not only help the agent to understand the task completion, but also reflect the load state and the utilization of computing resources. By comprehensively considering these factors, the agent can better balance the scheduling of tasks and the allocation of resources, avoid excessive dependence on the optimization goal of one aspect, and thus effectively solve the problem of reward sparsity. This reward function design can accelerate the learning process of the agent and improve the convergence of training.

[0155] Further, after modeling the task scheduling problem as a partially observable Markov decision process, the pre-set deep reinforcement learning agent can be trained using offline simulation data to obtain a trained strategy model.

[0156] Next, the pre-set deep reinforcement learning agent will be introduced.

[0157] Specifically, the deep reinforcement learning agent includes a decision model and an evaluation model. In the training phase, both the decision model and the evaluation model participate in the training, and in the deployment phase, only the decision model is used for task scheduling and decision making. This allows the agent to obtain more rich information during training, thereby making better decisions in actual deployment.

[0158] It should be noted that in one possible implementation, both the strategy model and the evaluation model include an RNN network to capture historical observation states through the RNN network, enhancing the model's understanding of long-term dependencies and temporal relationships.

[0159] It can be understood that the RNN network is a recurrent neural network, which is a neural network specially designed for processing sequence data. Unlike traditional neural networks, RNN networks retain previous input information when processing current data, achieving memory of historical data through recurrent connections in the hidden layer, thereby capturing temporal relationships and long-term dependencies in the data. For example, in this example, the RNN network can combine previous observation states to help the model understand the association between the current state and the historical state.

[0160] Specifically, for the strategy model, the RNN can fuse the current local observation state with the historical local observation state, so that the output task scheduling scheme not only adapts to the current state, but also considers the influence of historical decisions, such as avoiding repeated selection of edge nodes with high load. For the evaluation model, the RNN can combine the historical sequence of global observation states, such as network load distribution and task completion in different time slots, to more accurately evaluate the value of the current global state and provide a more reliable reference for strategy optimization.

[0161] The training process of the deep reinforcement learning agent will be described in detail in Embodiment Two, which will not be repeated here.

[0162] In S104, the policy model is deployed in each perception device, so that each perception device invokes the policy model for distributed online inference according to the local observation state, to obtain a task scheduling scheme.

[0163] Specifically, the distributed online inference refers to a process in which, after the trained policy model is deployed in each perception device in the sea perception network, each perception device acts as an independent agent and invokes the policy model for task scheduling decision in real time based on only the local observation state of the perception device.

[0164] It can be understood that, using the distributed inference, each perception device directly invokes the policy model to generate a scheduling scheme based on the local observation state, without the need to upload data to a centralized node for processing and then return the result. The long-distance data transmission delay caused by the vast sea area and the scattered nodes can be avoided, and this method is particularly suitable for the marine environment where the coverage of the near-shore base station is limited and the delay of satellite communication is high, and can quickly respond to tasks with high real-time requirements.

[0165] The task scheduling method for sea perception tasks provided in this embodiment effectively optimizes the process of task scheduling and resource management by constructing a task scheduling model, a communication transmission model and a computing processing model of the sea perception system. The method models the task scheduling problem as a partially observable Markov decision process, and uses deep reinforcement learning to train the agent offline, so that the system can make adaptive task scheduling decisions in a dynamic and complex sea environment. By maximizing the number of completed tasks and optimizing under multiple constraints, the task scheduling efficiency can be significantly improved, the computing delay can be reduced, the resource utilization can be optimized, and the overall performance of the sea perception network can be improved. In addition, by deploying the trained policy model in each perception device, the agent can perform distributed online inference based on the local observation, so that the task scheduling decision is more flexible and real-time, thereby improving the user service completion quality and the response speed of the system. This method improves the robustness, efficiency and scalability of the system, and meets the needs of efficient and reliable task scheduling for sea perception tasks.

[0166] Figure 3 The flowchart of Embodiment Two of the task scheduling method for sea perception tasks provided in this application is shown in FIG. 2. Figure 3On the basis of the above-mentioned embodiments, the method provided in the embodiment comprises a decision model and an evaluation model; the decision model is used to output a task scheduling scheme according to a local observation state; the evaluation model is used to output a state value of the global observation state according to a global observation state; the preset deep reinforcement learning agent is trained by using offline simulation data to obtain a trained scheduling strategy model, which comprises the following steps:

[0167] S301, a simulation environment corresponding to the sea area perception system is constructed, and a plurality of groups of offline simulation data are collected based on the simulation environment.

[0168] Specifically, each group of offline simulation data comprises a current local observation state, a current task scheduling scheme, a current reward value and a next-time local observation state; wherein the current task scheduling scheme is determined by the decision model based on the current local observation state; when deciding the current task scheduling scheme, the decision model is based on the currently effective actions in the action space.

[0169] In specific implementation, the simulation environment corresponding to the sea area perception system is first constructed, which comprises a buoy, a small unmanned ship, a large unmanned ship mother ship, a sea operation platform, a high-altitude communication base station, a drone and a shore-based macro base station and other sensing devices. In the simulation environment, the time domain is discretized into a set of time slots, the length of a single time slot is fixed, the wireless channel and the network topology do not change in a single time slot and independently change in different time slots, the node coordinates are modeled by using a three-dimensional Cartesian coordinate system, and the movement models of the drone and the small unmanned ship are modeled by using a truncated Gaussian distribution, and a quasi-static fading wireless channel model is used for the channel.

[0170] In specific implementation, the simulation environment constructed in the embodiment is deployed with 8 buoys, 4 small unmanned ships, a sea operation platform, a large unmanned ship, a drone, a high-altitude communication base station and 2 shore-based macro base stations. The simulation time unit is 10 ms. The movement models of the small unmanned ship and the drone are truncated Gaussian distribution, and the mean and variance are 60 and 21 0.5 , respectively. The computing resource capacity of each buoy is FLOPS. The number of virtual machines of each small unmanned ship, sea operation platform, large unmanned ship, macro base station and high-altitude communication base station is 3, 4, 4, 4 and 5, respectively, and the computing resource capacity of the virtual machine of each small unmanned ship is FLOPS; the computing resource capacity of the virtual machine of the remaining edge perception device is FLOPS. The storage resource of each small unmanned ship, sea operation platform, large unmanned ship, macro base station and high-altitude communication base station is 500, 1000, 1000, 1500 and 1500 Mbits, respectively. Three function types are set in the simulation, and the task input data volume set is Mbits; the storage resource requirement set is Mbits; FLOPS; the task latency threshold set is ms. The cold start latency is 100 ms. The queue length is 8. There are three types of edge container resource configurations, which are Take the first type as an example, the storage resource capacity is 64 Mbits, and the computing resource capacity is FLOPS. The simulation uses Python 3.9.19, Pytorch 2.6.0, CUDA 12.8 and NVIDIA RTX A4000.

[0171] It should be noted that the current local observation state can be obtained, which is input into the decision model, and the decision model outputs the current task scheduling scheme. Further, the simulation system executes the task scheduling scheme to obtain the local observation state at the next time. Further, at a certain time, based on the local observation state of all perception devices, the global observation state of the perception system is obtained.

[0172] It should be noted that the decision model decides the current task scheduling scheme based on the currently effective actions in the action space.

[0173] Specifically, due to the dynamic characteristics of network topology structure and task generation, the action space of each agent changes under different time slots, and the action space may contain invalid actions, which may trigger unnecessary exploration behavior and lead to distortion of advantage function estimation.

[0174] To solve this problem, in this embodiment, a dynamic action mask layer is introduced to filter the invalid actions of each agent in real time. Specifically, the agent The action mask function of the agent at time slot is defined as:

[0175] ;

[0176] Then the action probability function based on the mask function is:

[0177] ;

[0178] S302, collect data from the multiple sets of offline simulation data to train the deep reinforcement learning agent, and obtain a trained strategy model.

[0179] In training the deep reinforcement learning agent, the decision model is updated using gradient ascent; the loss function of the decision model is determined based on a target proportion and an advantage function; the target proportion is the minimum of an importance proportion, a preset upper limit proportion, and a preset lower limit proportion; the importance proportion is the ratio of the decision probability of the current task option to the decision probability of the previous task option in the previous time slot; the advantage function is determined based on the reward value and the temporal difference error of the state value; the temporal difference error of the state value is determined based on the current state value and the state value of the previous time slot.

[0180] In practical implementation, each intelligent agent Deploy a parameter as strategy model and a parameter is Evaluation model During each update, the policy model adjusts its parameters based on the policy loss, while the evaluation model updates its parameters using the loss from the global state value estimate. The loss function for the decision model is shown below:

[0181] ;

[0182] Specifically, among them, This indicates the batch size of data used in training the neural network. This represents the target policy function, i.e., the decision probability of the current task plan; This represents the behavioral policy function, which is the decision probability of the previous task plan in the previous time slot; Represents the dominance function; Represents the clipping function; Indicates the cutting factor; Represents the entropy coefficient; Entropy, representing the behavioral policy, is used to measure the randomness of the policy and can enhance the policy exploration capability. Furthermore, by summing and averaging the losses across the number of agents, it can be ensured that the policy losses of different agents contribute equally to the overall loss.

[0183] Specifically, the parameterized strategy function is defined as follows:

[0184] ;

[0185] The optimization objective of the policy-based reinforcement learning algorithm is as follows:

[0186] ;

[0187] The policy gradient theorem is a fundamental theory supporting policy gradient-based reinforcement learning. This theorem establishes a direct gradient relationship between policy parameters and expected cumulative reward, enabling gradient ascent optimization of stochastic policies by maximizing the reward trajectory without requiring a dynamic environment model. Its mathematical formulation is as follows:

[0188] ;

[0189] Policy gradient estimation directly depends on accumulated rewards, which leads to high variance in gradient updates and convergence instability. State-value function It can represent the strategy starting from a given state. Policy performance is measured by evaluating the intrinsic value of states. (Introduction) As a baseline, determining whether the reward obtained from an action exceeds expectations can effectively reduce variance. Dominance function It can measure the relative advantage of an action with respect to the average value of the state, highlighting actions that generate rewards exceeding the baseline to optimize decision-making. By using... Instead of the original reward, the policy gradient reinforces actions superior to the baseline while suppressing noisy reward signals. This alternative approach generates gradient estimates with lower variance, significantly improving training stability and sample efficiency in complex environments.

[0190] Advantage function Action value function and state value function The dual dependency relationship introduces structural complexity to the decision-making model design. Furthermore, due to the higher dimensionality of the action space, action value estimation faces greater challenges than state value prediction. To address these issues, the advantage function is reconstructed using the Bellman expectation equation, i.e. This transforms the action-value dependency into state-value temporal difference (TD) error. This transformation allows the advantage function to be expressed in TD residual form: This eliminates the need to build an explicit action value network architecture.

[0191] To determine the composition of the loss function, this embodiment introduces the advantage function, a crucial component of the loss function. The transformed form of the policy gradient function is as follows:

[0192]

[0193] In summary, the advantage function constructed in this paper is:

[0194]

[0195] in, The temporal difference error is the state value. are the current state value and the state value of the previous time slot, respectively.

[0196] Specifically, the change of the policy parameter directly changes the action probability distribution, which may lead to sampling different actions in the same state. Therefore, the policy gradient algorithm is essentially a on-policy method, and the behavior policy (used for data collection, denoted as ) and the target policy (the policy to be optimized, denoted as ) must be consistent. This strong coupling brings two key limitations: one is low sample efficiency, and each policy update will invalidate the existing collected data, and the environment interaction must be performed again; the other is unstable learning process, and the correlation samples generated by a single policy will exacerbate the gradient variance and delay the convergence speed.

[0197] Specifically, to decouple the exploration and optimization processes, the importance sampling technique is introduced in this embodiment, so that the trajectory generated by the old policy or the auxiliary behavior policy can be used to update the current target policy. Importance sampling can effectively utilize policy data while ensuring the accuracy of gradient estimation. By reweighting the action probability according to the importance ratio of the two policies, we have:

[0198] ;

[0199] It can be understood that, however, importance sampling has a key defect: when the behavior policy differs significantly from the target policy , the importance weight will have a high variance problem. Extreme weights will destroy the stability of policy update and reduce the learning efficiency. In this embodiment, a truncation mechanism is used to directly limit the policy update within a fixed threshold range. The optimization objective can be expressed as:

[0200] ;

[0201] where the truncation function expression is:

[0202] ;

[0203] Specifically, obtaining the trained policy model will train the network through offline simulation data until convergence. During the training process, the generalized advantage estimation is used to handle the temporal difference error, balance the bias and variance, and improve the training stability. Finally, the trained scheduling policy model is obtained, so that each agent can make distributed task scheduling decisions based on local observations.

[0204] In this embodiment, by limiting the effective range of the importance ratio to the interval, the truncation function can prevent excessive policy updates from destroying the training stability. When the target policy The probability estimate for a certain action is significantly higher or lower than the behavior policy The corresponding advantage contribution will be truncated. This method reduces the computational complexity while maintaining convergence, achieving efficient and robust policy optimization.

[0205] Further, the decision model is updated by gradient ascent method, specifically as follows:

[0206] ;

[0207] Wherein, Indicates the learning rate of the decision model training.

[0208] Further, the loss function of the global evaluation model is as follows:

[0209] ;

[0210] Wherein, Indicates the current cumulative discounted reward. The parameter update of the evaluation model is as follows:

[0211] ;

[0212] Wherein, Indicates the learning rate of the evaluation model training. After the algorithm training is completed, each perception device agent can make task scheduling decisions based on the algorithm model.

[0213] In specific implementation, the parameter settings in the training process are as follows: the total number of training steps is ; the total length of each round is 250; the number of training data updates is 15; the data batch size is 1; the number of parallel training environments is 30; the coefficient of generalized advantage function estimation is 0.95; the neural network is 3 layers; the number of hidden layer neurons is 128; the number of recurrent neural network layers is 1; the number of recurrent neural network neurons is 10; the reward discount coefficient is 0.99; the policy clipping coefficient is 0.2; the optimization coefficient of RMSprop is 0.00001; the learning rates of the decision model and the evaluation model are 0.001 and 0.003 respectively; the entropy coefficient is 0.01; and the loss coefficient of the value function is 0.5.

[0214] The method provided by the embodiment significantly improves the learning efficiency and decision stability of the agent in a complex environment by introducing techniques such as policy gradient, advantage function and importance sampling. First, by introducing the advantage function and the baseline policy, the high-variance problem caused by reward fluctuation is reduced, and the stability of gradient update is improved. Second, the importance sampling technique enables the agent to use historical data for policy update, avoiding the inefficient problem of re-collecting data for each policy update. In addition, by optimizing the entropy of the policy, the agent can increase the exploratory nature and avoid falling into a local optimal solution, further enhancing the breadth of learning. Finally, the truncation mechanism is used to limit the amplitude of policy update, ensuring the stability of the training process. In summary, the training method balances the bias and variance, and improves the training stability, sample efficiency and policy optimization ability of the agent.

[0215] Figure 4 The flowchart of the third embodiment of the task scheduling method for sea area perception tasks provided in the present application is provided. Please refer to Figure 4 On the basis of the above-mentioned embodiments, when a plurality of sets of offline simulation data are collected based on the simulation environment, the method can further include:

[0216] S401, after obtaining a task scheduling scheme based on a set of local observation states, determining the total execution time delay corresponding to the task scheduling scheme based on the computing processing model.

[0217] Specifically, after generating a task scheduling scheme based on a set of local observation states, the total execution time delay of the task scheduling scheme can be calculated using the computing processing model. Through quantitative analysis of the total execution time delay, potential bottleneck problems can be found in time.

[0218] For specific implementation process and implementation principle of this step, please refer to the description in the previous embodiments, which will not be repeated here.

[0219] S402, when the total execution time delay is greater than the time delay threshold, the task scheduling scheme is not added to the corresponding task queue.

[0220] Specifically, the total execution time delay is compared with the time delay threshold of the task. If the total execution time delay exceeds the threshold, the filtering mechanism is triggered, i.e. the task scheduling scheme is marked as invalid, and it is not added to the task queue. For example, if the time delay threshold is 500ms, and the total execution time delay of the task scheduling scheme is 600ms, the task scheduling scheme is directly discarded, avoiding the occupation of storage resources and computing resources.

[0221] Specifically, local computing is performed in a serial manner, so that the waiting time of a request before starting computing can be accurately known when determining whether to add the request to the computing waiting queue. However, edge computing is performed in a parallel manner, which causes the agent to be unable to estimate the edge computing waiting time of a function request. However, the method can be known through experiments that a specific component of the edge computing waiting time can provide key information for scheduling decisions. Specifically, when edge computing resources are allocated to a function request, the agent can calculate the computing time delay required by the function request; in addition, when deciding to add the function request to the computing waiting queue in the edge node, if there is no available hot state container, the cold start time delay is also a known part of the waiting time. By using the prior knowledge of these specific waiting times, not only can the action space design be optimized, but also the queue updating mechanism can be improved, thereby significantly improving the scheduling efficiency.

[0222] The method provided in the embodiment can accelerate the convergence of the training process.

[0223] Corresponding to the foregoing embodiment of the task scheduling device for the sea area perception task, the present application also provides an embodiment of a task scheduling device for a sea area perception task.

[0224] Figure 5 A structural schematic diagram of the embodiment one of the task scheduling device for the sea area perception task provided in the present application is provided. Please refer to Figure 5 The device provided in the embodiment includes a construction module 510, a modeling module 520, a training module 530, and an application module 540; wherein,

[0225] The construction module 510 is configured to construct a task scheduling model, a communication transmission model, and a computing processing model corresponding to the sea area perception system for the pre-built sea area perception system; wherein, the task scheduling model is configured to describe the computing offloading decision, the edge node selection decision, the relay node selection strategy, the container resource level decision, and the container survival time level decision in the task scheduling process; the communication transmission model is configured to describe the link transmission characteristics of the sea area perception system; and the computing processing model is configured to describe the relationship between the total execution time delay of a task and the computing offloading decision, the edge node selection decision, the local computing time delay, and the transmission time delay of the task.

[0226] The modeling module 520 is configured to construct a task scheduling problem corresponding to the sea area perception system according to the task request model, the communication transmission model, and the computing processing model.

[0227] The training module 530 is configured to model the task scheduling problem as a partially observable Markov decision process, and train a preset deep reinforcement learning agent using offline simulation data to obtain a trained strategy model.

[0228] The application module 540 is configured to deploy the strategy model in each perception device, so that each perception device invokes the strategy model for distributed online inference according to a local observation state to obtain a task scheduling scheme.

[0229] The implementation process of the functions and roles of each unit in the above apparatus is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0230] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the part of the method embodiment. The above described device embodiment is only illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to the actual needs, part or all of the modules can be selected to achieve the purpose of the present application. Those skilled in the art can understand and implement without creative labor.

[0231] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A task scheduling method for maritime perception tasks, characterized in that, The method includes: For a pre-built marine sensing system, a task scheduling model, a communication transmission model, and a computational processing model are constructed. The task scheduling model describes the computation offloading decision, edge node selection decision, relay node selection strategy, container resource level decision, and container lifetime level decision during the task scheduling process. The communication transmission model describes the link transmission characteristics of the marine sensing system. The computational processing model describes the relationship between the total execution latency of the task and the computation offloading decision, edge node selection decision, local computation latency, and task transmission latency. Based on the task scheduling model, the communication transmission model, and the computation processing model, construct the task scheduling problem corresponding to the marine sensing system; The task scheduling problem is modeled as a partially observable Markov decision process, and a pre-defined deep reinforcement learning agent is trained using offline simulation data to obtain a trained policy model; wherein, the partially observable Markov decision process includes the local observation state of each sensing device in the sensing system, the global state of the sensing system, the action space, and the reward function. The action space is constructed based on decision variables; The reward function is constructed based on the optimization objective; the reward items of the reward function include completing the task within a specified latency threshold, the number of cold starts of the edge computing node, the warm start of the edge computing node, the edge computing node performing local computation, and the number of function requests in the computation queue of the edge computing node; the penalty items of the reward function include failing to complete the task within the specified latency threshold, the memory requirement of the task exceeding the available memory capacity of the edge node, the communication requirement of the task exceeding the communication capacity of the edge node, the signal-to-noise ratio between the task and the edge node being lower than a preset threshold, and the computation waiting queue of the edge node exceeding a preset queue length; The sensing system includes a first type of sensing device and a second type of sensing device. The first type of sensing device only has local computing capabilities, while the second type of sensing device has both local computing capabilities and the ability to assist other nodes in processing tasks. The local observation state of the first type of sensing device includes the computational resource requirement vector of all function requests in the local computation waiting queue of the node, the function type number of the function request at the head of the queue in the local computation waiting queue of the node, and the computational resource requirement to be processed. The local observation status of the second type of sensing device includes the remaining storage resources of the node, the remaining computing resource vector of all virtual machines in the node, the type number vector of all function requests in the edge computing waiting queue of the node, the container lifetime vector of all function requests in the edge computing waiting queue of the node, the type number vector of all function requests in the edge cold start waiting queue of the node, the type number vector of all function requests in the edge computing queue of the node, the type number vector of all function requests in the edge hot state container queue of the node, the computing resource demand vector of all function requests in the edge computing queue of the node, and the container lifetime vector of all function requests in the edge computing queue of the node. The global state of the sensing system includes a portion of the local observation state of the first type of sensing device and a portion of the local observation state of the second type of sensing device. The strategy model is deployed in each sensing device so that each sensing device can call the strategy model to perform distributed online inference based on its local observation status and obtain a task scheduling scheme.

2. The method according to claim 1, characterized in that, The task scheduling model for constructing the marine sensing system includes: Construct a computational offloading decision model corresponding to the marine sensing system; wherein, the computational offloading decision model is used to decide whether the current task is offloaded to the edge node; Construct an edge node selection decision model corresponding to the marine sensing system; wherein, the edge node selection decision model is used to decide the specific edge node to which the current task is unloaded when the current task is unloaded to an edge node; Construct a relay node selection strategy model corresponding to the marine sensing system; wherein, the relay node selection strategy model is used to describe whether the current task requires a relay node, and the specific relay node required; Construct a container resource level model corresponding to the marine sensing system; wherein, the container resource level model is used to describe the resource level of each container in each edge node; the resource level of each container is determined based on the storage resources and computing resources of the edge node where the container is located, and different combinations of storage resources and computing resources correspond to different resource levels; Construct a container survival time level model corresponding to the marine sensing system; wherein, the container survival time level model is used to describe the survival time level of each container in each edge node.

3. The method according to claim 1, characterized in that, The construction of the computational processing model corresponding to the marine sensing system includes: Determine the local waiting latency calculation model for tasks in local computation and the edge execution latency calculation model when tasks are processed at edge nodes; Based on the local waiting latency calculation model and the edge execution latency calculation model, the total task execution latency calculation model is determined.

4. The method according to claim 1, characterized in that, The deep reinforcement learning agent includes a decision model and an evaluation model; the decision model is used to output a task scheduling scheme based on the local observation state; the evaluation model is used to output the state value of the global observation state based on the global observation state. The step of training a pre-defined deep reinforcement learning agent using offline simulation data to obtain a trained scheduling policy model includes: A simulation environment corresponding to the marine perception system is constructed, and multiple sets of offline simulation data are collected based on the simulation environment. Each set of offline simulation data includes the current local observation state, the current task scheduling scheme, the current reward value, and the local observation state at the next moment. The current task scheduling scheme is determined by the decision model based on the current local observation state. When making a decision on the current task scheduling scheme, the decision model makes a decision based on the currently effective actions in the action space. Data is collected from the multiple sets of offline simulation data to train the deep reinforcement learning agent, resulting in a trained policy model.

5. The method according to claim 4, characterized in that, When collecting multiple sets of offline simulation data based on the simulation environment, the method further includes: After obtaining a task scheduling scheme based on a set of local observation states, the total execution latency corresponding to the task scheduling scheme is determined based on the computational processing model. If the total execution latency exceeds the latency threshold, the task scheduling scheme will not be added to the corresponding task queue.

6. The method according to claim 1, characterized in that, The pre-built marine perception system includes multiple buoys, multiple small unmanned surface vessels (USVs), at least one large USV mother ship, at least one offshore operating platform, one high-altitude communication base station, at least one unmanned aerial vehicle (UAV), and multiple shore-based macro base stations. The coverage area of ​​the high-altitude communication base station is divided into multiple cells, and any one of the sensing devices among the multiple buoys, the multiple small USVs, the at least one large USV mother ship, the at least one offshore operating platform, the at least one UAV, and the multiple shore-based macro base stations is located in any cell. The buoy and the drone belong to the first type of sensing device; the small unmanned surface vessel, the large unmanned surface vessel mother ship, the offshore operation platform, the high-altitude communication base station and the shore-based macro base station belong to the second type of sensing device.

7. The method according to claim 4, characterized in that, Both the strategy model and the evaluation model include RNN networks to capture historical observation states and enhance the model's understanding of long-term dependencies and temporal relationships.

8. A task scheduling device for maritime sensing tasks, characterized in that, The device includes a construction module, a modeling module, a training module, and an application module; wherein, The construction module is used to build a task scheduling model, a communication transmission model, and a computational processing model corresponding to the pre-built marine sensing system. The task scheduling model describes the computation offloading decision, edge node selection decision, relay node selection strategy, container resource level decision, and container lifetime level decision during the task scheduling process. The communication transmission model describes the link transmission characteristics of the marine sensing system. The computational processing model describes the relationship between the total execution latency of the task and the computation offloading decision, edge node selection decision, local computation latency, and task transmission latency. The modeling module is used to construct the task scheduling problem corresponding to the marine sensing system based on the task scheduling model, the communication transmission model, and the computation processing model. The training module is used to model the task scheduling problem as a partially observable Markov decision process and train a preset deep reinforcement learning agent using offline simulation data to obtain a trained policy model; wherein, the partially observable Markov decision process includes the local observation state of each sensing device in the sensing system, the global state of the sensing system, the action space, and the reward function. The action space is constructed based on decision variables; The reward function is constructed based on the optimization objective; the reward items of the reward function include completing the task within a specified latency threshold, the number of cold starts of the edge computing node, the warm start of the edge computing node, the edge computing node performing local computation, and the number of function requests in the computation queue of the edge computing node; the penalty items of the reward function include failing to complete the task within the specified latency threshold, the memory requirement of the task exceeding the available memory capacity of the edge node, the communication requirement of the task exceeding the communication capacity of the edge node, the signal-to-noise ratio between the task and the edge node being lower than a preset threshold, and the computation waiting queue of the edge node exceeding a preset queue length; The sensing system includes a first type of sensing device and a second type of sensing device. The first type of sensing device only has local computing capabilities, while the second type of sensing device has both local computing capabilities and the ability to assist other nodes in processing tasks. The local observation state of the first type of sensing device includes the computational resource requirement vector of all function requests in the local computation waiting queue of the node, the function type number of the function request at the head of the queue in the local computation waiting queue of the node, and the computational resource requirement to be processed. The local observation status of the second type of sensing device includes the remaining storage resources of the node, the remaining computing resource vector of all virtual machines in the node, the type number vector of all function requests in the edge computing waiting queue of the node, the container lifetime vector of all function requests in the edge computing waiting queue of the node, the type number vector of all function requests in the edge cold start waiting queue of the node, the type number vector of all function requests in the edge computing queue of the node, the type number vector of all function requests in the edge hot state container queue of the node, the computing resource demand vector of all function requests in the edge computing queue of the node, and the container lifetime vector of all function requests in the edge computing queue of the node. The global state of the sensing system includes a portion of the local observation state of the first type of sensing device and a portion of the local observation state of the second type of sensing device. The application module is used to deploy the strategy model in each sensing device, so that each sensing device can call the strategy model to perform distributed online inference based on its local observation status and obtain a task scheduling scheme.

Images