A multi-source device linkage multi-modal task scheduling method, system, device and medium

Abstract

The application discloses a multi-source device linkage multi-modal task scheduling method, system, device and medium, and belongs to the technical field of computer data processing. The method comprises the following steps: realizing intelligent distribution by checking and evaluating the task quantity and hardware indexes, finely routing according to task metadata including device identification and data format, adopting a differentiated execution strategy for three types of queues, namely, a conventional queue, a high-resource queue and a high-priority queue, and finally mapping and integrating multi-source alarm results into a standardized report for pushing. The application realizes dynamic balancing of computing resources and accurate scheduling of task classification, effectively avoids cluster overload while ensuring the real-time performance of high-priority tasks, and greatly improves the concurrent processing efficiency of multi-modal data and the integration output efficiency of multi-source alarms.

Description

Technical Field

[0001] This application belongs to the field of computer data processing technology, and specifically relates to a multimodal task scheduling method, system, device and medium for multi-source device linkage. Background Technology

[0002] With the deep integration of drone technology and artificial intelligence algorithms, drones linked with cameras and satellites are widely used in fields such as forest fire prevention and land monitoring. Cloud computing clusters need to handle massive image, video, and remote sensing data processing requests from multiple tenants. However, existing scheduling strategies have significant shortcomings: traditional scheduling relies on single-dimensional or static rules, which cannot accurately perceive the overall pressure on the cluster; during high concurrency, the lack of buffering mechanisms can easily lead to node downtime, while during off-peak hours, scattered tasks cause resource waste; the resource consumption of multimodal data (real-time streams, remote sensing images, etc.) varies greatly, and general strategies are difficult to adapt, resulting in a tradeoff between service stability and resource utilization, which has become an urgent problem to be solved in the industry. For example, in existing technical solutions, such as patent document CN116643893A "A scheduling method, scheduling device, storage medium and server for computing power tasks", only the task and resource level matching is realized, lacking differentiated execution and life cycle management, and the permanent guarantee mechanism for high-priority tasks is not disclosed, which cannot adapt to the needs of multimodal data processing. Patent document CN112486642A, "Resource Scheduling Method, Apparatus, Electronic Equipment and Computer-Readable Storage Medium", adopts a passive resource preemption strategy, which only triggers resource allocation when high-priority task resources are insufficient. There is no mechanism to actively increase resource deployment density, resulting in low resource utilization. Patent document CN116385857A, "A computing power scheduling method and system based on AI task scenarios", only identifies AI task scenarios, lacks a closed loop of "classification-scheduling-differentiated execution-intelligent scaling", does not involve multi-source alarm integration and device linkage, and has limited adaptability.

[0003] Therefore, to address the shortcomings of existing technologies in simultaneously ensuring the stability and resource utilization of multi-tenant high concurrency and multimodal heterogeneous task scheduling, a differentiated scheduling method is proposed. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a multi-modal task scheduling method, system, device, and medium with multi-source device linkage. By employing dual-threshold traffic distribution, multi-device dedicated engines, vertical / horizontal elastic scaling, and spatiotemporal clustering alarm integration, it achieves a balance between stable operation of high-priority tasks and efficient resource utilization, thereby improving scheduling accuracy and alarm reliability.

[0005] Firstly, this application provides a multi-modal task scheduling method for multi-source device linkage, the method comprising, Upon receiving an algorithm analysis request, after verifying the algorithm analysis request, the task metadata is obtained, the total number of tasks within the same time window as the algorithm analysis request is counted, and the hardware indicators of the computing cluster are obtained to calculate the comprehensive load value. The total number of tasks is compared with preset high and low thresholds, the comprehensive load value is compared with preset resource load warning thresholds, and the algorithm analysis request is distributed to the target physical computing node or persistent message queue according to the comparison results. The algorithm analysis requests distributed to the target physical computing node or pulled from the persistent message queue are parsed, and the device identification field and data stream format field are extracted from the task metadata. Based on the extraction results, the task is routed to the regular comparison queue, the high-resource serial queue, or the high-priority real-time queue. For the regular comparison queue, the high-resource serial queue, and the high-priority real-time queue, a differentiated strategy corresponding to each queue is adopted to execute the task and obtain the original analysis results of each queue. The raw analysis results of each queue are mapped to a unified alarm data structure template, and multi-source alarms within the same monitoring area are integrated to generate standardized reports and push them to the tenant's business system.

[0006] Furthermore, The process of obtaining the hardware metrics of the computing cluster to calculate the overall load value specifically includes: The average utilization of the central processing unit, average memory utilization, average graphics processing unit memory utilization, and disk I / O throughput utilization of the computing cluster are multiplied by their respective weighting coefficients and then summed in a weighted manner.

[0007] Furthermore, The step of distributing the algorithm analysis request to the target physical computing node or persistent message queue based on the comparison results specifically includes: If the total number of tasks is greater than the high threshold and the comprehensive load value is greater than the resource load warning threshold, the algorithm analysis request is serialized and written into the persistent message queue, and the number of gap computing power nodes is calculated by dividing the difference between the total number of tasks and the current maximum concurrent processing capacity of the cluster by the rated processing capacity of a single standard computing node to trigger horizontal scaling. If the total number of tasks is greater than the low threshold and less than or equal to the high threshold, or if the total number of tasks is greater than the high threshold but the comprehensive load value is less than the resource load warning threshold, then the device identifier in the task metadata is extracted as a hash key, and the nearest virtual node is found clockwise on the preset hash ring using a consistent hash algorithm to determine the target physical computing node. If the total number of tasks is less than or equal to the low threshold and the comprehensive load value is less than or equal to the resource load warning threshold, then multiple algorithm analysis requests belonging to the same tenant identifier within the time window are encapsulated into a task batch list and sent to a single computing node for batch processing.

[0008] Furthermore, For the high-resource serial queue, a differentiated strategy corresponding to each queue is adopted to execute tasks, specifically including: Acquire the remote sensing image data associated with the tasks in the high-resource serial queue; The header information of the remote sensing image data is read to obtain a six-parameter affine transformation matrix, and the remote sensing image data is divided into blocks according to the set block window size and overlap buffer width. For each data block, the Siamese neural network change detection model is called, inputting the current temporal block and the baseline base map block to output a change region mask. The center pixel coordinates of the change region in the change region mask are extracted, and the center pixel coordinates are converted into geospatial coordinates using the six-parameter affine transformation matrix.

[0009] Furthermore, For the high-resource serial queue, a differentiated strategy corresponding to each queue is adopted to execute tasks, which specifically includes: Use the obtained geospatial coordinates as the alarm targets to create a global results list; Calculate the intersection-union ratio (IUU) of the geographic bounding boxes of any two alarm targets in the global results list; If the crossover-union ratio is greater than a preset threshold, the target with the highest confidence is retained and redundant results are eliminated to perform global nonmaximum suppression.

[0010] Furthermore, For the high-priority real-time queues, differentiated strategies corresponding to each queue are used to execute tasks, specifically including: The task in the high-priority real-time queue is obtained and a long connection is established with the UAV image transmission system. The real-time video stream is pulled from the UAV image transmission system and the real-time video stream is decoded to obtain video frames. The video frames are written into a fixed-length circular buffer in timestamp order, and the received flight status telemetry data packets are stored in a time-series database. Based on the circular buffer, two telemetry data points adjacent to the timestamp of the video frame are retrieved from the time-series database. The UAV attitude data corresponding to the video frame is calculated by weighting the time difference using linear interpolation and input into the target detection model. Combined with monocular geolocation, the estimated latitude and longitude of the ground target are calculated.

[0011] Furthermore, For the aforementioned regular comparison queues, a differentiated strategy corresponding to each queue is employed to perform the task, specifically including: The task in the conventional comparison queue is analyzed to obtain the storage address of the baseline background image and the address of the real-time video stream; According to the reference background image storage address, the reference background image corresponding to the monitoring point is retrieved from the object storage system, and the current video frame is obtained as the current frame to be tested according to the real-time video stream address. The reference background image and the current frame to be tested are scaled to the input size specified by the feature extraction network and then input into the feature extraction layer of the convolutional neural network for forward inference to obtain the reference vector and the current vector. Calculate the cosine similarity between two vectors. If the cosine similarity is less than a preset change threshold, input the current test frame into the target detection model to identify the abnormal object category.

[0012] Furthermore, The integration of multi-source alarms within the same monitoring area specifically includes: Extract the target location, event category, and trigger time from the original analysis results of each queue output, and fill the target location, event category, and trigger time into the corresponding fields of the preset unified alarm data structure template to complete the structured encapsulation. Based on the integration results of multi-source alarms within the same monitoring area, the encapsulated alarm data is assembled into a standardized report according to the preset report generation rules, and the standardized report is pushed to the tenant's business system through the application programming interface.

[0013] Secondly, based on the same inventive concept, this application provides a multi-modal task scheduling system with multi-source device linkage. The system includes: The load assessment module receives algorithm analysis requests, obtains task metadata after verifying the algorithm analysis requests, counts the total number of tasks within the same time window as the algorithm analysis requests, and obtains the hardware indicators of the computing cluster to calculate the comprehensive load value. The intelligent distribution module compares the total number of tasks with preset high and low thresholds, compares the comprehensive load value with preset resource load warning thresholds, and distributes the algorithm analysis request to the target physical computing node or persistent message queue based on the comparison results. The dynamic routing module parses the algorithm analysis requests distributed to the target physical computing node or pulled from the persistent message queue, extracts the device identification field and data stream format field from the task metadata, and routes the task to the regular comparison queue, the high-resource serial queue or the high-priority real-time queue according to the extraction results. The differentiated execution module executes tasks using differentiated strategies corresponding to each of the regular comparison queue, the high-resource serial queue, and the high-priority real-time queue, and obtains the original analysis results for each queue. The report generation module maps the raw analysis results of each queue to a unified alarm data structure template, integrates multi-source alarms within the same monitoring area, generates standardized reports, and pushes them to the tenant's business system.

[0014] Thirdly, this application also provides an electronic device, including at least one processor and at least one memory electrically connected; The memory is electrically connected to the processor, wherein the memory stores instructions that can be executed by at least one of the processors, the instructions being executed by at least one of the processors to enable at least one of the processors to perform any of the multi-source device linkage multimodal task scheduling methods described above.

[0015] Fourthly, this application also provides a computer storage medium, wherein a computer program is stored within the computer-readable storage medium; When the computer program is executed by the processor, it implements any of the multi-modal task scheduling methods described above for multi-source device linkage.

[0016] Fifthly, this application also provides a computer program product, which is stored in at least one storage medium; The computer program product includes several instructions to cause at least one electronic device to execute any of the multi-source device linkage multimodal task scheduling methods described above.

[0017] Compared with the prior art, this application has the following advantages: 1. Through multi-dimensional resource monitoring and dual-threshold traffic distribution mechanism, the comprehensive pressure of heterogeneous computing clusters is quantified in real time and the distribution strategy is dynamically matched according to high, medium and low load scenarios. Specifically, under normal load, consistent hashing is used to ensure routing stability. During high-pressure congestion, buffer queuing is implemented and resource expansion is linked to prevent service avalanche. During idle periods, tasks are aggregated for batch processing to reduce process overhead. This effectively balances the contradiction between the impact of high-concurrency tasks and resource utilization efficiency, avoids the downtime or resource waste caused by uneven load of computing nodes, and thus significantly improves the task processing throughput in multi-tenant environment. 2. By using a dedicated execution engine designed for satellite remote sensing, UAV real-time streaming and fixed monitoring data, and adopting differentiated processing strategies such as geographic sliding window segmentation, ring buffer combined with telemetry time alignment and deep feature vector comparison, the problem of memory overflow caused by large-format satellite imagery and the technical pain point of video frames and telemetry data not being synchronized in UAV image transmission are solved. This ensures that multi-source heterogeneous data can be analyzed in parallel with high precision under a unified architecture, and realizes accurate coordinate back calculation and abnormal status identification of various monitoring targets in geospatial space. 3. By using a spatiotemporal clustering and weighted fusion verification mechanism based on geohashing and time slots, alarm information triggered by different devices in the same area is effectively integrated. Redundant data is merged and low-confidence noise is filtered out by merging spatial, temporal, and category similarities. Based on the fusion results, a linkage rule engine is triggered to automatically dispatch drones to conduct close-range verification of high-priority events discovered by satellites. This constructs an automated closed-loop workflow from wide-area discovery to precise verification, eliminating the risk of false alarms from a single data source and providing users with standardized and highly reliable monitoring reports containing a complete chain of evidence and accurate positioning.

[0018] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a multi-modal task scheduling method for multi-source device linkage according to an embodiment of this application is shown. Figure 2 A detailed flowchart of an embodiment of this application is shown; Figure 3 A flowchart illustrating the task traffic distribution process of an embodiment of this application is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Figure 1 A multimodal task scheduling method for multi-source device linkage according to an embodiment of this application is illustrated. For example... Figure 1 As shown in the figure, the multi-modal task scheduling method for multi-source device linkage in this application includes the following steps: S1, Receive algorithm analysis request, after the algorithm analysis request is verified, obtain task metadata, count the total number of tasks within the same time window as the algorithm analysis request, and obtain the hardware indicators of the computing cluster to calculate the comprehensive load value. S2, compare the total number of tasks with preset high and low thresholds, compare the comprehensive load value with preset resource load warning thresholds, and distribute the algorithm analysis request to the target physical computing node or persistent message queue according to the comparison results. S3, parse the algorithm analysis requests distributed to the target physical computing node or pulled from the persistent message queue, extract the device identification field and data stream format field from the task metadata, and route the task to the regular comparison queue, high resource serial queue or high priority real-time queue according to the extraction results; S4, For the regular comparison queue, the high-resource serial queue and the high-priority real-time queue, the task is executed using a differentiated strategy corresponding to each queue to obtain the original analysis results of each queue; S5 maps the raw analysis results of each queue to a unified alarm data structure template, integrates multi-source alarms within the same monitoring area, generates standardized reports, and pushes them to the tenant's business system.

[0023] In the embodiments of this application, step S1 specifically includes: S11: Multiply the average utilization of the central processing unit, the average utilization of memory, the average utilization of graphics processing unit memory, and the disk I / O throughput utilization of the computing cluster by their respective weighting coefficients and then sum them up in a weighted manner.

[0024] In the specific implementation process, algorithm analysis requests are received from different tenants. The data packets of the algorithm analysis requests contain authentication information and task metadata. First, the authentication information is verified to ensure the legality and validity of the task source and adapt to the business needs of multi-tenants.

[0025] like Figure 2 As shown in the embodiments of this application, step S2 specifically includes: Receive task real-time traffic That is, the total number of tasks within the time window, and the overall load value. Then, the built-in dual-threshold judgment logic is combined with the resource load warning line. The real-time resource operation status of the computing cluster is divided into three scenarios: high-pressure congestion, normal load, and low-load idle, and a differentiated distribution strategy is executed.

[0026] In the embodiments of this application, step S3 specifically includes: S31, if the total number of tasks is greater than the high threshold and the comprehensive load value is greater than the resource load warning threshold, then the algorithm analysis request is serialized and written into the persistent message queue, and the number of gap computing power nodes is calculated by dividing the difference between the total number of tasks and the current maximum concurrent processing capacity of the cluster by the rated processing capacity of a single standard computing node to trigger horizontal scaling. S32, if the total number of tasks is greater than the low threshold and less than or equal to the high threshold, or if the total number of tasks is greater than the high threshold but the comprehensive load value is less than the resource load warning threshold, then extract the device identifier in the task metadata as a hash key, and use a consistent hashing algorithm to find the nearest virtual node clockwise on a preset hash ring to determine the target physical computing node; S33, if the total number of tasks is less than or equal to the low threshold and the comprehensive load value is less than or equal to the resource load warning threshold, then multiple algorithm analysis requests belonging to the same tenant identifier within the time window are encapsulated into a task batch list and sent to a single computing node for batch processing.

[0027] like Figure 2 As shown, in the specific implementation process, under high-pressure congestion scenarios, newly arriving task requests are serialized and written to the buffer topic of a persistent message queue, their execution is temporarily suspended, and a capacity expansion trigger signal is generated. At the same time, an independent consumer thread is started to poll and check the overall load value, waiting for it to drop to a certain level. When the success rate is below 90%, task requests are retrieved in batches from the persistent message queue and re-injected into the distribution process. Under normal load scenarios, a consistent hashing distribution strategy is executed to build a system containing... A hash ring is created at each location, mapping existing compute nodes onto the hash ring and generating a hash ring for each physical compute node. Each virtual node extracts the device ID from the task request as a hash key to calculate the hash value and then searches for the nearest virtual node clockwise to determine the target physical computing node. In low-load, idle scenarios, an aggregation and reuse strategy is executed. A task buffer list protected by a mutex lock is built in memory to temporarily store incoming task requests. When the number of tasks reaches a preset batch size, the aggregation and reuse strategy is implemented. Or the dwell time of the first task exceeds the maximum waiting time. At that time, the tasks in the list are packaged into composite task packages and sent to a single computing node for batch processing.

[0028] In the embodiments of this application, step S4 specifically includes: S411, Obtain remote sensing image data associated with tasks in the high-resource serial queue; S412, Read the header information of the remote sensing image data to obtain the six-parameter affine transformation matrix, and divide the remote sensing image data into blocks according to the set block window size and overlap buffer width; S413, for each data block, call the Siamese neural network change detection model to input the current temporal block and the baseline map block to output a change area mask, extract the center pixel coordinates of the change area in the change area mask, and use the six-parameter affine transformation matrix to convert the center pixel coordinates into geospatial coordinates.

[0029] In the embodiments of this application, step S4 further includes: S421, Use the obtained geospatial coordinates as alarm targets to create a global results list; S422, Calculate the intersection-union ratio of the geographic bounding boxes of any two alarm targets in the global result list; S423, if the intersection-union ratio is greater than a preset threshold, then retain the target with the highest confidence and remove redundant results to perform global nonmaximum suppression.

[0030] In the embodiments of this application, step S4 further includes: S431, Obtain the tasks in the high-priority real-time queue and establish a long connection with the UAV image transmission system, pull the real-time video stream from the UAV image transmission system and decode the real-time video stream to obtain video frames. S432, the video frames are written into a fixed-length circular buffer in timestamp order, and the received flight status telemetry data packets are stored in a time-series database; S433, based on the circular buffer, retrieve two telemetry data points adjacent to the timestamp of the video frame in the time-series database, use linear interpolation to calculate the UAV attitude data corresponding to the video frame based on the time difference weighting, and input it into the target detection model to calculate the estimated latitude and longitude of the ground target in combination with monocular geolocation.

[0031] like Figure 2As shown, in the specific implementation process, independent execution engines are used for different task types, and each execution engine works in parallel; For tasks in the regular comparison queue, the execution engine calls the object storage service interface to obtain the baseline map corresponding to the current monitoring point. The feature vectors of the baseline map and the current image to be tested (or video key frame) are extracted by the feature extraction algorithm, and the feature difference is calculated to identify abnormal targets. For tasks in the high-resource serial queue, since the remote sensing image data is large and contains geospatial information, the execution engine adopts a block reading strategy based on the geospatial library. The task scheduling system allocates independent Docker containers or Kubernetes Pods to the high-resource serial queue and sets exclusive memory limit parameters to perform pixel-by-pixel comparison of the images and extract the surface change areas. For tasks in the high-priority real-time queue, the execution engine establishes a long TCP (Transmission Control Protocol) or WebSocket connection with the UAV image transmission system. The task scheduling system continuously reads real-time video stream data, uses streaming media frame extraction technology to obtain analysis frames, and inputs the UAV's real-time status data (latitude, longitude, altitude, gimbal angle) as parameters into the target detection model, marking alarm boxes on the video frames in real time. In this embodiment, the task scheduling system dynamically activates one or more corresponding execution engines based on the type of the received task request. Specifically, when only a satellite remote sensing image analysis task is received, only the satellite image execution engine is activated, while the UAV streaming media execution engine and the conventional monitoring execution engine are in a dormant or standby state. When a joint task request of "satellite + UAV" is received, the corresponding two engines are started in parallel. This mechanism ensures that it can adapt to any combination of tasks in a single scenario (such as a single UAV inspection) or a complex scenario (such as integrated air-ground-space monitoring), and realizes on-demand allocation of computing resources. When receiving a satellite remote sensing image analysis task, given that remote sensing images usually have ultra-high resolution and contain geographic coordinate information, directly reading them into memory would cause memory overflow. Therefore, the satellite image execution engine adopts a geographic reference-based sliding window block strategy. The satellite imagery execution engine first reads the header information of the remote sensing image file to obtain the image width. ,high and the six-parameter affine transformation matrix The satellite imagery execution engine is set to have a tile window size of [size missing]. And set the overlap buffer width To cover the detection area at the edge of the block; The satellite image execution engine generates a series of read requests according to the set window size. Using virtual memory mapping technology, it loads only the pixel data in the current window into memory. For each block of data, the satellite image execution engine calls the pre-loaded Siamese neural network change detection model, inputs the current temporal block and the baseline base map block, and outputs a mask of the changed area. For the center pixel coordinates of the detected changing region The satellite imagery execution engine uses an affine transformation matrix to convert it into geospatial coordinates. The conversion formula is as follows:

[0032] in, The pixel resolution is in the X direction. The rotation coefficient is in the Y direction. The rotation coefficient is in the X direction. The pixel resolution is in the Y direction. and These are the geographic X and Y coordinates of the top-left pixel, respectively; To remove duplicate alarms caused by overlapping buffers, the satellite imagery execution engine establishes a global result list. After all blocks are processed, the satellite imagery execution engine performs global non-maximum suppression on all targets in the global result list. Specifically, it calculates the intersection-union ratio (IoU) of the geographic bounding boxes of any two alarm targets. If the IoU is greater than a preset threshold (e.g., 0.5), the target with higher confidence is retained, and redundant results are removed, thereby ensuring that the final output alarm information is unique and has accurate latitude and longitude positioning. When receiving a real-time streaming task from a drone, it is necessary to solve the problem of unstable frame rate caused by network jitter and the time synchronization problem between video frames and remote sensing parameters; the drone streaming media execution engine allocates a fixed-length circular buffer in memory; The drone streaming engine retrieves the video stream via a streaming media transport protocol, decodes it into a sequence of raw frames using a decoder, and writes the raw frames into a circular buffer in timestamp order. Simultaneously, the drone streaming engine listens to the data link port to receive flight status telemetry data packets (including the current altitude) transmitted from the drone. Camera pitch angle θ pitch (UAV latitude and longitude), and store the telemetry data in an in-memory time-series database; The inference thread reads the latest frame image and its associated frame generation timestamp from the circular buffer. Because the telemetry data and the video stream are not synchronized in frequency, the drone streaming engine uses linear interpolation to acquire the data. The corresponding UAV attitude data at a given time; specifically: retrieving the timestamp from the time-series database. Two telemetry data points before and after and Calculated based on time difference weighting Estimated attitude data at each moment; The drone streaming execution engine inputs images into a lightweight object detection model; to calculate the actual location of the target in geospatial space, the drone streaming execution engine performs monocular geolocation calculations; assuming the ordinate of the center point of the bottom edge of the detection box in the image coordinate system is... The vertical coordinate of the image center is The vertical field of view is α vfov The horizontal distance of the target relative to the drone The calculation formula is as follows:

[0033] The drone streaming engine combines the drone's own latitude and longitude with the calculated horizontal distance. This allows us to calculate the estimated latitude and longitude of the ground target. When receiving images or short video tasks from fixed surveillance cameras, the conventional monitoring execution engine mainly performs anomaly state comparison; the conventional monitoring execution engine first retrieves the pre-stored baseline background image of the fixed surveillance camera from the object storage system; To improve the robustness of the comparison, the conventional monitoring execution engine employs a deep feature vector comparison method. First, the engine uniformly scales the baseline background image and the current test frame to the input size specified by the feature extraction network (e.g., 224x224 pixels). Then, the engine calls the feature extraction layer of a convolutional neural network to perform forward inference on both the baseline background image and the current test frame, obtaining two high-dimensional feature vectors: the baseline vector and the current test frame. and the current vector ; The standard monitoring execution engine calculates the cosine similarity between two vectors. To quantify the degree of change in the scenario:

[0034] in, The dimension of the feature vector; if the calculated similarity If the change is less than the preset threshold, the scene is determined to be significantly abnormal; the regular monitoring execution engine then inputs the frame to be tested into the target detection model to further identify the specific abnormal object category; The three execution engines work in parallel, producing raw result data with different structures. The result normalization component is responsible for mapping the heterogeneous output data to a unified standard output object. The result normalization component instantiates a standard alert class object and assigns values ​​to each field. Assign the task's unique identifier to TaskID; Assign the absolute time of the alarm occurrence to Timestamp; Assign the converted WGS84 coordinates to GeoLocation; Assign the target category text to TargetClass; Assign the confidence score to Confidence; Assign the storage address of the supporting images or video clips to EvidenceURL; For drone streaming inference, the result normalization component will also add a trajectory point sequence field. After encapsulation, the standard output object is sent to the "processing complete" topic of the message queue for downstream modules to consume.

[0035] In the embodiments of this application, step S5 specifically includes: S51, extract the target location, event category and trigger time from the original analysis results of each queue output, and fill the target location, event category and trigger time into the corresponding fields of the preset unified alarm data structure template to complete the structured encapsulation; S52, based on the integration results of multi-source alarms in the same monitoring area, the packaged alarm data is assembled into a standardized report according to the preset report generation rules, and the standardized report is pushed to the tenant business system through the application programming interface.

[0036] In the specific implementation process, the raw analysis results output by each execution engine in step S4 are collected. These raw analysis results may contain different data structures. First, data normalization is performed. Specifically, a unified alarm data structure template is established, containing fields such as "Event ID," "Timestamp," "Latitude and Longitude Coordinates," "Confidence Level," "Alarm Type," and "Evidence Link." Each raw analysis result is parsed, and the output fields from different devices are filled into the corresponding fields of the data structure template through key-value mapping. Subsequently, the task scheduling system processes the data according to preset priorities. The rules integrate multi-source alarms within the same monitoring area. When alarm information from multiple types of devices is detected within the same spatiotemporal range, the task scheduling system executes alarm fusion logic. For example, let Acam, Asat, and Auav be the alarm event sets from cameras, satellites, and drones, respectively. The final reported alarm set Afinal is then merged and sorted by priority. For alarms in overlapping areas, device alarm information with high confidence or strong data timeliness is prioritized. Finally, the task scheduling system generates a standardized report containing alarm type, occurrence time, geographical coordinates, on-site evidence data, and confidence score, which is pushed to the tenant's business system through a standard RESTful API interface, completing the task scheduling and processing closed loop.

[0037] Based on the same inventive concept, this application also provides a multi-modal task scheduling system with multi-source device linkage corresponding to the above method; The system includes: The load assessment module receives algorithm analysis requests, obtains task metadata after verifying the algorithm analysis requests, counts the total number of tasks within the same time window as the algorithm analysis requests, and obtains the hardware indicators of the computing cluster to calculate the comprehensive load value. The intelligent distribution module compares the total number of tasks with preset high and low thresholds, compares the comprehensive load value with preset resource load warning thresholds, and distributes the algorithm analysis request to the target physical computing node or persistent message queue based on the comparison results. The dynamic routing module parses the algorithm analysis requests distributed to the target physical computing node or pulled from the persistent message queue, extracts the device identification field and data stream format field from the task metadata, and routes the task to the regular comparison queue, the high-resource serial queue or the high-priority real-time queue according to the extraction results. The differentiated execution module executes tasks using differentiated strategies corresponding to each of the regular comparison queue, the high-resource serial queue, and the high-priority real-time queue, and obtains the original analysis results for each queue. The report generation module maps the raw analysis results of each queue to a unified alarm data structure template, integrates multi-source alarms within the same monitoring area, generates standardized reports, and pushes them to the tenant's business system.

[0038] like Figure 3 As shown, the specific operation of the task traffic distribution process during implementation is as follows: Tenant task acceptance and load status monitoring: Receive algorithm analysis requests from different tenants. The data packet of the algorithm analysis request contains authentication information and task metadata. The load assessment module first verifies the authentication information. After the verification is passed, the algorithm analysis request is included in the pending buffer pool. While receiving tasks, the total number of tasks Q arriving within the current time window is counted according to the preset time window Δt. total Simultaneously, current resource load indicators are collected through resident monitoring programs deployed on each computing node. To quantify the current system resource pressure, this invention employs a weighted load calculation model to calculate the current comprehensive load value L. load .

[0039] Overall load value L load The calculation formula is as follows: L load =α×U cpu +β×U mem +γ×U gpu +δ×U io Among them, U cpu U represents the average CPU utilization of the computing cluster. mem Indicates average memory utilization, U gpu This represents the average utilization of GPU memory, U. io This represents disk I / O throughput utilization; α, β, γ, and δ are the corresponding weight coefficients, and satisfy α+β+γ+δ=1; the specific values ​​of the weight coefficients are set according to the characteristics of the currently running algorithm model. For example, for deep learning algorithms that rely on GPU inference, the value of γ is greater than other coefficients; through the above formula, the task scheduling system can abstract multi-dimensional hardware indicators into a unified load status value, providing a quantitative basis for subsequent traffic distribution; Task traffic distribution based on dual thresholds and load status: The intelligent distribution module reads the total number of tasks Q from the statistics. total and comprehensive load value L load Based on the preset high threshold T of the task volume high Low task threshold T low and resource load warning threshold L limit The distribution strategy is executed; the specific distribution logic includes the following sub-steps: Determine the total current task quantity Qtotal Is it greater than the high threshold T? high And the overall load value L load Is it greater than the resource load warning threshold L? limit If both conditions are met, the system is determined to be in a high-load congestion state. The intelligent distribution module will push the task into the global waiting queue and send a horizontal expansion command to the dynamic routing module. To ensure the accuracy of the scaling, the number of missing computing nodes, Nadd, in the horizontal scaling instruction is calculated using the following formula:

[0040] Among them, Q capacity P represents the maximum concurrent processing capacity of the current cluster. node The rated processing capacity of a single standard computing node. This indicates rounding up; the dynamic routing module rounds up based on N. add The value is used to call the cloud platform interface to start the corresponding number of virtual machine instances, and the task will be distributed after the expansion is completed. If the above conditions are not met, continue to check whether the total task amount Q is satisfied. total At low threshold T low With high threshold T high "between" or "total task Q" total Greater than T high But the overall load value L load Less than L limit If any of the above conditions are met, it is determined that the current state is under normal load. The intelligent distribution module will directly distribute the task to the existing computing nodes for processing according to the weighted round-robin algorithm. If the above conditions are not met, continue to determine the total current task quantity Q. total Is it less than or equal to the low threshold T? low And the overall load value L load If the overall load value is less than or equal to the resource load warning threshold, the system is considered to be in a low-load idle state. To improve resource utilization, the intelligent distribution module executes a task merging strategy. Specifically, the task merging strategy is as follows: multiple task requests belonging to the same tenant ID are encapsulated into a task batch list in the buffer pool. This task batch list is sent as a whole scheduling unit to the computing node in the idle state. The computing node processes the tasks in the list sequentially in a single process startup to reduce the overhead of frequent process creation and destruction. Multi-device, multi-modal differentiated execution: The differentiated execution module includes independent execution engines for different task types, and each execution engine works in parallel. For tasks in the regular comparison queue, the execution engine calls the object storage service interface to obtain the baseline map corresponding to the current monitoring point. Feature vectors of the baseline map and the current image to be tested (or video keyframe) are extracted separately through feature extraction algorithms, and feature differences are calculated to identify abnormal targets. For tasks in the high-resource serial queue, due to the large amount of remote sensing image data and the presence of geospatial information, the execution engine adopts a block-based reading strategy based on a geospatial library (such as GDAL). The task scheduling system allocates independent Docker containers or Kubernetes Pods to the high-resource serial queue and sets exclusive memory limit parameters to perform pixel-by-pixel comparison of the images and extract the surface change areas. For tasks in the high-priority real-time queue, the execution engine establishes a long TCP or WebSocket connection with the UAV image transmission system; the task scheduling system continuously reads real-time video stream data, uses streaming media frame extraction technology to obtain analysis frames, and inputs the UAV's real-time status data (latitude, longitude, altitude, gimbal angle) as parameters into the target detection model, and annotates alarm boxes on the video frames in real time.

[0041] Based on the same inventive concept, this application also provides an electronic device. The electronic device of this application includes at least one processor and at least one memory electrically connected to each other. The memory is electrically connected to the processor, wherein the memory stores instructions executable by the at least one processor. These instructions are executed by the at least one processor to enable the at least one processor to perform the multi-modal task scheduling method for multi-source device linkage as described above.

[0042] It should be noted that the electrical connections between the various units mentioned above do not necessarily represent the connections between lines. Any indirect connection method can be applied to the embodiments of this application as long as it achieves the purpose of this application.

[0043] Based on the same inventive concept, this application also provides a computer storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it implements the multi-modal task scheduling method for multi-source device linkage as described above.

[0044] Based on the same inventive concept, this application also provides a computer program product, which is stored in at least one storage medium; the computer program product includes several instructions to cause at least one computer device to execute the multi-source device linkage multimodal task scheduling method described above.

[0045] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal task scheduling method for multi-source device linkage, characterized in that, Upon receiving an algorithm analysis request, after verifying the algorithm analysis request, the task metadata is obtained, the total number of tasks within the same time window as the algorithm analysis request is counted, and the hardware indicators of the computing cluster are obtained to calculate the comprehensive load value. The total number of tasks is compared with preset high and low thresholds, the comprehensive load value is compared with preset resource load warning thresholds, and the algorithm analysis request is distributed to the target physical computing node or persistent message queue according to the comparison results. The algorithm analysis requests distributed to the target physical computing node or pulled from the persistent message queue are parsed, and the device identification field and data stream format field are extracted from the task metadata. Based on the extraction results, the task is routed to the regular comparison queue, the high-resource serial queue, or the high-priority real-time queue. For the regular comparison queue, the high-resource serial queue, and the high-priority real-time queue, a differentiated strategy corresponding to each queue is adopted to execute the task and obtain the original analysis results of each queue. The raw analysis results of each queue are mapped to a unified alarm data structure template, and multi-source alarms within the same monitoring area are integrated to generate standardized reports and push them to the tenant's business system.

2. The method according to claim 1, characterized in that, The process of obtaining the hardware metrics of the computing cluster to calculate the overall load value specifically includes: The average utilization of the central processing unit, average memory utilization, average graphics processing unit memory utilization, and disk I / O throughput utilization of the computing cluster are multiplied by their respective weighting coefficients and then summed in a weighted manner.

3. The method according to claim 1, characterized in that, The step of distributing the algorithm analysis request to the target physical computing node or persistent message queue based on the comparison results specifically includes: If the total number of tasks is greater than the high threshold and the comprehensive load value is greater than the resource load warning threshold, the algorithm analysis request is serialized and written into the persistent message queue, and the number of gap computing power nodes is calculated by dividing the difference between the total number of tasks and the current maximum concurrent processing capacity of the cluster by the rated processing capacity of a single standard computing node to trigger horizontal scaling. If the total number of tasks is greater than the low threshold and less than or equal to the high threshold, or if the total number of tasks is greater than the high threshold but the comprehensive load value is less than the resource load warning threshold, then the device identifier in the task metadata is extracted as a hash key, and the nearest virtual node is found clockwise on the preset hash ring using a consistent hash algorithm to determine the target physical computing node. If the total number of tasks is less than or equal to the low threshold and the comprehensive load value is less than or equal to the resource load warning threshold, then multiple algorithm analysis requests belonging to the same tenant identifier within the time window are encapsulated into a task batch list and sent to a single computing node for batch processing.

4. The method according to claim 1, characterized in that, For the high-resource serial queue, a differentiated strategy corresponding to each queue is adopted to execute tasks, specifically including: Acquire the remote sensing image data associated with the tasks in the high-resource serial queue; The header information of the remote sensing image data is read to obtain a six-parameter affine transformation matrix, and the remote sensing image data is divided into blocks according to the set block window size and overlap buffer width. For each data block, the Siamese neural network change detection model is called, inputting the current temporal block and the baseline base map block to output a change region mask. The center pixel coordinates of the change region in the change region mask are extracted, and the center pixel coordinates are converted into geospatial coordinates using the six-parameter affine transformation matrix.

5. The method according to claim 4, characterized in that, For the high-resource serial queue, a differentiated strategy corresponding to each queue is adopted to execute tasks, which specifically includes: Use the obtained geospatial coordinates as the alarm targets to create a global results list; Calculate the intersection-union ratio (IUU) of the geographic bounding boxes of any two alarm targets in the global results list; If the crossover-union ratio is greater than a preset threshold, the target with the highest confidence is retained and redundant results are eliminated to perform global nonmaximum suppression.

6. The method according to claim 1, characterized in that, For the high-priority real-time queues, differentiated strategies corresponding to each queue are used to execute tasks, specifically including: The task in the high-priority real-time queue is obtained and a long connection is established with the UAV image transmission system. The real-time video stream is pulled from the UAV image transmission system and the real-time video stream is decoded to obtain video frames. The video frames are written into a fixed-length circular buffer in timestamp order, and the received flight status telemetry data packets are stored in a time-series database. Based on the circular buffer, two telemetry data points adjacent to the timestamp of the video frame are retrieved from the time-series database. The UAV attitude data corresponding to the video frame is calculated by weighting the time difference using linear interpolation and input into the target detection model. Combined with monocular geolocation, the estimated latitude and longitude of the ground target are calculated.

7. The method according to claim 1, characterized in that, For the aforementioned regular comparison queues, a differentiated strategy corresponding to each queue is employed to perform the task, specifically including: The task in the conventional comparison queue is analyzed to obtain the storage address of the baseline background image and the address of the real-time video stream; According to the reference background image storage address, the reference background image corresponding to the monitoring point is retrieved from the object storage system, and the current video frame is obtained as the current frame to be tested according to the real-time video stream address. The reference background image and the current frame to be tested are scaled to the input size specified by the feature extraction network and then input into the feature extraction layer of the convolutional neural network for forward inference to obtain the reference vector and the current vector. Calculate the cosine similarity between two vectors. If the cosine similarity is less than a preset change threshold, input the current test frame into the target detection model to identify the abnormal object category.

8. The method according to claim 1, characterized in that, The integration of multi-source alarms within the same monitoring area specifically includes: Extract the target location, event category, and trigger time from the original analysis results of each queue output, and fill the target location, event category, and trigger time into the corresponding fields of the preset unified alarm data structure template to complete the structured encapsulation. Based on the integration results of multi-source alarms within the same monitoring area, the encapsulated alarm data is assembled into a standardized report according to the preset report generation rules, and the standardized report is pushed to the tenant's business system through the application programming interface.

9. A multi-modal task scheduling system with multi-source device linkage, characterized in that, The system includes: The load assessment module receives algorithm analysis requests, obtains task metadata after verifying the algorithm analysis requests, counts the total number of tasks within the same time window as the algorithm analysis requests, and obtains the hardware indicators of the computing cluster to calculate the comprehensive load value. The intelligent distribution module compares the total number of tasks with preset high and low thresholds, compares the comprehensive load value with preset resource load warning thresholds, and distributes the algorithm analysis request to the target physical computing node or persistent message queue based on the comparison results. The dynamic routing module parses the algorithm analysis requests distributed to the target physical computing node or pulled from the persistent message queue, extracts the device identification field and data stream format field from the task metadata, and routes the task to the regular comparison queue, the high-resource serial queue or the high-priority real-time queue according to the extraction results. The differentiated execution module executes tasks using differentiated strategies corresponding to each of the regular comparison queue, the high-resource serial queue, and the high-priority real-time queue, and obtains the original analysis results for each queue. The report generation module maps the raw analysis results of each queue to a unified alarm data structure template, integrates multi-source alarms within the same monitoring area, generates standardized reports, and pushes them to the tenant's business system.

10. An electronic device, characterized in that, Includes at least one processor and at least one memory electrically connected; The memory is electrically connected to the processor, wherein the memory stores instructions that can be executed by at least one of the processors, the instructions being executed by at least one of the processors to enable at least one of the processors to perform the multi-modal task scheduling method for multi-source device linkage as described in any one of claims 1-8.

11. A computer storage medium, characterized in that, The computer-readable storage medium stores a computer program. When the computer program is executed by the processor, it implements the multimodal task scheduling method for multi-source device linkage as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product is stored in at least one storage medium; The computer program product includes several instructions for causing at least one electronic device to execute the multi-modal task scheduling method for multi-source device linkage as described in any one of claims 1-8.

Images