An emergency video intelligent monitoring system based on multi-source data fusion

Abstract

The present invention relates to the field of emergency video monitoring technology, and specifically to an emergency video intelligent monitoring system based on multi-source data fusion, comprising: a data sensing unit, the data sensing unit is used to sense data based on a plurality of pre-set client terminals to obtain multi-source data; a prioritization unit, the prioritization unit is used to prioritize the multi-source data based on the importance of the client terminal and the bandwidth consumption of the multi-source data through a pre-set edge center node to obtain a data priority queue. The present invention prioritizes the multi-source data through the prioritization unit, and can give priority to the transmission of key data according to the bandwidth consumption and importance of the data. This enables the system to ensure that important data is transmitted in a timely manner when the bandwidth is limited, thereby effectively improving the efficiency of data transmission.

Description

Technical Field

[0001] The present invention relates to the technical field of emergency video monitoring, and in particular to an emergency video intelligent monitoring system based on multi-source data fusion. Background Art

[0002] When dealing with emergencies, many departments need to coordinate and communicate, so the communication systems involved are scattered, such as video conferencing systems, video surveillance systems, cluster communication systems, public telephone systems and other communication systems. It is often difficult to integrate communication resources. The command center is unable to uniformly dispatch various audio and video resources, resulting in a lack of interconnection means in important links of the emergency response process. The command form is single, and the existing network facilities are insufficient to support the transmission of hundreds of high-definition videos from each branch station to the centralized control center. The command and dispatch center can only review and store videos of important parts of each dispersed station, resulting in the command and dispatch center's emergency dispatch and post-event analysis capabilities for each branch station being greatly weakened.

[0003] When bandwidth is limited, traditional video surveillance systems usually transmit all data at a fixed rate without any priority division, which may lead to a waste of bandwidth resources. Especially when network conditions are poor, the timely transmission of important data cannot be guaranteed, affecting the response speed at critical moments. In addition, traditional systems usually use static transmission strategies and cannot be dynamically adjusted according to changes in the network environment or real-time needs. Whether it is changes in network load or the type and urgency of the system, traditional systems may not be able to adapt flexibly, resulting in data transmission delays or losses. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide an emergency video intelligent monitoring system based on multi-source data fusion.

[0005] The technical solution adopted to solve the above technical problems is: an emergency video intelligent monitoring system based on multi-source data fusion, including:

[0006] A data sensing unit, the data sensing unit is used to perform data sensing based on a plurality of pre-set client terminals to obtain multi-source data;

[0007] A prioritization unit, the prioritization unit is used to prioritize the multi-source data based on the importance of the client terminal and the bandwidth consumption of the multi-source data through a preset edge center node to obtain a data priority queue;

[0008] A transmission optimization unit, the transmission optimization unit is used to perform transmission optimization modeling based on the data priority queue through the preset edge center node to obtain a Markov decision process for transmission optimization, and solve the Markov decision process based on deep reinforcement learning to obtain a data transmission strategy;

[0009] A data transmission unit, the data transmission unit being used to sequentially transmit the multi-source data to a pre-built edge computing network through the pre-set edge center node based on the data transmission strategy, wherein the edge computing network includes a plurality of edge computing nodes;

[0010] An edge service unit, the edge service unit being used to compress and store the multi-source data through the pre-built edge computing network to obtain multi-source compressed data;

[0011] A cloud service unit, wherein the cloud service unit is used to obtain multi-source compressed data corresponding to the target scheduling task from the edge computing network based on the received target scheduling task, and store and display the multi-source compressed data.

[0012] Preferably, the multi-source data is prioritized based on the importance of the client terminal and the bandwidth consumption of the multi-source data to obtain a data priority queue, including:

[0013] The importance of the multi-source data is defined based on the importance of the client terminal, wherein the definition expression of the importance of the multi-source data is as follows:

[0014] ;

[0015] in, Indicated in Time The importance of multi-source data uploaded by each client terminal, Indicated in Time The click prevalence of multi-source data uploaded by client terminals, Indicated in Time The key area attention rate of multi-source data uploaded by client terminals, Indicated in Time Target information monitoring rate of multi-source data uploaded by client terminals;

[0016] Grouping the multi-source data based on the importance of the multi-source data to divide the multi-source data into different data sets, wherein the different data sets include a primary importance set, a secondary importance set, and a tertiary importance set;

[0017] The importance of the client terminal and the bandwidth consumption of the multi-source data construct a data priority queue, wherein the definition expression of the data priority queue is as follows:

[0018] ;

[0019] in, express The first in the importance set Data priority queue for multi-source data uploaded by client terminals, Indicates The bandwidth consumption of multi-source data uploaded by each client terminal, Indicates The transmission smoothness of multi-source data uploaded by each client terminal, and , Indicates that the edge center node provides The bit rate of each client terminal, Indicates The maximum network rate at which a client terminal uploads multi-source data.

[0020] Preferably, transmission optimization modeling is performed based on the data priority queue to obtain a Markov decision process for transmission optimization, including:

[0021] The state variables of the Markov decision process are defined based on the data priority queue, wherein the definition expression of the state variables is as follows:

[0022] ;

[0023] in, represents the state variable of the Markov decision process, Indicates the playback delay of multi-source data, and , Indicates the buffering time of multi-source data. represents the queuing delay of multi-source data, Indicates the quality of data provided by the edge center node to the client terminal, and , Represents the transmission cost of multi-source data;

[0024] The action variable of the Markov decision process is defined based on the data priority queue, wherein the definition expression of the action variable is as follows:

[0025] ;

[0026] in, represents the action variable of the Markov decision process, Indicates The level of importance set of data priority queues of multi-source data uploaded by a client terminal;

[0027] The Markov decision process reward function is defined based on the data priority queue, wherein the definition expression of the reward function is as follows:

[0028] ;

[0029] in, represents the reward function of the Markov decision process, represents the multi-source data transmission accuracy, and , represents the number of decisions for multi-source data transmission, represents the indicator function, and , represents the three-level importance set, Represents the first-level importance set and the second-level importance set.

[0030] Preferably, sequentially transmitting the multi-source data to a pre-built edge computing network based on the data transmission strategy includes:

[0031] The calculation delay of the edge computing node executing the multi-source data compression task is calculated in sequence by the preset edge center node, wherein the calculation formula of the calculation delay of the multi-source data compression task is as follows:

[0032] ;

[0033] in, Indicated in At time t The computational delay of edge computing nodes performing multi-source data compression tasks, Indicates the edge computing network The number of local tasks in the multi-source data compression task performed by the edge computing node, Indicated in At time t The edge computing node performs the multi-source data compression task. The computation delay of local tasks, and , Indicated in Time The edge computing node performs the multi-source data compression task. The computational requirements of a local task, Represents the computing resources of the edge computing node;

[0034] The transmission delay of multi-source data from the edge center node to the edge computing node in the edge computing network is calculated in sequence by the preset edge center node, wherein the calculation formula of the transmission delay is as follows:

[0035] ;

[0036] in, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission delay of edge computing nodes, Indicated in At time t The amount of data output by the layer, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission rate of edge computing nodes, and , Indicated in At time t The available bandwidth of edge computing nodes, Indicated in At time t The channel gain of the edge computing node, Indicated in At time t The upload power of edge computing nodes, represents the noise power spectral density;

[0037] The download delay of multi-source data from the edge center node to the edge computing node in the edge computing network is calculated in sequence by the preset edge center node, wherein the calculation formula of the download delay is as follows:

[0038] ;

[0039] in, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The download delay of edge computing nodes, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The download rate of edge computing nodes, and , Indicated in At time t Download power of edge computing nodes.

[0040] Preferably, sequentially transmitting the multi-source data to a pre-built edge computing network based on the data transmission strategy further includes:

[0041] The computing energy consumption of the edge computing node for executing the multi-source data compression task is calculated in sequence by the preset edge center node, wherein the calculation formula of the computing energy consumption is as follows:

[0042] ;

[0043] in, Indicated in At time t The computing energy consumption of edge computing nodes performing multi-source data compression tasks, Indicated in At time t The computing power consumption of an edge computing node performing multi-source data compression tasks;

[0044] The transmission energy consumption of multi-source data from the edge center node to the edge computing nodes in the edge computing network is calculated in sequence by the preset edge center node, wherein the calculation formula of the transmission energy consumption is as follows:

[0045] ;

[0046] in, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission energy consumption of edge computing nodes, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission power of edge computing nodes, Indicates the preset time threshold.

[0047] Preferably, sequentially transmitting the multi-source data to a pre-built edge computing network based on the data transmission strategy further includes:

[0048] The total delay of executing the multi-source data compression task is calculated based on the calculation delay, the transmission delay and the download delay, wherein the calculation formula of the total delay is as follows:

[0049] ;

[0050] in, Indicated in At time t The total delay of edge computing nodes in performing multi-source data compression tasks;

[0051] The total energy consumption of executing the multi-source data compression task is calculated based on the computing energy consumption and the transmission energy consumption, wherein the calculation formula of the total energy consumption is as follows:

[0052] ;

[0053] in, Indicated in At time t The total energy consumption of edge computing nodes performing multi-source data compression tasks;

[0054] Based on the minimization of the total delay and the total energy consumption as the objective function, based on the maximum and minimum values ​​of the total energy consumption and the total delay as constraints, a task optimization model is constructed based on the objective function and the constraints, and the task optimization model is solved based on the optimization algorithm to obtain an edge computing node scheduling strategy.

[0055] Preferably, compressing and storing the multi-source data through the pre-built edge computing network to obtain multi-source compressed data includes:

[0056] The multi-source data is processed based on the trained foreground extraction network to obtain a foreground detection box and an instance mask set, wherein the foreground detection box and the instance mask are expressed as follows:

[0057] ;

[0058] in, Indicates the first The set of target detection boxes of the frame, and , Indicates the first Frame No. A detection frame, Indicates the first The set of instance masks for the frame, and , Indicates the first Frame No. instance masks;

[0059] All foreground detection frames in the multi-source data are tracked to obtain the motion trajectory of each foreground. The trajectory set of all trajectories is recorded as , obtaining the matching relationship between each foreground and the upper and lower frames based on the trajectory set, and calculating the intersection-over-union ratio of each foreground detection frame between two adjacent frames in the multi-source data;

[0060] If the IoU is greater than the preset IoU threshold, the instance masks of the two foregrounds are aligned, and the two instance masks are XORed, and the area of ​​the XOR result is compared with the area of ​​the instance mask of the previous frame to obtain a difference value, wherein the calculation formula of the difference value is as follows:

[0061] ;

[0062] in, represents the difference value, and represents a pair of instance masks, Indicates The transmission smoothness of multi-source data uploaded by each client terminal, Represents the exclusive OR operation;

[0063] If the difference value is less than a preset difference value threshold, the corresponding target detection frame and instance mask are removed from the foreground extraction result to obtain a new foreground detection frame and instance mask set.

[0064] Preferably, compressing and storing the multi-source data through the pre-built edge computing network to obtain multi-source compressed data also includes:

[0065] Using the foreground detection frame and instance mask set of each frame in the multi-source data, each foreground is intercepted from the video frame corresponding to the multi-source data as a foreground image;

[0066] A foreground image is stored in the trajectory of each foreground target to retain the original color and texture features of the foreground target, and the remaining foreground images are compressed to obtain a foreground detection frame and a foreground image set, and the foreground detection frame and the foreground image set are the compressed data of the foreground target.

[0067] The beneficial effects of the present invention are as follows: (1) The present invention prioritizes multi-source data through a prioritization unit, and can give priority to transmitting key data based on the bandwidth consumption and importance of the data. This enables the system to ensure that important data is transmitted in a timely manner when bandwidth is limited, thereby effectively improving data transmission efficiency. The transmission optimization unit performs transmission optimization modeling on the data, and combines deep reinforcement learning to solve the Markov decision process, which can dynamically adjust the data transmission strategy to ensure the optimal decision during the data transmission process, thereby minimizing data loss or delay; (2) The present invention uses an edge computing network for data transmission and compressed storage, which can reduce the need to directly transmit large amounts of data to the cloud. The edge computing node performs preliminary processing and compressed storage of multi-source data, which can effectively reduce the amount of data transmitted, thereby reducing network load and avoiding bandwidth bottlenecks. , and the edge service unit is responsible for compressing and storing multi-source data, reducing the storage space requirements and transmission burden of data, and improving the efficiency of system processing and transmission; (3) The present invention can respond to emergency situations in real time through multi-source data fusion and intelligent optimization based on edge computing. The system can not only quickly obtain data from multiple client terminals, but also optimize transmission according to preset strategies, and promptly transmit key data to cloud services for further processing and display, thereby improving the real-time nature of emergency monitoring. By optimizing the transmission strategy through deep reinforcement learning, the system can dynamically adapt to different network environments and load conditions, and automatically adjust the transmission strategy, thereby improving the intelligence and adaptability of decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A schematic diagram of the system architecture of an overall system in an embodiment of the present invention.

[0069] Figure numerals: 1. Data perception unit; 2. Prioritization unit; 3. Transmission optimization unit; 4. Data transmission unit; 5. Edge service unit; 6. Cloud service unit. DETAILED DESCRIPTION

[0070] Embodiment 1, as Figure 1 As shown, the present invention proposes an emergency video intelligent monitoring system based on multi-source data fusion, comprising:

[0071] The data sensing unit 1 is used to sense data based on a plurality of pre-set client terminals to obtain multi-source data;

[0072] The prioritization unit 2 is used to prioritize the multi-source data based on the importance of the client terminal and the bandwidth consumption of the multi-source data through a preset edge center node to obtain a data priority queue;

[0073] The transmission optimization unit 3 is used to perform transmission optimization modeling based on the data priority queue through the pre-set edge center node to obtain a Markov decision process for transmission optimization, and solve the Markov decision process based on deep reinforcement learning to obtain a data transmission strategy;

[0074] The data transmission unit 4 is used to sequentially transmit multi-source data to a pre-built edge computing network through a pre-set edge center node based on a data transmission strategy, wherein the edge computing network includes a plurality of edge computing nodes;

[0075] The edge service unit 5 is used to compress and store multi-source data through a pre-built edge computing network to obtain multi-source compressed data;

[0076] The cloud service unit 6 is used to obtain multi-source compressed data corresponding to the target scheduling task from the edge computing network based on the received target scheduling task, and store and display the multi-source compressed data.

[0077] In the present invention, multiple client terminals include but are not limited to video conferencing systems, video surveillance systems, cluster communication systems, public network telephone systems and other communication systems; multi-source data includes video data uploaded by multiple client terminals; Markov decision process (MDP) is a mathematical model used to represent how to choose the optimal action under different states in the framework of reinforcement learning and decision-making process. The process takes into account the current state, the selected action and the reward or cost it brings; the edge center node is usually located close to the data source or user end and is responsible for processing and forwarding data; the edge computing network refers to the computing nodes distributed at the edge of the network, which are responsible for processing data closer to the user or data source to reduce the burden on the cloud and reduce latency; compressed storage is to compress and store data to save storage space, especially when bandwidth or storage is limited. The storage in edge computing network refers to the data storage on edge computing nodes. Compared with traditional centralized cloud storage, edge storage can process local data faster and reduce delay. The target scheduling task is a task scheduling system that arranges tasks in edge computing nodes by receiving scheduling requests. The target scheduling task may refer to transmission. The cloud server stores the data obtained from the edge computing nodes and provides a visual display to help users view and analyze the data. The data transmission strategy refers to the order, path and method of scheduling data transmission by prioritizing and optimizing data transmission in the process of multi-source data transmission. It aims to optimize the data transmission process under limited bandwidth, computing resources and time constraints. It usually relies on Markov decision processes and reinforcement learning methods to dynamically adjust the strategy. Its goal is to ensure that key data is prioritized, reduce transmission delays and energy consumption, and ensure data integrity and system efficiency.

[0078] Embodiment 2, an emergency video intelligent monitoring system based on multi-source data fusion proposed by the present invention, compared with embodiment 1, this embodiment also includes: prioritizing multi-source data based on the importance of the client terminal and the bandwidth consumption of the multi-source data to obtain a data priority queue, including:

[0079] The importance of multi-source data is defined based on the importance of the client terminal, where the definition expression of the importance of multi-source data is as follows:

[0080] ;

[0081] in, Indicated in Time The importance of multi-source data uploaded by each client terminal, Indicated in Time The click prevalence of multi-source data uploaded by client terminals, Indicated in Time The key area attention rate of multi-source data uploaded by client terminals, Indicated in Time Target information monitoring rate of multi-source data uploaded by client terminals;

[0082] The multi-source data are grouped based on the importance of the multi-source data to divide the multi-source data into different data sets, wherein the different data sets include a primary importance set, a secondary importance set, and a tertiary importance set;

[0083] The importance of the client terminal and the bandwidth consumption of multi-source data build a data priority queue, where the definition expression of the data priority queue is as follows:

[0084] ;

[0085] in, express The first in the importance set Data priority queue for multi-source data uploaded by client terminals, Indicates The bandwidth consumption of multi-source data uploaded by each client terminal, Indicates The transmission smoothness of multi-source data uploaded by each client terminal, and , Indicates that the edge center node provides The bit rate of each client terminal, Indicates The maximum network rate at which a client terminal uploads multi-source data.

[0086] In an optional embodiment, transmission optimization modeling is performed based on the data priority queue to obtain a Markov decision process for transmission optimization, including:

[0087] The state variables of the Markov decision process are defined based on the data priority queue, where the definition expression of the state variables is as follows:

[0088] ;

[0089] in, represents the state variable of the Markov decision process, Indicates the playback delay of multi-source data, and , Indicates the buffering time of multi-source data. represents the queuing delay of multi-source data, Indicates the quality of data provided by the edge center node to the client terminal, and , Represents the transmission cost of multi-source data;

[0090] The action variables of the Markov decision process are defined based on the data priority queue, where the definition expression of the action variable is as follows:

[0091] ;

[0092] in, represents the action variable of the Markov decision process, Indicates The level of importance set of data priority queues of multi-source data uploaded by a client terminal;

[0093] The Markov decision process reward function is defined based on the data priority queue, where the definition expression of the reward function is as follows:

[0094] ;

[0095] in, represents the reward function of the Markov decision process, represents the multi-source data transmission accuracy, and , represents the number of decisions for multi-source data transmission, represents the indicator function, and , represents the three-level importance set, Represents the first-level importance set and the second-level importance set.

[0096] In an optional embodiment, transmitting multi-source data to a pre-built edge computing network in sequence based on a data transmission strategy includes:

[0097] The calculation delay of the edge computing node executing the multi-source data compression task is calculated in sequence by the pre-set edge center node, wherein the calculation formula of the calculation delay of the multi-source data compression task is as follows:

[0098] ;

[0099] in, Indicated in At time t The computational delay of edge computing nodes performing multi-source data compression tasks, Indicates the edge computing network The number of local tasks in the multi-source data compression task performed by the edge computing node, Indicated in At time t The edge computing node performs the multi-source data compression task. The computation delay of local tasks, and , Indicated in Time The edge computing node performs the multi-source data compression task. The computational requirements of a local task, Represents the computing resources of the edge computing node;

[0100] The transmission delay of multi-source data from the edge center node to the edge computing node in the edge computing network is calculated in sequence through the pre-set edge center node, where the calculation formula of the transmission delay is as follows:

[0101] ;

[0102] in, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission delay of edge computing nodes, Indicated in At time t The amount of data output by the layer, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission rate of edge computing nodes, and , Indicated in At time t The available bandwidth of edge computing nodes, Indicated in At time t The channel gain of the edge computing node, Indicated in At time t The upload power of edge computing nodes, represents the noise power spectral density;

[0103] The download delay of multi-source data from the edge center node to the edge computing node in the edge computing network is calculated in sequence through the pre-set edge center node, where the calculation formula of the download delay is as follows:

[0104] ;

[0105] in, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The download delay of edge computing nodes, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The download rate of edge computing nodes, and , Indicated in At time t Download power of edge computing nodes.

[0106] In an optional embodiment, transmitting multi-source data to a pre-built edge computing network in sequence based on a data transmission strategy further includes:

[0107] The computing energy consumption of edge computing nodes performing multi-source data compression tasks is calculated in sequence through the pre-set edge center node, where the calculation formula for computing energy consumption is as follows:

[0108] ;

[0109] in, Indicated in At time t The computing energy consumption of edge computing nodes performing multi-source data compression tasks, Indicated in At time t The computing power consumption of an edge computing node performing multi-source data compression tasks;

[0110] The transmission energy consumption of multi-source data from the edge center node to the edge computing node in the edge computing network is calculated in sequence through the pre-set edge center node, where the calculation formula of the transmission energy consumption is as follows:

[0111] ;

[0112] in, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission energy consumption of edge computing nodes, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission power of edge computing nodes, Indicates the preset time threshold.

[0113] In an optional embodiment, transmitting multi-source data to a pre-built edge computing network in sequence based on a data transmission strategy further includes:

[0114] The total delay of executing the multi-source data compression task is calculated based on the calculation delay, transmission delay and download delay, where the calculation formula of the total delay is as follows:

[0115] ;

[0116] in, Indicated in At time t The total delay of edge computing nodes in performing multi-source data compression tasks;

[0117] The total energy consumption of executing the multi-source data compression task is calculated based on the computing energy consumption and the transmission energy consumption, where the total energy consumption is calculated as follows:

[0118] ;

[0119] in, Indicated in At time t The total energy consumption of edge computing nodes performing multi-source data compression tasks;

[0120] Based on the minimization of total delay and total energy consumption as the objective function, and the maximum and minimum values ​​of total energy consumption and total delay as constraints, a task optimization model is constructed based on the objective function and constraints, and the task optimization model is solved based on the optimization algorithm to obtain the edge computing node scheduling strategy.

[0121] It should be noted that the core goal of the edge computing node scheduling strategy is to optimize the performance of the entire multi-source data processing process while meeting the constraints of delay and energy consumption by reasonably allocating computing resources, network bandwidth, and transmission power. By solving the task optimization model, a specific scheduling plan can be obtained to achieve the best system performance.

[0122] In an optional embodiment, multi-source data is compressed and stored through a pre-built edge computing network to obtain multi-source compressed data, including:

[0123] Based on the trained foreground extraction network, multi-source data is processed to obtain a set of foreground detection boxes and instance masks, where the expressions of the foreground detection boxes and instance masks are as follows:

[0124] ;

[0125] in, Indicates the first The set of target detection boxes of the frame, and , Indicates the first Frame No. A detection frame, Indicates the first The set of instance masks for the frame, and , Indicates the first Frame No. instance masks;

[0126] All foreground detection boxes in multi-source data are tracked to obtain the motion trajectory of each foreground. The trajectory set of all trajectories is recorded as , based on the trajectory set, obtain the matching relationship between each foreground and the upper and lower frames, and calculate the intersection-over-union ratio of each foreground detection box between two adjacent frames in the multi-source data;

[0127] If the IoU is greater than the preset IoU threshold, the instance masks of the two foregrounds are aligned, and the two instance masks are XORed, and the area of ​​the XOR result is compared with the area of ​​the instance mask of the previous frame to obtain the difference value, where the calculation formula of the difference value is as follows:

[0128] ;

[0129] in, represents the difference value, and represents a pair of instance masks, Indicates The transmission smoothness of multi-source data uploaded by each client terminal, Represents the exclusive OR operation;

[0130] If the difference value is less than the preset difference value threshold, the corresponding target detection box and instance mask are removed from the foreground extraction result to obtain a new foreground detection box and instance mask set.

[0131] It should be noted that the foreground extraction network is a trained deep learning model, which is usually used to extract foreground objects of interest from videos or image sequences. In multi-source data, these foreground objects usually refer to moving objects or important areas in images or videos; the foreground detection box (also called a bounding box or target box) is a rectangular box used to surround the target object in an image or video frame and identify its position in the image; the instance mask is a binary image used to represent the shape and area of ​​a specific target object in the image. Each target object usually corresponds to a mask, whose value is 1 for the area of ​​the object and 0 for the background area; the motion trajectory refers to the trajectory of the position change of a target object in consecutive video frames or image frames, which is usually defined by the moving path of the target detection frame.

[0132] In an optional embodiment, compressing and storing multi-source data through a pre-built edge computing network to obtain multi-source compressed data also includes:

[0133] Using the foreground detection frame and instance mask set of each frame in the multi-source data, each foreground is intercepted from the video frame corresponding to the multi-source data as a foreground image;

[0134] A foreground image is stored in the trajectory of each foreground target to retain the original color and texture features of the foreground target, and the remaining foreground images are compressed to obtain a foreground detection frame and a foreground image set, and the foreground detection frame and the foreground image set are the compressed data of the foreground target.

[0135] It should be noted that color features refer to the color information of pixels in an image, while texture features describe the details of the image surface (such as roughness, smoothness, etc.). These features can be extracted through image processing algorithms.

[0136] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. An emergency video intelligent monitoring system based on multi-source data fusion, characterized in that: include: A data sensing unit (1), the data sensing unit (1) is used to perform data sensing based on a plurality of pre-set client terminals to obtain multi-source data; A prioritization unit (2), the prioritization unit (2) being used to prioritize the multi-source data based on the importance of the client terminal and the bandwidth consumption of the multi-source data through a preset edge center node, so as to obtain a data priority queue; A transmission optimization unit (3), the transmission optimization unit (3) being used for performing transmission optimization modeling based on the data priority queue through the pre-set edge center node to obtain a Markov decision process for transmission optimization, and solving the Markov decision process based on deep reinforcement learning to obtain a data transmission strategy; A data transmission unit (4), the data transmission unit (4) being used to sequentially transmit the multi-source data to a pre-built edge computing network through the pre-set edge center node based on the data transmission strategy, wherein the edge computing network comprises a plurality of edge computing nodes; An edge service unit (5), the edge service unit (5) being used to compress and store the multi-source data through the pre-built edge computing network to obtain multi-source compressed data; A cloud service unit (6), the cloud service unit (6) being used for acquiring multi-source compressed data corresponding to the target scheduling task from the edge computing network based on the received target scheduling task, and storing and displaying the multi-source compressed data; Prioritizing the multi-source data based on the importance of the client terminal and the bandwidth consumption of the multi-source data to obtain a data priority queue, including: defining the importance of the multi-source data based on the importance of the client terminal; Grouping the multi-source data based on the importance of the multi-source data to divide the multi-source data into different data sets, wherein the different data sets include a primary importance set, a secondary importance set, and a tertiary importance set; The importance of the client terminal and the bandwidth consumption of the multi-source data construct a data priority queue.

2. According to claim 1, an emergency video intelligent monitoring system based on multi-source data fusion is characterized in that: The definition expression of the importance of the multi-source data is as follows: ； in, Indicated in Time The importance of multi-source data uploaded by each client terminal, Indicated in Time The click prevalence of multi-source data uploaded by client terminals, Indicated in Time The key area attention rate of multi-source data uploaded by client terminals, Indicated in Time Target information monitoring rate of multi-source data uploaded by client terminals; The definition expression of the data priority queue is as follows: ； in, express The first in the importance set Data priority queue for multi-source data uploaded by client terminals, Indicates The bandwidth consumption of multi-source data uploaded by each client terminal, Indicates The transmission smoothness of multi-source data uploaded by each client terminal, and , Indicates that the edge center node provides The bit rate of each client terminal, Indicates The maximum network rate at which a client terminal uploads multi-source data.

3. The emergency video intelligent monitoring system based on multi-source data fusion according to claim 2 is characterized in that: Transmission optimization modeling is performed based on the data priority queue to obtain a Markov decision process for transmission optimization, including: The state variables of the Markov decision process are defined based on the data priority queue, wherein the definition expression of the state variables is as follows: ； in, represents the state variable of the Markov decision process, Indicates the playback delay of multi-source data, and , Indicates the buffering time of multi-source data. represents the queuing delay of multi-source data, Indicates the quality of data provided by the edge center node to the client terminal, and , Represents the transmission cost of multi-source data; The action variable of the Markov decision process is defined based on the data priority queue, wherein the definition expression of the action variable is as follows: ； in, represents the action variable of the Markov decision process, Indicates The level of importance set of data priority queues of multi-source data uploaded by a client terminal; The Markov decision process reward function is defined based on the data priority queue, wherein the definition expression of the reward function is as follows: ； in, represents the reward function of the Markov decision process, represents the multi-source data transmission accuracy, and , represents the number of decisions for multi-source data transmission, represents the indicator function, and , represents the three-level importance set, Represents the first-level importance set and the second-level importance set.

4. The emergency video intelligent monitoring system based on multi-source data fusion according to claim 3 is characterized in that: Transmitting the multi-source data to a pre-built edge computing network in sequence based on the data transmission strategy includes: The calculation delay of the edge computing node executing the multi-source data compression task is calculated in sequence by the preset edge center node, wherein the calculation formula of the calculation delay of the multi-source data compression task is as follows: ； in, Indicated in At time t The computational delay of edge computing nodes performing multi-source data compression tasks, Indicates the edge computing network The number of local tasks in the multi-source data compression task performed by the edge computing node, Indicated in At time t The edge computing node performs the multi-source data compression task. The computation delay of local tasks, and , Indicated in Time The edge computing node performs the multi-source data compression task. The computational requirements of a local task, Represents the computing resources of the edge computing node; The transmission delay of multi-source data from the edge center node to the edge computing node in the edge computing network is calculated in sequence by the preset edge center node, wherein the calculation formula of the transmission delay is as follows: ； in, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission delay of edge computing nodes, Indicated in At time t The amount of data output by the layer, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission rate of edge computing nodes, and , Indicated in At time t The available bandwidth of edge computing nodes, Indicated in At time t The channel gain of the edge computing node, Indicated in At time t The upload power of edge computing nodes, represents the noise power spectral density; The download delay of multi-source data from the edge center node to the edge computing node in the edge computing network is calculated in sequence by the preset edge center node, wherein the calculation formula of the download delay is as follows: ； in, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The download delay of edge computing nodes, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The download rate of edge computing nodes, and , Indicated in At time t Download power of edge computing nodes.

5. The emergency video intelligent monitoring system based on multi-source data fusion according to claim 4 is characterized in that: Transmitting the multi-source data to a pre-built edge computing network in sequence based on the data transmission strategy, further comprising: The computing energy consumption of the edge computing node for executing the multi-source data compression task is calculated in sequence by the preset edge center node, wherein the calculation formula of the computing energy consumption is as follows: ； in, Indicated in At time t The computing energy consumption of edge computing nodes performing multi-source data compression tasks, Indicated in At time t The computing power consumption of an edge computing node performing multi-source data compression tasks; The transmission energy consumption of multi-source data from the edge center node to the edge computing nodes in the edge computing network is calculated in sequence by the preset edge center node, wherein the calculation formula of the transmission energy consumption is as follows: ； in, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission energy consumption of edge computing nodes, Indicated in At any time, multi-source data is sent from the edge center node to the edge computing network. The transmission power of edge computing nodes, Indicates the preset time threshold.

6. The emergency video intelligent monitoring system based on multi-source data fusion according to claim 5 is characterized in that: Transmitting the multi-source data to a pre-built edge computing network in sequence based on the data transmission strategy, further comprising: The total delay of executing the multi-source data compression task is calculated based on the calculation delay, the transmission delay and the download delay, wherein the calculation formula of the total delay is as follows: ； in, Indicated in At time t The total delay of edge computing nodes in performing multi-source data compression tasks; The total energy consumption of executing the multi-source data compression task is calculated based on the computing energy consumption and the transmission energy consumption, wherein the calculation formula of the total energy consumption is as follows: ； in, Indicated in At time t The total energy consumption of edge computing nodes performing multi-source data compression tasks; Based on the minimization of the total delay and the total energy consumption as the objective function, based on the maximum and minimum values ​​of the total energy consumption and the total delay as constraints, a task optimization model is constructed based on the objective function and the constraints, and the task optimization model is solved based on the optimization algorithm to obtain an edge computing node scheduling strategy.

7. The emergency video intelligent monitoring system based on multi-source data fusion according to claim 1 is characterized in that: Compressing and storing the multi-source data through the pre-built edge computing network to obtain multi-source compressed data includes: The multi-source data is processed based on the trained foreground extraction network to obtain a foreground detection box and an instance mask set, wherein the foreground detection box and the instance mask are expressed as follows: ； in, Indicates the first The set of target detection boxes of the frame, and , Indicates the first Frame No. A detection frame, Indicates the first The set of instance masks for the frame, and , Indicates the first Frame No. instance masks; All foreground detection frames in the multi-source data are tracked to obtain the motion trajectory of each foreground. The trajectory set of all trajectories is recorded as , obtaining the matching relationship between each foreground and the upper and lower frames based on the trajectory set, and calculating the intersection-over-union ratio of each foreground detection frame between two adjacent frames in the multi-source data; If the IoU is greater than the preset IoU threshold, the instance masks of the two foregrounds are aligned, and the two instance masks are XORed, and the area of ​​the XOR result is compared with the area of ​​the instance mask of the previous frame to obtain a difference value, wherein the calculation formula of the difference value is as follows: ； in, represents the difference value, and represents a pair of instance masks Represents the exclusive OR operation; If the difference value is less than a preset difference value threshold, the corresponding target detection frame and instance mask are removed from the foreground extraction result to obtain a new foreground detection frame and instance mask set.

8. The emergency video intelligent monitoring system based on multi-source data fusion according to claim 7 is characterized in that: The multi-source data is compressed and stored through the pre-built edge computing network to obtain multi-source compressed data, and further includes: Using the foreground detection frame and instance mask set of each frame in the multi-source data, each foreground is intercepted from the video frame corresponding to the multi-source data as a foreground image; A foreground image is stored in the trajectory of each foreground target to retain the original color and texture features of the foreground target, and the remaining foreground images are compressed to obtain a foreground detection frame and a foreground image set, and the foreground detection frame and the foreground image set are the compressed data of the foreground target.

Images