On-site judgment method based on multi-modal information matching

Abstract

The invention discloses a field judgment method based on multi-modal information matching, and the method comprises the steps: carrying out the multi-view environment collection of an uncertain region based on a first unmanned plane cluster, so as to obtain a multi-view environment image set of the uncertain region, carrying out the target detection of the multi-view environment image set based on a pre-trained target detection model, and obtaining a target detection result. A target detection result is obtained; and moving the second unmanned aerial vehicle cluster to the vicinity of the first unmanned aerial vehicle which detects that the suspicious target exists in the first unmanned aerial vehicle cluster. According to the method, strict target detection and tracking are carried out between the uncertain area and the protection area, suspicious targets which possibly threaten the protection area can be recognized in real time, potential threats can be quickly recognized and responded through dynamic monitoring and evaluation in the uncertain area in combination with target positioning and moving track analysis, and the safety of the protection area is improved. Therefore, the security of the security area is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scene generation, in particular to a live adjudication method based on multi-modal information matching. BACKGROUND

[0002] Traditional methods usually rely on manual judgment or relatively simple technical means, lacking real-time, dynamic monitoring and response mechanisms. When a suspicious target enters a protected area or a threat occurs, traditional methods may not be able to discover and respond in time, resulting in reduced security. Moreover, traditional methods often rely on a single detection method or low-precision target tracking technology, which is prone to false positives or false negatives, and cannot fully assess the behavior patterns and potential threat levels of targets, thus their threat assessment and target tracking accuracy is poor. Furthermore, traditional methods require human involvement in adjudication and decision-making, which not only increases decision-making time, but also can lead to incorrect measures due to human factors such as judgment errors or fatigue, affecting overall security protection effectiveness. In addition, traditional methods may lack comprehensive processing capabilities for multi-modal data, making it difficult to efficiently integrate information from different sensors and different perspectives, thus reducing the comprehensiveness and accuracy of decision-making. Effective fusion and real-time analysis of multi-modal data is key to improving protection capabilities, and traditional methods have limited capabilities in this regard. Finally, traditional methods may struggle to cope with multiple targets, multiple paths, and dynamic changes, such as in uncertain areas or complex terrain, where traditional methods may not effectively track the simultaneous movement of multiple targets, resulting in failure to identify real threats. SUMMARY

[0003] The purpose of the present application is to provide a live adjudication method based on multi-modal information matching to solve the problems in the background art.

[0004] Technical solution: The live adjudication method based on multi-modal information matching comprises the following steps: (1) Based on the first drone swarm, multi-view environment acquisition is performed on the uncertain area to obtain a multi-view environment image set of the uncertain area; (2) Based on a pre-trained target detection model, target detection is performed on the multi-view environment image set to obtain a target detection result including the presence of a suspicious target, the absence of a suspicious target, and the target type. If a suspicious target is detected, move the second drone swarm to the vicinity of the first drone in the first drone swarm that detected the suspicious target; (3) Based on the second drone swarm, target positioning is performed on the suspicious target to obtain suspicious target position coordinates, wherein the second drone swarm is located in the determined area; (4) Obtain the suspicious target position coordinates according to a preset sampling step to generate a sequence of suspicious target position coordinates; (5) determining whether the moving direction of the suspicious target is toward the protection area based on the sequence of position coordinates, to obtain a moving type; (6) evaluating a threat level of the suspicious target according to the target type and the moving type, and constructing a moving track of the suspicious target based on the sequence of position coordinates; (7) constructing a region track graph based on the moving tracks and the threat levels of all suspicious targets in the uncertain region; performing abnormality detection on the region track graph by a pre-trained region abnormality detection model to obtain a region detection label of the uncertain region; and matching the region detection label with a preset decision table to generate on-site decision information of the uncertain region.

[0005] Further, in step (2), the target detection model is specifically as follows: the backbone network sequentially passes through a 7x7 convolution kernel, a 3x3 convolution kernel with 16 channels, and a 3x3 convolution layer to extract features, and generates multi-scale feature maps of the original image with scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 through FPN; the decoding network fuses the multi-scale feature maps through upsampling and deformable convolution operations, and outputs a fused feature map; the four output heads include a center point prediction head, a detection frame regression head, a center point bias regression head, and an appearance embedding feature extraction head.

[0006] Further, in step (3), the target positioning is specifically as follows: the positions of the unmanned aerial vehicles in the second unmanned aerial vehicle cluster are located through GPS to form a position coordinate set; a distance set between each second unmanned aerial vehicle and the suspicious target is calculated based on the position coordinate set; a set of arrival angles of the radiation signal of each second unmanned aerial vehicle to the suspicious target is calculated; a time delay set of each second unmanned aerial vehicle receiving the radiation source signal is calculated based on the distance set; an observation equation set is constructed based on the time delay set and the arrival angle set; and the observation equation set is solved by using a nonlinear least squares method to obtain the position coordinates of the suspicious target.

[0007] Further, in step (5), the moving direction determination is specifically as follows: a distance sequence of each point in the sequence of position coordinates to the protection area is calculated; if the distance sequence is arranged in descending order, it is determined that the moving direction is toward the protection area; and if the length of a continuous descending sub-sequence exceeds a preset threshold, it is determined that the moving direction is toward the protection area.

[0008] Further, in step (7), the region track graph includes a node set and an edge set; wherein the node set includes the position coordinates of the suspicious target, the threat level, and the time stamp of each node; and the edge set connects two nodes and includes the spatial distance and the moving direction between the nodes.

[0009] Furthermore, in step (7), the regional anomaly detection model includes an embedding layer, a graph attention layer, a fusion gate, a propagation aggregation layer, and a prediction layer; wherein, the embedding layer embeds node features into a feature matrix; the graph attention layer calculates the correlation between nodes through attention weights to generate first-order information; the fusion gate fuses the first-order information to generate a fused embedding vector; the propagation aggregation layer generates a multi-layer representation through graph neural network message passing; and the prediction layer multiplies the multi-layer representation with the reference vector to output the anomaly detection result.

[0010] The present invention discloses a field adjudication system based on multimodal information matching, comprising: Acquisition module: used to acquire multi-view environmental data of uncertain areas based on the first UAV cluster, and obtain a set of multi-view environmental images of the uncertain areas; Target detection module: Used to perform target detection on multi-view environmental image sets based on a pre-trained target detection model, and obtain target detection results including the presence of suspicious targets, the absence of suspicious targets, and the target type; if a suspicious target is detected, the second UAV cluster is moved to the vicinity of the first UAV in the first UAV cluster that detected the suspicious target; Target localization module: used to locate suspicious targets based on the second UAV cluster, and obtain the location coordinates of the suspicious targets, wherein the second UAV cluster is located in a defined area; Location coordinate module: used to acquire the location coordinates of suspicious targets according to a preset sampling step size and generate a sequence of location coordinates of suspicious targets; The movement module is used to determine whether the movement direction of a suspicious target is toward the protected area based on the sequence of location coordinates, and to obtain the movement type. Trajectory module: Used to assess the threat level of suspicious targets based on target type and movement type, and to construct the movement trajectory of suspicious targets based on the sequence of location coordinates; Matching module: used to construct a regional trajectory map based on the movement trajectory and threat level of all suspicious targets in the uncertain area; to perform anomaly detection on the regional trajectory map through a pre-trained regional anomaly detection model to obtain regional detection labels for the uncertain area; and to match the regional detection labels with a preset adjudication table to generate on-site adjudication information for the uncertain area.

[0011] An electronic device according to the present invention includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the steps of any of the methods described herein.

[0012] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described herein.

[0013] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) The present invention can identify suspicious targets that may pose a threat to the protected area in real time by performing strict target detection and tracking between uncertain areas and protected areas. By dynamically monitoring and evaluating uncertain areas, and combining the positioning and movement trajectory analysis of targets, potential threats can be quickly identified and responded to, thereby improving the security of the safe area. Moreover, by using a second UAV cluster, after the first UAV cluster detects a suspicious target, it can quickly go to the area and locate the target, thereby quickly determining the position coordinates of the suspicious target and accurately tracking the target's movement trajectory. The generation of sampling step length and position coordinate sequence can provide detailed time series data, help determine whether the target is heading towards the protected area, and provide more information to assess the threat; (2) The present invention can assess the potential threat level by combining the type of suspicious target with its movement type. This not only considers Considering the attributes of the target itself, the behavioral patterns of the target are also taken into account, such as whether there is abnormal behavior, so as to provide a more comprehensive and accurate threat assessment. Moreover, by constructing a regional trajectory map, anomaly detection is performed based on the movement trajectory and threat level of multiple suspicious targets, which can identify abnormal behavior or potential threats in uncertain areas. The regional trajectory map integrates the behavioral patterns of all targets, making anomaly detection more efficient and enabling timely identification of anomalies and corresponding disposal measures. (3) This invention achieves automated adjudication by matching regional detection labels with a preset adjudication table. Based on the regional trajectory map and the results of anomaly detection, the on-site adjudication information of uncertain areas can be quickly determined, providing support for rapid response and decision-making, avoiding delays and errors caused by manual intervention. By combining multimodal data (such as multi-view images, target detection, location coordinates, trajectory analysis, etc.), this method not only improves the comprehensiveness and accuracy of the data, but also provides stronger support for adjudication. Through a systematic process, the adjudication information has high scientificity and accuracy, thereby improving the quality of decision-making. Attached Figure Description

[0014] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0015] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0016] like Figure 1 As shown, this embodiment of the invention provides a field adjudication method based on multimodal information matching, including: S1. Based on the first UAV cluster, multi-view environmental data is collected in the uncertain area to obtain a multi-view environmental image set of the uncertain area. Based on the pre-trained target detection model, target detection is performed on the multi-view environmental image set to obtain target detection results. The target detection results include whether there are suspicious targets, whether there are no suspicious targets, and the target type. S2. Move the second UAV cluster to the vicinity of the first UAV cluster in which a suspicious target was detected, and locate the suspicious target in the uncertain area based on the second UAV cluster to obtain the location coordinates of the suspicious target, wherein the second UAV cluster is located in the determined area; S3. Obtain the location coordinates of the suspicious target according to the preset sampling step size to obtain the location coordinate sequence of the suspicious target. Based on the location coordinate sequence, determine whether the movement direction of the suspicious target is towards the protected area to obtain the movement type of the suspicious target. S4. Assess the threat level of suspicious targets based on their target type and movement type, and construct their movement trajectory based on their location coordinate sequence. S5. Construct a regional trajectory map based on the movement trajectory and threat level of all suspicious targets in the uncertain area, and perform anomaly detection on the regional trajectory map based on a pre-trained regional anomaly detection model to obtain the regional detection label of the uncertain area. S6. Match the area detection label of the uncertain area with the preset adjudication table to obtain the on-site adjudication information of the uncertain area.

[0017] In this invention, the multi-view environmental image set refers to image data collected from multiple angles and altitudes by a first drone cluster. These images integrate different perspectives of uncertain areas, helping to achieve more comprehensive target detection. Multiple perspectives can provide more details, reduce blind spots, and improve detection accuracy. A defined area refers to an area where the behavior or state of a target can be accurately predicted and controlled. It has clear boundaries and known conditions, and the environmental information is usually stable. An uncertain area refers to an area where the behavior or state of a target is not easily predicted, making control and monitoring of the area more difficult. The sampling step size refers to the time interval for acquiring target location data. The threat level is a standard for assessing the potential threat of a suspicious target to security. The assessment of the threat level comprehensively considers the target type (such as drones, armed personnel, etc.) and movement behavior (such as whether it is rapidly approaching the protected area). Generally, the more abnormal the target type and behavior, the higher the threat level. The decision table is a decision table based on security policies, rules, or standards. It matches detection labels (such as "threat exists") with corresponding security response measures. For example, when a threat is detected in an area, the decision table may indicate the need to initiate security protection, evacuate personnel, or carry out other emergency responses.

[0018] The target detection model includes a backbone network, a decoding network, and an output head. The backbone network first passes through a 7×7 convolutional kernel with 16 channels, followed by two 3×3 convolutional kernels with 16 and 32 channels respectively. Finally, the feature maps are passed through the FPN to extract features from the input feature maps. The extracted feature maps of different sizes are then subjected to deep feature aggregation, resulting in four multi-scale feature maps with sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original input image. The decoding network first performs feature fusion on the multi-scale feature maps through upsampling and deformable convolution operations to obtain a fused feature map of the multi-scale feature maps, and then feeds the fused feature map into four different output heads. The output heads are respectively the center point prediction head, the detection box regression head, the center point offset regression head, and the appearance embedding feature extraction head.

[0019] In this embodiment, the center point prediction head is used to predict the center position of the target in the feature map, which is usually determined by regressing the response at each position; the detection box regression head is used to regress and predict the bounding box of the target, that is, the location information of the detected target, which describes the target by the four coordinates of the predicted box (such as center position, width and height); the center point offset regression head is used to further regress and correct the offset to ensure the accurate positioning of the detection box; the appearance embedding feature extraction head is used to extract the appearance features of the target, which is usually used for target recognition or multi-target tracking tasks, so that the model can distinguish different targets by appearance features; FPN mainly uses a feature pyramid to enhance the representation of multi-scale features, which solves the shortcomings of traditional CNN when facing targets of different sizes. Its key idea is to enhance the detection effect by combining information from multiple scales through top-down feature fusion and bottom-up feature enhancement.

[0020] In an optional embodiment, the location of a suspicious target in an uncertain area is determined based on a second drone swarm, to obtain the location coordinates of the suspicious target, including: A1. Based on GPS, the position coordinates of each second UAV in the second UAV cluster are located to obtain the set of position coordinates of the second UAV cluster. Each second UAV in the second UAV cluster emits radiation towards the suspicious target in the uncertain area. A2. Calculate the distance between each second UAV in the second UAV cluster and the suspicious target based on the set of position coordinates to obtain the set of distances between the second UAV cluster and the suspicious target; A3. Calculate the angle of arrival between the radiation emitted by each second UAV in the second UAV cluster and the suspected target based on the set of position coordinates, so as to obtain the set of angles of arrival between the radiation emitted by the second UAV cluster and the suspected target; A4. Calculate the time delay of each second UAV in the second UAV cluster receiving radiation source signals based on the distance set, so as to obtain the time delay set of the second UAV cluster. A5. Construct a set of observation equations for defining suspicious targets using a second UAV cluster based on the time delay set and the arrival angle set; A6. Solve the observation equations using the nonlinear least squares method to obtain the coordinates of the suspected target.

[0021] It should be noted that nonlinear least squares is a method for solving nonlinear optimization problems. It is widely used in fitting models, estimating parameters, and solving systems of equations containing nonlinear relationships, especially in applications such as target localization, data fitting, and curve fitting.

[0022] In an optional embodiment, the observation equations are expressed as follows: ; Indicates the second drone cluster The time delay between the second UAV and the first second UAV in receiving the radiation source signal. Indicates the second drone cluster The angle of arrival between the radiation emitted by the second drone and the suspected target. Indicates the second drone cluster The location coordinates of the second drone. This represents the location coordinates of the suspicious target to be calculated. Indicates the second drone cluster The measurement error of the time delay between the second UAV and the first UAV in receiving the radiation source signal. Indicates the second drone cluster The measurement error of the angle of arrival between the radiation emitted by the second UAV and the suspected target.

[0023] In an optional embodiment, determining whether the movement direction of a suspicious target is toward the protected area based on a sequence of location coordinates includes: B1. Calculate the distance between the suspected target and the protected area based on the location coordinate sequence to obtain the distance sequence corresponding to the location coordinate sequence; B2. Determine whether the distance sequence is arranged in descending order. If the distance sequence is arranged in descending order, determine that the suspicious target is moving towards the protected area. B3. Calculate the length of the sub-distance sequence arranged in descending order. If the length is greater than the preset length threshold, determine that the suspicious target is moving towards the protected area.

[0024] In an optional embodiment, the regional trajectory map includes a set of nodes and a set of edges, wherein the nodes include the location coordinates, threat level, and timestamp of the suspected target, and the edges are used to connect two nodes in the regional trajectory map, including the spatial distance and direction of movement between the two nodes.

[0025] In an optional embodiment, the region anomaly detection model includes an embedding layer, a graph attention layer, a propagation aggregation layer, and a prediction layer. The embedding layer is used to embed node features in the region trajectory graph to obtain a node feature embedding matrix. The graph attention layer is used to calculate the attention weights between nodes in the region trajectory graph through an attention mechanism. Based on the attention weights between nodes in the region trajectory graph, the first-order information of the region trajectory graph is captured through a weighted representation, and the first-order information of the region trajectory graph is fused through a fusion gate to obtain the fused embedding vector corresponding to the region trajectory graph. The propagation aggregation layer is used to perform message passing on the fused embedding vector corresponding to the region trajectory graph using a graph neural network to obtain the layer representations corresponding to the region trajectory graph. The prediction layer is used to concatenate the layer representations with a single vector and then perform an inner product to obtain the prediction result.

[0026] It should be noted that the embedding layer is used to map the node features in the regional trajectory map to a new low-dimensional space. Node features are usually high-dimensional, such as representing information like location, time, and velocity. The role of the embedding layer is to transform these high-dimensional features into a more easily processed low-dimensional vector representation, which can effectively capture the relationships between nodes. The graph attention layer uses an attention mechanism to calculate a weight for each pair of nodes, representing their relative importance or correlation. The propagation and aggregation layer uses a graph neural network (GNN) to message-pass the node features in order to capture more complex structural information in the graph. The fusion gate is used to combine features from different information sources or layers, usually used to fuse first-order information from the regional trajectory map with other information. The fusion embedding vector is a vector obtained by fusing multi-level information or multiple features, which can represent the overall features of the regional trajectory map. The prediction layer is the last part of the model, and its goal is to make predictions based on the previously obtained node representations.

[0027] In an optional embodiment, the expression for the attention weights is as follows: ; in, Indicates attention weights. express Activation function , Indicates that the node is composed of Dimensional node space projection to 3D relation space, Indicates the first Feature embedding matrix of each node via edge Transformation matrix Perform a linear transformation, then add edges. relation vector offset, Represents the hyperbolic tangent activation function; The expression for the first-order information of the region trajectory map is as follows: ; in, Indicates category node First-order information.

[0028] In an optional embodiment, the expression for fusing the embedding vector is as follows: ; in, Represents the fused embedding vector. This represents the activation function. and This represents the learnable transformation parameters. Indicates category node First-order information; The expressions representing each layer of the region trajectory map are as follows: ; in, The region trajectory map represents the first Layer representation, express Activation function The region trajectory map represents the first Layer fusion embedding vector, The region trajectory map represents the first The fused embedding vector after message passing in the layer.

Claims

1. A field adjudication method based on multimodal information matching, characterized in that, Includes the following steps: (1) Based on the first UAV cluster, multi-view environmental data collection is performed on the uncertain area to obtain a multi-view environmental image set of the uncertain area; (2) Target detection is performed on a multi-view environmental image set based on a pre-trained target detection model. The target detection results include the presence of suspicious targets, the absence of suspicious targets, and the target type. If a suspicious target is detected, the second drone cluster will be moved to the vicinity of the first drone in the first drone cluster that detected the suspicious target; (3) Based on the second UAV cluster, the suspicious target is located and the coordinates of the suspicious target are obtained, wherein the second UAV cluster is located in a defined area; (4) Obtain the location coordinates of the suspicious target according to the preset sampling step size, and generate a sequence of location coordinates of the suspicious target; (5) Determine whether the movement direction of the suspicious target is toward the protected area based on the position coordinate sequence to obtain the movement type; (6) Assess the threat level of suspicious targets based on target type and movement type, and construct the movement trajectory of suspicious targets based on location coordinate sequence; (7) Construct a regional trajectory map based on the movement trajectory and threat level of all suspicious targets in the uncertain area; perform anomaly detection on the regional trajectory map through a pre-trained regional anomaly detection model to obtain regional detection labels for the uncertain area; match the regional detection labels with a preset adjudication table to generate on-site adjudication information for the uncertain area.

2. The on-site adjudication method based on multimodal information matching according to claim 1, characterized in that, In step (2), the target detection model is as follows: the backbone network extracts features through convolutional layers with 7×7 kernels, 3×3 kernels (16 channels), and 3×3 kernels in sequence, and generates multi-scale feature maps of the original image at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 through FPN; the decoding network fuses the multi-scale feature maps through upsampling and deformable convolution operations, and outputs the fused feature map; the four output heads include a center point prediction head, a detection box regression head, a center point offset regression head, and an appearance embedding feature extraction head.

3. The on-site adjudication method based on multimodal information matching according to claim 1, characterized in that, In step (3), the target localization is specifically as follows: the position coordinates of each UAV in the second UAV cluster are located by GPS to form a set of position coordinates; the distance set between each second UAV and the suspected target is calculated based on the set of position coordinates. Calculate the set of arrival angles of each second UAV's radiated signal to the suspected target; calculate the set of time delays of each second UAV receiving the radiated source signal based on the distance set; construct an observation equation set based on the time delay set and the arrival angle set; solve the observation equation set using the nonlinear least squares method to obtain the location coordinates of the suspected target.

4. The on-site adjudication method based on multimodal information matching according to claim 1, characterized in that, In step (5), the movement direction is determined as follows: calculate the distance sequence between each point in the position coordinate sequence and the protected area; if the distance sequence is arranged in descending order, determine that the movement direction is towards the protected area; if the length of the continuous descending subsequence exceeds the preset threshold, determine that the movement direction is towards the protected area.

5. The on-site adjudication method based on multimodal information matching according to claim 1, characterized in that, In step (7), the regional trajectory map includes a set of nodes and a set of edges; the set of nodes is each node including the location coordinates of the suspected target, the threat level and the timestamp; the set of edges connects two nodes and includes the spatial distance between the nodes and the direction of movement.

6. The on-site adjudication method based on multimodal information matching according to claim 1, characterized in that, In step (7), the regional anomaly detection model includes an embedding layer, a graph attention layer, a fusion gate, a propagation aggregation layer, and a prediction layer. The embedding layer embeds node features into a feature matrix. The graph attention layer calculates the correlation between nodes through attention weights to generate first-order information. The fusion gate fuses the first-order information to generate a fused embedding vector. The propagation aggregation layer generates a multi-layer representation through graph neural network message passing. The prediction layer multiplies the multi-layer representation with the reference vector and outputs the anomaly detection result.

7. A field adjudication system based on multimodal information matching, characterized in that, include: Acquisition module: used to acquire multi-view environmental data of uncertain areas based on the first UAV cluster, and obtain a set of multi-view environmental images of the uncertain areas; The target detection module is used to perform target detection on a multi-view environmental image set based on a pre-trained target detection model, and the target detection results include whether there are suspicious targets, whether there are no suspicious targets, and the target type. If a suspicious target is detected, the second drone cluster will be moved to the vicinity of the first drone in the first drone cluster that detected the suspicious target; Target localization module: used to locate suspicious targets based on the second UAV cluster, and obtain the location coordinates of the suspicious targets, wherein the second UAV cluster is located in a defined area; Location coordinate module: used to acquire the location coordinates of suspicious targets according to a preset sampling step size and generate a sequence of location coordinates of suspicious targets; The movement module is used to determine whether the movement direction of a suspicious target is toward the protected area based on the sequence of location coordinates, and to obtain the movement type. Trajectory module: Used to assess the threat level of suspicious targets based on target type and movement type, and to construct the movement trajectory of suspicious targets based on the sequence of location coordinates; Matching module: Used to construct a regional trajectory map based on the movement trajectories and threat levels of all suspicious targets within an uncertain area; Anomaly detection is performed on the regional trajectory map using a pre-trained regional anomaly detection model to obtain regional detection labels for uncertain regions; the regional detection labels are then matched with a preset adjudication table to generate on-site adjudication information for uncertain regions.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the program to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images