Global image recognition situation awareness system based on convolutional neural network

Abstract

The application discloses a global image recognition situation awareness system based on a convolutional neural network, a bidirectional mapping function of a visible light and a thermal infrared feature space is obtained through training of a cross-modal feature translation network, and the bidirectional mapping function is compressed into a lightweight convolutional translation network through knowledge distillation and pre-deployed to each edge device; an edge collaborative mode is triggered when cloud connection is interrupted, cross-modal translation features are generated by each edge device using the lightweight convolutional translation network, cross-modal cross-view target correlation is performed by a coordination node, global unified target trajectories are generated, and continuous situation awareness results are output.

Description

Technical Field

[0001] This invention relates to the field of image recognition and situational awareness technology, and more specifically, to a global image recognition and situational awareness system based on convolutional neural networks. Background Technology

[0002] In the scenario of full-domain image recognition and situational awareness in urban surveillance networks, two types of heterogeneous imaging devices, visible light cameras and thermal infrared cameras, are deployed in different areas. Each edge device is equipped with a convolutional neural network to perform local target detection. The detection results are uploaded to the cloud for full-domain cross-view target association and situational aggregation analysis. The cloud uses a complete cross-modal convolutional neural network to uniformly process the visible light and thermal infrared images, maintaining the temporal continuity of full-domain situational awareness.

[0003] In existing technologies, complete cross-modal convolutional neural networks are deployed in the cloud, relying on cloud computing power to perform unified feature extraction and cross-modal association processing on images of two modalities. Edge devices only perform single-modal object detection, do not have the ability to transform cross-modal feature spaces, and lack autonomous collaboration mechanisms between edge devices.

[0004] The aforementioned existing technologies have the following drawbacks: When the day-night cycle and cloud connection interruption occur simultaneously, adjacent edge devices operate in visible light mode and thermal infrared mode, respectively. Visible light features rely on reflected light to present texture details, while thermal infrared features rely on target thermal radiation to present temperature distribution contours. The two are in different feature spaces with significant semantic differences, and direct calculation of feature similarity between edge devices results in a large number of mismatches. At the same time, edge devices with limited computing power cannot run complete cross-modal convolutional neural networks and lack cross-device collaboration mechanisms, leading to failure of cross-view target association, severe degradation of global image recognition situational awareness capabilities, and interruption of continuous tracking of key situational events. Summary of the Invention

[0005] This invention provides a global image recognition situational awareness system based on convolutional neural networks, which solves the technical problem in related technologies that heterogeneous modal edge devices cannot effectively perform cross-modal target association and situational awareness when cloud connection is interrupted.

[0006] This invention discloses a global image recognition situational awareness system based on convolutional neural networks, comprising: extracting paired convolutional feature samples from visible light images and thermal infrared images acquired in the same scene; training a cross-modal feature translation network to obtain a bidirectional mapping function between the visible light feature space and the thermal infrared feature space; using the cross-modal feature translation network as a teacher network, compressing the cross-modal feature translation network using a knowledge distillation algorithm to obtain a lightweight convolutional translation network; and pre-deploying the lightweight convolutional translation network to each edge device; each edge device monitoring the connection status with the cloud server, triggering an edge collaboration mode when a connection interruption is detected, and exchanging device identifiers and current working information of each edge device via a local area network. Modal identification is performed, and a coordination node is determined based on the network connection topology. Each edge device performs target detection on locally acquired image frames to obtain an initial feature vector. The initial feature vector is then input into the corresponding mapping path of the lightweight convolutional translation network to generate cross-modal translation features. Each edge device transmits the initial feature vector, cross-modal translation features, bounding box coordinates, and modal identification to the coordination node. The coordination node calculates bidirectional similarity in the corresponding feature space based on the modal identification of the target pair to be associated and takes the average as the comprehensive cross-modal similarity. Combined with the mapping relationship of the camera coverage area, target pairs exceeding the preset association threshold are determined to be the same target, generating a unified target trajectory across the entire domain and outputting the situational awareness result.

[0007] Furthermore, the cross-modal feature translation network includes a visible light modal encoder, a thermal infrared modal encoder, a shared latent alignment layer, a visible light to thermal infrared decoder, and a thermal infrared to visible light decoder. The visible light modal encoder receives visible light convolutional feature vectors and outputs intermediate encoded representations to the shared latent alignment layer. The thermal infrared modal encoder receives thermal infrared convolutional feature vectors and outputs intermediate encoded representations to the shared latent alignment layer. The shared latent alignment layer maps the intermediate encoded representations of the two modalities to a unified latent semantic space. The visible light to thermal infrared decoder receives the intermediate encoded representations of the two modalities... The visible light mode encoder receives the latent representation of the path input and outputs the thermal infrared mode translation features; the thermal infrared to visible light decoder receives the latent representation of the path input through the thermal infrared mode encoder and outputs the visible light mode translation features; the bidirectional mapping function is as follows: the mapping function from the visible light feature space to the thermal infrared feature space is composed of the visible light mode encoder, the shared latent alignment layer and the visible light to thermal infrared decoder cascaded in sequence, and the mapping function from the thermal infrared feature space to the visible light feature space is composed of the thermal infrared mode encoder, the shared latent alignment layer and the thermal infrared to visible light decoder cascaded in sequence.

[0008] Furthermore, the total loss function used when training the cross-modal feature translation network is composed of a weighted sum of reconstruction loss and distribution alignment loss; the reconstruction loss is the sum of the mean square error between the output after mapping visible light features to thermal infrared and the real thermal infrared features, and the mean square error between the output after mapping thermal infrared features to visible light and the real visible light features; the distribution alignment loss uses the maximum mean difference algorithm to calculate the maximum mean difference between two latent representation sets output through a shared latent alignment layer after being encoded by the visible light modal encoder and the thermal infrared modal encoder, respectively, within a training batch.

[0009] Furthermore, the lightweight convolutional translation network, as the student network, retains the same bidirectional mapping structure as the cross-modal feature translation network. The number of hidden layer neurons and layers in each component unit are reduced compared to the cross-modal feature translation network. The distillation loss of the student network is composed of the mean squared error between the final translation feature output of the student network and the final translation feature output of the teacher network, and the mean squared error between the output of the lightweight shared latent alignment layer of the student network and the output of the shared latent alignment layer of the teacher network, in a weighted summation manner.

[0010] Furthermore, the determination of the coordination node based on the network connection topology relationship includes: each edge device calculates the centrality index of each edge device in the local area network topology according to the network connection topology relationship stored locally. The centrality index is the ratio of the total number of edge devices participating in edge collaboration minus one to the sum of the shortest path hops from the current edge device to all other edge devices. The edge device with the largest centrality index value is selected as the coordination node.

[0011] Furthermore, the coordinating node calculates bidirectional similarity in the corresponding feature space based on the modal identifiers of the target pair to be associated, including: when the two targets to be associated come from a visible light modal device and a thermal infrared modal device, respectively, the cross-modal translation features reported by the visible light modal device and the initial feature vector reported by the thermal infrared modal device are selected in the thermal infrared feature space and the cosine similarity algorithm is used to calculate the first similarity; the cross-modal translation features reported by the thermal infrared modal device and the initial feature vector reported by the visible light modal device are selected in the visible light feature space and the cosine similarity algorithm is used to calculate the second similarity; the average of the first similarity and the second similarity is taken as the comprehensive cross-modal similarity. When the two targets to be associated come from edge devices of the same modality, the initial feature vectors of the two targets to be associated are directly used in the same modal feature space and the cosine similarity algorithm is used to calculate the similarity.

[0012] Furthermore, each edge device concatenates the initial feature vector with the cross-modal translation features along the channel dimension to obtain a concatenated feature vector. This concatenated feature vector is then input into the cross-modal fusion module to generate a unified feature vector. The cross-modal fusion module includes a channel concatenation layer, a 1×1 convolutional dimensionality reduction layer, a batch normalization layer, and a non-linear activation layer. The channel concatenation layer concatenates the initial feature vector with the cross-modal translation features along the channel dimension. The 1×1 convolutional dimensionality reduction layer reduces the concatenated feature vector to the same dimension as the initial feature vector. The batch normalization layer normalizes the dimensionality-reduced features. The non-linear activation layer outputs a unified feature vector. The coordinating node weights and fuses the overall cross-modal similarity with the unified feature vector using a cosine similarity algorithm to obtain an enhanced association similarity. This enhanced association similarity is then used to replace the overall cross-modal similarity in association determination.

[0013] Furthermore, when each edge device detects that the cloud connection has been restored, it uploads the multimodal detection results accumulated during edge collaboration and the global unified target trajectory generated by the coordination node to the cloud server in batches. The cloud server re-inputs the original image regions of each target into a complete cross-modal convolutional neural network for feature extraction to obtain a high-precision cross-modal feature representation. Based on the high-precision cross-modal feature representation, it recalculates the target association similarity and refines and corrects the association results during edge collaboration.

[0014] Furthermore, the cloud server performs time-series alignment and fusion of the refined and corrected target trajectory with the historical situation data stored in the cloud. Using the timestamp and target state of the last frame of cloud situation data at the moment of cloud connection interruption as the alignment anchor point, the refined and corrected target trajectory during edge collaboration is inserted into the corresponding time interval in the historical situation sequence according to the timestamp order. The target state at the alignment anchor point and at the moment of connection restoration is smoothed using a linear interpolation algorithm.

[0015] This invention discloses a global image recognition situational awareness system based on convolutional neural networks, comprising: a cross-modal mapping training module, used to extract paired convolutional feature samples based on paired acquired visible light images and thermal infrared images, and train a cross-modal feature translation network to obtain a bidirectional mapping function between the visible light feature space and the thermal infrared feature space; a network compression and deployment module, used to obtain a lightweight convolutional translation network using the cross-modal feature translation network as the teacher network and employing a knowledge distillation algorithm, and pre-deploying the lightweight convolutional translation network to each edge device; and a connection monitoring and collaborative triggering module, used to monitor the connection status between each edge device and the cloud server, and trigger the edge device when a connection interruption is detected. The edge collaboration mode interacts with the device identifiers and current working mode identifiers of each edge device, and determines the coordination node based on the network connection topology. The feature extraction and translation module is used by each edge device to perform target detection on locally acquired image frames to obtain initial feature vectors, and inputs the initial feature vectors into a lightweight convolutional translation network to generate cross-modal translation features. The cross-modal association and situational awareness output module is used to aggregate the feature data of each edge device to the coordination node, calculate the bidirectional similarity in the corresponding feature space according to the modality identifier, take the average value as the comprehensive cross-modal similarity, perform target association determination in combination with the mapping relationship of the camera coverage area, generate a unified target trajectory across the entire domain, and output the situational awareness results.

[0016] This invention addresses the technical problem of insufficient computing power for edge devices to run a complete cross-modal convolutional neural network and complete cross-modal feature space transformation under combined fault scenarios of day-night alternation and cloud connection interruption. It trains a cross-modal feature translation network and compresses it into a lightweight convolutional translation network via knowledge distillation, which is then pre-deployed to various edge devices. This achieves the technical effect of enabling each edge device to map its own modality features to the modality feature space of the peer device during cloud connection interruption, and supporting similarity calculation of features extracted by different modal devices in the same feature space. Through LAN broadcasting, coordination node aggregation, and cross-modal and cross-view target association mechanisms in the edge collaboration mode, it solves the technical problem of a single edge device being unable to complete cross-device, cross-modal, and cross-view target association when cloud coordination is lacking. This achieves the technical effect of maintaining the continuous operation of unified target trajectory generation and regional situation aggregation analysis across the entire domain during network outages. After cloud connection is restored, the edge association results are refined and corrected using a complete cross-modal convolutional neural network and time-series aligned and fused with historical situation data. This achieves the technical effect of compensating for the accuracy loss of the lightweight convolutional translation network and eliminating the time-series discontinuity of situation data before and after network outages. Attached Figure Description

[0017] Figure 1 This is a flowchart of the global image recognition situational awareness method provided in the embodiments of the present invention; Figure 2This is a schematic diagram of the convergence curve of the training loss of the cross-modal feature translation network provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the comparison of paired feature samples provided in the embodiments of the present invention: visible light, thermal infrared and translation features (first 8 dimensions of pedestrian target); Figure 4 This is a schematic diagram of the translation mean square error distribution of each feature dimension provided in the embodiments of the present invention; Figure 5 This is a schematic diagram comparing the structural parameters of the teacher network and the student (lightweight) network provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the edge device network topology hop count heatmap provided in an embodiment of the present invention; Figure 7 This is a schematic diagram comparing the network centrality of various edge devices provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the cross-modal target pair association similarity decomposition and comparison provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical problem to be solved by this embodiment is: In a global image recognition-based situational awareness scenario, the urban surveillance network deploys two types of heterogeneous imaging devices in different areas: visible light cameras and thermal infrared cameras. Each edge device is equipped with a domain-specific normalized convolutional neural network to perform image feature extraction and target detection. The detection results are uploaded to the cloud for global cross-view target association and situational aggregation analysis. During the day, visible light image recognition is the primary method, while at night, thermal infrared image recognition is used. The cloud uses a complete cross-modal convolutional neural network to process both modalities of images uniformly, maintaining the temporal continuity of global situational awareness.

[0019] When the day-night cycle coincides with a cloud connection interruption, adjacent edge devices operate in visible light and thermal infrared modes, respectively. The feature vectors extracted by the visible light convolutional neural network rely on reflected light to represent texture details, while the feature vectors extracted by the thermal infrared convolutional neural network rely on target thermal radiation to represent temperature distribution contours. These two feature spaces exist in significantly different semantic spaces. Directly calculating feature similarity between edge devices results in numerous mismatches, making cross-view target association impossible. Simultaneously, a complete cross-modal convolutional neural network capable of bridging the semantic gap between the two modes is deployed in the cloud. The edge devices lack the computing power to run a complete cross-modal convolutional neural network, and the lack of collaboration mechanisms between edge devices leads to a severe degradation of the overall image recognition situational awareness capability in this combined fault scenario, disrupting continuous tracking of critical situational events.

[0020] Therefore, the technical problem that this implementation method needs to solve is: in the case of a combined fault scenario of day-night alternation and cloud connection interruption, how to enable edge devices with limited computing power to have cross-modal feature space conversion capabilities, and to complete cross-device, cross-modal, and cross-view target association and situational awareness in the absence of cloud coordination.

[0021] It should be understood that the method of this embodiment is executed by processors deployed in the edge devices and their local area network interconnection environment within the urban surveillance network. Each edge device is equipped with an embedded computing unit and pre-installed with a domain-specific normalized convolutional neural network for local object detection. Edge devices communicate with each other via a local area network, and each edge device communicates with the cloud server via a wide area network.

[0022] Step 1: Train paired visible light-thermal infrared feature samples to obtain a bidirectional mapping function between the visible light feature space and the thermal infrared feature space. An image dataset consisting of paired visible light images and thermal infrared images acquired in the same scene is obtained. The paired images are then input into pre-trained visible light convolutional neural networks and thermal infrared convolutional neural networks, respectively, to extract paired convolutional feature samples of the same target in both modalities. Based on these paired convolutional feature samples, a cross-modal feature translation network is trained to learn a bidirectional mapping function between the visible light feature space and the thermal infrared feature space.

[0023] The cross-modal feature translation network consists of the following components: visible light modal encoder. Thermal infrared mode encoder Shared potential alignment layer Z, visible light to thermal infrared decoder and thermal infrared to visible light decoder The data transmission relationships between the various components are as follows: Visible light modal encoder Receive visible light convolution feature vectors The intermediate encoded representation is then output to the shared latent alignment layer Z by the thermal infrared modal encoder. Received thermal infrared convolutional feature vector The intermediate encoded representation is output to the shared latent alignment layer Z; the shared latent alignment layer Z maps the intermediate encoded representations of the two modalities to a unified latent semantic space; visible light to thermal infrared decoder. Receive from the visible light mode encoder in the shared potential alignment layer Z The latent representation of the path input is used to output the translation features in the thermal infrared modal space; thermal infrared to visible light decoder. Receive thermal infrared mode encoder from shared potential alignment layer Z The latent representation of the path input is used to output the translation features in the visible light modal space.

[0024] Furthermore, visible light mode encoders and thermal infrared mode encoder All are multilayer perceptron structures. The input layer receives convolutional feature vectors of their respective modalities, contains several hidden layers, and the output layer outputs the intermediate encoded representation. The shared latent alignment layer Z is a single fully connected layer; the input layer receives the intermediate encoded representation, and the output layer outputs the latent semantic representation. Visible light to thermal infrared decoder. and thermal infrared to visible light decoder All are multilayer perceptron structures. The input layer receives the latent semantic representation, contains several hidden layers, and the output layer outputs thermal infrared mode translation features and visible light mode translation features, respectively. The output layer is a fully connected layer.

[0025] Therefore, the bidirectional mapping function is expressed as: ; ; in, The convolutional feature vector in the visible light mode, The convolutional feature vector in the thermal infrared mode, Let be the mapping function from the visible light feature space to the thermal infrared feature space. Let be the mapping function from the thermal infrared feature space to the visible light feature space.

[0026] The total loss function used when training the cross-modal feature translation network is: ; in, The reconstruction loss is used to constrain the element-wise differences between the translated features and the true features in the target modality; The distribution alignment loss is used to constrain the latent representation distributions of the two modes to tend to be consistent after mapping through the shared latent alignment layer Z; This is the balance coefficient, and its value range is... .

[0027] Furthermore, reconstruction losses The calculation method is as follows: for each pair of samples in the paired convolutional feature samples, the visible light features are... through The mapped output corresponds to the true thermal infrared features of the target. Calculate the mean square error between them, and combine the thermal infrared features through The mapped output corresponds to the true visible light features of the target. The mean square error is calculated between the two directions, and the sum of the mean square errors in the two directions is taken as the reconstruction loss.

[0028] Furthermore, distribution alignment loss The maximum mean difference algorithm is used for calculation. Specifically, for all visible light features within a training batch... After encoding, the latent representation set output through the shared latent alignment layer Z is combined with all thermal infrared features. After encoding, the maximum mean difference between the two sets is calculated by sharing the latent representation set output by the latent alignment layer Z, and the maximum mean difference value is used as the distribution alignment loss.

[0029] Furthermore, the Adam optimization algorithm is used when training the cross-modal feature translation network, and the training mode is batch training, using a training set composed of paired convolutional feature samples for supervised learning.

[0030] Step 2: Compress the cross-modal feature translation network to generate a lightweight convolutional translation network, and pre-deploy the lightweight convolutional translation network to various edge devices. Using the trained cross-modal feature translation network as the teacher network, a knowledge distillation algorithm is employed to compress the cross-modal feature translation network. The input to the knowledge distillation algorithm consists of the teacher network and the student network, and the output is the trained lightweight convolutional translation network.

[0031] Specifically, a lightweight convolutional translation network with fewer parameters and less computational cost than a cross-modal feature translation network is predefined as the student network. The student network retains the same bidirectional mapping structure as the teacher network and includes a lightweight visible light modality encoder. Lightweight thermal infrared mode encoder Lightweight shared potential alignment layer Lightweight visible light to thermal infrared decoder and a lightweight thermal infrared to visible light decoder The number of hidden layer neurons and the number of layers in each component unit are reduced compared to the teacher network.

[0032] Furthermore, lightweight visible light mode encoder and lightweight thermal infrared mode encoder Both are multilayer perceptron architectures. The input layer receives convolutional feature vectors from their respective modalities, contains several hidden layers, and the output layer outputs the intermediate encoded representations. A lightweight shared latent alignment layer is also included. It is a single-layer fully connected layer. The input layer receives the intermediate encoded representation, and the output layer outputs the latent semantic representation. Lightweight visible light to thermal infrared decoder. and lightweight thermal infrared to visible light decoder All are multilayer perceptron structures. The input layer receives the latent semantic representation, contains several hidden layers, and the output layer outputs thermal infrared mode translation features and visible light mode translation features, respectively. The output layer is a fully connected layer.

[0033] Pairs of convolutional feature samples are simultaneously input into the teacher network and the student network. The intermediate representations output by each component of the teacher network and the final translation features are used as supervision signals to train the student network's mapping ability to approximate the teacher network, resulting in a lightweight convolutional translation network adapted to the computing power constraints of edge devices. The lightweight convolutional translation network is then pre-deployed to various edge devices in the urban surveillance network.

[0034] Furthermore, the distillation loss of the student network consists of two parts: one part is the mean squared error between the final translation feature output of the student network and the final translation feature output of the teacher network; the other part is the lightweight shared latent alignment layer of the student network. The mean squared error between the output of the student network and the output of the teacher network's shared latent alignment layer Z is used to ensure that the student network's representation in the latent semantic space is consistent with that of the teacher network. The two loss components are combined in a weighted summation manner, with the weight coefficients ranging from [value missing]. .

[0035] Furthermore, the Adam optimization algorithm is used when training the student network, and the training mode is batch training. Knowledge distillation learning is performed using the same training set consisting of paired convolutional feature samples as the teacher network.

[0036] Furthermore, while pre-deploying the lightweight convolutional translation network to each edge device, the network connection topology between the edge devices and the coverage area mapping of the cameras mounted on each edge device are also acquired, and these relationships are stored in the local storage of each edge device. The network connection topology records the connection paths and communication latency information of each edge device in the local area network; the coverage area mapping records the projected area of ​​the field of view of each camera in the geographic coordinate system and the overlapping area of ​​the field of view between adjacent cameras. The network connection topology and coverage area mapping are used for subsequent determination of coordination nodes and cross-field-of-view target association in edge collaboration mode.

[0037] Step 3: Monitor the connection status between each edge device and the cloud. When a connection interruption is detected, trigger the edge collaboration mode and obtain the current working mode identifier of each edge device. Each edge device sends heartbeat probe messages to the cloud server at preset time intervals and monitors the cloud server's response. If an edge device does not receive a response message from the cloud server within a preset number of consecutive heartbeat cycles, the connection between the edge device and the cloud is determined to be interrupted. Edge devices in the interrupted connection state trigger edge collaboration mode, broadcasting a collaboration request message containing their own device identifier and current operating mode identifier to neighboring edge devices via the local area network. Each edge device that receives the collaboration request message responds to the collaboration request message, sending back its own device identifier and current operating mode identifier. In this way, each edge device obtains the operating mode identifiers of all devices participating in edge collaboration within the local area network.

[0038] Furthermore, the current operating mode identifier is determined based on the current time period and camera type: when the edge device is equipped with a visible light camera and it is daytime, the current operating mode identifier is visible light mode; when the edge device is equipped with a thermal infrared camera or it is nighttime, the current operating mode identifier is thermal infrared mode.

[0039] Furthermore, to determine the coordinating node responsible for cross-device information aggregation and target association in the edge collaboration mode, each edge device calculates its centrality index within the local area network topology based on its locally stored network connection topology. The centrality index is calculated based on the shortest path hop count between each edge device and other edge devices: for edge device m, its centrality... Calculated as: ; Where N is the total number of edge devices participating in edge collaboration, m is the index of the edge device currently calculating centrality, and n is the index of other edge devices besides edge device m. This represents the number of hops along the shortest path between edge device m and edge device n. Centrality The edge device with the highest centrality value is selected as the coordinating node. After each edge device completes its coordination request interaction, they are ranked according to their centrality. The calculation results confirm the coordinating node, and the device identifier of the coordinating node is broadcast to all participating edge devices in the local area network.

[0040] Step 4: Detect targets in the local image, extract the initial feature vector for the current modality, and generate cross-modal translation features using a lightweight convolutional translation network. Each edge device inputs image frames captured in real time by its local camera into a domain-specific normalized convolutional neural network to perform target detection, obtaining the bounding box coordinates and initial feature vectors of each target detected in the current modality. The domain-specific normalized convolutional neural network refers to a convolutional neural network with normalized parameters configured for either the visible light modality or the thermal infrared modality. Devices using the visible light modality employ normalized parameters adapted to the statistical characteristics of visible light images, while devices using the thermal infrared modality employ normalized parameters adapted to the statistical characteristics of thermal infrared images.

[0041] Furthermore, the domain-specific normalized convolutional neural network takes an image frame as input and outputs the bounding box coordinates and initial feature vectors of each detected object. The domain-specific normalized convolutional neural network contains convolutional layers, normalization layers, activation layers, and fully connected layers; the fully connected layers output the initial feature vectors.

[0042] For each detection target, the initial feature vector is input into a pre-deployed lightweight convolutional translation network. Specifically, if the current device operating mode is identified as the visible light mode, the initial feature vector is... The visible light to thermal infrared mapping path is input to a lightweight convolutional translation network. Generate cross-modal translation features in the corresponding thermal infrared modal space. If the current operating mode of the device is identified as the thermal infrared mode, then the initial feature vector will be... Input the thermal infrared to visible light mapping path of the lightweight convolutional translation network Generate corresponding cross-modal translation features in the visible light modal space. .

[0043] Furthermore, and These are the visible light to thermal infrared mapping function and the thermal infrared to visible light mapping function obtained through knowledge distillation in a lightweight convolutional translation network, respectively. and The structure is consistent with the bidirectional mapping function defined in step 1, but the number of parameters in each component unit has been compressed, specifically as follows: ; ; Step 5: Aggregate feature data from various edge devices, perform cross-modal and cross-view target association, generate a unified target trajectory across the entire domain, and output the situational awareness results. Each edge device transmits the initial feature vector, cross-modal translation features, bounding box coordinates, modal identifier, and device identifier of each detected target to the coordination node determined in step 3 via the local area network.

[0044] After receiving data reported by each edge device, the coordination node performs the following processing on target pairs requiring cross-view association: It determines the target association strategy based on the modal identifiers of the targets to be associated. When two targets to be associated come from a visible light modal device and a thermal infrared modal device respectively, the coordination node selects the cross-modal translation features reported by the visible light modal device. The initial feature vector reported by the thermal infrared modal device The cosine similarity algorithm is used to calculate the similarity in the thermal infrared feature space. Simultaneously, cross-modal translation features reported by the thermal infrared modal device were selected. The initial feature vector reported by the visible light modal device Similarity is calculated using a cosine similarity algorithm within the visible light feature space. The average similarity in both directions is taken as the comprehensive cross-modal similarity of the target pair. ; in, Cosine similarity in thermal infrared feature space, It represents the cosine similarity in the visible light feature space.

[0045] The coordinating node combines the locally stored camera coverage area mapping relationship to perform comprehensive cross-modal similarity analysis. Target pairs exceeding a preset association threshold and whose bounding box coordinates fall within the overlapping areas of adjacent camera fields of view are identified as observations of the same target in different modalities. The coordination node performs cross-field association processing on the data reported by all participating edge devices, aggregates the association results from each edge device, assigns a unified identifier to each successfully associated target, and concatenates the timestamped target locations detected by each edge device according to the unified identifier to generate a unified global target trajectory under heterogeneous modalities. Based on the unified global target trajectory, the coordination node performs regional-level situational aggregation analysis on the target density and movement trends within each monitoring area, outputting continuous situational awareness results during edge collaboration.

[0046] Furthermore, the preset correlation threshold range is as follows: The preset association threshold is determined based on the similarity distribution of paired target features in the training dataset. Specifically, it is the minimum similarity value that makes the recall rate reach more than 95% when the similarity of the real paired targets is higher than the preset association threshold.

[0047] Furthermore, when two targets to be associated come from edge devices of the same modality, the coordinating node directly uses the initial feature vectors of the two targets to be associated to calculate the similarity using the cosine similarity algorithm in the same modal feature space, without needing to call cross-modal translation features.

[0048] Furthermore, in order to improve the robustness of cross-modal target association under the condition of limited feature translation accuracy, each edge device concatenates the initial feature vector with the cross-modal translation features along the channel dimension to obtain a concatenated feature vector, and inputs the concatenated feature vector into the cross-modal fusion module to generate a modally robust unified feature vector.

[0049] Furthermore, the cross-modal fusion module comprises the following units: a channel concatenation layer, a 1×1 convolutional dimensionality reduction layer, a batch normalization layer, and a nonlinear activation layer. The channel concatenation layer concatenates the initial feature vector with the cross-modal translation features along the channel dimension, obtaining a concatenated feature vector with a dimension twice that of the original feature vector. The 1×1 convolutional dimensionality reduction layer performs channel compression on the concatenated feature vector, reducing it to the same dimension as the initial feature vector. The batch normalization layer normalizes the dimensionality-reduced features. The nonlinear activation layer performs a nonlinear transformation on the normalized features, outputting a unified feature vector.

[0050] Furthermore, the cross-modal fusion module takes a concatenated feature vector as input and outputs a unified feature vector. The output layer of the 1×1 convolutional dimensionality reduction layer is a fully connected layer. The Adam optimization algorithm is used to train the cross-modal fusion module, with end-to-end training mode and the loss function being the mean squared error between the unified feature vector and the true target features.

[0051] Furthermore, each edge device simultaneously reports a unified feature vector when transmitting data to the coordination node. When performing cross-modal target association, the coordination node will integrate cross-modal similarity. Similarity between the vector and the target vector calculated using the cosine similarity algorithm Weighted fusion is performed to obtain enhanced correlation similarity: ; in, The similarity between the unified feature vectors of two targets to be associated is calculated using the cosine similarity algorithm. The weighting coefficient has a range of values. Coordinating nodes to enhance association similarity. Alternative comprehensive cross-modal similarity Perform association determination.

[0052] Furthermore, to compensate for the accuracy loss of the lightweight convolutional translation network relative to the full cross-modal convolutional neural network during edge collaboration after cloud connectivity is restored, and to restore the temporal continuity of global situational awareness, each edge device continuously monitors the connection status between itself and the cloud server. When the heartbeat detection message receives a response from the cloud server again, it is determined that the cloud connection has been restored. Each edge device uploads the multimodal detection results accumulated during edge collaboration (including the initial feature vectors of each target, cross-modal translation features, bounding box coordinates, timestamps, and modal identifiers) and the global unified target trajectory generated by the coordination node to the cloud server in batches.

[0053] Furthermore, after receiving the multimodal detection results and the unified target trajectory across the entire domain, the cloud server re-inputs the original image regions of each target into a complete cross-modal convolutional neural network for feature extraction, obtaining a high-precision cross-modal feature representation. Based on the high-precision cross-modal feature representation, the target association similarity is recalculated, and the association results during edge collaboration are refined and corrected, correcting any erroneous or missed associations caused by insufficient accuracy of the lightweight convolutional translation network.

[0054] Furthermore, the input to the complete cross-modal convolutional neural network is the original image region, and the output is a high-precision cross-modal feature representation. The complete cross-modal convolutional neural network includes convolutional layers, normalization layers, activation layers, and fully connected layers, with the fully connected layers outputting a high-precision cross-modal feature representation.

[0055] Furthermore, the cloud server performs time-series alignment and fusion of the refined and corrected target trajectory with the historical situational data stored in the cloud. Specifically, using the timestamp and target state of the last frame of cloud situational data at the moment of cloud connection interruption as alignment anchors, the refined and corrected target trajectory during edge collaboration is inserted into the corresponding time interval in the historical situational sequence according to the timestamp order. A linear interpolation algorithm is used to smooth the target state at the alignment anchor and at the moment of connection restoration, eliminating the discontinuous jumps in the temporal sequence of situational data before and after the network outage, and restoring the continuity of global image recognition situational awareness.

[0056] Technical effects of this embodiment: This implementation addresses the combined fault scenario of day-night transition and cloud connection interruption, employing a technique combining a cross-modal feature translation network and knowledge distillation. Because the cross-modal feature translation network learns a bidirectional mapping function between the visible light feature space and the thermal infrared feature space through a shared latent alignment layer, and this bidirectional mapping function is compressed into a lightweight convolutional translation network adapted to edge computing power and pre-deployed to each edge device via knowledge distillation, each edge device can still map its own modal features to the counterpart modal feature space during cloud connection interruption. This overcomes the semantic differences in feature space caused by visible light features depending on reflected light texture details and thermal infrared features depending on the target's thermal radiation temperature distribution contour, enabling similarity calculations for features extracted by different modal devices within the same feature space.

[0057] Furthermore, this implementation employs a local area network broadcasting, coordination node aggregation, and cross-modal, cross-view target association mechanism within an edge collaboration mode. Since each edge device broadcasts its modality identifier and identifies a coordination node via the local area network after detecting a cloud connection interruption, the coordination node performs bidirectional similarity calculations in the corresponding feature space based on the modality identifiers of the target pair and combines this with the mapping relationship of the camera coverage area for association determination. This compensates for the lack of a single edge device's full-domain field of view and cross-device coordination capabilities, achieving cross-device information exchange and cross-view target association during network outages, maintaining the continuous operation of generating a unified target trajectory across the entire domain and regional situational aggregation analysis.

[0058] Furthermore, the modal robust unified feature vector generated by channel-dimensional splicing and fusion integrates complementary information from the original modal features and translated features, enhancing the stability of cross-modal target association under the condition of limited accuracy of the lightweight convolutional translation network. After the cloud connection is restored, the edge association results are refined and corrected using a complete cross-modal convolutional neural network and temporally aligned and fused with historical situational data, compensating for the accuracy loss of the lightweight convolutional translation network and eliminating the temporal discontinuity of situational data before and after the network outage. Therefore, this implementation maintains the continuous operation capability of global image recognition situational awareness under the combined fault scenario of day-night alternation and cloud connection interruption.

[0059] A city's security monitoring network experienced a wide area network (WAN) outage during dusk (approximately 6:30 PM to 7:15 PM) on a certain day in 20XX. The network deployed five edge devices in the city's core area: devices A, C, and E equipped with visible light cameras, and devices B and D equipped with thermal infrared cameras. Dusk falls during the critical period between day and night, meaning both visible light and thermal infrared cameras were online simultaneously, with their extracted feature vectors residing in different semantic spaces. The cloud connection outage rendered the complete cross-modal convolutional neural network unavailable, forcing the edge devices to rely on a pre-deployed lightweight convolutional translation network to independently perform cross-modal target association and situational awareness.

[0060] Step 1 Implementation Example: Training paired visible light-thermal infrared feature samples to obtain a bidirectional mapping function. During the offline phase, the system acquires paired images from a group of visible light and thermal infrared binocular cameras installed at the same site in the city's core area, covering three types of targets: pedestrians, vehicles, and non-motorized vehicles, obtaining a total of 42,000 pairs of paired image samples. The paired images are then input into pre-trained visible light convolutional neural networks and thermal infrared convolutional neural networks, respectively, to extract a 512-dimensional initial feature vector for each target, forming 42,000 pairs of paired convolutional feature samples.

[0061] Taking a pair of pedestrian target feature samples as an example, the first 8 components of the visible light feature vector reflect texture details (clothing color, edge gradient, etc.), while the first 8 components of the thermal infrared feature vector reflect the temperature distribution profile (torso thermal radiation intensity, etc.). The distribution characteristics of the two are significantly different.

[0062] The cross-modal feature translation network structure is set as follows: the visible light modal encoder and the thermal infrared modal encoder are both 3-layer perceptrons with hidden layer dimensions of 256 and 128, respectively; the shared latent alignment layer is a single fully connected layer with a latent space dimension of 64; the two decoders are each 3-layer perceptrons with an output dimension of 512.

[0063] Take the balance coefficient during training Based on the reconstruction losses of a certain batch Distribution alignment loss For example, the total loss is: ; After 80 training rounds, the bidirectional mapping function converged, and the reconstruction loss stabilized below 0.018.

[0064] Table 1. Examples of paired feature samples (pedestrian targets, first 8 dimensions of feature vector):

[0065] Step 2 Implementation Example: Knowledge distillation and compression to generate a lightweight convolutional translation network and pre-deploy it to edge devices: The trained cross-modal feature translation network was used as the teacher network. The student network (lightweight convolutional translation network) compressed the hidden layer dimensions of each encoder to half that of the teacher network. The encoder hidden layer dimensions were 128 and 64, respectively, and the latent space dimension was compressed to 32. The decoder hidden layer dimensions were reduced accordingly, and the total number of parameters was approximately 23% of that of the teacher network.

[0066] The distillation loss is a weighted sum of two parts, with a weighting coefficient of 0.5: one part is the mean squared error of the student network's translation feature output and the teacher network's translation feature output, and the other part is the mean squared error of the student network's lightweight shared latent alignment layer output and the teacher network's shared latent alignment layer output. Taking a certain batch as an example, the former is 0.0241, and the latter is 0.0189, so the distillation loss is: ; Simultaneously, the network connection topology between each edge device and the camera coverage area mapping are stored locally. The network topology and camera coverage information for the 5 edge devices are as follows: Table 2. Basic Information and Network Topology of Edge Devices:

[0067] Step 3 Implementation Example: Detect cloud connection interruption, trigger edge collaboration mode, and determine the coordination node: At 18:47:23 on a certain day in 20XX, device C did not receive a response from the cloud for 5 consecutive heartbeat cycles (12 seconds per cycle, 60 seconds in total), and determined that the cloud connection was interrupted, triggering the edge collaboration mode. Device C broadcast a collaboration request message through the local area network, and devices A, B, D, and E all returned their respective device identifiers and current working mode identifiers.

[0068] The current time period is dusk. Devices A, C, and E (visible light cameras) are identified as being in visible light mode, while devices B and D (thermal infrared cameras) are identified as being in thermal infrared mode.

[0069] The centrality of each device is calculated based on the hop count matrix in Table 2. Taking device C as an example, the total number of devices N=5:

[0070] The centerness calculation results for each piece of equipment are as follows: Table 3: Centrality calculation results for each edge device:

[0071] Device C has the highest centrality (1.000) and is selected as the coordinating node, broadcasting the result to all participating devices.

[0072] Step 4 Implementation Example: Object Detection, Initial Feature Extraction, and Cross-Modal Translation Feature Generation: At 18:48:05, each edge device performs target detection on the current image frame. Taking pedestrian target A-P01 detected by device A (visible light mode) and target B-P02, which is suspected to be the same pedestrian, detected by device B (thermal infrared mode), as examples, the generation process of cross-modal translation features is explained.

[0073] Device A extracts the 512-dimensional initial feature vector of A-P01 and inputs it into the visible light to thermal infrared mapping path of a pre-deployed lightweight convolutional translation network to generate thermal infrared modal translation features. Device B extracts the 512-dimensional initial feature vector of B-P02 and inputs it into the thermal infrared to visible light mapping path to generate visible light modal translation features.

[0074] Meanwhile, device A concatenates the initial feature vector of A-P01 with the generated thermal infrared modal translation features to obtain a 1024-dimensional concatenated feature vector. After 1×1 convolutional dimensionality reduction, batch normalization and nonlinear activation by the cross-modal fusion module, a 512-dimensional unified feature vector is output. Device B generates a 512-dimensional unified feature vector for the B-P02 target in the same way.

[0075] Table 4. Target detection and feature generation results for each edge device (at 18:48:05):

[0076] Step 5 Implementation Example: Cross-modal and cross-view target association, global unified trajectory generation, and situational awareness output: Each edge device reports the above data to the coordination node (device C). There is an overlapping area between the north side of the East Gate Plaza view of device A and the south side of the East Gate Plaza view of device B (the passageway connection area, whose geographical coordinates are already stored locally). The coordination node performs correlation calculations on the cross-modal target pair A-P01 and B-P02.

[0077] Taking A-P01 and B-P02 as examples: The thermal infrared modal translation features reported by device A (already mapped to thermal infrared space) and the initial feature vector reported by device B are selected. Cosine similarity is calculated in the thermal infrared feature space, yielding a thermal infrared similarity SR = 0.8731. The visible light modal translation features reported by device B (already mapped to visible light space) and the initial feature vector reported by device A are selected. Cosine similarity is calculated in the visible light feature space, yielding a visible light similarity SV = 0.8612. The overall cross-modal similarity is: ; Meanwhile, the cosine similarity between the two target unified feature vectors, Suni = 0.8814, with a weighting coefficient α = 0.6, enhances the association similarity as follows: ; The preset association threshold is 0.75, Senhanced=0.8730>0.75, and the bounding box coordinates of A-P01 and B-P02 both fall within the overlapping area of ​​the fields of view of device A and device B. Therefore, they are determined to be the same pedestrian target and are assigned a unified identifier UID-001.

[0078] Table 5. Results of correlation calculations for major cross-modal target pairs:

[0079] The coordination node aggregates the timestamped location sequences of all successfully associated targets, generating a unified target trajectory across the entire region. Taking UID-001 as an example, its trajectory records a continuous movement path in the East Gate Square area from 18:47:58 to 18:49:22. Based on the unified target trajectory across the entire region, the coordination node calculates the pedestrian density in the East Gate Square area to be 3.2 people / 100 square meters, with a movement trend of converging towards the central passage, and outputs continuous situational awareness results.

[0080] At 19:02:47, the heartbeat detection messages from each device received a response from the cloud server, and the cloud connection was restored. Each device uploaded the multimodal detection results and unified target trajectories accumulated during edge collaboration in batches. The cloud server input the original image regions of each target into a complete cross-modal convolutional neural network to extract high-precision cross-modal features, recalculate the association similarity, and refine and correct the association results such as UID-001. After correction, using the last frame of cloud situational awareness data at 18:47:23 (the time of network outage) as the alignment anchor, the refined target trajectories from the edge collaboration period were inserted into the historical situational awareness sequence, and linear interpolation smoothing was applied to the target states before and after the network outage to restore the temporal continuity of the global situational awareness.

[0081] The entire data flow starts with offline pairing feature training in step 1. Through knowledge distillation in step 2, the bidirectional mapping capability is compressed to the edge, laying the foundation for subsequent online processing. Centrality calculation in step 3 relies on the pre-stored network topology data from step 2, ensuring the rationality of the selected coordination nodes. The translation and unified features in step 4 are directly generated by the compressed mapping function trained in step 1, with strict consistency in feature dimensions and semantic space across steps. The comprehensive similarity calculation in step 5 directly consumes the translation and unified features output from step 4 and combines them with the pre-stored field-of-view overlap region information from step 2 to complete the association determination. The unified target trajectory across the entire domain is formed by concatenating the association results at each time step. Finally, after cloud refinement and temporal alignment, it regresses to a complete historical situational sequence, demonstrating the vertical transmission and logical closure of data across each step.

[0082] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A global image recognition and situational awareness system based on convolutional neural networks, characterized in that, Includes the following steps: Based on paired visible light and thermal infrared images acquired in the same scene, paired convolutional feature samples are extracted to train a cross-modal feature translation network, obtaining a bidirectional mapping function between the visible light feature space and the thermal infrared feature space. Using the cross-modal feature translation network as the teacher network, a knowledge distillation algorithm is used to compress the cross-modal feature translation network to obtain a lightweight convolutional translation network, which is pre-deployed to each edge device. Each edge device monitors the connection status with the cloud server. When a connection interruption is detected, an edge collaboration mode is triggered. The device identifier and current working mode identifier of each edge device are exchanged through the local area network, and a coordination node is determined based on the network connection topology. Each edge device performs target detection on the locally acquired image frames to obtain an initial feature vector, and inputs the initial feature vector into the corresponding mapping path of the lightweight convolutional translation network to generate cross-modal translation features. Each edge device transmits its initial feature vector, cross-modal translation features, bounding box coordinates, and modal identifier to the coordination node. The coordination node calculates bidirectional similarity in the corresponding feature space based on the modal identifier of the target pair to be associated and takes the average as the comprehensive cross-modal similarity. Combined with the mapping relationship of the camera coverage area, target pairs exceeding the preset association threshold are determined to be the same target, generating a unified target trajectory across the entire domain and outputting the situational awareness result.

2. The method according to claim 1, characterized in that, The cross-modal feature translation network includes a visible light modal encoder, a thermal infrared modal encoder, a shared latent alignment layer, a visible light to thermal infrared decoder, and a thermal infrared to visible light decoder. The visible light modal encoder receives visible light convolutional feature vectors and outputs intermediate encoded representations to the shared latent alignment layer. The thermal infrared modal encoder receives thermal infrared convolutional feature vectors and outputs intermediate encoded representations to the shared latent alignment layer. The shared latent alignment layer maps the intermediate encoded representations of the two modalities to a unified latent semantic space. The visible light to thermal infrared decoder receives the latent representations input through the path of the visible light modal encoder and outputs thermal infrared modal translation features. The thermal infrared to visible light decoder receives the latent representations input through the path of the thermal infrared modal encoder and outputs visible light modal translation features. The bidirectional mapping function is as follows: the mapping function from the visible light feature space to the thermal infrared feature space is composed of the visible light modal encoder, the shared latent alignment layer, and the visible light to thermal infrared decoder cascaded sequentially; the mapping function from the thermal infrared feature space to the visible light feature space is composed of the thermal infrared modal encoder, the shared latent alignment layer, and the thermal infrared to visible light decoder cascaded sequentially.

3. The method according to claim 2, characterized in that, The total loss function used when training the cross-modal feature translation network consists of a weighted sum of reconstruction loss and distribution alignment loss. The reconstruction loss is the sum of the mean square error between the output of the visible light feature mapped to thermal infrared and the real thermal infrared feature, and the mean square error between the output of the thermal infrared feature mapped to visible light and the real visible light feature. The distribution alignment loss uses the maximum mean difference algorithm to calculate the maximum mean difference between two latent representation sets output through a shared latent alignment layer after being encoded by the visible light modal encoder and the thermal infrared modal encoder, respectively, within a training batch.

4. The method according to claim 1, characterized in that, The lightweight convolutional translation network, as the student network, retains the same bidirectional mapping structure as the cross-modal feature translation network. The number of hidden layer neurons and layers in each component unit are reduced compared to the cross-modal feature translation network. The distillation loss of the student network is composed of the mean squared error between the final translation feature output of the student network and the final translation feature output of the teacher network, and the mean squared error between the output of the lightweight shared latent alignment layer of the student network and the output of the shared latent alignment layer of the teacher network, in a weighted summation.

5. The method according to claim 1, characterized in that, The determination of the coordination node based on the network connection topology relationship includes: each edge device calculates the centrality index of each edge device in the local area network topology according to the network connection topology relationship stored locally. The centrality index is the ratio of the total number of edge devices participating in edge collaboration minus one to the sum of the shortest path hops from the current edge device to all other edge devices. The edge device with the largest centrality index value is selected as the coordination node.

6. The method according to claim 1, characterized in that, The coordination node calculates bidirectional similarity in the corresponding feature space based on the modal identifiers of the target pair to be associated, including: when the two targets to be associated come from a visible light modal device and a thermal infrared modal device, respectively, the cross-modal translation features reported by the visible light modal device and the initial feature vector reported by the thermal infrared modal device are selected in the thermal infrared feature space and the cosine similarity algorithm is used to calculate the first similarity; the cross-modal translation features reported by the thermal infrared modal device and the initial feature vector reported by the visible light modal device are selected in the visible light feature space and the cosine similarity algorithm is used to calculate the second similarity; the average of the first similarity and the second similarity is taken as the comprehensive cross-modal similarity. When the two targets to be associated come from edge devices of the same modality, the initial feature vectors of the two targets to be associated are directly used in the same modal feature space and the cosine similarity algorithm is used to calculate the similarity.

7. The method according to claim 1, characterized in that, Each edge device concatenates the initial feature vector with the cross-modal translation features along the channel dimension to obtain a concatenated feature vector, and inputs the concatenated feature vector into the cross-modal fusion module to generate a unified feature vector; The cross-modal fusion module includes a channel concatenation layer, a 1×1 convolutional dimensionality reduction layer, a batch normalization layer, and a non-linear activation layer. The channel concatenation layer concatenates the initial feature vector with the cross-modal translation features along the channel dimension. The 1×1 convolutional dimensionality reduction layer reduces the concatenated feature vector to the same dimension as the initial feature vector. The batch normalization layer normalizes the dimensionality-reduced features. The non-linear activation layer outputs a unified feature vector. The coordination node performs a weighted fusion of the comprehensive cross-modal similarity and the similarity calculated using the cosine similarity algorithm between the unified feature vector to obtain an enhanced association similarity. This enhanced association similarity is used to replace the comprehensive cross-modal similarity in the association determination.

8. The method according to claim 1, characterized in that, When each edge device detects that the cloud connection has been restored, it will upload the multimodal detection results accumulated during the edge collaboration and the unified target trajectory generated by the coordination node to the cloud server in batches. The cloud server re-inputs the original image regions of each target into a complete cross-modal convolutional neural network for feature extraction to obtain a high-precision cross-modal feature representation. Based on the high-precision cross-modal feature representation, the target association similarity is recalculated, and the association results during edge collaboration are refined and corrected.

9. The method according to claim 8, characterized in that, The cloud server performs time-series alignment and fusion of the refined and corrected target trajectory with the historical situation data stored in the cloud. Using the timestamp and target state of the last frame of cloud situation data at the moment of cloud connection interruption as the alignment anchor point, the refined and corrected target trajectory during edge collaboration is inserted into the corresponding time interval in the historical situation sequence according to the timestamp order. The target state at the alignment anchor point and at the moment of connection restoration is smoothed using a linear interpolation algorithm.

10. A global image recognition and situational awareness system based on a convolutional neural network, used to execute the method according to any one of claims 1 to 9, characterized in that, include: The cross-modal mapping training module is used to extract paired convolutional feature samples based on paired acquisition of visible light images and thermal infrared images, and train the cross-modal feature translation network to obtain a bidirectional mapping function between the visible light feature space and the thermal infrared feature space; the network compression and deployment module is used to obtain a lightweight convolutional translation network by using the cross-modal feature translation network as the teacher network and employing a knowledge distillation algorithm, and pre-deploy the lightweight convolutional translation network to each edge device; The connection monitoring and coordination triggering module is used to monitor the connection status between each edge device and the cloud server. When a connection interruption is detected, the edge coordination mode is triggered, the device identifier and current working mode identifier of each edge device are exchanged, and the coordination node is determined based on the network connection topology. The feature extraction and translation module is used by each edge device to perform target detection on locally acquired image frames to obtain initial feature vectors, and input the initial feature vectors into a lightweight convolutional translation network to generate cross-modal translation features. The cross-modal association and situational awareness output module is used to aggregate feature data from various edge devices to the coordination node. Based on the modal identifier, it calculates the bidirectional similarity in the corresponding feature space and takes the average as the comprehensive cross-modal similarity. Combined with the mapping relationship of the camera coverage area, it performs target association determination, generates a unified target trajectory across the entire domain, and outputs the situational awareness results.

Images