Target detection method and device fusing multi-source semantic information, medium and equipment

Abstract

The invention relates to a target detection method and device fusing multi-source semantic information, a medium and equipment, and aims to solve the problems of low target detection precision, poor environmental adaptability and the like in a complex natural scene. The method comprises the following steps: receiving a monitoring image and extracting meta-information such as shooting time, shooting place and PTZ (Pan / Tilt / Zoom); performing space-time coding on the meta-information to generate a space-time embedded vector; generating a static semantic mask graph containing semantic confidence; and fusing the image, the semantic mask graph and the space-time embedded vector, inputting the fused image, the semantic mask graph and the space-time embedded vector into an improved YOLOv11 model for detection, and outputting a result. The device comprises a text detection module, a space-time coding module, a PTZ coding module, a static semantic segmentation module, an improved YOLOv11 network module and a training module. Through multi-source information fusion and structural improvement, complex scene detection precision is remarkably improved, space-time visual angle adaptability is enhanced, multi-scale adaptive detection is supported, dependence on auxiliary information is weakened, training efficiency is high, extra hardware is not needed, and the method is suitable for large-scale monitoring scenes of forestry, agriculture and the like.

Description

Technical Field

[0001] This disclosure belongs to the field of target detection technology, and specifically relates to a target detection method, apparatus, medium and equipment that integrates multi-source semantic information. Background Technology

[0002] With the widespread deployment of intelligent monitoring equipment and the popularization of high-definition pan-tilt-zoom (PTZ) cameras, the acquisition of high-resolution images in natural environments (such as forest areas, farmland, and the Loess Plateau) has become highly feasible. These images provide a rich visual data foundation for target detection tasks in complex scenes, powerfully promoting the intelligentization process in fields such as smart forestry and ecological environment monitoring. Meanwhile, computer vision algorithms (such as target detection, image segmentation, and semantic recognition) have made significant breakthroughs in recent years, with detection models represented by the YOLO series and Faster R-CNN being widely used in practical scenarios such as human and vehicle target recognition. However, the dynamic characteristics of the natural environment (such as light intensity, seasonal changes, and diurnal variations) pose a severe challenge to the generalization ability of models. Existing models often experience a significant decline in detection performance due to changes in target appearance in environments with significant seasonal, temporal, or lighting differences.

[0003] Current mainstream methods are mostly based on deep learning models (such as YOLO and Faster R-CNN for object detection; DeepLab, PSPNet, and Segment Anything for semantic segmentation), improving detection accuracy and speed by optimizing model structure (such as introducing attention mechanisms, lightweight modules, and multi-scale fusion structures (BiFPN, CSP, etc.)). For example, the published patent CN116681646A enhances small object detection capabilities by embedding CoordConv and a multi-head prediction module into YOLOv5. Although such methods perform well in standard scenarios, they still rely on the visual features of the image ontology and do not fully exploit prior environmental information (such as camera spatial location, temporal attributes (season / time period), historical semantic information, etc.). This leads to missed detections and false detections in scenarios with drastic environmental changes (such as the growth cycle of forest vegetation, seasonal cover of bare soil in farmland, lodging of tall crops, etc.), severely limiting generalization ability.

[0004] Based on the above analysis, the current object detection technology has the following main shortcomings in the application of complex natural scenes: (1) Lack of environmental perception: Single-modal detection and segmentation methods rely too much on image visual features and do not make full use of the spatial location information, temporal attributes (season / time period) or historical semantic information of the camera, resulting in insufficient generalization ability of the model when environmental conditions change significantly.

[0005] (2) Diminishing marginal benefits of structural optimization: As model structure optimization matures (such as stacking deeper networks and designing more complex attention mechanisms), the performance improvement space of simply relying on model structure improvement in non-ideal scenarios is very limited.

[0006] (3) High cost of multi-source fusion: Although multi-sensor fusion solutions can improve robustness, the hardware deployment and system maintenance costs are too high, making it difficult to adapt to the needs of low-cost, large-scale monitoring.

[0007] To address the aforementioned technical bottlenecks, there is an urgent need for a target detection method that requires no additional hardware investment and can effectively integrate multi-dimensional prior information. Summary of the Invention

[0008] To address the aforementioned issues, this disclosure proposes a novel target detection method, apparatus, medium, and device that integrates multi-source semantic information. It aims to solve the problems of low recognition accuracy, poor environmental adaptability, lack of prior modeling capabilities, limited structural optimization effects, and high deployment costs inherent in existing target detection algorithms in complex natural scenes. This disclosure integrates temporal information (such as season and time period), spatial information (such as camera location and PTZ parameters), and static semantic information (such as background segmentation maps) from image acquisition to construct a spatial-temporal encoding. Simultaneously, it utilizes static semantic segmentation maps to assist the model in perceiving scene semantics, achieving contextual understanding and robust detection of surveillance images, thus improving the model's contextual understanding and robustness in complex environments. This disclosure significantly improves generalization ability in simple detection models (such as YOLOv11) without increasing hardware costs, making it particularly suitable for time- and space-sensitive target recognition scenarios such as forestry, agriculture, and bare surface monitoring.

[0009] Specifically, this disclosure aims to achieve the following technical objectives: 1. Improve detection robustness under different seasons, time periods, and changing shooting angles.

[0010] To address the problem that traditional YOLO-like object detection algorithms are highly sensitive to changes in lighting and background complexity in natural environments (such as woodlands, farmland, and bare soil areas), and are prone to false negatives and missed detections, this disclosure constructs spatial-temporal coding features by introducing shooting time (season, time of day), location, and PTZ parameters. This effectively enhances the model's environmental perception capabilities and maintains a high recognition accuracy under complex spatiotemporal conditions.

[0011] 2. Enhance the model's ability to understand static semantic context.

[0012] A static semantic map is generated from the first frame of a surveillance video after removing dynamic targets. This map serves as auxiliary information input to the detection model, guiding the model to understand the scene structure and stable background elements, thereby improving the accuracy of dynamic target detection. This method is particularly suitable for fixed monitoring points where the background remains unchanged for a long time but the lighting and climate changes drastically.

[0013] 3. Reduce reliance on complex model structure optimization and improve the performance of general-purpose models.

[0014] Compared to existing technologies that rely on complex structures such as attention mechanisms and multi-scale convolutions to improve performance, this disclosure focuses on semantic priors and spatial-temporal coding, and improves the model's generalization ability through information fusion, thus possessing higher structural compatibility and deployment flexibility.

[0015] 4. Improve overall detection performance without increasing hardware costs.

[0016] To address the drawback of high deployment costs associated with infrared / multimodal fusion solutions, which offer good performance, this disclosure utilizes extractable prior information (temporal, spatial, and static semantics) from images to achieve significant performance improvements without requiring additional hardware, demonstrating strong potential for engineering applications.

[0017] To achieve the above objectives, this disclosure provides the following technical solutions: The first aspect of this disclosure provides a target detection method that integrates multi-source semantic information, the method comprising: S1: Receive the monitoring image and extract the metadata of the monitoring image, including the shooting time, shooting location, and PTZ parameters (camera attitude parameters). S2: Perform spatial-temporal encoding on the metadata to generate a spatiotemporal embedding vector; S3: Generate a static semantic mask image corresponding to the monitoring image; S4: The monitoring image, static semantic mask image and spatiotemporal embedding vector are fused together, and then input into the target detection model for target detection, and the detection result is output.

[0018] Optionally, in the method of this disclosure, the extraction of metadata from the surveillance image in step S1 includes: The text regions in the surveillance image are detected by OCR technology, the text content is identified by CRNN model, and then the text content is classified by image-text joint classification network to extract structured time, location, and PTZ parameters. The time parameter supports the normalization of various date formats, the location parameter generates a complete administrative address by merging multiple lines of text, and the PTZ parameter parses the horizontal angle, pitch angle, and zoom of the camera.

[0019] Optionally, in the method of this disclosure, step S2, which involves spatial-temporal encoding of the metadata to generate a spatiotemporal embedding vector, includes: The shooting location is converted into a geographic region code to generate a spatial code, and the shooting time is converted into a season and time period code to generate a time code. The PTZ parameters are normalized and then concatenated with the spatial and time codes. The spatiotemporal embedding vector is generated by mapping through a lightweight fully connected network. Among them, the geographic region code is based on the camera name and mapped to the provincial administrative region and the default position. The season code is divided into spring, summer, autumn, winter and the default category. The time period code is divided into morning, noon, afternoon, evening and the default category.

[0020] Optionally, in the method of this disclosure, the step S3 of generating a static semantic mask map corresponding to the monitoring image includes: S31: For multiple frames of images captured from the same camera location, a target detection model is used to identify dynamic targets and generate dynamic masks. Multiple dynamic masks are then fused to generate a pseudo-static background image. S32: Perform semantic segmentation on the pseudo-static background image and extract the semantic confidence map as a soft prior; S33: The pixel difference between the original image and the pseudo-static background image is used as an additional channel input to the semantic segmentation model to enhance the semantic boundary alignment capability of the repaired region and output a static semantic mask image.

[0021] Optionally, in the method of this disclosure, the target detection model in step S4 is an improved YOLOv11 network structure, and step S4 includes: S41: A semantic segmentation confidence map guidance module is introduced at the output of the backbone feature extraction network, and the foreground enhancement and background suppression information of the semantic mask map are fused through the Cross-Attention mechanism; S42: The spatiotemporal embedding vector is expanded and added to the Cross-Attention fusion feature to achieve global context control over local detection; S43: A dynamic scale-aware fusion module (DSAF) is introduced in the detection head. Multi-scale features are extracted through global average pooling, the fusion weights of each detection branch are calculated and adjusted, and adaptive multi-scale detection is achieved.

[0022] Optionally, in the method of this disclosure, the target detection model is trained using a collaborative training strategy of multi-source information consistency alignment, including: (1) Construct an auxiliary information consistency regularization term, generate prediction results through dual input paths (complete auxiliary input and image input only), and introduce KL divergence loss to constrain the consistency of the two prediction distributions; (2) A two-stage hybrid label training mechanism is adopted. In the first stage, the backbone network is pre-trained, and in the second stage, auxiliary inputs are introduced and weighted consistency loss is applied to enhance the ability to utilize spatiotemporal context.

[0023] Optionally, in the method disclosed herein, the backbone network of the target detection model is selected from any one of RT-DETR, DETR, Faster R-CNN, EfficientDet, CenterNet, and RetinaNet; The static semantic mask image is generated using any one of the following segmentation models: SAM (Segment Anything Model), DeepLabV3+, HRNet, SegFormer, UpperNet, and BiSeNet. The feature fusion method can be selected from any one of the following: Feature Guided Fusion Module (FGFM), Gating Unit, Conditional Convolution (CondConv), SE Module (Feature Selection Module), and ECA.

[0024] A second aspect of this disclosure provides a target detection apparatus that integrates multi-source semantic information, the apparatus comprising: The text detection and recognition module is used to receive monitoring images and extract metadata from the monitoring images. The metadata includes shooting time, shooting location, and PTZ parameters (camera attitude parameters). The space-time encoding module is used to perform space-time encoding on the metadata to generate a spatiotemporal embedding vector; The PTZ information feature encoding module is used to normalize the PTZ parameters and map them into higher-order semantic representations; A static image semantic segmentation module is used to generate a static semantic mask image corresponding to the monitoring image; An improved YOLOv11 network structure module is used to fuse the monitoring image, static semantic mask map, and spatiotemporal embedding vector, and then perform target detection and output the detection result; An improved YOLOv11 model training module is used to train the model using a multi-source information consistency alignment strategy.

[0025] The device implements the steps of the aforementioned target detection method that integrates multi-source semantic information during operation.

[0026] A third aspect of this disclosure provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned target detection method that integrates multi-source semantic information.

[0027] A fourth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned target detection method that integrates multi-source semantic information.

[0028] In summary, this disclosure proposes a novel target detection method, apparatus, medium, and device that integrates multi-source semantic information. Through multi-source information fusion and structured improvement, this disclosure achieves the following beneficial technical effects: 1. Significantly improved detection accuracy in complex scenes: By constructing a multi-source detection system that integrates image, temporal, spatial, PTZ parameters and static semantic segmentation maps with confidence, and combining channel attention and Cross-Attention mechanisms to dynamically integrate spatiotemporal semantic information into the YOLOv11 backbone network, and using the DSAF module to enhance multi-scale feature expression, the system effectively suppresses background interference and false target detection. It is particularly suitable for target detection tasks in static background complex and texture similar scenes such as mountains, farmland and forest areas.

[0029] 2. Enhanced Adaptability to Spatiotemporal and Perspective Changes: A spatial-temporal encoding module and a PTZ information feature encoding module are designed to integrate high-level semantic information such as season, time of day, and camera rotation angle into the YOLO feature stream through feature embedding. This guides the network to perform scene perception and feature reweighting, thereby improving the model's stability in recognizing image differences under different times, angles, and regions. Simultaneously, multi-frame background fusion and LAMA inpainting are used to generate pseudo-static images, which are then semantically segmented to obtain high-confidence auxiliary images, achieving effective identification of occluded areas and suppression of false detections.

[0030] 3. Achieve scale-adaptive detection capability: Based on the Dynamic Scale Aware Fusion Module (DSAF), global scale features are extracted through Global Average Pooling (GAP) and the fusion weights of the three-scale detection heads are calculated. The feature response can be adaptively adjusted according to the target size distribution (such as the ratio of far-small / near-large targets), which significantly enhances the model's bidirectional perception capability for far-small and near-large targets.

[0031] 4. Improve the robustness of the model in scenarios with weak annotations / missing auxiliary information: In the training phase, a collaborative training strategy with consistent alignment of multi-source information is introduced. An auxiliary information regularization term and a two-stage training mechanism are constructed. The DBNet+CRNN structure is adopted, and a joint image-text classification network is designed in combination with text distribution features. This effectively extracts and classifies multimodal information such as time, location, and PTZ. This enables the model to maintain the stability of the prediction output even when some samples lack semantic graphs or PTZ information, thus solving the problem of excessive reliance on auxiliary data in traditional object detection models.

[0032] 5. Accelerate model convergence and reduce labor costs: By using structured auxiliary information to guide training (such as semantic graphs, time labels, and PTZ parameters), the model's ability to understand image context is significantly enhanced, making it easier to converge during training. This reduces the dependence on large-scale, finely labeled data and requires no additional hardware investment (relying only on image metadata and single-frame semantic segmentation). It has high engineering adaptability in large-scale, low-cost monitoring scenarios such as forestry and agriculture.

[0033] In summary, the method disclosed herein is applicable to seasonal and time-sensitive scenarios such as loess exposure monitoring and tall tree crop identification. After deployment, it can improve target detection accuracy, significantly reduce false alarm and false negative rates, decrease the frequency of manual inspections, and enhance the system's stability and automation level in complex environments, demonstrating promising prospects for industry promotion and commercial application value.

[0034] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the techniques pointed out in the description, claims and drawings. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is the overall design architecture of the target detection method that integrates multi-source semantic information provided in the embodiments of this disclosure; Figure 2 A flowchart illustrating the target detection method that integrates multi-source semantic information provided in this embodiment of the disclosure; Figure 3 This is a flowchart illustrating the workflow of the text detection and recognition module in the target detection device of this disclosure; Figure 4 This is a flowchart illustrating the spatial-temporal encoding module in the target detection device of this disclosure. Figure 5 This is a flowchart illustrating the workflow of the PTZ information feature encoding module in the target detection device of this disclosure. Figure 6 This is a flowchart illustrating the workflow of the static image semantic segmentation module in the target detection device of this disclosure; Figure 7 This is a diagram of the improved YOLOv11 network structure in the target detection device of this disclosure; Figure 8This is a structural diagram of the target detection device disclosed herein; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the protection scope of this disclosure.

[0038] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0039] In this disclosure, the terms "a" and "a plurality of" are used to modify related elements or features in an illustrative rather than restrictive manner. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0040] Example: A target detection method that integrates multi-source semantic information Figure 1 The figure shows the overall design architecture of the target detection method that integrates multi-source semantic information disclosed in this paper. Figure 2 The diagram shown is a flowchart illustrating the target detection method that integrates multi-source semantic information provided in this embodiment of the present disclosure. Figure 2 As shown, this method includes the following steps: S1: Receive the monitoring image and extract the metadata of the monitoring image, including the shooting time, shooting location, and PTZ parameters (camera attitude parameters). S2: Perform spatial-temporal encoding on the metadata to generate a spatiotemporal embedding vector; S3: Generate a static semantic mask image corresponding to the monitoring image; S4: The monitoring image, static semantic mask image and spatiotemporal embedding vector are fused together, and then input into the target detection model for target detection, and the detection result is output.

[0041] To better understand the technical solutions of this disclosure, further explanation is provided using embodiments in the following scenarios.

[0042] This embodiment provides a target detection method that integrates multi-source semantic information, aiming to improve the generalization ability and detection accuracy of target detection models in complex natural environments. The design process of this method is as follows: Figure 1 As shown: First, the system receives monitoring images and extracts metadata such as shooting time, location, and PTZ parameters through OCR recognition. This information is then encoded to form a spatial-temporal feature vector. Simultaneously, one image is selected from each camera location for dynamic target removal and image restoration to generate a static background image. A static semantic mask image is then extracted using a semantic segmentation model as a global prior. Next, the current image frame, semantic map, and spatial-temporal encoding are fused and input into the YOLOv11 target detection model to enhance its adaptability to background, lighting, and temporal changes, thereby improving detection performance. After the model outputs the detection results, an abnormal behavior recognition module can be optionally connected to enable monitoring and early warning of specific events. The entire solution significantly improves the robustness and practicality of the target detection system in real-world scenarios through information fusion without increasing hardware costs.

[0043] This embodiment also provides a target detection device that fuses multi-source semantic information. When this device operates, it implements the steps of the target detection method that fuses multi-source semantic information disclosed in this invention. The structure of this device is as follows: Figure 8 As shown, it includes: a text detection and recognition module, a space-time encoding module, a PTZ information feature encoding module, a static image semantic segmentation module, an improved YOLOv11 network structure module, and an improved YOLOv11 model training module.

[0044] (I) Text Detection and Recognition Module: The text detection and recognition module is the core component of this device used to parse the semantic information of offline monitoring images. It aims to extract embedded text content from the images, including shooting time, monitoring point location information, and camera pose parameters (PTZ), and convert it into structured, standardized input that can be used for model training and inference. Compared to online images, whose metadata can be directly obtained from cloud platforms, offline images, due to their dispersed sources, mixed information, and lack of labels, are difficult to automate in batch parsing using traditional methods. Therefore, this module introduces an OCR mechanism for complete semantic restoration. The workflow of this module is as follows: Figure 3As shown: First, the DBNet model is used to detect all text regions in the image, automatically identifying high-contrast embedded text boxes and excluding irrelevant text content naturally appearing in the image background. Then, CRNN is used to perform character recognition on each detected text region, outputting text content containing a mixture of Chinese, English, and numbers. Next, the system inputs the recognition results into a multimodal classification network that combines text and image for precise classification of the text's semantic category: This network uses Tiny-BERT to semantically encode the recognized text (outputting 312-dimensional features) and inputs the image region containing the text (uniformly 128×128 pixels) into a ResNet18 network to extract visual features (outputting 512-dimensional features). These two are concatenated into an 824-dimensional fusion vector, which is then mapped through two fully connected layers and classified into four categories—"Time," "Location," "PTZ," or "Other"—using Softmax. After classification, the module employs customized parsing strategies based on different categories: the time field supports multiple date formats (such as "2024-08-25 15:39:48" or "2024-06-03"). The module performs standardized processing of the timeline (16:39:15); the location field supports automatic merging of multi-line text, integrating complete administrative address information according to the order and positional relationship of the OCR detection boxes (e.g., "Zhuhai / Doumen / Baijiao / Jinheshi" merged into "Zhuhai City Doumen District Baijiao Town Jinheshi Industrial Park"); the PTZ field can identify parameter lines containing prefixes such as "P:", "T:", and "Z:", parsing out the corresponding horizontal angle, pitch angle, and zoom magnification; other useless fields such as "preset point 3" or "advertising slogan" are categorized as "other" and automatically removed; finally, the module outputs standardized structured metadata objects, including three key fields: timestamp, address description, and PTZ parameters, providing accurate, unified, and high-quality input for subsequent spatial-temporal coding modules. Without relying on EXIF ​​metadata, camera-attached sensors, or manual annotation, this module can automatically parse large-scale offline image datasets to achieve key tasks such as image tracing, semantic enhancement, and training set construction, demonstrating outstanding advantages such as strong robustness, good scalability, and flexible deployment.

[0045] (II) Spatial-Temporal Encoding Module: This module encodes the shooting time, season, and geographic location information extracted from images into a unified semantic vector, thereby providing environmental context priors for the object detection model and enhancing its generalization ability under different natural conditions. The workflow of this module is as follows: Figure 4As shown, the system first extracts the camera name, shooting month, and specific hour from the text recognition results, corresponding to spatial, seasonal, and time period information, respectively. The camera name is mapped to the 34 provincial-level administrative regions in China, with a placeholder for missing data, forming a 35-dimensional one-hot spatial code. The month information is divided into spring, summer, autumn, and winter, also with a placeholder for missing data, forming a 5-dimensional seasonal code. The time period is divided into morning, noon, afternoon, and evening based on the hour, with a default category, forming a 4-dimensional time period code. The three types of codes are B×35, B×5, and B×4, respectively, and are concatenated to obtain a unified B×44-dimensional semantic feature vector. To improve semantic expressiveness and compress dimensionality, this vector is fed into a lightweight mapping network containing fully connected layers, ReLU activation, and BatchNorm, outputting a final B×256-dimensional time-space embedding vector. This vector can be used in conjunction with other module outputs (such as PTZ encoding, semantic graph features, etc.) to enhance the contextual understanding capability of the YOLO detection backbone network, significantly improving detection accuracy and stability under different time periods, seasonal transitions, or geographical conditions.

[0046] (III) PTZ Information Feature Encoding Module: This module converts the camera's gimbal control parameters (i.e., horizontal angle Pan, pitch angle Tilt, zoom magnification) into semantic embedding vectors recognizable by a deep neural network, and fuses them with spatial-temporal encoded features to jointly construct a high-dimensional prior feature input YOLO detection backbone network. The workflow of this module is as follows: Figure 5 As shown: First, input the original PTZ 3D parameters (B×3) obtained from OCR recognition, namely pan, tilt, and zoom. These parameters have different value ranges; directly inputting them will lead to inconsistent feature distributions. Therefore, the first step of this module is to normalize the 3D PTZ vector, scaling it to a uniform range. The normalization formulas are as follows:

[0047] When the image does not contain PTZ information, the default P / T / Z values ​​are 0 / 0 / 1, respectively. The normalized features are fed into a lightweight fully connected layer (Linear+ReLU), mapping them to a 256-dimensional high-order semantic representation, with an output size of B×256. Simultaneously, the semantic vector (B×256) generated by the spatial-temporal encoding module is used as parallel input and concatenated with the PTZ embedding features in the dimensional direction, forming a fused feature vector of B×512. To further enhance the interaction and alignment between different semantic dimensions, this fused vector passes through an encoding layer containing a fully connected layer, a ReLU activation function, and BatchNorm normalization, maintaining an output dimension of B×512 to ensure consistency with the intermediate feature dimensions of the YOLO backbone network. Finally, to achieve feature alignment with the convolutional backbone network, the semantic vector is reshaped into a four-dimensional tensor format of B×512×1×1, which can be directly injected into the YOLO network as an additional semantic channel, such as through feature concatenation or weighted fusion, and fused with multi-scale features in the Backbone or Neck modules. This module has advantages such as lightweight structure, full information utilization, and flexible deployment, and can significantly improve the detection stability of the model under varying perspectives (such as high-angle view and long-range zoom), and is an important component of the spatiotemporal semantic enhancement target detection system constructed in this disclosure.

[0048] (iv) Static Image Semantic Segmentation Module: To improve the device's environmental understanding and target detection robustness in fixed camera scenarios, especially in outdoor environments with complex backgrounds, frequent lighting changes, or dynamic interference, the static image semantic segmentation module not only extracts semantic background based on the basic workflow of "dynamic target detection + mask extraction + image inpainting + semantic segmentation," but also introduces three key innovative mechanisms to enhance the stability, accuracy, and multidimensional controllability of the static semantic graph. The workflow of this module is as follows: Figure 6As shown: First, this disclosure proposes a pseudo-static image generation strategy based on multi-frame fusion to address the problem that static background extraction from a single-frame image is easily interfered with under short-term perturbations. Specifically, at each fixed point, the system continuously acquires multiple frames of images, identifies dynamic targets (such as pedestrians and vehicles) in each frame using the YOLO object detection model, and extracts the target bounding boxes to generate a mask. Subsequently, through mechanisms such as inter-frame IoU matching, time window voting, and dynamic confidence ranking, a more stable dynamic mask with a low false detection rate is fused and input into the LaMa image inpainting algorithm to generate a pseudo-static background image without dynamic objects, effectively avoiding the problem of "pseudo-dynamic regions" interfering with the semantic map due to single-frame noise. Secondly, in the semantic segmentation stage, this disclosure not only uses the Segment Anything v2 model to perform semantic parsing on the repaired image, but also introduces a semantic guidance mechanism based on a confidence map: for each pixel, the system not only records its semantic category, but also extracts the attention weights or final category confidence calculated internally by the model to form a semantic confidence map. This confidence map, as a soft prior, can be fused with the Neck layer of YOLOv11 to guide the detection model to enhance feature responses in high-confidence regions (such as buildings, ground, and farmland) and adaptively adjust the threshold in low-confidence regions (such as distant occlusions, water edges, and repaired areas), thereby improving the robustness of the detection task in easily confused regions and significantly reducing the background false activation rate. Finally, to further improve the quality of segmentation boundaries and region consistency, this disclosure designs a semantic segmentation strategy that uses the original image and the repaired image as joint inputs. Specifically, during semantic segmentation, the system not only inputs the pseudo-static image repaired by LaMa, but also sends the pixel difference between the original image and the repaired image as an additional channel to the segmentation model, thereby explicitly telling the model "which regions have just undergone inpainting". This mechanism effectively guides the model to learn stronger boundary alignment and context completion capabilities in the repaired area, showing significant improvement, especially in addressing issues such as blurred textures and unnatural edge transitions left after small object removal. In summary, the static image semantic segmentation module disclosed in this paper significantly improves the accuracy and spatiotemporal consistency of semantic perception in fixed-location scenes through multi-frame consistency fusion, confidence enhancement guidance, and image-residual joint modeling. It provides a more reliable and stable environmental semantic prior for subsequent object detection, and is suitable for various long-term deployment applications such as urban monitoring, farmland inspection, flood control, and field station monitoring, demonstrating strong practical value and technological advancement.

[0049] (v) Improved YOLOv11 Network Structure Module: The detailed network structure of this module is as follows: Figure 7As shown, the improved YOLOv11 network structure, based on the original backbone feature extraction network, FPN structure, and three-branch detection head, systematically introduces three types of enhancement modules to optimize the target detection process from three perspectives: semantic guidance, spatiotemporal prior, and target scale. This aims to improve the model's robustness and detection accuracy in real-world complex scenarios, making it particularly suitable for scenarios such as drone aerial photography, zoom PTZ camera monitoring, and seasonally sensitive areas (e.g., forests, farmland, and exposed mountain areas) where the perception of small targets, large-viewpoint targets, and multi-scale targets is crucial. First, after the output of the backbone's SPPF module, a semantic segmentation confidence map guidance module is introduced. This module receives semantic segmentation results (including foreground and background category confidence maps, with a size of B×4×640×640) provided externally or obtained through parallel training. It then compresses these results into a B×256×20×20 semantic space feature map through five consecutive convolutional layers and downsampling operations. Meanwhile, the image features output by the SPPF module, B×512×20×20, are reduced to B×256×20×20 through 1×1 convolution. These two feature streams are then input into the Cross-Attention module to achieve target perception guidance capabilities such as foreground region enhancement, background region suppression, and boundary semantic completion. The fused features are then restored back to B×512×20×20 via channels and fed into the C2PSA module of the YOLOv11 backbone, maintaining the original structural compatibility. Secondly, the model introduces a spatiotemporal + PTZ global prior encoding module. As mentioned in the previous steps, the fused feature size is B×512×1×1. This feature is spatially expanded to B×512×20×20 and added element-wise with the backbone features fused by the Cross-Attention module, enabling the global context information to regulate local detection results. This module effectively improves the model's adaptability to seasonal differences, viewpoint changes, and geographical location differences, and enhances the ability to distinguish between targets and background through the guided attention mechanism. Finally, to address the lack of adaptability in multi-scale object detection using fixed-scale FPN, a Dynamic Scale-Aware Fusion (DSAF) module is designed. Global Average Pooling (GAP) is performed on the feature maps of the three detection branches (B×128×80×80, B×256×40×40, and B×512×20×20) to extract global channel semantic representations, resulting in three vectors B×128, B×256, and B×512. These vectors are concatenated and passed through a two-layer fully connected network to output three weights α1, α2, and α3. After Softmax normalization, these weights serve as dynamic fusion factors, which are used to adjust the feature maps of Head1, Head2, and Head3 accordingly. This allows the model to adaptively adjust the detection strategy based on the distribution characteristics of the target scale in the current image.For example, in long-range panoramic images, the model will enhance the response of the high-resolution branch Head1; while in top-down close-range images, it will favor the low-resolution Head3 for large target detection, thereby achieving comprehensive coverage and adaptive detection of small targets, large targets, and occluded areas.

[0050] (vi) Improved YOLOv11 Model Training Module: This disclosure proposes a multi-source consistency-guided joint training strategy to address the problems of large performance fluctuations and weak generalization ability of models under multi-source heterogeneous data conditions (such as the presence or absence of PTZ information, the presence or absence of semantic segmentation maps, different shooting times and perspectives, etc.). This strategy includes two key mechanisms: one is the auxiliary information consistency loss, and the other is the two-stage-wise hybrid supervision mechanism. Specifically, the auxiliary information consistency loss constructs a dual input path to generate two forms for each image: "complete auxiliary input" (image + semantic map + PTZ encoding) and "image only input". The prediction results are obtained by the model for each form, and KL divergence loss is introduced into the output layer to ensure that the prediction distribution remains consistent with the "with auxiliary input" version even when auxiliary information is missing, thereby improving the robustness of the model to the lack of auxiliary channels. The second mechanism achieves collaborative optimization of the backbone and auxiliary channels through phased training: In the first phase (warm-up), the backbone network is stably pre-trained using only images and basic detection loss. In the second phase, auxiliary inputs such as PTZ semantic features and semantic segmentation maps are gradually introduced, along with consistency loss weights, to enhance the model's ability to utilize spatiotemporal context and improve cross-scene adaptability. Overall, this training strategy not only effectively alleviates training fluctuations caused by incomplete or unstable auxiliary information, but also improves the model's detection consistency and generalization performance under different regions, time periods, and shooting settings, reducing dependence on precise semantic maps or metadata. It is particularly suitable for large-scale surveillance video automatic learning, deployment, and migration application scenarios.

[0051] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of the present disclosure, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0052] like Figure 9 As shown, embodiments of this disclosure also disclose an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned target detection method that integrates multi-source semantic information.

[0053] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0054] The operations of this disclosure can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0055] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0056] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0057] Furthermore, this disclosure also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the target detection method that integrates multi-source semantic information disclosed herein.

[0058] In the context of this disclosure, a computer-readable storage medium can be a tangible medium, and more specific examples include portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0059] In particular, according to embodiments of this disclosure, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the target detection method for fusing multi-source semantic information disclosed in this disclosure. When the computer program is executed by a processing device, it performs the functions defined in the methods of embodiments of this disclosure.

[0060] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept.

[0061] Those skilled in the art should also understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A target detection method that integrates multi-source semantic information, characterized in that, The method includes: S1: Receive the monitoring image and extract the metadata of the monitoring image, including the shooting time, shooting location, and PTZ parameters; S2: Perform spatial-temporal encoding on the metadata to generate a spatiotemporal embedding vector; S3: Generate a static semantic mask image corresponding to the monitoring image; S4: The monitoring image, static semantic mask image and spatiotemporal embedding vector are fused together, and then input into the target detection model for target detection, and the detection result is output.

2. The method according to claim 1, characterized in that, The extraction of metadata from the surveillance image in step S1 includes: The text regions in the surveillance image are detected by OCR technology, the text content is identified by CRNN model, and then the text content is classified by image-text joint classification network to extract structured time, location, and PTZ parameters. The time parameter supports the normalization of various date formats, the location parameter generates a complete administrative address by merging multiple lines of text, and the PTZ parameter parses the horizontal angle, pitch angle, and zoom of the camera.

3. The method according to claim 2, characterized in that, Step S2, which involves spatial-temporal encoding of the metadata to generate a spatiotemporal embedding vector, includes: The shooting location is converted into a geographic region code to generate a spatial code, and the shooting time is converted into a season and time period code to generate a time code. The PTZ parameters are normalized and then concatenated with the spatial and time codes. The spatiotemporal embedding vector is generated by mapping through a lightweight fully connected network. Among them, the geographic region code is based on the camera name and mapped to the provincial administrative region and the default position. The season code is divided into spring, summer, autumn, winter and the default category. The time period code is divided into morning, noon, afternoon, evening and the default category.

4. The method according to claim 3, characterized in that, Step S3, generating a static semantic mask image corresponding to the monitoring image, includes: S31: For multiple frames of images captured from the same camera location, a target detection model is used to identify dynamic targets and generate dynamic masks. Multiple dynamic masks are then fused to generate a pseudo-static background image. S32: Perform semantic segmentation on the pseudo-static background image and extract the semantic confidence map as a soft prior; S33: The pixel difference between the original image and the pseudo-static background image is used as an additional channel input to the semantic segmentation model to enhance the semantic boundary alignment capability of the repaired region and output a static semantic mask image.

5. The method according to claim 4, characterized in that, The target detection model mentioned in step S4 is an improved YOLOv11 network structure. Step S4 includes: S41: A semantic segmentation confidence map guidance module is introduced at the output of the backbone feature extraction network, and the foreground enhancement and background suppression information of the semantic mask map are fused through the Cross-Attention mechanism; S42: The spatiotemporal embedding vector is expanded and added to the Cross-Attention fusion feature to achieve global context control over local detection; S43: A dynamic scale-aware fusion module is introduced into the detection head. Multi-scale features are extracted through global average pooling, the fusion weights of each detection branch are calculated and adjusted, and adaptive multi-scale detection is achieved.

6. The method according to claim 5, characterized in that, The target detection model is trained using a collaborative training strategy that aligns multiple sources of information, including: (1) Construct an auxiliary information consistency regularization term, generate prediction results through dual input paths, and introduce KL divergence loss to constrain the consistency of the two prediction distributions; (2) A two-stage hybrid label training mechanism is adopted. In the first stage, the backbone network is pre-trained, and in the second stage, auxiliary inputs are introduced and weighted consistency loss is applied to enhance the ability to utilize spatiotemporal context.

7. The method according to claim 6, characterized in that, The backbone network of the target detection model is selected from any one of RT-DETR, DETR, Faster R-CNN, EfficientDet, CenterNet, and RetinaNet; The static semantic mask image is generated using any one of the segmentation models: SAM, DeepLabV3+, HRNet, SegFormer, UpperNet, and BiSeNet. The feature fusion method can be selected from any one of the following: feature-guided fusion module, gating mechanism, conditional convolution, SE module, and ECA.

8. A target detection device that integrates multi-source semantic information, characterized in that, The device includes: The text detection and recognition module is used to receive surveillance images and extract metadata from the surveillance images, including shooting time, shooting location, and PTZ parameters. The space-time encoding module is used to perform space-time encoding on the metadata to generate a spatiotemporal embedding vector; The PTZ information feature encoding module is used to normalize the PTZ parameters and map them into higher-order semantic representations; A static image semantic segmentation module is used to generate a static semantic mask image corresponding to the monitoring image; An improved YOLOv11 network structure module is used to fuse the monitoring image, static semantic mask map, and spatiotemporal embedding vector, and then perform target detection and output the detection result; An improved YOLOv11 model training module is used to train the model using a multi-source information consistency alignment strategy.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the target detection method that fuses multi-source semantic information as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing the computer program to implement the steps of the target detection method that fuses multi-source semantic information as described in any one of claims 1-7.

Images