Target monitoring method and device and electronic equipment

Through multimodal data acquisition and self-supervised learning, virtual scenes are constructed, and the gimbal adjustment strategy is generated, which solves the adjustment accuracy problem of the gimbal monitoring system in complex environments, and achieves high-precision target monitoring and tracking.

CN120510561AInactive Publication Date: 2025-08-19TIANJIAN GRP CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202510418389.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-04-03
Publication Date
2025-08-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing gimbal monitoring system has lagged or accumulated control commands due to environmental noise, target occlusion and multi-sensor data fusion errors in complex dynamic scenarios, affecting the adjustment accuracy.

Method used

By integrating multimodal data acquisition, digital twin modeling and self-supervised learning, a virtual scene is constructed, multi-dimensional data is obtained using infrared cameras, lidar and event cameras, and a self-supervised neural network model is used to monitor the changes in target characteristics in real time to generate a gimbal adjustment strategy.

Benefits of technology

It improves the adjustment accuracy and response speed of the gimbal in complex environments, ensures that the monitoring screen is always focused on the target, and improves the stability and robustness of the target monitoring.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120510561A_ABST
    Figure CN120510561A_ABST
Patent Text Reader

Abstract

The invention provides a target monitoring method and device and electronic equipment, and relates to the field of data processing. The method comprises the following steps: acquiring multi-modal data for a target object; obtaining a digital twin model of a target area where the target object is located; adding the multi-modal data into the digital twin model to obtain a virtual scene; processing the multi-modal data by adopting a self-supervised neural network model, and determining a target feature corresponding to the target object; and if it is determined that the state of the target feature changes in the virtual scene, generating a pan-tilt adjustment strategy which is used for controlling the pan-tilt to monitor the target object. By implementing the technical scheme provided by the invention, the adjustment precision of the holder during target monitoring can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a target monitoring method, device and electronic equipment. Background Art

[0002] Currently, pan-tilt monitoring systems are widely used in video surveillance, target tracking, intelligent security and other fields. However, the existing technology still has many shortcomings in the pan-tilt control accuracy.

[0003] At present, traditional gimbal control solutions mostly rely on PID control, Kalman filtering and simple data association algorithms. Although these methods have certain guarantees in terms of stability and real-time performance, when faced with complex dynamic scenes, they often lead to control command lag or error accumulation due to factors such as environmental noise, target occlusion, incomplete sensor data and multi-sensor data fusion errors, which in turn affects the adjustment accuracy of the gimbal.

[0004] Therefore, there is an urgent need for a target monitoring method, device and electronic equipment. Summary of the Invention

[0005] The present application provides a target monitoring method, device and electronic equipment, which are convenient for improving the adjustment accuracy of the pan-tilt head when performing target monitoring.

[0006] In a first aspect of the present application, a target monitoring method is provided, which includes: obtaining multimodal data for a target object; obtaining a digital twin model of a target area where the target object is located; adding the multimodal data to the digital twin model to obtain a virtual scene; using a self-supervised neural network model to process the multimodal data to determine the target features corresponding to the target object; if it is determined that the target features have undergone a state change in the virtual scene, generating a pan-tilt adjustment strategy, and the pan-tilt adjustment strategy is used to control the pan-tilt to monitor the target object.

[0007] By adopting the above-mentioned technical solutions, through the integration of multimodal data acquisition, digital twin modeling, and self-supervised learning, all-round, multi-angle information perception and precise feature extraction of the target object are achieved. Specifically, by acquiring data from multiple sensors and constructing a virtual replica of the target area, the system can reproduce the actual scene in a digital environment, providing a high-fidelity platform for subsequent target behavior analysis; the self-supervised neural network does not need to rely on a large amount of manual annotation, automatically extracts the core features of the target, and monitors changes in these features in real time. When the state of the target in the virtual scene changes, the system can instantly generate and adjust the pan-tilt control strategy, thereby significantly improving the adjustment accuracy and response speed of the pan-tilt, ensuring that the monitoring image is always focused on the target. This method effectively solves the control error problem caused by incomplete sensor data, environmental interference, and insufficient target motion prediction in traditional monitoring, and facilitates improving the adjustment accuracy of the pan-tilt during target monitoring.

[0008] Optionally, the acquiring of multimodal data for the target object specifically includes: controlling an infrared camera, a lidar, and an event camera to shoot the target object to obtain an original data packet, wherein the original data packet includes an image, depth information, and motion data; denoising, correcting, and unifying the format of the original data packet to obtain the multimodal data.

[0009] By adopting the above technical solution, a multi-angle, multi-layered information acquisition system is constructed by controlling infrared cameras, lidar, and event cameras to collect images, depth information, and motion data of target objects. First, by utilizing multiple sensors to collect raw data packets, the advantages of each can be fully utilized: infrared cameras are suitable for low-light or nighttime scenes, lidar provides accurate depth information, and event cameras can capture details of high-speed motion. Second, by denoising, correcting, and formatting the raw data, high-quality multimodal data fusion is achieved, significantly improving data accuracy and robustness. This approach provides a solid data foundation for subsequent target feature extraction and dynamic monitoring, effectively improving the overall performance of the monitoring system in complex environments.

[0010] Optionally, obtaining the digital twin model of the target area where the target object is located specifically includes: controlling the lidar and the panoramic image sensor to scan and photograph the target area to obtain fused data; constructing an initial model through the fused data; and inputting the static structure and environmental parameters corresponding to the target area into the initial model to obtain the digital twin model.

[0011] By employing this technical solution, a comprehensive scan of the target area is performed using LiDAR and panoramic image sensors. An initial model is constructed through data fusion, and then static structural and environmental parameters within the area are incorporated to generate a high-fidelity digital twin model. This method not only accurately reproduces the geometry and environmental characteristics of the actual monitored area, but also provides a detailed and accurate virtual scene for subsequent target monitoring, behavior prediction, and pan / tilt control, significantly improving the system's response speed and overall monitoring performance in complex environments.

[0012] Optionally, adding the multimodal data to the digital twin model to obtain a virtual scene specifically includes: embedding the multimodal data into the digital twin model to obtain the virtual state and virtual motion of the target object in the target area; analyzing the multimodal data to obtain the actual state and actual motion of the target object in the target area; and using the actual state and actual motion to correct the virtual state and virtual motion to obtain the virtual scene.

[0013] By adopting the above technical solution and embedding multimodal data into the digital twin model, the precise construction and real-time correction of the state and motion of the target object in the virtual scene are achieved. Specifically, the multidimensional data collected by the sensor is first integrated into the digital twin model to form the virtual state and motion trajectory of the target object; then, this data is deeply analyzed to extract the real state and motion pattern of the target in the actual scene; finally, the virtual state is dynamically corrected using the actual state to ensure that the virtual scene is highly consistent with the real environment. This process not only improves the realism and accuracy of the virtual scene, but also provides reliable data support for subsequent target monitoring and pan-tilt control, enabling the system to achieve more precise and efficient monitoring and response in complex environments.

[0014] Optionally, the self-supervised neural network model is used to process the multimodal data to determine the target features corresponding to the target object, specifically including: constructing the self-supervised neural network model using a self-supervised learning mechanism; obtaining the appearance, motion features and environmental context information of the target object through continuous frame reconstruction loss; combining the appearance, the motion features and the environmental context information, and extracting the target features through comparative learning and feature alignment mechanism.

[0015] By employing these technical solutions and utilizing self-supervised learning to build a neural network model, the team automatically extracts the target's appearance, motion characteristics, and environmental context through a continuous frame reconstruction loss, capturing the target's multi-dimensional features without extensive manual annotation. Combining contrastive learning with feature alignment mechanisms further integrates these features, ensuring the extracted target features are more robust and discernible, significantly improving the accuracy and adaptability of target monitoring and tracking in complex environments.

[0016] Optionally, if it is determined that the target feature undergoes a state change in the virtual scene, a gimbal adjustment strategy is generated, specifically including: real-time monitoring of the target feature in the virtual scene; if it is determined that the target feature undergoes a state change, the gimbal adjustment strategy is generated, and the state change includes target position offset, speed change, and sudden occlusion; the gimbal adjustment strategy is sent down to the gimbal to control the gimbal to make real-time adjustments according to the gimbal adjustment strategy.

[0017] By adopting this technical solution, the target's characteristics are monitored in real time within the virtual scene. Once a state change such as position deviation, speed change, or sudden occlusion is detected, a gimbal adjustment strategy is immediately generated and sent to the gimbal for real-time adjustment, significantly improving the real-time responsiveness and control accuracy of target tracking. This mechanism ensures that the gimbal can quickly adapt to dynamic changes in target motion, keeping the target at the center of monitoring, and improving the system's robustness and reliability in complex environments.

[0018] Optionally, the method further includes: controlling the pan-tilt adjustment strategy to adjust the lens angle and focal length in real time; and controlling the target object to remain in the center of the monitoring screen corresponding to the digital twin model according to the pan-tilt adjustment strategy.

[0019] By employing this technical solution, the lens angle and focal length are precisely controlled through a pan / tilt adjustment strategy, ensuring that the target object is always centered within the monitoring image corresponding to the digital twin model. This not only improves target tracking stability but also reduces monitoring blind spots caused by target offset, improving the effectiveness and clarity of the monitoring image. Furthermore, real-time adjustment of lens parameters adapts to changes in target motion, enabling the system to maintain high-precision target monitoring even in complex and dynamic environments, enhancing the adaptability and practicality of intelligent monitoring.

[0020] In a second aspect of the present application, a target monitoring device is provided, which includes an acquisition module and a processing module, wherein the acquisition module is used to acquire multimodal data for a target object; the acquisition module is also used to acquire a digital twin model of a target area where the target object is located; the processing module is used to add the multimodal data to the digital twin model to obtain a virtual scene; the processing module is also used to process the multimodal data using a self-supervised neural network model to determine the target features corresponding to the target object; the processing module is also used to generate a pan-tilt adjustment strategy if it is determined that the target feature has a state change in the virtual scene, and the pan-tilt adjustment strategy is used to control the pan-tilt to monitor the target object.

[0021] In a third aspect of the present application, an electronic device is provided, which includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs the method described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions. When the instructions are executed, the method described above is executed.

[0023] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: By integrating multimodal data acquisition, digital twin modeling, and self-supervised learning, all-round, multi-angle information perception and precise feature extraction of the target object are achieved. Specifically, by acquiring data from multiple sensors and constructing a virtual replica of the target area, the system can reproduce the actual scene in a digital environment, providing a high-fidelity platform for subsequent target behavior analysis; the self-supervised neural network does not need to rely on a large amount of manual annotation, automatically extracts the core features of the target, and monitors changes in these features in real time. When the target state in the virtual scene changes, the system can instantly generate and adjust the pan-tilt control strategy, thereby greatly improving the adjustment accuracy and response speed of the pan-tilt, ensuring that the monitoring screen is always focused on the target. This method effectively solves the control error problem caused by incomplete sensor data, environmental interference, and insufficient target motion prediction in traditional monitoring, and facilitates improving the adjustment accuracy of the pan-tilt during target monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a target monitoring method provided in an embodiment of the present application; Figure 2 Another flowchart of a target monitoring method provided in an embodiment of the present application; Figure 3 A schematic diagram of a module of a target monitoring device provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0025] Explanation of the reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0027] In the description of the embodiments of this application, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a concrete manner.

[0028] In the description of the embodiments of the present application, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0029] Currently, pan-tilt monitoring systems have been widely used in video surveillance, target tracking, intelligent security and other fields, but existing technologies still have many limitations in terms of pan-tilt control accuracy.

[0030] At present, traditional pan-tilt control schemes mainly rely on PID control, Kalman filtering and simple data association algorithms. Although these methods have certain advantages in stability and real-time performance, they are susceptible to environmental noise, target occlusion, sensor data loss and multi-sensor fusion errors in complex dynamic scenes, resulting in control command response lag or error accumulation, thereby reducing the adjustment accuracy of the pan-tilt and affecting the continuous tracking and monitoring effect of the target. In order to solve the above technical problems, this application provides a target monitoring method, referring to Figure 1 , Figure 1 This is a flow chart of a target monitoring method provided in an embodiment of the present application. The target monitoring method is applied to a server and includes steps S110 to S150, which are as follows: S110: Acquire multimodal data for the target object.

[0031] Specifically, the server acquires multimodal data about a target object. This means the server collects multidimensional information about the target object through multiple different types of sensors or data sources, enabling more comprehensive and accurate target perception and analysis. Multimodal data typically includes, but is not limited to, visual information, depth information, motion trajectory, thermal imaging data, etc., collected by different sensors and transmitted to the server for processing, fusion, and analysis. After acquiring this data, the server can use algorithms for data preprocessing, target monitoring, feature extraction, and status tracking, thereby improving the accuracy and adaptability of the monitoring system.

[0032] For example, in a smart factory, the server needs to monitor specific personnel in the factory in real time to ensure safety. The server can obtain the following multimodal data: RGB camera: collects visible light images to identify appearance features such as faces and clothing. Infrared thermal imager: monitors the body temperature of personnel to determine whether there are any abnormal conditions, such as high temperature warnings. Laser radar (LiDAR): obtains the three-dimensional position and movement trajectory of personnel to ensure that the target can still be tracked in low-light or obscured environments. Event camera: records the high-speed motion characteristics of the target, such as running or sudden acceleration, in order to monitor abnormal behavior. After receiving this data, the server first performs denoising, correction and format unification, and then combines deep learning algorithms to analyze target features to achieve identity recognition, behavior analysis and anomaly monitoring.

[0033] In one possible implementation, obtaining multimodal data for a target object specifically includes: controlling an infrared camera, a lidar, and an event camera to photograph the target object to obtain an original data packet, which includes an image, depth information, and motion data; and denoising, correcting, and unifying the format of the original data packet to obtain multimodal data.

[0034] Specifically, infrared cameras are used to capture thermal imaging data of target objects in low-light or nighttime environments, providing information on the target's temperature or contours. Laser ranging technology is used to obtain three-dimensional depth information of the target object, generating high-precision point cloud data. Unlike traditional cameras, event cameras only generate data when pixel brightness changes, effectively capturing the trajectory of high-speed moving targets and reducing redundant frame data. The combined use of these sensors ensures complete information about the target object in various environments, thereby improving the accuracy and stability of monitoring and tracking.

[0035] The data collected by sensors may contain noise, such as thermal interference from infrared cameras, multipath reflection errors from lidars, or random noise from event cameras. Therefore, filtering algorithms are required to reduce noise. Because different sensors have different coordinate systems, viewing angles, and resolutions, spatial alignment is necessary. For example, lidar point cloud data is spatially calibrated with RGB images to ensure that depth information and image information are correctly matched. Data formats may vary between sensors. For example, RGB cameras provide 2D images, lidars provide 3D point clouds, and event cameras provide time-series event streams. These data formats need to be converted into a unified data structure for subsequent fusion analysis. After these processing steps, the resulting multimodal data is more accurate and stable, laying a solid foundation for subsequent target recognition, tracking, and analysis.

[0036] S120: Obtain a digital twin model of the target area where the target object is located.

[0037] Specifically, the server controls multiple sensors (such as lidar, panoramic cameras, and structured light sensors) to scan and capture the target area, acquiring information about the environmental structure and features. By fusing sensor data, the server generates a 3D model of the target area and models static structures in the environment (such as buildings, roads, and equipment), ensuring that the model accurately reflects the real world. The server combines the target object's motion data with dynamic adjustments to the digital twin model to keep it synchronized with the real world, ensuring accurate monitoring and analysis.

[0038] For example, in a smart manufacturing factory, a server needs to monitor the production workshop to ensure personnel safety and equipment operation. The server controls the lidar to scan the workshop's spatial layout, acquiring precise 3D point cloud data. A panoramic camera captures high-resolution images, supplementing visual details such as equipment labels and operating panels. A structured light sensor precisely measures the dimensions of key areas, ensuring the geometric accuracy of the model. Based on this data, the server constructs a complete digital twin model of the workshop, including production lines, robots, storage areas, and more. Static structures such as fixed equipment, passageways, and emergency exits are annotated to provide a foundation for subsequent analysis. The server continuously receives sensor data and, if it detects changes in the location of production equipment or personnel, it synchronously updates the digital twin model to ensure the accuracy of monitoring information. This digital twin model can be used for automated inspections and anomaly detection (such as equipment overheating or personnel entering a hazardous area), enhancing the intelligence of factory management.

[0039] The server acquires a digital twin model of the target area and, through multi-sensor fusion, accurately constructs a virtual mapping of the physical world, which is updated in real time. This approach not only improves the accuracy of PTZ monitoring but also provides efficient data support for fields such as intelligent security, industrial monitoring, and smart transportation, making the system more adaptable and intelligent.

[0040] In one possible implementation, obtaining a digital twin model of a target area where a target object is located specifically includes: controlling a lidar and a panoramic image sensor to scan and photograph the target area to obtain fused data; constructing an initial model through the fused data; and inputting the static structure and environmental parameters corresponding to the target area into the initial model to obtain a digital twin model.

[0041] Specifically, LiDAR is used to perform spatial scanning of the target area, generating high-precision 3D point cloud data, including terrain, obstacles, and building structures. A panoramic camera or high-resolution camera is used to obtain panoramic images of the target area, capturing details such as texture, color, and identification information. The LiDAR 3D point cloud is aligned and fused with the panoramic image to obtain a complete dataset that includes both spatial information (LiDAR) and visual details (camera). The server preprocesses the fused data through denoising, alignment, and format conversion to ensure data consistency. A preliminary 3D environment model is generated using 3D reconstruction algorithms (such as Poisson reconstruction and TSDF voxel modeling), including basic structures such as the ground, walls, and buildings.

[0042] The server incorporates fixed features of the target area (such as building frames, road boundaries, and fixed equipment) into the model to improve the stability and accuracy of the digital twin. Environmental variables such as lighting, temperature, humidity, and air flow are also input, enabling the digital twin model to not only simulate geometry but also reflect the physical characteristics of the real environment. Ultimately, after this processing, the server generates a digital twin model that closely resembles the real world and can be used for subsequent simulation, prediction, analysis, and control tasks.

[0043] For example, in large logistics warehouses, high-precision environmental modeling is required to optimize the navigation paths of unmanned forklifts and improve cargo management efficiency. A server controls a lidar system to scan the warehouse's internal structure, acquiring point cloud data such as shelf locations, wall boundaries, and aisle widths. A panoramic camera captures the entire warehouse, recording visual information such as item labels, colors, and floor markings. The server integrates this data to ensure precise alignment of shelves, items, and robot navigation routes. Based on this point cloud data, the server generates a 3D map of the warehouse, including shelf placement and aisle layout. Using machine learning algorithms, the server analyzes camera images, extracts item label information, and maps it to the 3D model. Fixed warehouse structures, such as load-bearing columns, ceiling height limits, and immovable areas, are recorded to ensure forklifts do not stray into hazardous areas. Environmental parameters such as temperature, humidity, and ventilation are incorporated to monitor the storage status of perishable goods. This digital twin model can be used for intelligent management, including unmanned forklift path optimization, warehouse layout planning, and fire risk assessment, to improve operational efficiency and safety.

[0044] For example, a municipal government department wants to create a detailed model of a congested intersection to optimize traffic light scheduling and improve traffic efficiency. A server controls a LiDAR scanner to scan the road environment, generating a high-precision 3D road structure, including lane widths, street sign locations, and traffic lights. A panoramic camera records real-time vehicle flow and extracts data such as lane occupancy, pedestrian distribution, and weather conditions. The server integrates this data to construct a comprehensive perception model of the intersection. The server generates a 3D traffic scene, including elements such as roads, buildings, traffic lights, and signage. An AI algorithm analyzes the camera data to identify common congestion patterns, such as uneven traffic light durations during peak hours and excessive pedestrian wait times. Fixed traffic infrastructure, such as traffic light poles, zebra crossings, and sidewalk guardrails, is incorporated to ensure the simulation system is accurate. Environmental variables, such as peak-hour vehicle density and weather factors (such as increased sliding distances in rainy and snowy conditions), are incorporated to optimize traffic light control strategies. This digital twin model can be used for intelligent traffic light scheduling, traffic congestion prediction, and accident simulation, improving urban traffic management.

[0045] By fusing data from LiDAR and panoramic cameras, high-precision environmental modeling is achieved. By inputting static structural and environmental parameters, a more complete intelligent monitoring and simulation system can be constructed. This technology has broad applications in fields such as smart warehousing, intelligent transportation, industrial automation, and intelligent security, significantly improving perception accuracy and monitoring efficiency while providing a solid data foundation for intelligent decision-making and automated control.

[0046] S130: Add the multimodal data to the digital twin model to obtain a virtual scene.

[0047] Specifically, the server acquires multimodal data from sensors (such as infrared cameras, lidar, and event cameras). This data includes RGB image data (providing information about the target object's appearance), depth information (providing information about its spatial position and structure), and motion data (providing the target's motion trajectory from event cameras or inertial sensors). Data from different sensors comes in different formats (e.g., RGB images and 3D coordinate points from lidar point clouds), requiring a unified coordinate system and timestamp alignment. This eliminates noise from the multimodal data, such as visual interference caused by varying lighting and lidar scanning errors. Deep learning algorithms (such as the GNN or Transformer architecture shown in the figure) extract and match features from the data from different modalities to obtain a more accurate representation of the target object's state. The server embeds this fused data into a digital twin model, generating a virtual representation of the target object's state and motion in a 3D environment. The server analyzes the target object's actual state (e.g., its posture and motion trends) and then modifies the target state in the virtual scene to ensure that the virtual environment accurately reflects the real world. This virtual scene is updated in real time and can be used to predict the target object's next behavior, providing a basis for intelligent control.

[0048] For example, in an industrial production workshop, automated inspection robots need to monitor the status of mechanical equipment in real time and issue alerts for abnormalities. The robot's LiDAR scans the production equipment, acquiring 3D structural and positional data. A thermal imaging camera records the equipment's temperature distribution to detect overheating or abnormalities. A vibration sensor collects the equipment's vibration frequency to determine whether it is operating abnormally. The server fuses multimodal data (3D point cloud, temperature, and vibration information), corrects the data format, and removes noise. The server analyzes the data using a deep learning model and extracts key features, such as whether the equipment has hot spots, abnormal vibration, and normal operating status. The server maps this fused data into a digital twin factory, generating a virtual representation of the equipment's status. If a temperature abnormality is detected for a piece of equipment, the device is highlighted in a warning color in the virtual scene, and an alert is sent to maintenance personnel. Maintenance personnel can use the digital twin system to remotely monitor the equipment's status and even predict future failure trends, allowing them to proactively schedule repairs. By incorporating multimodal data into the digital twin model, factories can achieve remote monitoring and intelligent early warning of equipment, improving production safety and efficiency.

[0049] In one possible implementation, multimodal data is added to a digital twin model to obtain a virtual scene, specifically including: embedding the multimodal data into the digital twin model to obtain the virtual state and virtual motion of the target object in the target area; analyzing the multimodal data to obtain the actual state and actual motion of the target object in the target area; and using the actual state and actual motion to correct the virtual state and virtual motion to obtain a virtual scene.

[0050] Specifically, the server inputs multimodal data (such as RGB images, depth information, LiDAR point clouds, and motion trajectory data) acquired by various sensors into the digital twin model. Within the digital twin environment, the initial virtual state (such as position, appearance, size, and shape) and virtual motion (such as speed, direction, and trajectory) of the target object are created. This process relies on computer vision, graphics rendering, and simulation modeling technologies to visualize and simulate the target object's motion in the virtual space. The server extracts features and analyzes the multimodal data collected by the sensors to determine the target object's true state in the real world, including its position, speed, direction, motion trajectory, posture changes, and its interactions with the environment (such as occlusion and collision). Self-supervised learning, deep learning, and data fusion algorithms are used to align, match, and optimize the data from various sensors to ensure the authenticity and reliability of the extracted information. The server compares the actual state and motion with the virtual state and motion, calculating errors (such as position offset and motion trajectory deviation). Optimization algorithms (such as Kalman filtering, Bayesian reasoning, and deep learning correction models) are used to dynamically adjust the virtual scene to make it more realistic. Update the digital twin model in real time to ensure that the system can always provide accurate monitoring, control and prediction capabilities.

[0051] For example, an autonomous driving company wants to test the driving capabilities of an autonomous vehicle in a virtual simulation environment and use real-world data to correct the simulation system to improve test reliability. LiDAR is used to scan the road environment and build a digital twin city model, including roads, traffic lights, pedestrians, and vehicles. A digital model of the autonomous vehicle is created in the virtual scene, simulating its sensor data and driving behavior. The real autonomous vehicle drives on the test field, recording multimodal data such as onboard camera video, LiDAR point clouds, IMU inertial data, and GPS positioning information. A server analyzes this data to extract information such as the vehicle's actual position, speed, braking points, and steering angles. Comparing the trajectory of the virtual and real autonomous vehicles reveals discrepancies (e.g., different turning radii). The server adjusts the vehicle's dynamic model and sensor error compensation in the virtual scene to more closely align with real-world driving conditions. The corrected virtual scene can be used for autonomous driving AI training and simulation testing, reducing the cost and risk of real-world road testing. The digital twin scene closely matches the real world, making the autonomous driving system's performance in the simulation environment more similar to actual road conditions and improving test reliability.

[0052] For example, a large shopping mall has deployed an intelligent security system, hoping to monitor human behavior in real time within a digital twin environment and promptly detect anomalies (such as theft and violence). The server uses panoramic cameras and lidar to scan the mall's layout, creating a 3D digital twin model of the mall, including counters, aisles, and elevators. Customers and security personnel are mapped into the twin model as virtual objects, preliminarily simulating their normal behavior patterns (such as shopping, walking, and queuing). Using cameras, infrared sensors, RFID tags, and other devices, the system captures data such as people's real-world movement trajectories, dwell time, and body movements. The server analyzes this data to identify people's gait, movements, and abnormal behavior (such as prolonged stays in sensitive areas, unusual running, and items being taken without checkout). The server compares normal behavior patterns in the digital twin with real-world monitoring data to identify discrepancies (such as someone staying for an extended period of time but not reflected in the virtual twin). The server automatically adjusts the state of target objects in the virtual twin to better reflect actual conditions and predicts their next behavior. If the target object's behavior pattern is abnormal, the system automatically triggers a security alarm, notifying management to intervene. Through real-time data correction, digital twin scenarios can accurately reflect human behavior, improve the shopping mall’s security monitoring capabilities, and effectively prevent theft and emergencies.

[0053] S140: Process the multimodal data using a self-supervised neural network model to determine target features corresponding to the target object.

[0054] Specifically, the server trains a multimodal fusion neural network model based on a self-supervised learning mechanism. This model can autonomously learn the internal structure and patterns of data without manual annotation. Using methods such as contrastive learning, feature alignment, and time series prediction, the model can autonomously identify key features of target objects in data from multiple sensors. The server analyzes consecutive frames of sensor data, including RGB images, infrared images, LiDAR point clouds, and event camera data, to learn features such as the appearance, shape, and motion trajectory of target objects. Through self-supervised inter-frame prediction, the model automatically learns the motion patterns of target objects and optimizes itself using a loss function to reduce errors and improve feature extraction accuracy. Appearance features extract information such as color, texture, and edges from RGB images to distinguish different targets. Motion features extract information such as the target's motion trajectory and velocity changes from LiDAR point clouds and event camera data to determine its motion pattern. Environmental context information analyzes the target's surroundings from panoramic camera data and infrared images to identify influencing factors such as occlusion and lighting changes. The server uses contrastive learning to compare features of the same target object acquired by different sensors, enabling the model to better understand cross-modal information about the target object. Through the feature alignment mechanism, the differences between different sensor data are eliminated, ensuring that the features of the target object remain consistent under different modalities, and improving the generalization ability of the model.

[0055] In one possible implementation, a self-supervised neural network model is used to process multimodal data to determine target features corresponding to the target object, specifically including: constructing a self-supervised neural network model using a self-supervised learning mechanism; obtaining the appearance, motion features and environmental context information of the target object through continuous frame reconstruction loss; combining the appearance, motion features and environmental context information, and extracting the target features through comparative learning and feature alignment mechanisms.

[0056] Specifically, self-supervised learning is a label-free learning method that, by designing proxy tasks, enables the model to utilize the data's own information for training, thereby learning the characteristics of the target object. In this method, the self-supervised neural network constructed by the server is primarily based on contrastive learning: it aligns data from different times, different perspectives, and different sensors to improve cross-modal data consistency. The feature alignment mechanism eliminates feature deviations between sensors during the multimodal data fusion process, improving the accuracy of feature extraction.

[0057] The server acquires data from multiple sensors (such as RGB cameras, infrared cameras, LiDAR, and event cameras) and feeds it into a self-supervised model. Through time series feature learning, a feature representation of multimodal data is established, enabling the model to automatically learn the common features of target objects across different viewpoints and modalities. Using contrastive learning, the model learns to identify corresponding relationships between the same object in different modalities, improving cross-modal matching capabilities.

[0058] The continuous frame reconstruction loss is a method in self-supervised learning. It uses video sequences or time series data to enable the model to predict future data from past data, thereby learning the characteristics of the target object. The server uses data from RGB cameras and infrared cameras and employs a convolutional neural network (CNN) to extract information such as the target's color, texture, and shape. By comparing multiple frames, the model learns how the target's appearance changes from different angles and lighting conditions, enabling it to adapt to complex scenarios. For example, in intelligent security surveillance, even if the target changes its coat, it can still be identified based on facial features or body shape. The server uses event camera and lidar data to calculate the target's trajectory, angular velocity, acceleration, and other information. Time series models (such as LSTM and Transformer) are used to analyze the target's motion patterns across different time frames. Using the inter-frame reconstruction loss, the model learns the target's motion pattern, enabling it to predict its trajectory even when the target is partially occluded or temporarily disappears. In autonomous driving scenarios, even if a pedestrian is briefly obscured by a vehicle, the system can still predict their next direction of movement and avoid collisions. The server analyzes data from panoramic cameras and infrared cameras to extract environmental features surrounding the target. Deep neural networks (such as Transformers) are used to analyze the interaction between the target and surrounding objects, such as whether the target is obscured by other objects or whether there is dynamic background interference around the target. In intelligent monitoring systems, even if the target briefly enters a shadowed area, the system can still predict its likely direction of movement based on environmental information. After completing the above feature extraction, the server uses contrastive learning and feature alignment mechanisms to enable the model to extract stable target features from multiple sensor data.

[0059] The server acquires data from different modalities of the target object during the same time period and constructs positive sample pairs (same target, different perspectives) and negative sample pairs (different targets). The model learns to minimize the feature distance of the positive samples and maximize the feature distance of the negative samples. In smart city surveillance, even if the color of the same vehicle varies slightly when viewed from different cameras, the comparative learning model can still identify it as the same target. Data from different sensors has different distributions (for example, RGB and infrared cameras have different data features), necessitating a feature alignment mechanism to represent data from different modalities in a unified feature space. The server utilizes deep neural networks to map the feature layers of data from different modalities, matching the target's features across different modalities and minimizing information loss. In security monitoring, even if a target appears in color on a visible light camera and in black and white thermal image on an infrared camera, the system can still identify it as the same target.

[0060] S150: If it is determined that the target feature has a state change in the virtual scene, a pan-tilt adjustment strategy is generated, where the pan-tilt adjustment strategy is used to control the pan-tilt to monitor the target object.

[0061] Specifically, the virtual scene is constructed based on the target object's multimodal data and digital twin model, reflecting the target object's state and movement in the real environment. Position coordinates (X, Y, Z): Whether the target moves within the monitoring area. Movement direction and speed: Whether the target changes direction or accelerates. Appearance characteristics: Whether the target is difficult to identify due to changes in lighting or angle. Occlusion: Whether the target is obscured by other objects or partially disappears. If a change in the target's characteristics is detected, the next step is to generate a gimbal adjustment strategy.

[0062] When the state of the target object changes in the virtual scene, the server will calculate the optimal adjustment strategy based on the mechanical parameters of the gimbal and the target changes to ensure that the target is always in the best monitoring state. Rotation angle (Pan): Adjust the horizontal rotation direction of the gimbal to ensure that the target is still within the monitoring range. Pitch angle (Tilt): Adjust the vertical angle of the gimbal to ensure that the target does not exceed the field of view due to height changes. Zoom ratio (Zoom): Automatically adjust the focal length according to the target distance to ensure that the target is clearly visible. Automatic compensation (Compensation): When the target is temporarily obscured, its possible position is predicted based on the historical trajectory, and the gimbal viewing angle is adjusted in advance.

[0063] In one possible implementation, if it is determined that the target feature has undergone a state change in the virtual scene, a gimbal adjustment strategy is generated, specifically including: real-time monitoring of the target feature in the virtual scene; if it is determined that the target feature has undergone a state change, a gimbal adjustment strategy is generated, and the state change includes target position offset, speed change, and sudden occlusion; the gimbal adjustment strategy is sent to the gimbal to control the gimbal to make real-time adjustments according to the gimbal adjustment strategy.

[0064] Specifically, the server constructs a virtual scene based on multimodal data and digital twin technology that reflects the state of the target object (such as position, motion trajectory, speed, and appearance) in real time. Within this virtual scene, target characteristics (such as position, speed, and appearance) are continuously monitored to determine whether the target is abnormal or changes. When the target undergoes any of the following state changes, the server triggers a gimbal adjustment strategy: Position offset: The target's spatial coordinates change, such as when the target moves from one area to another. Speed change: The target's movement suddenly speeds up or slows down, such as when a pedestrian suddenly runs or a vehicle suddenly slows down. Sudden occlusion: The target is temporarily blocked by an obstacle (such as a pillar, tree, or other moving object), preventing the camera from directly viewing the target. Based on the target's changes, the server calculates the optimal gimbal adjustment strategy, including: Roll adjustment: Adjusting the horizontal angle of the gimbal to realign the target. Pitch adjustment: Adjusting the vertical angle of the gimbal to accommodate changes in the target's height. Zoom adjustment: Automatically increasing the focus if the target moves farther from the camera, and decreasing the focus if the target moves farther away. Prediction Compensation: If a target is briefly obscured, the server predicts its likely trajectory and adjusts the gimbal angle in advance to ensure the target remains within the monitoring range when it regains visibility. Once the server calculates the target, it sends the adjustment policy to the gimbal control system, which then implements the appropriate adjustments to keep the target within the monitoring range and positioned as close to the center of the image as possible.

[0065] For example, a surveillance camera on a highway is tracking a car traveling normally. Suddenly, the car accelerates, exceeding the speed limit, and attempts to change lanes to evade surveillance. The server monitors the car's speed and trajectory in a virtual scene. The server detects that the target car's speed has suddenly increased, exceeding the normal threshold. The target car deviates from its original trajectory and begins to change lanes. The server calculates the horizontal rotation angle of the pan / tilt system (PTZ) so that the camera can adjust its direction as the target car changes lanes. The camera also calculates the appropriate zoom factor to ensure that the vehicle remains clearly visible. The server predicts the car's likely next trajectory and adjusts the PTZ angle in advance. Based on the strategy, the PTZ camera adjusts its direction and zoom in real time to ensure that the speeding vehicle remains within monitoring range and captures high-definition images for subsequent processing.

[0066] In one possible implementation, refer to Figure 2 , Figure 2Another flow chart of a target monitoring method provided for an embodiment of the present application includes steps S210 to S220, and the above steps are as follows: S210, according to the pan-tilt adjustment strategy, control the pan-tilt to adjust the lens angle and focal length in real time; S220, according to the pan-tilt adjustment strategy, control the target object to remain in the center of the monitoring screen corresponding to the digital twin model.

[0067] Specifically, a pan / tilt adjustment strategy is a set of instructions or policies that guide the pan / tilt system (PTZ) based on the real-time state of the target. This includes: if the spatial position of the target changes, the PTZ adjusts the lens angle (pitch and roll) to ensure the target remains centered in the frame. If the target's distance changes, or if more detailed monitoring is required, the PTZ automatically adjusts the focus to ensure the target remains clearly visible. The model is a virtualized 3D scene that accurately represents the real-world monitoring area. By synchronizing with real-world data (such as position, size, and environmental information), the digital twin model provides an accurate virtual representation of the target's position and motion. The PTZ control system uses this model to obtain the target's precise position information and guide PTZ adjustments. The center of the monitoring frame refers to the target's position within the video surveillance image. Through real-time PTZ adjustments, the target object is always kept centered in the frame, ensuring it remains clearly visible and not out of view.

[0068] This application also provides a target monitoring device, referring to Figure 3 , Figure 3 A schematic diagram of a target monitoring device provided in an embodiment of the present application. The target monitoring device is a server comprising an acquisition module 31 and a processing module 32. The acquisition module 31 acquires multimodal data for a target object; the acquisition module 31 acquires a digital twin model of a target area where the target object is located; the processing module 32 adds the multimodal data to the digital twin model to obtain a virtual scene; the processing module 32 processes the multimodal data using a self-supervised neural network model to determine target features corresponding to the target object; and if the processing module 32 determines that the target features have changed state in the virtual scene, it generates a pan-tilt adjustment strategy, which is used to control the pan-tilt to monitor the target object.

[0069] In one possible implementation, the acquisition module 31 acquires multimodal data for the target object, specifically including: the processing module 32 controls the infrared camera, lidar, and event camera to shoot the target object to obtain an original data packet, which includes an image, depth information, and motion data; the processing module 32 denoises, corrects, and unifies the format of the original data packet to obtain multimodal data.

[0070] In one possible implementation, the acquisition module 31 acquires a digital twin model of the target area where the target object is located, specifically including: the processing module 32 controls the lidar and panoramic image sensor to scan and photograph the target area to obtain fused data; the processing module 32 constructs an initial model by fusing the data; the processing module 32 inputs the static structure and environmental parameters corresponding to the target area into the initial model to obtain a digital twin model.

[0071] In one possible implementation, the processing module 32 adds multimodal data to the digital twin model to obtain a virtual scene, specifically including: the processing module 32 embeds the multimodal data into the digital twin model to obtain the virtual state and virtual motion of the target object in the target area; the processing module 32 analyzes the multimodal data to obtain the actual state and actual motion of the target object in the target area; the processing module 32 uses the actual state and actual motion to correct the virtual state and virtual motion to obtain a virtual scene.

[0072] In one possible implementation, a self-supervised neural network model is used to process multimodal data to determine target features corresponding to the target object, specifically including: the processing module 32 uses a self-supervised learning mechanism to construct a self-supervised neural network model; the processing module 32 obtains the appearance, motion features and environmental context information of the target object through continuous frame reconstruction loss; the processing module 32 combines the appearance, motion features and environmental context information, and extracts the target features through comparative learning and feature alignment mechanism.

[0073] In one possible implementation, if the processing module 32 determines that the target feature has undergone a state change in the virtual scene, a gimbal adjustment strategy is generated, specifically including: the processing module 32 monitors the target feature in real time in the virtual scene; if the processing module 32 determines that the target feature has undergone a state change, a gimbal adjustment strategy is generated, and the state change includes target position offset, speed change, and sudden occlusion; the processing module 32 sends the gimbal adjustment strategy to the gimbal to control the gimbal to make real-time adjustments according to the gimbal adjustment strategy.

[0074] In one possible implementation, the processing module 32 controls the pan-tilt adjustment strategy to adjust the lens angle and focal length in real time; the processing module 32 controls the target object to remain in the center of the monitoring screen corresponding to the digital twin model according to the pan-tilt adjustment strategy.

[0075] It should be noted that the above embodiments provide devices that implement their functions using only the division of the above functional modules as examples. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0076] This application also provides an electronic device, referring to Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: at least one processor 41, at least one network interface 44, a user interface 43, a memory 45, and at least one communication bus 42.

[0077] The communication bus 42 is used to realize the connection and communication between these components.

[0078] The user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may also include a standard wired interface and a wireless interface.

[0079] The network interface 44 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0080] The processor 41 may include one or more processing cores. Using various interfaces and circuits, the processor 41 connects to various components within the server. It executes instructions, programs, code sets, or instruction sets stored in the memory 45, as well as accesses data stored in the memory 45, to perform various server functions and process data. Optionally, the processor 41 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 41 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the display screen; and the modem handles wireless communications. It is understood that the modem may also be implemented as a separate chip, rather than integrated into the processor 41.

[0081] Among them, the memory 45 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 45 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 45 may also be optionally at least one storage device located away from the aforementioned processor 41. As Figure 4 As shown, the memory 45 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a target monitoring method.

[0082] exist Figure 4 In the electronic device shown, the user interface 43 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 41 can be used to call an application program for a target monitoring method stored in the memory 45. When executed by one or more processors, the electronic device executes one or more methods in the above embodiments.

[0083] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0084] The present application also provides a computer-readable storage medium storing instructions, which, when executed by one or more processors, enable an electronic device to execute one or more of the methods described in the above embodiments.

[0085] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic, such as the division of units, which is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0087] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0088] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of this application. The aforementioned memory includes various media that can store program code, such as USB flash drives, mobile hard drives, magnetic disks, or optical disks.

[0090] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any variation, use or adaptive change of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art that are not recorded in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A target monitoring method, characterized in that: The method comprises: Acquire multimodal data for the target object; Obtaining a digital twin model of a target area where the target object is located; Adding the multimodal data to the digital twin model to obtain a virtual scene; Processing the multimodal data using a self-supervised neural network model to determine target features corresponding to the target object; If it is determined that the state of the target feature changes in the virtual scene, a pan-tilt adjustment strategy is generated, and the pan-tilt adjustment strategy is used to control the pan-tilt to monitor the target object.

2. The target monitoring method according to claim 1, characterized in that: The acquiring of multimodal data for the target object specifically includes: Controlling an infrared camera, a laser radar, and an event camera to shoot the target object to obtain an original data packet, wherein the original data packet includes an image, depth information, and motion data; De-noising, correction and formatting are performed on the original data packet to obtain the multimodal data.

3. The target monitoring method according to claim 1, characterized in that: The obtaining of the digital twin model of the target area where the target object is located specifically includes: Controlling the laser radar and the panoramic image sensor to scan and photograph the target area to obtain fused data; By fusing data, an initial model is constructed; The static structure and environmental parameters corresponding to the target area are input into the initial model to obtain the digital twin model.

4. The target monitoring method according to claim 1, characterized in that: Adding the multimodal data to the digital twin model to obtain a virtual scene specifically includes: Embedding the multimodal data into the digital twin model to obtain a virtual state and a virtual motion of a target object in the target area; Analyzing the multimodal data to obtain an actual state and actual movement of the target object in the target area; The virtual state and virtual movement situation are corrected using the actual state and actual movement situation to obtain the virtual scene.

5. The target monitoring method according to claim 1, characterized in that: The self-supervised neural network model is used to process the multimodal data to determine the target features corresponding to the target object, specifically including: Adopting a self-supervised learning mechanism to construct the self-supervised neural network model; Obtain the appearance, motion characteristics and environmental context information of the target object through continuous frame reconstruction loss; The target features are extracted by combining the appearance, the motion features and the environmental context information through contrastive learning and feature alignment mechanism.

6. The target monitoring method according to claim 1, characterized in that: If it is determined that the target feature has a state change in the virtual scene, generating a pan / tilt adjustment strategy specifically includes: In the virtual scene, real-time monitoring of the target features is performed; If it is determined that the target feature has a state change, generating the gimbal adjustment strategy, wherein the state change includes target position offset, speed change, and sudden occlusion; The pan-tilt adjustment strategy is sent to the pan-tilt to control the pan-tilt to make real-time adjustments according to the pan-tilt adjustment strategy.

7. The target monitoring method according to claim 1, characterized in that: The method further comprises: According to the pan / tilt adjustment strategy, control the pan / tilt to adjust the lens angle and focal length in real time; According to the pan-tilt adjustment strategy, the target object is controlled to remain in the center of the monitoring screen corresponding to the digital twin model.

8. A target monitoring device, characterized in that: The target monitoring device includes an acquisition module (31) and a processing module (32), wherein: The acquisition module (31) is used to acquire multimodal data for the target object; The acquisition module (31) is further used to acquire a digital twin model of the target area where the target object is located; The processing module (32) is used to add the multimodal data to the digital twin model to obtain a virtual scene; The processing module (32) is further configured to process the multimodal data using a self-supervised neural network model to determine target features corresponding to the target object; The processing module (32) is further configured to generate a pan-tilt adjustment strategy if it is determined that a state change occurs in the target feature in the virtual scene, wherein the pan-tilt adjustment strategy is used to control the pan-tilt to monitor the target object.

9. An electronic device, characterized in that: The electronic device comprises a processor (41), a memory (45), a user interface (43) and a network interface (44), wherein the memory (45) is used to store instructions, the user interface (43) and the network interface (44) are both used to communicate with other devices, and the processor (41) is used to execute the instructions stored in the memory (45) so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Citation Information

Cited By

  • Method for reminding abnormal passage of airport security check

    CN120766493A