Security check opening and checking strategy intelligent generation method and system based on multi-modal perception and deep reinforcement learning

Abstract

The application discloses a kind of based on multi-modal perception and deep reinforcement learning's security check opening detection strategy intelligent generation method and system.The method includes: the X-ray, infrared and millimeter wave data of the luggage to be checked are synchronously collected, and multi-modal structured data set is fused and constructed;Three-dimensional voxel field containing semantic and physical properties is generated by neural network;Virtual physical graph model is constructed using graph neural network;Run physics engine in digital twin simulation environment, evaluate operational risk;Train reinforcement learning agent, automatically generate opening detection strategy that meets risk constraints and is most efficient in simulation environment;Finally, the strategy is converted into visual operation guide through augmented reality device.The application solves the problem that opening detection decision relies on artificial experience and lacks risk quantification evaluation, and realizes intelligent, safe and efficient automatic opening detection decision.

Description

Technical Field

[0001] This invention relates to the fields of intelligent security inspection and industrial artificial intelligence technology, specifically to an intelligent decision-making method and system that integrates multimodal sensing, digital twin simulation and deep reinforcement learning to generate security inspection opening operation strategies. Background Technology

[0002] During security checks at transportation hubs such as airports and train stations, when suspicious items are detected in luggage using equipment such as X-ray machines, manual baggage inspection by security personnel is usually required. However, this process has significant technical flaws and security risks: First, current baggage inspection procedures heavily rely on the personal experience of security personnel. Faced with luggage of unknown internal structure and complex overlapping items, security personnel can only operate based on static images from two-dimensional X-rays and their own judgment, unable to predict the chain of mechanical effects their actions (such as cutting, rummaging, or moving) will have on the items inside. For example, when handling suspected explosive devices, improper cutting or moving could accidentally trigger hidden fuses or pressure switches, causing a serious safety accident. This "operational blind spot" is a pain point that current technology cannot solve.

[0003] Secondly, although existing technical solutions (such as patent application CN104488002A) have proposed the concept of "virtual inspection" based on three-dimensional volume data, which can visualize the items inside luggage and simulate the removal process, their technical essence remains at the level of visualizing and replaying the inspection process. This solution solves the problem of "how to see the inspection results," but it does not solve the core decision-making problem of "how to perform the inspection safely and efficiently." It lacks the ability to quantitatively assess and proactively plan for the safety and efficiency of the inspection operation itself, as well as the risks that different operational sequences may bring. In other words, it tells the operator "what's inside," but does not guide the operator "how to do it."

[0004] Furthermore, there is currently a lack of a systematic solution that can combine multi-source sensing data (such as X-rays, infrared, and millimeter waves), high-fidelity physical simulation, and advanced artificial intelligence decision-making models. Existing systems struggle to construct a computationally achievable, interactive, and learnable digital twin decision-making environment, enabling intelligent agents to autonomously evolve a complex inspection strategy that balances "maximizing target exposure," "minimizing operational risk," and "optimizing execution steps" through trial and error learning. This results in low levels of intelligence and standardization in security inspections, and poor adaptability to responding to new and covert threats.

[0005] Therefore, there is an urgent need to propose a new technical solution to overcome the shortcomings of existing technologies, such as reliance on human experience for inspection decisions, lack of risk prediction and quantitative assessment, and insufficient level of intelligence. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for intelligent generation of security inspection opening and inspection strategies based on multimodal perception and deep reinforcement learning, as well as electronic devices and computer-readable storage media, to solve the problems of existing security inspection decisions relying on human experience, which are inefficient and risky.

[0007] This invention provides an intelligent generation method for security check opening strategies based on multimodal perception and deep reinforcement learning. The method includes the following steps: A method for intelligently generating security check opening strategies based on multimodal perception and deep reinforcement learning includes the following steps: Step 1. Synchronously acquire multimodal sensing data of the inspected luggage, including X-ray transmission images, infrared thermal imaging, and millimeter-wave radar data; perform time-stamp alignment, spatial registration, and feature fusion on the multimodal sensing data to construct a three-dimensional spatial voxel grid, where each voxel is associated with a feature vector, forming a multimodal structured dataset; the feature vector is composed of fused physical feature values ​​from different modalities, and the physical feature values ​​include at least the equivalent atomic number, temperature value, and backscattering intensity value; Step 2. Using the multimodal structured dataset as input, decode and semantically fuse the feature vector associated with each voxel through a cross-modal attention coding network to generate semantic category probability and physical attribute vector for each voxel, thereby constructing a multi-attribute three-dimensional voxel field; Step 3. Based on the multi-attribute three-dimensional voxel field, a graph neural network is used to reason and model the physical components inside the luggage and their connection relationships, generating a virtual physical graph model that represents the physical components inside the luggage and their interaction relationships. Step 4. Based on the virtual physical graph model, construct a digital twin simulation environment, run a virtual physical engine in the environment to respond to the opening and inspection operation command and simulate the mechanical impact and chain risks of the opening and inspection operation corresponding to the command on the internal structure of the luggage; Step 5. Based on the reinforcement learning algorithm, train the agent to explore and learn in the digital twin simulation environment; apply the trained agent to the digital twin simulation environment to generate an opening and inspection strategy that satisfies risk constraints and is most efficient according to the initial simulation state configured by the virtual physical graph model. Step 6. Transform the optimal strategy into visual instructions for the augmented reality device; display safe operating paths and high-risk area warnings in real time.

[0008] The intelligent generation method for security check opening and closing strategies based on multimodal perception and deep reinforcement learning, as described above, includes the following steps in step 1: timestamp alignment, spatial registration, and feature fusion of the multimodal perception data. The central timing controller sends synchronous trigger pulses to the acquisition circuits that acquire X-ray transmission images, infrared thermal imaging, and millimeter-wave radar data to achieve timestamp alignment. Based on the multimodal joint calibration body, all sensor data are unified under the same world coordinate system to achieve spatial registration; The registered data is sampled onto a uniform 3D voxel grid, and a feature vector is constructed for each voxel.

[0009] As described above, in the intelligent generation method for security check opening and closing strategies based on multimodal perception and deep reinforcement learning, step 2, which involves decoding and semantically fusing the multimodal feature vectors associated with each voxel using a cross-modal attention encoding network, specifically includes: The multimodal structured dataset is input into an encoder-decoder architecture that combines a three-dimensional convolutional neural network with a cross-modal attention module; In the middle and higher levels of the encoder, the weights between features from different modalities are dynamically calculated through the cross-modal attention module to achieve adaptive feature fusion; the middle and higher levels refer to the middle or later stages of feature extraction in the encoder. The decoder outputs two parallel output heads at the end, which output a semantic category probability vector and a physical property vector containing standardized density, thermal conductivity, initial risk base value and mechanical stiffness identifier for each voxel; wherein the initial risk base value is a scalar preset according to the semantic category and abnormal thermal features.

[0010] As described above, the intelligent generation method for security check opening and closing strategies based on multimodal perception and deep reinforcement learning, specifically includes step 3 as follows: Based on the multi-attribute three-dimensional voxel field, independent physical components are identified through a three-dimensional connected component analysis algorithm and used as nodes in the virtual physical graph model. A graph neural network is used to infer the physical connection relationships between the nodes, predict whether there is a connection between the nodes, the type of connection, and the strength of the connection, and use it as the edge of the virtual physical graph model; The node attributes of the virtual physical graph model include the geometric attributes, semantic attributes, and physical attributes of the components, while the edge attributes include the connection type and the connection strength coefficient.

[0011] As described above, the intelligent generation method for security check opening and closing strategies based on multimodal perception and deep reinforcement learning, specifically includes step 4 as follows: The nodes in the virtual physical graph model are instantiated as rigid bodies in the physics engine, and their edges are instantiated as corresponding physical constraints. In the digital twin simulation environment, a structured opening and closing operation command is received. The command includes the operation type, target and parameters, and the command is mapped to the dynamic control of the corresponding rigid body or constraint in the physics engine. Advance physical simulation to monitor and record connection breakage events, component movement speed exceeding limits events, and disturbance events where the component's comprehensive risk base value exceeds a preset threshold, and quantify the risk value of a single operation based on a preset risk assessment function.

[0012] As described above, in the intelligent generation method for security inspection and opening strategies based on multimodal perception and deep reinforcement learning, step 5 involves training the agent to explore and learn in the digital twin simulation environment, including: The inspection process is modeled as a Markov decision process, defining the state space, action space, and reward function. Based on the defined state space, action space, and reward function, an interaction framework between the intelligent agent and the digital twin simulation environment is constructed. The reward function is a multi-objective function, and its output value includes a first component that is positively correlated with the increase in the visibility ratio of the target suspicious object, a second component that is positively correlated with the negative value of the operational risk assessment value, and a third component that is negatively correlated with the number of operational steps. A proximal policy optimization algorithm is used to enable the agent to explore the environment through trial and error based on the interaction framework, with the goal of maximizing long-term cumulative rewards. The agent is trained by including a policy network and a value network.

[0013] As described above, the intelligent generation method for security check opening and closing strategies based on multimodal perception and deep reinforcement learning, specifically includes step 6 as follows: Based on visual tags, spatial registration is performed on physical luggage, augmented reality devices, and the digital twin simulation environment; The optimal opening and inspection strategy is analyzed to generate a corresponding three-dimensional guide graphic, including a cutting path guide line, a highlight box of the operation target, and a risk area warning body; The three-dimensional guiding graphic is superimposed and rendered onto the display screen of the augmented reality device in real time based on the spatial registration result.

[0014] This invention also provides an intelligent generation system for security check and opening strategies based on multimodal perception and deep reinforcement learning. This system executes the aforementioned intelligent generation method for security check and opening strategies based on multimodal perception and deep reinforcement learning. The system includes a multimodal perception data acquisition and structuring module, a three-dimensional voxel field construction module, a virtual physical modeling module, a strategy simulation and deduction module, a strategy intelligent generation module, and an AR-guided execution module. Multimodal sensing data acquisition and structuring module: used to synchronously acquire multimodal sensing data of the inspected luggage, including X-ray transmission images, infrared thermal imaging, and millimeter-wave radar data; the multimodal sensing data is time-stamp aligned, spatially registered, and feature fused to construct a three-dimensional spatial voxel grid, where each voxel is associated with a feature vector, which is composed of fused physical feature values ​​from different modalities, including at least equivalent atomic number, temperature value, and backscattering intensity value; The three-dimensional voxel field construction module is used to take the multimodal structured dataset as input, decode and semantically fuse the multimodal feature vector associated with each voxel through a cross-modal attention coding network, generate semantic category probability and physical attribute vector for each voxel, and thus construct a multi-attribute three-dimensional voxel field. Virtual physical modeling module: Based on the multi-attribute three-dimensional voxel field, it uses a graph neural network to reason and model the physical components inside the luggage and their connection relationships, generating a virtual physical graph model that represents the physical components inside the luggage and their interaction relationships. Strategy simulation and deduction module: Based on the virtual physical graph model, it is used to build a digital twin simulation environment, run a virtual physical engine in the environment, respond to the opening and inspection operation command, and simulate the mechanical impact and chain risks of the opening and inspection operation corresponding to the command on the internal structure of the luggage. The strategy intelligent generation module is used to train an agent to explore and learn in the digital twin simulation environment based on reinforcement learning algorithms; and to apply the trained agent to the digital twin simulation environment to generate an opening and inspection strategy that meets risk constraints and is most efficient, based on the initial simulation state configured by the virtual physical graph model. AR-guided execution module: used to translate optimal strategies into visual instructions for augmented reality devices, displaying safe operation paths and high-risk area warnings in real time.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the method as described in any of the above.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform any of the methods described above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The system achieves automatic generation and optimization of inspection strategies: By training and making decisions in a digital twin simulation environment using deep reinforcement learning algorithms, the system can automatically generate inspection strategies that meet risk constraints and are most efficient, fundamentally solving the problems of inspection decisions relying on human experience and lacking quantitative evaluation and global optimization.

[0018] Significantly improves the safety of baggage inspection operations: By constructing a virtual physical graph model that integrates multimodal perception data and running a physics engine in a high-fidelity digital twin simulation environment, it is possible to accurately simulate the chain mechanical effects of baggage inspection operations on the internal structure of the baggage, realize the early prediction and quantitative assessment of operational risks, and effectively avoid the "operational blind spots" in traditional baggage inspection.

[0019] The system's adaptability to complex scenarios has been enhanced: the deep reinforcement learning agent can learn and optimize strategies autonomously through trial and error in a simulation environment, enabling the system to adapt to different internal structures and threat types without redesigning fixed rules, thus improving the system's intelligence and generalization ability in handling unknown and complex scenarios.

[0020] It improves the accuracy and efficiency of on-site operations: By using augmented reality (AR) technology, the optimal strategy is transformed into an intuitive visual operation guide and presented to security inspectors in real time in a virtual-real overlay manner. This can effectively avoid misoperation caused by unclear instructions or misunderstandings, realize efficient human-machine collaboration, and ensure the accurate and safe execution of strategies. Attached Figure Description

[0021] Figure 1 A flowchart illustrating the intelligent generation method for security inspection and opening strategies based on multimodal perception and deep reinforcement learning provided in an embodiment of the present invention; Figure 2 A schematic diagram of the architecture of the intelligent generation system for security inspection and opening strategies based on multimodal perception and deep reinforcement learning provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention. Detailed Implementation

[0022] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the intelligent generation method and system for security inspection and opening strategies based on multimodal perception and deep reinforcement learning, electronic equipment, and computer-readable storage medium proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The following describes in detail, with reference to the accompanying drawings, the specific solutions of the intelligent generation method and system for security inspection and opening strategies based on multimodal perception and deep reinforcement learning, electronic devices, and computer-readable storage media provided by the present invention.

[0023] Example 1: This embodiment describes in detail the specific technical implementation of each step in the method. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating the intelligent generation method for security inspection and opening strategies based on multimodal perception and deep reinforcement learning provided in this application embodiment. The core innovative steps include multimodal data acquisition and fusion, virtual physical modeling, and intelligent strategy generation.

[0024] The present invention discloses an intelligent generation method for security inspection opening strategies based on multimodal perception and deep reinforcement learning, the implementation process of which is as follows: Step 1. Multimodal sensing data acquisition and structuring: Simultaneously acquire multimodal sensing data of the inspected luggage, including X-ray transmission images, infrared thermal imaging, and millimeter-wave radar data; perform time-stamp alignment, spatial registration, and feature fusion on the multimodal sensing data to construct a three-dimensional spatial voxel grid, where each voxel is associated with a feature vector, which is composed of fused physical feature values ​​from different modalities, including at least equivalent atomic number, temperature value, and backscattering intensity value; It should be noted that the purpose of this step is to address the existing technological deficiency of "virtual inspection decisions lacking precise perception of the multi-dimensional attributes of luggage's internal physical, chemical, and structural properties" by providing a collaborative and calibrable multimodal perception data acquisition and fusion solution. Its core is not merely "reconstructing a visual model," but rather providing calculable, physically rich raw data input for subsequent virtual physical simulation and intelligent decision-making steps. This is the starting point and cornerstone for realizing the technology of "data-driven decision-making and simulation-verified decision-making."

[0025] The implementation of the above steps specifically includes the following two sub-steps: Step 1.1: Simultaneous acquisition of X-ray transmission images, infrared thermal imaging, and millimeter-wave radar data of the inspected luggage is achieved through an integrated sensor data acquisition unit. This sensor data acquisition unit includes: a dual-energy X-ray transmission imaging module, an uncooled infrared focal plane array (IR-FPA) module, and a frequency-modulated continuous wave (FMCW) millimeter-wave radar module. Among them, Dual-energy X-ray transmission imaging module: Employing a high dynamic range detector, it simultaneously acquires two spatially strictly registered high- and low-energy grayscale images through single-tube rapid voltage switching (80kV / 140kV) or dual-tube time-division triggering. and Based on the two images above and the energy spectrum response curves obtained beforehand using standard samples (such as aluminum, polyethylene, and polytetrafluoroethylene modules with known equivalent atomic numbers), the equivalent atomic number matrix was calculated. The pixels are initially classified based on the Zeff value to provide material feature input for subsequent identification of contraband. For example, areas with Zeff values ​​less than a first preset threshold (such as 10) are initially classified as organic feature areas, and areas with Zeff values ​​greater than or equal to the first preset threshold are initially classified as inorganic feature areas.

[0026] Uncooled infrared focal plane array (IR-FPA) module: Coaxial or side-by-side with the X-ray source, it acquires the thermal radiation intensity distribution on the surface of luggage in real time, outputting a raw grayscale image reflecting the thermal radiation intensity of the luggage surface. This image is then calibrated based on a blackbody radiation source and converted into an absolute temperature distribution matrix. The temperature resolution is better than 0.05°C, and the spatial resolution matches that of X-ray images (e.g., 640×512 pixels). Further calculation of the spatial gradient of the above temperature distribution matrix yields the temperature gradient matrix. It is used to identify areas of abnormal temperature change.

[0027] Frequency Modulated Continuous Wave (FMCW) millimeter-wave radar module: Employing a multi-transmitter, multi-receiver (MIMO) antenna array, operating in the W-band (e.g., 94 GHz), it transmits frequency-modulated continuous waves and receives backscattered signals from different media interfaces inside the luggage. Using a synthetic aperture radar (SAR) imaging algorithm, it generates three-dimensional point cloud data reflecting the distribution of scattering intensity inside the luggage. Each data point in this point cloud data contains its three-dimensional spatial coordinates. and the corresponding backscattering intensity value The scattering intensity value directly reflects the strength of the medium's ability to reflect millimeter-wave signals at that location. High scattering intensity usually corresponds to materials with high dielectric constant (such as metals or substances with high water content), while low scattering intensity corresponds to non-metallic materials with low dielectric constant or low surface roughness.

[0028] The outputs of the above three modules are not merely "images" for display, but structured data matrices with clear physical meaning and alignable coordinates. These data form the basis for subsequent processing.

[0029] Step 1.2: Timestamp Alignment, Spatial Registration, and Feature Fusion: To achieve unified processing of multimodal sensing data, this sub-step performs the following operations: Hardware synchronization and timestamp alignment: The central timing controller (e.g., based on FPGA) sends synchronization trigger pulses to the acquisition circuits of the above three modules to ensure that the start time of acquisition of all data in each scan cycle is strictly aligned and the synchronization error is less than 1 millisecond; Spatial registration: During system installation, a multimodal joint calibration body (which has feature points with known spatial coordinates clearly distinguishable under X-ray, infrared, and millimeter-wave conditions, such as a heating material block with embedded metal spheres) is used for one-time calibration. Through calibration, the rigid transformation parameters (rotation matrix and translation vector) between the X-ray image coordinate system, the infrared image coordinate system, and the millimeter-wave radar coordinate system are calculated. In subsequent operation, these parameters are used to unify all sensor data into a world coordinate system with the baggage scanning center as the origin.

[0030] Feature Fusion: The "fusion" in this step is not a pixel-weighted average of the original image, but rather, under a unified spatial reference system, extracting and associating complementary physical features from each modal sensing data to construct a multi-dimensional feature vector for each spatial unit (voxel). Specific operations: 1. Voxel Meshing: The previously registered 3D space in the world coordinate system is discretized into a regular 3D mesh, i.e., a 3D spatial voxel mesh, where each smallest unit is called a voxel. 2. Feature vector construction: For each voxel in the world coordinate system... a) Obtain the equivalent atomic number of the voxel position from the registered Zeff matrix through interpolation; b) Obtain the equivalent atomic number from the registered temperature distribution matrix T and its gradient matrix. In step T, the absolute temperature and temperature gradient values ​​at the voxel location are obtained through interpolation; c, in the registered millimeter-wave three-dimensional scattering intensity point cloud, the point that falls into or is closest to the voxel is found, and its backscattering intensity value is obtained. As a representation of this voxel. 3. Output definition: Each voxel Corresponding to a fusion feature vector All voxels and their associated eigenvectors Together, these constitute a complete, structured dataset, namely a multimodal structured dataset. This dataset uses a three-dimensional voxel grid as its carrier, with each grid point carrying physical features from different sensors. It should be noted that the "interpolation" operation here refers to sampling modal data of different spatial resolutions, already unified to the world coordinate system, onto a unified three-dimensional voxel grid using conventional methods in the field, such as trilinear interpolation. At the vertices, feature vectors for each voxel are constructed.

[0031] Through the above operations, this step achieves the following effects: (1) It overcomes the limitation of a single X-ray sensor being insensitive to non-metallic and temperature-abnormal items, and provides a representation of the physical world that approximates reality for the virtual environment. (2) The output is not an "image", but a multi-dimensional feature vector with clear physical meaning and unified coordinates, which provides a structured input for subsequent algorithm processing (deep learning, physical simulation).

[0032] Step 2. Construction of 3D voxel field: Using the multimodal structured dataset formed in Step 1 as input, the feature vector associated with each voxel is decoded and semantically fused through a cross-modal attention coding network to generate semantic category probability and physical attribute vector for each voxel, thereby constructing a multi-attribute 3D voxel field; It should be noted that the purpose of this step is to transform the multimodal structured dataset output from step 1.2 into a three-dimensional environment model with high-level semantic understanding and quantified physical properties, namely a multi-attribute three-dimensional voxel field, through a deep learning model. This step achieves a leap from the perception of heterogeneous physical signals at the lower level to the cognition of a unified environment at the higher level, providing a scene foundation that can be directly calculated and interacted with for subsequent virtual physics simulation and intelligent decision-making.

[0033] The construction of the 3D voxel field is fundamentally based on an end-to-end deep learning-based feature decoding and semanticization process. Specifically, it employs a 3D convolutional encoder-decoder network that integrates cross-modal attention mechanisms. This model takes the aforementioned multimodal structured dataset as input and directly outputs the semantic category and physical properties of each voxel. The specific implementation is as follows: 1. Network Input: Represent the multimodal structured dataset generated in step 1.2 as a dataset with dimension [dimension not specified]. A four-dimensional tensor. Among them, Let C represent the dimensions of the 3D voxel mesh in three spatial directions, and C be the dimension of each voxel feature vector. In this embodiment, C=4, corresponding to four physical features from X-ray, infrared, and millimeter-wave sensors: equivalent atomic number Zeff, absolute temperature T, and temperature gradient. Backscattering intensity Each element (i, j, k, c) of this tensor represents the c-th eigenvalue of voxel V(i, j, k).

[0034] 2. Network architecture: An encoder-decoder architecture combining a three-dimensional convolutional neural network (3D-CNN) with a cross-modal attention module is adopted.

[0035] Encoder: Used to extract multi-level, multi-scale spatial features from input tensors. It can consist of multiple sequentially connected modules, each typically containing a 3D convolutional layer, a batch normalization layer, an activation function (such as ReLU), and a 3D pooling layer. By stacking such modules, the spatial size of the feature map decreases progressively, while the number of feature channels and the level of semantic abstraction increase progressively.

[0036] The cross-modal attention module is the core of the network, used to achieve adaptive feature fusion. This module is configured to be embedded in the mid-to-high levels of the encoder. "Mid-to-high levels" refers to the middle or later stages of feature extraction in the encoder; specifically, it refers to the network layers after the encoder has completed at least two downsampling (pooling) operations, but before the final feature map is compressed to its minimum size. For example, in an encoder with four downsampling stages, the attention module can be embedded after the second or third downsampling stage. Embedding at this stage is chosen because the feature map at this point retains sufficient spatial detail for fine-grained alignment of features from different modalities, and has already undergone a certain degree of abstraction, containing rich semantic information, making it suitable for modeling global correlations between modalities. Its working principle is: the multi-channel feature map output from the current stage is processed according to the feature source modality (i.e., corresponding to Zeff, T / ...). The feature channels of T and σ are grouped, and then each group of features is mapped to a query vector and a key vector through a learnable linear transformation. By calculating the similarity between different groups of key vectors and query vectors (e.g., using dot product), an attention weight matrix representing the importance between modalities is obtained. This weight matrix is ​​used to dynamically reweight the original features, allowing the network to autonomously decide which modality's information to emphasize during fusion based on the current context. For example, the network can learn to assign higher weights to T and Zeff features when identifying "high-temperature metal parts," while focusing more on the combination patterns of σ and T when analyzing "internal liquids." This data-driven, context-sensitive feature fusion mechanism is the core of this step in achieving high-precision semantic understanding.

[0037] The decoder is responsible for upsampling the high-level semantic features extracted by the encoder back to the original input resolution, achieving dense voxel-level prediction. It typically consists of multiple 3D deconvolutional layers and can be fused with feature maps of the same scale from the encoder via skip connections to preserve detailed information.

[0038] 3. Network output and generation of "multi-attribute three-dimensional voxel field": At the end of the decoder, there are two parallel output heads, both of which are three-dimensional convolutional layers with a kernel size of 1*1*1, used for semantic classification and physical attribute regression, respectively.

[0039] Semantic category prediction: The first output head is for each voxel Output a (C) s +1) dimensional probability vector C s The number of semantic categories is a preset limit, where each category is a refinement and expansion of the initial material classification in step 1, for example, C. s =6 corresponds to: background, textiles, plastic / rubber, metal, electronic components, and liquid; +1 represents the "unknown" category. This vector is normalized using the Softmax function, representing the probability distribution of the voxel belonging to each category. The increase in the number of categories here stems from the network's comprehensive utilization of all multimodal features and their spatial context information, achieving more refined and accurate material recognition than a single Zeff threshold judgment.

[0040] Physical property prediction: The second output head is for each voxel Regress a physical property vector This vector contains several quantifiable physical parameters, which may include, for example, a normalized density used for mechanical simulation. Examples of components include the thermal conductivity coefficient K used for thermal simulation, the initial risk baseline value R0 used for risk assessment, and the mechanical stiffness indicator S.

[0041] The parameters in the physical property vector are determined in the following way: Standardized density The electron density calculated from X-ray dual-energy data is mapped to the [0,1] interval, where 1 represents the densest known material (such as lead). Thermal conductivity coefficient K and mechanical stiffness label S: both are categorical values. The system has a built-in material property lookup table. This is used to generate physical property vectors predicted from voxels. At that time, the system determines the semantic category (i.e., P) that has the highest probability predicted by the voxel in step 2. sem (The table shows the category with the highest probability), and K and S are assigned corresponding preset classification values. For example, for the "metal" category, S is assigned the value "hard" and K is assigned the value "high"; for "liquid", S is assigned the value "no fixed shape" and K is assigned the value "medium".

[0042] The initial risk baseline value R0 is a scalar value predefined based on semantic category and abnormal thermal characteristics (high temperature). For example, a component with the semantic category of "known explosive" has R0=1.0; a component with the semantic category of "electronic component" and accompanied by abnormal high temperature has R0=0.7; and a component with the semantic category of "ordinary clothing" has R0=0.1. This mapping relationship is predefined based on knowledge in the security inspection domain.

[0043] In one embodiment, the physical attribute vector A phy The parameters K, S, R0, etc., can be determined using the aforementioned rule lookup table method. In a more preferred embodiment, to improve model accuracy and adaptability, these attributes can also be learned implicitly by the network from the input multimodal features and learned semantic representations through supervised training. In this mode, the training phase requires the use of some data with ground truth labels for physical attributes, and by designing a loss function that includes regression loss, the network is guided to learn a complex mapping from features to attributes. Regardless of the method used, the network will output a complete physical attribute vector A for each voxel. phy .

[0044] Finally, the generated multi-attribute three-dimensional voxel field can be denoted as: This voxel field is the unique and unified environmental representation for all subsequent steps (virtual physical modeling, policy simulation, intelligent decision-making).

[0045] Through this step, the following effects were achieved: (1) Intelligent understanding from features to semantics was completed, and multimodal sensor data was transformed into a three-dimensional environment model rich in semantic information and physical parameters; (2) The optimal fusion problem of heterogeneous features was solved through the data-driven attention mechanism, and the modeling accuracy in complex scenes was improved; (3) The output voxel field provides a standardized interface for subsequent steps, and the semantic and physical information contained therein can be directly used for the simulation parameter setting of the virtual physics engine and the evaluation of intelligent decision-making algorithms.

[0046] Step 3. Virtual physical modeling: Based on the multi-attribute three-dimensional voxel field, a graph neural network is used to reason and model the physical components inside the luggage and their connection relationships, generating a virtual physical graph model that represents the physical components inside the luggage and their interaction relationships; It should be noted that the purpose of this step is to construct a dynamic, structured model, or "virtual physical graph model," based on the multi-attribute 3D voxel field generated in step 2, capable of mechanical interactive simulation. Step 2 clarified "what it is" and "what attributes it possesses," while this step defines "how they are connected and how they interact," which forms the logical foundation for subsequent physical simulation. Specifically, this step needs to address: how to identify "object components" with independent mechanical behavior from a massive number of voxels; how to define the connection relationships between these components (such as screw tightening, adhesion, electrical connection); and how to assign physical parameters to these components and connections that conform to their semantics. This provides calculable physical rules for "pre-simulating the chain effects of opening and closing operations in a digital twin simulation environment."

[0047] The virtual physics modeling takes the voxel field output from step 2 as its input. The core processing method involves using a Graph Neural Network (GNN) for relational reasoning, outputting a virtual physical graph model represented by a graph structure. The specific implementation includes the following sub-steps: Step 3.1 Object instance segmentation and graph node generation: Input: The multi-attribute 3D voxel field output from step 2, especially the semantic category probability vector of each voxel. and physical property vector .

[0048] Processing steps: First, based on the semantic categories of voxels (choosing the category with the highest probability), a 3D connected component analysis algorithm is used to cluster the voxel field. Voxels that are spatially adjacent and have the same (or similar) semantic categories are grouped into an independent "physical component". Each identified object instance constitutes a node in the virtual object graph.

[0049] Node attribute definition: Define node attributes for each graph node n. The node attributes are a collection of all the information necessary to describe the node in subsequent physical simulation and decision-making, specifically including the following: (1) Geometric properties: Based on the spatial coordinates of all voxels that make up the node, calculate its bounding box, centroid position, and approximate shape (e.g., approximately cuboid or cylinder).

[0050] (2) Semantic attributes: describe the material or functional category of a node. In this application, this is defined as the dominant semantic category, denoted as... The method for determining this is as follows: First, based on the semantic category probability vector predicted for each voxel in step 2... First, determine the semantic category of each voxel (taking the category with the highest probability). Then, count the semantic categories of all voxels within that node, and determine the category with the highest frequency (i.e., the mode) as the dominant semantic category of that node. For example, if most voxels of a node are classified as "metal", then its It means "metal".

[0051] (3) Physical properties: Quantization parameters describing the node required in physical simulation. These are obtained by analyzing the physical property vectors of all voxels constituting the node. This is obtained through aggregation calculations. Specifically, it includes: Average Standardized Density For all voxels within a node The value is taken as the arithmetic mean; Main material stiffness marking : Take the mode (the most frequent value) of the S-label of all voxels within the node. Comprehensive risk base value For all voxels within a node The value is taken as an arithmetic mean, or a weighted calculation based on the proportion of high-risk voxels.

[0052] Based on the node attributes (geometric, semantic, and physical attributes) defined above, the subsequent simulation environment initialization step (step 4.1) will calculate the precise simulation parameters required by the physics engine for this node. For example, the mass of the node is calculated from its geometric attributes (volume) and the average normalized density in its physical attributes; its material attributes (such as elastic coefficient and friction coefficient) are obtained by querying a preset physical parameter mapping library, based on its semantic attributes (dominant semantic category). ) and physical properties (dominant rigidity identifier) The initial risk label is determined by its physical attributes; the initial risk label is directly associated with the comprehensive risk base value in its physical attributes. All physical component nodes identified and whose attributes have been calculated through the above process together constitute the node set of the virtual physical graph model, denoted as N. That is, each node in set N... Each of these is a data structure that carries complete geometric, semantic, and physical attributes.

[0053] Step 3.2 Connection Reasoning and Graph Edge Generation: This sub-step employs a graph neural network model specifically designed for relational reasoning. Its specific implementation is as follows: a. Graph construction and node characterization: For each node n in the node set N obtained in the previous step, use its aggregated geometric and physical attribute vectors as the initial features of that node.

[0054] b. Model Architecture and Training: The GNN adopts either a Graph Attention Network (GAT) or a Graph Convolutional Network (GCN) architecture. During the offline training phase, a training dataset needs to be prepared. This dataset is constructed as follows: using 3D modeling software or procedural generation rules, a large number of virtual luggage models containing different internal structures and component connections (such as screw connections, binding, and welding) are created. For each model, its accurate virtual physical graph model G can be automatically exported. gt As a truth value, the node properties are known, as are the edges and their types (rigid, flexible, electrical) and strength coefficients. All have been determined. During training, a graph containing only the geometric and physical properties of nodes (excluding connectivity information) is input into the GNN. The GNN uses multi-layer message passing; at each layer, each node aggregates the features of its neighbors (e.g., taking the mean of the feature vectors) and updates it by combining it with its own features, ultimately generating a feature representation for each node that contains its local connectivity context. Subsequently, the edge classification head is applied to any pair of nodes. The final features are used to predict the connection probabilities between them. By minimizing the cross-entropy loss between the predicted probabilities and the true connection labels, the GNN learns to infer connection patterns from "node self-attributes" and "graph topology". After training, the network parameters are fixed. For a new piece of luggage, the GNN can predict the connections based on its component attributes. The edge classification head is typically implemented by a multilayer perceptron (MLP).

[0055] c. Loss function and learning objective: The loss function used for training is the cross-entropy loss of edge classification, with the aim of making the connection probability distribution predicted by GNN as close as possible to the true label.

[0056] After training, the network parameters are fixed. For a new baggage voxel field, after obtaining the node set and extracting features in step 3.1, it is input into the trained GNN. The edge classification head of the GNN outputs the connection probability for each pair of nodes. By setting a threshold (e.g., maximum probability > 0.5), the existence and type of an edge can be determined. Simultaneously, each edge... A connection strength coefficient is assigned. (A scalar between 0 and 1, where 1 represents the strongest connection) and a connection type label, serving as edge attributes. The connection types may include, but are not limited to: rigid connections (simulating welding, screw fastening, with almost no displacement before breakage), flexible connections (simulating binding, hinges, allowing a certain range of relative movement), and electrical connections (simulating wire connection, affecting risk transmission). In subsequent simulations, breaking a rigid connection requires a greater force, while breaking a flexible connection may first involve deformation. This will directly affect the force required to separate the two objects in the simulation.

[0057] Step 3.3 Construction of the Virtual Physical Graph Model: Finally, a virtual physical graph model is generated, whose graph structure can be represented as follows: Where: N is the set of nodes, each node represents a physical instance; E is the set of edges, each edge represents a predicted physical connection; It is a collection of node attributes, containing the geometric and physical properties of each node; This is a set of edge attributes, containing the connection strength and connection type label (such as rigid, flexible, electrical connection) for each edge. This virtual physical graph model G is a significant simplification and abstraction of the complex internal structure of luggage, but it fully retains key mass, connection, and risk information, making it an ideal intermediate representation for efficient and stable simulation. This graph model G is the direct input to the virtual physics engine in step 4 (strategy simulation derivation). The physics engine will simulate the movement of nodes, the breakage of edges, and the resulting chain reactions when external forces (cutting, grasping) are applied, based on the node attributes (mass, stiffness) and edge attributes (connection strength) in G.

[0058] After generating the virtual physical graph model G, based on preset rules (such as semantic categories like "explosives", "liquids", and "specific electronic devices") and risk baseline values... Alternatively, upon receiving external security inspection instructions, one or more nodes in the node set N of G are marked as "target suspicious object nodes," and this marking information is stored as part of the node attributes. .

[0059] This step achieves the following technical effects: it simplifies the fine-grained field of millions of voxels into a graph of tens to hundreds of nodes, significantly reducing the computational complexity of subsequent physical simulations and enabling them to run in real time; it clarifies the mechanical constraints between objects through "edges," which is a key prerequisite for predicting chain reactions such as "cutting A causes B to fall," surpassing traditional simulations that can only handle single objects; and the attributes of nodes and edges provide direct parameter inputs for physics engines (such as Bullet and PhysX), enabling virtual simulations to be based on real physical laws.

[0060] Step 4. Strategy simulation and deduction: Based on the virtual physical graph model, construct a digital twin simulation environment, run the virtual physical engine in the environment to respond to the opening and inspection operation command and simulate the mechanical impact and chain risks of the opening and inspection operation corresponding to the command on the internal structure of the luggage; It should be noted that the purpose of this step is to construct a high-fidelity digital twin simulation environment using the virtual physical graph model built in step 3, which contains precise physical properties and connections. Within this environment, various opening and inspection operations (such as cutting, grabbing, and moving) that security personnel might perform can be simulated in real-time and interactively. Based on the laws of physics, the chain reactions caused by these operations on the internal structure of luggage, such as mechanical deformation, component displacement, and connection breakage, can be accurately calculated, thereby quantifying the potential risks that each operation may trigger. This provides a crucial, quantifiable "trial and error" and "evaluation" platform for the intelligent strategy generation in the subsequent step 5.

[0061] The core of the strategy simulation is a simulation system integrating a physics engine, an interactive interface, and a risk assessment module. The specific implementation is as follows: Step 4.1 Simulation Environment Initialization and Model Loading: Import the graph model G into a mature, high-precision physical simulation engine (such as NVIDIA PhysX, Bullet Physics, or Unity3D PhysX). During this process, the system has a built-in physical parameter mapping library. This mapping library is essentially a pre-defined, structured database. Its core data structure uses a combination of "material semantic category - hardness grade" as a composite primary key, mapping to a set of specific physical parameter values ​​used in the physical simulation. More specifically, each record in the database contains the above fields: 1. Semantic categories: such as "metal", "plastic", "liquid", "textile".

[0062] 2. Hardness / rigidity grade: such as "hard", "medium", "soft", "no fixed shape".

[0063] 3. Physical parameter set: including but not limited to density, elastic coefficient, static friction coefficient, and kinetic friction coefficient. These parameter values ​​are pre-measured, calibrated, and stored in this library through physical experiments on standard samples.

[0064] The mapping from the abstract graph model to the entities that the physics engine can compute is as follows: Mapping nodes to rigid bodies: Map each node Instantiated as a rigid body in the physics engine. Its mass is determined by the average normalized density of the nodes. The collider's shape is calculated from its bounding box and its approximate shape; its material properties (such as elastic coefficient, static friction coefficient, and kinetic friction coefficient) are determined by querying the physical parameter mapping library. This query is based on the node's semantic attributes (dominant semantic category). The primary key is defined by , and the dominant stiffness identifier in its physical properties is referenced. This is to obtain a set of typical physical parameters that conform to the material category and hardness characteristics. For example, for It is a "metal" and For "hard" nodes, the query returns typical parameters of the steel; for It is "plastic" and For nodes designated as "hard", the parameters typical for hard plastic are returned.

[0065] Edge mapping to physical constraints: mapping each edge Instantiated as a constraint connecting two rigid bodies in the physics engine. The type and parameters of the constraint are determined by the edge properties. Decision: A rigid connection instantiated as a fixed joint has a fracture threshold that is related to the connection strength coefficient. Positive correlation; flexible connections are instantiated as spring nodes or hinge joints, and their stiffness and damping parameters are related to... Relatedly, electrical connections, besides being instantiated as a very low-intensity connection constraint, are primarily used as a logical marker for subsequent risk assessment.

[0066] Step 4.2 Parsing and Applying Interactive Operation Instructions: This sub-step aims to build an intuitive and operable human-computer interaction interface for the virtual physical graph model G that has been loaded into the physics engine, and to parse and map the user's high-level operation intentions into precise dynamic events that the physics engine can execute.

[0067] Step 4.2.1 Visualization of the interactive scene: The system runs a 3D graphics rendering engine (such as the Unity or Unreal Engine rendering module that is compatible with the physics engine, or a renderer based on OpenGL / Vulkan). This rendering engine does not directly use the voxel field in step 2, but reconstructs and renders the 3D scene based on the virtual physical graph model G: (1) Node visualization: For each node n ∈ N in the graph model G, according to its geometric attributes (bounding box, centroid, approximate shape), a corresponding simplified geometric body (such as a cube, sphere, cylinder) is generated in the 3D space for representation. The surface color or texture of the geometric body can be based on its dominant semantic category. Perform mapping, for example Nodes labeled "metal" are displayed in gray, while nodes labeled "liquid" are displayed in translucent blue. (2) Scene assembly: All the visual geometry of the nodes are arranged according to their spatial position (centroid) to assemble a complete three-dimensional appearance model of the luggage. The resulting three-dimensional visualization scene is a graphical representation of G, which maintains strict spatial alignment and logical correspondence with the rigid body model inside the physics engine, providing users with an intuitive operation view.

[0068] Step 4.2.2 Definition and structured input of operation instructions: Users (security inspectors) or upper-level decision-making modules interact within the 3D visualization scene, inputting the intended inspection operation. The system parses these interactions into structured operation instructions containing the following core elements: Operation type: Select from a predefined set of operations. In this embodiment, these include: cutting, grabbing and moving, rotating, and pressing. Target specification: Users specify the target by clicking, selecting, or drawing a path in the visualization scene. The system maps these interactive coordinates back to the virtual physical model G, accurately locating the target node (corresponding to a physical component) or a local area on the node. The accuracy of this mapping is guaranteed by the strict alignment between the visualization scene and the physical model G.

[0069] Operational parameters: to give the operation quantified physical properties: (1) Cutting command: the physical properties of the virtual tool (such as mass, edge stiffness), cutting path (continuous trajectory in three-dimensional space), feed speed and downward pressure need to be defined; (2) Grasping and moving command: the maximum gripping force of the virtual gripper, the expected moving trajectory (displacement vector or path point sequence) and the movement speed need to be defined; Step 4.2.3 Precise Mapping of Instructions to the Physics Engine: The system transforms the parsed structured instructions into dynamic control of the corresponding rigid bodies (nodes) and constraints (edges) in the physics engine. For example, the cutting operation is represented in the physics engine as a "knife" rigid body with specific collision properties moving along the cutting path. It continuously detects collisions with luggage parts and calculates the cutting force and deformation based on the material properties of the collision points.

[0070] Step 4.3 Mechanical Simulation and Chain Effect Calculation: The physics engine advances the simulation at fixed time steps (e.g., 1 / 60 second). In each time step, the engine calculates the position, attitude, and velocity of all rigid bodies based on Newtonian mechanics and preset constraints, and handles collisions and constraint resolution. This process is automatically completed by the selected physics engine (e.g., PhysX) and is a standard technique in this field.

[0071] Critical Status Monitoring: During the simulation, the system monitors and records the following critical events that may indicate an escalation of risk to an event log list in real time: (1) Connection Break Event: Record the broken edge And the moment of fracture. (2) Component movement speed over-limit event: The movement speed of each component is calculated in real time. When the movement speed of any node exceeds the preset safe speed threshold, this event is triggered. Record the node n that exceeds the speed limit, the speed value and direction when the speed exceeds the speed limit. The safe speed threshold is a parameter that can be preset according to the safety specifications of the opening and inspection operation (e.g., 0.5 m / s). (3) High comprehensive risk base value component disturbance event: Real-time monitoring of components identified as having a high comprehensive risk base value (R avg ) nodes (e.g., R) avgThis event is triggered when the external force (such as collision force or extrusion force) acting on such a node exceeds the preset risk threshold (e.g., for components with a force greater than 0.7). Record the affected node n, the time of the event, and the force applied.

[0072] Step 4.4 Risk Prediction and Quantification Output: Based on the event log list, the system calls the risk assessment function. A quantitative assessment of this operational simulation is conducted. The risk assessment function is designed as follows:

[0073] in: The event's base risk value is a value related to the event type. Predefined constants related to events such as fracture and high-speed motion reflect the inherent danger level of such events; This is the event severity coefficient: a variable related to the specific attributes of the event. For example, for a "connection breakage event," its severity could be related to the connection strength coefficient of the broken edge. Inversely proportional ( The larger the size, the more difficult it is to break, and once it breaks, the consequences may be more severe. For "high-speed motion events," the severity can be directly proportional to the overspeed amplitude.

[0074] The event-based risk value and severity coefficient calculation rules in the risk assessment function, as well as the weight coefficients involved in the reward function in subsequent steps, are all adjustable system parameters. Their specific values ​​are determined according to the following engineering practices: During the system development phase, domain experts first predefine a set of baseline parameter values ​​based on historical security incident data and domain knowledge. Subsequently, in an offline simulation testing environment, the security performance of the system's overall decision-making behavior under different parameter values ​​is observed, and the set of parameters is iteratively fine-tuned until the system can stably output risk assessment results and operational strategies that conform to security specifications on a test set covering various typical threats. This parameter determination method, based on domain knowledge initialization and combined with simulation feedback optimization, is a standard practice in this field for implementing reliable systems.

[0075] Simulation Output: After a complete operation simulation is completed, the system outputs a structured simulation report, which includes at least: (1) the final position and orientation of all components, and a list of broken connections; (2) the total risk assessment value: that is, the output value of the risk assessment function for this operation. (3) Key event sequence: events such as connection breakage and high-speed movement recorded in chronological order; (4) Change in the visibility ratio of the target suspicious object: the percentage change in the visibility ratio of the target suspicious object node before and after the operation. The visibility ratio is calculated in the following way: in the digital twin simulation environment, starting from the preset virtual operator viewpoint, a set of sampling rays are emitted to the target suspicious object node through the ray casting algorithm; through geometric collision detection, the number of rays that hit the node and are not blocked by other component models on the path is counted, and the proportion of them to the total number of emitted rays is calculated as the visibility ratio. This is a standard and mature technical means for calculating "visibility" or "exposure" in the field of computer vision and graphics.

[0076] Through this step, the following technical effects are achieved: (1) The leap from static analysis to dynamic prediction enables the forward-looking assessment of operational consequences, filling the gap in existing technology; (2) It provides quantifiable risk assessment, transforming vague empirical judgments into specific values, and supporting scientific decision comparison; (3) It constructs a high-fidelity interactive environment with the physical world, providing a safe, inexpensive and efficient "digital test field" for the subsequent training and verification of AI agents.

[0077] Step 5. Intelligent strategy generation: Based on reinforcement learning algorithm, train the agent to explore and learn in the digital twin simulation environment; apply the trained agent to the digital twin simulation environment to generate an opening and inspection strategy that meets risk constraints and is most efficient according to the initial simulation state configured by the virtual physical graph model. It should be noted that the purpose of this step is to deploy a deep reinforcement learning agent, allowing the agent to learn autonomously in this environment through trial and error, and ultimately automatically generate a set of opening and inspection operation sequences that balance "target exposure," "operational safety," and "operational efficiency." This solves the fundamental shortcomings of traditional methods, which rely on manual rules, cannot handle complex scenarios, and are difficult to quantify and balance multiple objectives.

[0078] The core of the intelligent policy generation is a deep reinforcement learning framework based on the Proximal Policy Optimization (PPO) algorithm. This framework models the open / close decision problem as a Markov decision process and performs training and decision-making in the simulation environment provided in step 4. The specific implementation is as follows: Step 5.1 Modeling the Strong Chemistry Problem: State space: The state s observed by the agent at time t. t It is a high-dimensional feature representation that integrates the current dynamics of the virtual physical graph model G. Specifically, it may include: the position and velocity of all component nodes, the state of all connecting edges, the current visible area of ​​the target or suspicious object, and historical information from the previous operation.

[0079] Action space: The action a that an agent can execute at time t. t The executable operations are defined. In this embodiment, the action space is designed as a discrete-continuous hybrid space: (1) Discrete actions: select the operation type, such as {cut, grab and move, press, terminate}; (2) Continuous parameters: if the action is "cut" or "grab and move", then a continuous parameter vector is required, such as the starting / ending coordinates of the cutting path, the direction vector of the grab and move, the magnitude of the force, etc. These parameters are directly output by the policy network.

[0080] State transition: The agent performs action a t Afterwards, the environment (i.e., the simulation system in step 4) will advance the simulation, calculate the new physical state, and thus transition to the next state s. t+1 This process is entirely determined by the physics engine and simulation logic in step 4.

[0081] Step 5.2 Reward Function Design: Reward Function r t It guides the agent's learning direction; after each state transition, the environment provides the agent with an immediate reward r. t The reward function of this invention integrates multiple objectives: Target exposure reward r expose This is proportional to the increase in the visibility ratio of the target suspicious node. The calculation method for the visibility ratio is the same as defined in step 4.4. An increase in this ratio is given a positive reward, encouraging the agent to search for exposed targets.

[0082] Risk penalty r risk The risk assessment value of Roperation calculated in step 4.4 is directly proportional to the negative value (i.e., r) risk =-λ risk * R operation High-risk operations result in large negative rewards, forcing agents to avoid dangerous operations.

[0083] Step efficiency penalty r step A small, constant negative reward (e.g., -0.1). This is deducted for each step performed, encouraging the agent to complete tasks in fewer steps and improve efficiency.

[0084] Task completion reward r done When the target exposure reaches a preset threshold (e.g., 95%) or the agent actively chooses to "terminate", a large positive reward is given, indicating that the task has been successfully completed.

[0085] Dangerous termination penalty r failIf an unacceptable high-risk event is triggered during the simulation (such as the detonation of a component marked as an explosive), a large negative reward is given and the current training round is terminated, indicating that the mission has failed.

[0086] In summary, the single-step reward function can be designed as follows: And add r at a specific time done or r fail Weighting coefficients , , Used to adjust the priority of different objectives.

[0087] Step 5.3 Deep Reinforcement Learning Agent Training: The agent comprises a policy network and a value network. The policy network receives state s. t It outputs the probability distribution of actions (discrete actions) and specific parameters (continuous parameters). The value network is used to evaluate the quality of the current state and assist in policy updates.

[0088] Based on the state space, action space, and reward function defined in step 5.1, a standardized agent-environment interaction framework is constructed. Within this framework, the agent interacts with the environment through a policy network based on the current state s. t Generate action a t The environment (simulation system) performs this action and transitions to the new state s. t+1 At the same time, an instant reward r is given. t Thus forming (s t a t r t s t+1 (Empirical data).

[0089] Training Method: The agent is trained using a proximal policy optimization algorithm. This algorithm ensures stable convergence during training by limiting the magnitude of updates between the old and new policies. The goal of training is to optimize the parameters of the policy network, enabling the agent to learn to select action sequences that maximize long-term cumulative rewards through extensive interactions with the environment.

[0090] Training Process: In the offline phase, the agent is trained using a large number of randomly generated or historical virtual baggage scenarios. Through trial and error, the agent continuously adjusts its policy network, ultimately learning how to generate safe and efficient opening and inspection operation sequences when faced with various complex baggage structures.

[0091] Step 5.4: Generation, verification, and output of the optimal strategy: Policy Generation: For a new, unknown baggage (steps 1-3 already completed), the initial state is input into the trained policy network. The policy network outputs the current optimal action, which the system executes (step 4 simulation is called). After the environment updates the state, it is input into the network again. This process is repeated iteratively until the policy network outputs a "termination" action or the exposure target is reached.

[0092] Policy verification: Input the complete operation sequence generated by the policy network into the digital twin simulation environment of step 4 for independent, full-process simulation verification.

[0093] Report Generation: Based on the deduction results of strategy verification, the system generates a structured report on the optimal opening and inspection strategy. This report includes at least: (1) Operation sequence: each specific operation instruction (type, target, parameter) arranged in sequence.

[0094] (2) Expected performance indicators: The total time (cumulative simulation step size), final exposure (the percentage of the target's visible area at the end of the simulation), and cumulative risk value (R value at each step during the simulation) calculated through simulation verification. operation (The sum of all additions).

[0095] (3) Risk warning: Identify the key steps in the operation sequence that have high risk values ​​calculated by simulation.

[0096] This step achieves the following technical effects: it eliminates reliance on predefined rule bases and human experience, enabling it to handle unprecedented baggage internal structures and threat configurations; through a carefully designed reward function, the generated strategy achieves an algorithmically optimal balance between risk, efficiency, and effectiveness, which is difficult to achieve with manual planning; and the reinforcement learning framework allows for continuous fine-tuning and optimization of the policy network as more data and cases are input, making the system increasingly "intelligent".

[0097] Step 6. AR-guided execution: Transform the optimal strategy into visual instructions for the augmented reality device; display safe operation paths and high-risk area warnings in real time.

[0098] It should be noted that the purpose of this step is to safely, intuitively, and unambiguously guide the "optimal opening and inspection strategy" generated in step 5, which exists in the digital twin simulation environment, into actual operations in the physical world. It solves the key problem of "how to implement virtual strategies" by overlaying virtual guidance information onto the security inspector's real field of vision using augmented reality (AR) technology, achieving "what you see is what you are guided," greatly reducing the risk of misoperation and improving the accuracy of strategy execution and the efficiency of human-machine collaboration.

[0099] The core of the AR-guided execution is a virtual-real spatial registration and information overlay system. Its implementation includes the following key steps: 1. Spatial registration and coordinate system alignment: Registration objective: Establish a strict spatial coordinate correspondence between physical luggage, AR devices (such as AR glasses), and the digital twin simulation model. This is the foundation for all guidance information to be accurately "projected" to the correct location.

[0100] Implementation Method: A marker-based visual SLAM (Simultaneous Localization and Mapping) technique is employed. Pre-designed visual markers (such as ArUco codes) are placed on the inspection workbench. When the AR device is activated, its camera recognizes the marker, thereby determining its relative pose (camera coordinate system) to the workbench (world coordinate system). Simultaneously, the position of the luggage in the digital twin model is known (fixed to the workbench coordinate system). Through a one-time system calibration, spatial alignment among the "physical world, AR device, and virtual model" can be achieved.

[0101] 2. Strategy instruction parsing and 3D guided graphics generation: Strategy instruction parsing: Receive the structured optimal opening and inspection strategy report output from step 5 above, and parse the operation sequence (type, target, operation parameters) and risk warning information in it.

[0102] 3D Guided Graphics Generation: Based on the precise parameters analyzed above, corresponding 3D graphic models are dynamically created in the 3D space corresponding to the digital twin simulation environment. These models are collectively referred to as 3D guided graphics. The specific generation rules are as follows: (1) Cutting Path Guide Line: For the "cutting" operation, a 3D line model with specific visual effects (such as bright green, semi-transparent, and continuously glowing) is generated between the starting and ending points specified in the instruction. The geometry of the line strictly follows the path planning in the instruction. (2) Operation Target Highlighted Box: For the target component of the "grab and move" or "press" operation, a highlighted 3D border model is generated outside the 3D model (derived from the node geometry of the virtual physical graph model G) corresponding to the component. (3) Risk Area Warning Body: For the "high-risk area" marked in the strategy report, a semi-transparent, pulse-flickering red 3D cube box is generated at the corresponding spatial position as a warning. (4) Auxiliary Indicator Elements: 3D arrows indicating the direction of movement or text labels marking the operation step number can be generated simultaneously as auxiliary information.

[0103] Output: All generated 3D guide graphics, along with their precise world coordinate system position, pose, color, animation attributes, and other information, are packaged into an AR rendering data package. This data package is the direct data source driving AR devices for visualization overlay.

[0104] 3. Real-time guidance and interactive feedback: Information overlay: The AR device's display system overlays and renders the generated 3D guide graphics onto the real baggage image seen by the security inspector through the lens in real time and stably, based on the spatial registration results.

[0105] Operation confirmation: The system can integrate simple interactions (such as gesture recognition, voice commands or buttons) to allow security personnel to manually confirm after completing one step, and the system will then switch to the next step of guidance.

[0106] Abnormal alert: If the security inspector's actual operation deviates significantly from the guidance path (judged by real-time analysis of AR camera footage) or triggers a preset prohibited action, the system can issue an immediate warning through visual flashing, sound prompts, or device vibration.

[0107] This step achieves the following technical effects: intuitive spatial graphic guidance eliminates potential misunderstandings that may arise when conveying operational instructions based on text or verbal descriptions, enabling precise execution of complex operations. Continuous visual warnings of high-risk areas provide operators with proactive risk avoidance prompts, forming a double layer of protection. AR guidance delivers information in a non-intrusive manner, without interfering with the operator's hands or normal field of vision, conforming to ergonomics and improving operational fluency and acceptability. Here, the visual marker-based SLAM registration, 3D graphics rendering, and overlay are all mature technologies in the AR field. The innovation of this step lies in combining a strategy sequence that has undergone deep simulation and optimization verification with high-precision AR spatial registration technology, providing a standardized and visualized safety operation guidance system for high-risk opening and inspection operations.

[0108] Example 2: Please see Figure 2 This is a schematic diagram of the architecture of the intelligent generation system for security inspection and opening strategies based on multimodal perception and deep reinforcement learning provided in the embodiments of this application.

[0109] This invention provides an intelligent generation system for security check and opening strategies based on multimodal perception and deep reinforcement learning. This system executes the intelligent generation method for security check and opening strategies based on multimodal perception and deep reinforcement learning described in the above embodiments. The system includes a multimodal perception data acquisition and structured model, a three-dimensional voxel field construction module, a virtual physical modeling module, a strategy simulation and deduction module, a strategy intelligent generation module, and an AR-guided execution module. Multimodal sensing data acquisition and structuring module: used to synchronously acquire multimodal sensing data of the inspected luggage, including X-ray transmission images, infrared thermal imaging, and millimeter-wave radar data; the multimodal sensing data is time-stamp aligned, spatially registered, and feature fused to construct a three-dimensional spatial voxel grid, where each voxel is associated with a feature vector, which is composed of fused physical feature values ​​from different modalities, including at least equivalent atomic number, temperature value, and backscattering intensity value; The three-dimensional voxel field construction module is used to take the multimodal structured dataset as input, decode and semantically fuse the multimodal feature vector associated with each voxel through a cross-modal attention coding network, generate semantic category probability and physical attribute vector for each voxel, and thus construct a multi-attribute three-dimensional voxel field. Virtual physical modeling module: Based on the multi-attribute three-dimensional voxel field, it uses a graph neural network to reason and model the physical components inside the luggage and their connection relationships, generating a virtual physical graph model that represents the physical components inside the luggage and their interaction relationships. Strategy simulation and deduction module: Based on the virtual physical graph model, it is used to build a digital twin simulation environment, run a virtual physical engine in the environment, respond to the opening and inspection operation command, and simulate the mechanical impact and chain risks of the opening and inspection operation corresponding to the command on the internal structure of the luggage. The strategy intelligent generation module is used to train an agent to explore and learn in the digital twin simulation environment based on reinforcement learning algorithms; and to apply the trained agent to the digital twin simulation environment to generate an opening and inspection strategy that meets risk constraints and is most efficient, based on the initial simulation state configured by the virtual physical graph model. AR-guided execution module: used to translate optimal strategies into visual instructions for augmented reality devices, displaying safe operation paths and high-risk area warnings in real time.

[0110] Example 3: See Figure 3 This is a schematic diagram of the structure of an electronic device according to this application, such as... Figure 3 As shown, the electronic device 3000 includes a memory 3100, a processor 3200, and a computer program stored in the memory 3100 and executable by the processor. When the processor 3200 executes the program code of the method steps according to the present invention, it executes each step of the method in the present invention, thereby realizing an intelligent generation method for security check opening and closing strategies based on multimodal perception and deep reinforcement learning. The intelligent generation method for security check opening and closing strategies based on multimodal perception and deep reinforcement learning has been described in detail above and will not be repeated here. Example 4: Please see the appendix Figure 4This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention, as shown below. Figure 4 As shown, a computer-readable storage medium 4000 stores program code 4100 for executing the method steps according to the present invention. When executed by a processor, the program code 4100 for executing the method steps according to the present invention is used to implement the intelligent generation method for security inspection and opening strategies based on multimodal perception and deep reinforcement learning as described above. The intelligent surveillance and audit reasoning code generation method based on knowledge graphs and templated mapping has been described in detail above, and will not be repeated here.

[0111] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for intelligently generating security check opening and closing strategies based on multimodal perception and deep reinforcement learning, characterized in that, Includes the following steps: Step 1. Synchronously collect multimodal sensing data of the baggage being inspected. The multimodal sensing data includes X-ray transmission images, infrared thermal imaging, and millimeter-wave radar data. Perform time stamp alignment, spatial registration, and feature fusion on the multimodal sensing data to construct a three-dimensional spatial voxel grid, in which each voxel is associated with a feature vector, forming a multimodal structured dataset. The eigenvector is composed of fused physical eigenvalues ​​from different modes, which include at least the equivalent atomic number, temperature value, and backscattering intensity value. Step 2. Using the multimodal structured dataset as input, decode and semantically fuse the feature vector associated with each voxel through a cross-modal attention coding network to generate semantic category probability and physical attribute vector for each voxel, thereby constructing a multi-attribute three-dimensional voxel field; Step 3. Based on the multi-attribute three-dimensional voxel field, a graph neural network is used to reason and model the physical components inside the luggage and their connection relationships, generating a virtual physical graph model that represents the physical components inside the luggage and their interaction relationships. Specifically, it includes: Based on the multi-attribute three-dimensional voxel field, independent physical components are identified through a three-dimensional connected component analysis algorithm and used as nodes in the virtual physical graph model. A graph neural network is used to infer the physical connection relationships between the nodes, predict whether there is a connection between the nodes, the type of connection, and the strength of the connection, and use it as the edge of the virtual physical graph model; The node attributes of the virtual physical graph model include the geometric attributes, semantic attributes, and physical attributes of the components, and the edge attributes include the connection type and connection strength coefficient. Step 4. Based on the virtual physical graph model, construct a digital twin simulation environment, run a virtual physics engine in this environment to respond to inspection operation commands and simulate the mechanical impact and cascading risks of the corresponding inspection operation on the internal structure of the luggage; specifically including: The nodes in the virtual physical graph model are instantiated as rigid bodies in the physics engine, and their edges are instantiated as corresponding physical constraints. In the digital twin simulation environment, a structured opening and closing operation command is received. The command includes the operation type, target and parameters, and the command is mapped to the dynamic control of the corresponding rigid body or constraint in the physics engine. Advance physical simulation to monitor and record connection breakage events, component movement speed exceeding limits events, and disturbance events where the component's comprehensive risk base value exceeds a preset threshold, and quantify the risk value of a single operation based on a preset risk assessment function; Step 5. Based on the reinforcement learning algorithm, train the agent to explore and learn in the digital twin simulation environment; apply the trained agent to the digital twin simulation environment to generate an opening and inspection strategy that satisfies risk constraints and is most efficient according to the initial simulation state configured by the virtual physical graph model. Step 6. Transform the optimal strategy into visual instructions for the augmented reality device; display safe operating paths and high-risk area warnings in real time.

2. The method according to claim 1, characterized in that, In step 1, the multimodal sensing data undergoes timestamp alignment, spatial registration, and feature fusion, specifically including: The central timing controller sends synchronous trigger pulses to the acquisition circuits that acquire X-ray transmission images, infrared thermal imaging, and millimeter-wave radar data to achieve timestamp alignment. Based on the multimodal joint calibration body, all sensor data are unified under the same world coordinate system to achieve spatial registration; The registered data is sampled onto a uniform 3D voxel grid, and a feature vector is constructed for each voxel.

3. The method according to claim 1, characterized in that, In step 2, the decoding and semantic fusion of the feature vector associated with each voxel using a cross-modal attention encoding network specifically includes: The multimodal structured dataset is input into an encoder-decoder architecture that combines a three-dimensional convolutional neural network with a cross-modal attention module; In the middle and higher levels of the encoder, the weights between features from different modalities are dynamically calculated through the cross-modal attention module to achieve adaptive feature fusion; the middle and higher levels refer to the middle or later stages of feature extraction in the encoder. The decoder outputs two parallel output heads at the end, which output a semantic category probability vector and a physical property vector containing standardized density, thermal conductivity, initial risk base value and mechanical stiffness identifier for each voxel; wherein the initial risk base value is a scalar preset according to the semantic category and abnormal thermal features.

4. The method according to claim 1, characterized in that, In step 5, the trained agent explores and learns in the digital twin simulation environment, including: The inspection process is modeled as a Markov decision process, defining the state space, action space, and reward function. Based on the defined state space, action space, and reward function, an interaction framework between the agent and the digital twin simulation environment is constructed. The reward function is a multi-objective function, and its output value includes a first component that is positively correlated with the increase in the visibility ratio of the target suspicious object, a second component that is positively correlated with the negative value of the operational risk assessment value, and a third component that is negatively correlated with the number of operational steps. A proximal policy optimization algorithm is used to enable the agent to explore the environment through trial and error based on the interaction framework, with the goal of maximizing long-term cumulative rewards. The agent is trained by including a policy network and a value network.

5. The method according to claim 1, characterized in that, Step 6 specifically includes: Based on visual tags, spatial registration is performed on physical luggage, augmented reality devices, and the digital twin simulation environment; The optimal opening and inspection strategy is analyzed to generate a corresponding three-dimensional guiding graphic, including a cutting path guide line, a highlight box of the operation target, and a risk area warning body; The three-dimensional guiding graphic is superimposed and rendered onto the display screen of the augmented reality device in real time based on the spatial registration result.

6. A security inspection opening strategy intelligent generation system based on multimodal perception and deep reinforcement learning, characterized in that, The system executes the intelligent generation method for security check and opening strategies based on multimodal perception and deep reinforcement learning as described in claim 1. The system includes a multimodal perception data acquisition and structuring module, a three-dimensional voxel field construction module, a virtual physical modeling module, a strategy simulation and deduction module, a strategy intelligent generation module, and an AR-guided execution module; wherein... Multimodal sensing data acquisition and structuring module: used to synchronously acquire multimodal sensing data of the inspected luggage, including X-ray transmission images, infrared thermal imaging, and millimeter-wave radar data; the multimodal sensing data is time-stamp aligned, spatially registered, and feature fused to construct a three-dimensional spatial voxel grid, where each voxel is associated with a feature vector, which is composed of fused physical feature values ​​from different modalities, including at least equivalent atomic number, temperature value, and backscattering intensity value; The three-dimensional voxel field construction module is used to take the multimodal structured dataset as input, decode and semantically fuse the multimodal feature vector associated with each voxel through a cross-modal attention coding network, generate semantic category probability and physical attribute vector for each voxel, and thus construct a multi-attribute three-dimensional voxel field. Virtual physical modeling module: Based on the multi-attribute three-dimensional voxel field, it uses a graph neural network to reason and model the physical components inside the luggage and their connection relationships, generating a virtual physical graph model that represents the physical components inside the luggage and their interaction relationships. Strategy simulation and deduction module: Based on the virtual physical graph model, it is used to build a digital twin simulation environment, run a virtual physical engine in the environment, respond to the opening and inspection operation command, and simulate the mechanical impact and chain risks of the opening and inspection operation corresponding to the command on the internal structure of the luggage. The strategy intelligent generation module is used to train an agent to explore and learn in the digital twin simulation environment based on reinforcement learning algorithms; and to apply the trained agent to the digital twin simulation environment to generate an opening and inspection strategy that meets risk constraints and is most efficient, based on the initial simulation state configured by the virtual physical graph model. AR-guided execution module: used to translate optimal strategies into visual instructions for augmented reality devices, displaying safe operation paths and high-risk area warnings in real time.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the processor of the electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 5.

Images