A deep learning network construction method fusing attention mechanism and multi-modal data

Abstract

The application relates to the technical field of artificial intelligence and industrial data processing, and discloses a deep learning network construction method fusing an attention mechanism and multi-modal data, which comprises the following steps: acquiring first and second modal perception data and extracting transient electromechanical element data capturing an instant; combining learnable physical calibration parameters to construct a globally differentiable physical space offset vector; mapping the vector into a relative pixel offset vector through dimension reduction, generating a dynamic sparse feature index set based on a mechanical tolerance limit; generating an attention weight containing a space distance penalty term based on the same, performing a dot product operation on a feature matrix to output a cross-modal fusion result; calculating a target loss function, and jointly updating deep learning network parameters and learnable physical calibration parameters through back propagation. The technical scheme of synchronously extracting multi-modal sensor transient electromechanical element data and constructing a differentiable physical space offset vector is adopted, so that the technical effect of directly compensating dynamic mechanical displacement at a model bottom layer is achieved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and industrial data processing technology, specifically to a method for constructing a deep learning network that integrates attention mechanisms and multimodal data. Background Technology

[0002] Industrial automated production demands increasingly stringent quality control. Single-dimensional sensors are often insufficient to meet complex inspection requirements. In production lines, visible light cameras are needed to extract the surface morphology features of workpieces, while infrared thermal imagers are used to determine their internal thermodynamic states. The physical quantities represented by data acquired from different types of sensors vary significantly. Systems must integrate multi-source heterogeneous data for comprehensive judgment. Multimodal data fusion technology has become the foundation of intelligent industrial inspection.

[0003] Existing multimodal fusion solutions primarily employ pure software-level image registration algorithms and deep learning global attention networks. In the data preprocessing stage, the system relies on feature point extraction or template matching algorithms to achieve cross-modal image alignment. This pure image processing approach requires no modification to the underlying electromechanical control links, simplifying deployment. Moving to the network fusion stage, mainstream methods utilize a global cross-attention computation framework to establish global semantic associations across spatial distances and autonomously mine the macroscopic dependencies of multimodal features. To ensure spatial correspondence, engineers perform offline geometric calibration using a standard calibration board before the equipment goes live to obtain the relative position parameters of the initial state of the device.

[0004] However, a disconnect exists between the underlying physical state and digital algorithms in high-speed industrial environments. There is a time lag between the hardware triggering and photoelectric conversion of multimodal sensors. Within this extremely short timeframe, the high-speed vibration of the conveyor belt and equipment base causes real physical displacement of the workpiece. Pure visual registration algorithms cannot capture the real mechanical displacement data caused by traction wheel slippage or support vibration. Algorithms that force alignment based solely on image content directly cause misalignment in feature fusion at the source. At the feature calculation level, the global attention mechanism requires traversing and performing dot product operations on all nodes of the feature map. Local features are forcibly associated with the background of the distant production line. This traversal calculation not only consumes a large amount of computing power, but the mixing of distant background features also interferes with the expression of effective network features. Furthermore, the long-term continuous operation of industrial equipment inevitably leads to loosening of transmission belts and wear of mechanical parts. Mechanical degradation causes continuous spatial drift. The initial offline calibration parameters are static and fixed. Traditional neural networks only update numerical weights, and the theoretical calibration position gradually deviates from the actual physical position over time, ultimately causing the detection accuracy of the model to continuously decline after a period of production. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for constructing deep learning networks that integrates attention mechanisms and multimodal data. This method solves the technical problems in existing technologies, such as the inability of pure visual registration to perceive feature source misalignment caused by transient mechanical displacement, the waste of computing power and interference from distant backgrounds caused by global attention traversal calculations, and the inability of static offline calibration to adapt to the mechanical degradation caused by long-term operation of equipment, which ultimately leads to a continuous decline in model detection accuracy.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for constructing a deep learning network that integrates attention mechanisms and multimodal data, comprising the following steps:

[0007] Simultaneously acquire first modal sensing data and second modal sensing data, and extract transient electromechanical metadata at the moment of capture of the first modal sensing data and the second modal sensing data;

[0008] Based on the transient electromechanical data and the preset learnable physical calibration parameters, a globally differentiable physical space offset vector is constructed to characterize the spatial coordinate deviation.

[0009] The parameters of the deep learning network are obtained, the global differentiable physical space offset vector is reduced in dimension and mapped to the feature level, the relative pixel offset vector is determined, and a dynamic sparse feature index set is generated based on the relative pixel offset vector and the preset mechanical tolerance limit.

[0010] Based on the dynamic sparse feature index set, attention weights containing spatial distance penalty terms are generated in the corresponding local space to perform dot product operations on the feature matrices corresponding to the first modality perception data and the second modality perception data, and output cross-modality feature fusion results.

[0011] The target loss function is calculated based on the cross-modal feature fusion result, and the backpropagation algorithm is executed to jointly update the deep learning network parameters and the learnable physical calibration parameters.

[0012] Preferably, the step of simultaneously acquiring the first modal sensing data and the second modal sensing data, and extracting transient electromechanical data at the moment of acquisition of the first modal sensing data and the second modal sensing data, includes:

[0013] Extract the first timestamp of the first modal sensing data and the second timestamp of the second modal sensing data, and generate the time deviation between the first timestamp and the second timestamp through difference calculation;

[0014] Read encoder data from the electromechanical control bus to determine transient scalar linear velocity and extract the preset motion direction unit vector;

[0015] The three-axis analog signal output by the accelerometer at the instant of capturing the first modal sensing data and the second modal sensing data is acquired, and a transient acceleration vector is generated;

[0016] The time deviation, the transient scalar linear velocity, the unit vector of the direction of motion, and the transient acceleration vector are combined and encapsulated into the transient electromechanical data.

[0017] Preferably, the learnable physical calibration parameters include a velocity proportional drift coefficient and a vibration phase delay coefficient; the step of constructing a globally differentiable physical space offset vector to characterize spatial coordinate deviation based on the transient electromechanical data and the preset learnable physical calibration parameters includes:

[0018] The macroscopic translational motion offset compensation amount is determined by multiplying the transient scalar linear velocity, the time deviation, the unit vector of the motion direction, and the velocity proportional drift coefficient.

[0019] Multiply the transient acceleration vector by the vibration phase delay coefficient to determine the micro-vibration offset compensation amount;

[0020] The macroscopic translational offset compensation amount and the microscopic vibration offset compensation amount are algebraically summed to generate the global differentiable physical space offset vector.

[0021] Preferably, the step of obtaining deep learning network parameters, dimensionality-reducing and mapping the globally differentiable physical space offset vector to the feature level, and determining the relative pixel offset vector includes:

[0022] Extract the cumulative spatial downsampling step size of the deep learning network, and combine it with the preset original sensor physical resolution to determine the physical real side length of a single pixel in the feature map;

[0023] The global differentiable physical space offset vector is reduced in dimension and projected into a two-dimensional physical plane offset vector;

[0024] The relative pixel offset vector is generated by dividing the two-dimensional physical plane offset vector by the physical reality side length.

[0025] Preferably, the step of generating a dynamic sparse feature index set based on the relative pixel offset vector and a preset mechanical tolerance limit includes:

[0026] Divide the mechanical tolerance limit by the physical real side length to determine the pixel tolerance radius;

[0027] The sum of the two-dimensional spatial coordinate vector of the query node on the feature map of the deep learning network and the relative pixel offset vector is used as the expected drift center point;

[0028] Candidate key nodes whose Euclidean distance between the two-dimensional spatial coordinate vector and the expected drift center point is less than or equal to the pixel tolerance radius are selected and combined to generate the dynamic sparse feature index set.

[0029] Preferably, the step of generating attention weights containing spatial distance penalties in the corresponding local space based on the dynamic sparse feature index set includes:

[0030] Extract the feature vector of the query node as the query feature, and extract the feature vectors corresponding to all candidate key nodes in the dynamic sparse feature index set as candidate key features;

[0031] Calculate the square of the Euclidean distance between the two-dimensional spatial coordinate vectors of all the candidate key nodes and the expected drift center point;

[0032] Using a preset variance hyperparameter, a decay calculation is performed on the squared value of the Euclidean distance to generate a negative spatial distance penalty term.

[0033] Preferably, the step of performing a dot product operation on the feature matrices corresponding to the first modality sensing data and the second modality sensing data, and outputting a cross-modality feature fusion result, includes:

[0034] Perform an inner product operation on the transpose matrix of the query feature and the candidate key feature to obtain a dot product result, and then add the spatial distance penalty term to the dot product result;

[0035] Feature nodes that do not belong to the dynamic sparse feature index set are assigned a preset floating-point minimum value, and the dot product operation result after superimposing the spatial distance penalty term is normalized using an activation function to generate the attention weights.

[0036] The attention weights are weighted and summed with the feature vectors of the value matrix corresponding to the second modality perception data to generate the cross-modal feature fusion result.

[0037] Preferably, calculating the target loss function based on the cross-modal feature fusion result includes:

[0038] The cross-modal feature fusion result is input into the decoder network for feature dimensionality reduction and mapping processing, and the predicted probability distribution is output.

[0039] By comparing the predicted probability distribution with the preset real target label, the distribution divergence between the predicted probability distribution and the real target label is calculated using the cross-entropy loss function, and the target loss function is constructed.

[0040] Preferably, the execution of the backpropagation algorithm to jointly update the deep learning network parameters and the learnable physical calibration parameters includes:

[0041] The automatic differentiation mechanism of the network computing framework is triggered, so that the gradient of the target loss function is backpropagated along the computation graph to the parameters of the deep learning network.

[0042] The gradient of the target loss function is transmitted in reverse through the calculation path of the spatial distance penalty term and the relative pixel offset vector according to the chain rule, and then to the learnable physical calibration parameters.

[0043] Perform gradient descent calculations and simultaneously update the values ​​of the deep learning network parameters and the learnable physical calibration parameters.

[0044] Preferably, the step of synchronously updating the values ​​of the deep learning network parameters and the learnable physical calibration parameters includes:

[0045] Assign a base learning rate to the parameters of the deep learning network, and calculate the first parameter update amount based on the base learning rate and the gradient value corresponding to the parameters of the deep learning network.

[0046] An independent learning rate is assigned to the learnable physical calibration parameter, and a second parameter update amount is calculated based on the gradient value corresponding to the independent learning rate and the learnable physical calibration parameter, while limiting the value of the independent learning rate to be less than the value of the base learning rate;

[0047] Based on the first parameter update amount and the second parameter update amount, the numerical updates of the deep learning network parameters and the learnable physical calibration parameters are completed synchronously.

[0048] Preferably, after filtering out candidate key nodes whose Euclidean distance between the two-dimensional spatial coordinate vector and the expected drift center point is less than or equal to the pixel tolerance radius and combining them to generate the dynamic sparse feature index set, the method further includes:

[0049] Determine whether the dynamic sparse feature index set is an empty set;

[0050] When the dynamic sparse feature index set is empty, the boundary range of the feature map of the deep learning network is obtained, and the legal feature node index with the smallest Euclidean distance from the expected drift center point is extracted. The extracted legal feature node index is then filled into the dynamic sparse feature index set.

[0051] Preferably, the preset learnable physical calibration parameters are configured as leaf nodes of the network computation graph during the deep learning network initialization phase, and the automatic differentiation attribute is activated.

[0052] The initial value of the velocity proportional drift coefficient is set to a constant of 1.0, and the initial value of the vibration phase delay coefficient is set to a non-zero constant based on the preset main resonant frequency of the equipment.

[0053] Preferably, the step of inputting the cross-modal feature fusion result into the decoder network for feature dimensionality reduction and mapping processing, and outputting a predicted probability distribution, includes:

[0054] The cross-modal feature fusion result is reduced to a one-dimensional global feature vector using a global average pooling layer;

[0055] The one-dimensional global feature vector is input into a multilayer perceptron and mapped to generate a logical value array that is consistent with the number of preset target categories.

[0056] The normalized exponential function is used to activate the logical value array, and the predicted probability distribution is output.

[0057] This invention provides a method for constructing deep learning networks that integrates attention mechanisms and multimodal data. It offers the following advantages:

[0058] 1. This invention employs a technical solution of synchronously extracting transient electromechanical data from multimodal sensors and constructing a differentiable physical space offset vector, achieving the technical effect of directly compensating for dynamic mechanical displacement at the model's underlying level. Compared to existing technologies that rely solely on pure image pixel feature matching for registration, this invention overcomes the shortcomings of overcoming the spatiotemporal misalignment of hardware caused by high-frequency mechanical vibration and conveyor belt slippage.

[0059] 2. This invention employs a technical solution that generates a dynamic sparse feature index set and combines it with a spatial distance penalty term to restrict dot product operations. This achieves the technical effect of accurately filtering and fusing effective features within a local physical neighborhood. Compared to the existing technology that forces global attention traversal matching of all nodes in the graph, this invention overcomes the shortcomings of its serious waste of computational power and susceptibility to being misled by irrelevant distant background noise.

[0060] 3. This invention employs a technical solution that constructs a computational graph containing learnable physical calibration parameters and performs backpropagation iteration in conjunction with network parameters. This achieves the technical effect of online correction of underlying hardware errors driven by business data. Compared to existing technologies that rely on test reference components for manual offline calibration, this invention overcomes the inability to cope with the dynamic cumulative wear and tear caused by long-term operation of industrial field equipment. Attached Figure Description

[0061] Figure 1 This is a system architecture diagram of an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0063] Figure 3 This is a flowchart illustrating the synchronous acquisition process of sensing data and electromechanical metadata according to an embodiment of the present invention.

[0064] Figure 4 This is a schematic diagram illustrating the physical offset field construction principle of an embodiment of the present invention.

[0065] Figure 5 This is a schematic diagram illustrating the principle of feature receptive field mapping and sparse index generation in an embodiment of the present invention.

[0066] Figure 6 This is a schematic diagram illustrating the principle of cross-modal constrained attention forward propagation in an embodiment of the present invention.

[0067] Figure 7 This is a schematic diagram illustrating the end-to-end closed-loop reverse optimization principle of an embodiment of the present invention.

[0068] Figure 8 This is a scatter plot showing the spatial offset distribution of cross-modal feature nodes according to an embodiment of the present invention.

[0069] Figure 9 This is a three-dimensional surface diagram of the dynamic spatial constraint attention weight in an embodiment of the present invention.

[0070] Among them, 10 is the synchronous data acquisition module; 20 is the physical offset field construction module; 30 is the sparse index generation module; 40 is the constrained attention fusion module; and 50 is the closed-loop reverse optimization module. Detailed Implementation

[0071] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Reference Figure 1 The present invention provides a deep learning network construction system that integrates attention mechanism and multimodal data, which may include: a synchronous data acquisition module 10; a physical offset field construction module 20; a sparse index generation module 30; a constrained attention fusion module 40; and a closed-loop reverse optimization module 50.

[0073] The synchronous data acquisition module 10 is communicatively connected to the multimodal sensor group and electromechanical control bus in the industrial field. The synchronous data acquisition module 10 extracts the heterogeneous sensing data stream of the physical object and extracts the transient electromechanical metadata of the underlying hardware within the sensor trigger clock cycle.

[0074] The input of the physical offset field construction module 20 is connected to the synchronous data acquisition module 10. The physical offset field construction module 20 internally contains learnable physical calibration parameters independent of the deep learning network parameters. Based on classical kinematic models and input data, the physical offset field construction module 20 calculates the spatial coordinate offset relationship between heterogeneous sensing data streams.

[0075] The input of the sparse index generation module 30 is connected to the physical offset field construction module 20. The sparse index generation module 30 obtains the parameters of the deep learning network and, in conjunction with the spatial coordinate offset relationship output by the physical offset field construction module 20, determines the local interaction space range in the feature mapping process of each layer.

[0076] The constrained attention fusion module 40 is deployed inside the deep learning network. Based on the local interaction space range determined by the sparse index generation module 30, the constrained attention fusion module 40 performs dot product operations and normalization processing of the multimodal feature matrix within a restricted spatial neighborhood.

[0077] The closed-loop backward optimization module 50 acquires the forward propagation results output by the constrained attention fusion module 40. After calculating the target error, the closed-loop backward optimization module 50 performs gradient propagation calculations. The parameter update path of the closed-loop backward optimization module 50 simultaneously points to the deep learning network parameters and the learnable physical calibration parameters within the physical offset field construction module 20.

[0078] Reference Figure 2 This invention provides a method for constructing a deep learning network that integrates attention mechanisms and multimodal data, comprising the following steps:

[0079] S100, synchronously acquires the sensing data of the first mode sensor and the second mode sensor, and extracts the transient electromechanical metadata of the underlying hardware at the moment of sensing data capture;

[0080] S200: Based on transient electromechanical metadata and preset learnable physical calibration parameters, establish a global differentiable physical space offset vector of the first mode relative to the second mode;

[0081] S300, based on the physical receptive field size of the feature layer of the deep learning network, reduces the dimension of the global differentiable physical space offset vector to the pixel level and generates a dynamic sparse feature index set.

[0082] S400 generates attention weights containing spatial distance penalty terms in the corresponding local space based on the dynamic sparse feature index set, and outputs cross-modal feature fusion results.

[0083] S500 calculates the target loss function based on the cross-modal feature fusion results and executes the backpropagation algorithm to jointly update the deep learning network parameters and learnable physical calibration parameters.

[0084] The above method steps will be described in detail below with reference to specific implementation methods.

[0085] Reference Figure 3 Regarding step S100 above, this embodiment of the invention performs specific data extraction and state encapsulation operations through the synchronous data acquisition module 10. This step may specifically include the following sub-steps:

[0086] S110, the synchronous data acquisition module 10 triggers the multimodal sensor group to perform data acquisition through the industrial clock synchronization protocol and calculates the time deviation between the sensed data.

[0087] In this embodiment, as a preferred approach, the acquisition of multimodal sensing data and underlying electromechanical physical parameters is completed collaboratively within the same hardware clock domain. Typically, the root cause of misalignment in cross-modal image spatial features lies not only in the static mechanical tolerances of the sensor installation, but also in the dynamic spatial drift caused by the movement of the detected material along the conveyor belt and environmental vibrations. Therefore, this solution focuses on simultaneously capturing the underlying state parameters that cause this physical displacement while acquiring image content.

[0088] Specifically, the synchronous data acquisition module 10 establishes communication connections with the first and second modal sensors in the industrial field, and sends synchronization trigger commands to the relevant sensors based on the industrial clock synchronization protocol, so that each sensor outputs its corresponding sensing data stream. For the underlying clock node alignment mechanism of the industrial clock synchronization protocol, those skilled in the art can refer to existing industrial Ethernet communication standards for configuration and implementation. Its timestamp distribution principle is a well-known technology in the field and will not be elaborated here.

[0089] Due to the differences in photoelectric conversion mechanisms within heterogeneous sensors and the response fluctuations of the bus transmission link, multimodal data inevitably exhibits millisecond-level time asynchrony when arriving at the synchronous data acquisition module 10. The synchronous data acquisition module 10 extracts the timestamp information carried in the header of each sensing data frame. Let the acquisition timestamp of the first modal sensing data be... The capture timestamp of the second modality sensing data is The synchronous data acquisition module 10 performs a difference calculation based on the extracted timestamp values ​​to generate a time deviation. The formula for calculating time deviation is:

[0090] ;

[0091] in, It is a floating-point scalar with positive and negative attributes, and its basic dimension is set to milliseconds. It is used to characterize the time interval between the first mode sensor and the second mode sensor at the moment of actual exposure sampling. This parameter is the reference variable for subsequent derivation of longitudinal motion displacement.

[0092] S120, the synchronous data acquisition module 10 reads the transient scalar linear velocity of the conveyor belt at the moment of data acquisition via the electromechanical control bus.

[0093] Generation time deviation Simultaneously, the synchronous data acquisition module 10 listens to and parses the real-time operation messages in the underlying electromechanical control bus. The synchronous data acquisition module 10 extracts the absolute encoder values ​​of the servo motor controlling the conveyor belt, and, combined with the pre-calibrated mechanical reduction ratio and traction wheel circumference in the mechanical drawings, calculates the transient scalar linear velocity corresponding to the instant of data capture. Its unit of measurement is uniformly defined as millimeters per second. In addition, the synchronous data acquisition module 10 retrieves the preset motion direction unit vector stored in the system configuration register. This is to determine the reference orientation of the pipeline motion in three-dimensional physical space. The mechanism for parsing encoder messages and the basic rules for converting rotational speed and linear velocity can be implemented by those skilled in the art based on the servo motor communication protocol manual. The data decoding calculations are well-known techniques in the field and will not be elaborated upon here.

[0094] S130, the synchronous data acquisition module 10 extracts the transient acceleration vector at the moment of capture.

[0095] In addition to the macroscopic translational motion of the conveyor belt, high-frequency mechanical vibrations in industrial settings can cause nonlinear lateral or vertical offsets between the sensor support and the target being detected within an extremely short exposure window. To capture this interference parameter, the synchronous data acquisition module 10 communicates synchronously with a triaxial accelerometer rigidly fixed to the sensor mounting base or the conveyor belt support. This is done after acquiring the aforementioned timestamp. or At the corresponding moment, the synchronous data acquisition module 10 triggers the multi-channel data acquisition card to read the analog voltage signal output by the triaxial accelerometer, and generates a transient acceleration vector after analog-to-digital conversion. Transient acceleration vector The transient acceleration measurements are contained in three orthogonal directions in a Cartesian coordinate system, with dimensions in millimeters per square second. Although acceleration and transient displacement exhibit a complex quadratic integral relationship in strict physical kinematics, this embodiment selects transient acceleration as the original state variable characterizing physical vibration. This is intended to be used in conjunction with differentiable learning parameters in a deep learning network to implicitly fit a complex mechanical response function that includes damping and rigidity transmission effects.

[0096] Finally, the synchronous data acquisition module 10 will use the time deviation obtained above. Transient scalar linear velocity , unit vector of motion direction and high-frequency amplitude vector The data is aligned and merged according to a preset format and encapsulated into a transient electromechanical metadata tensor with a unified dimensional reference. This tensor will serve as the forward data source for the forward propagation network and will be input into the physical offset field construction module 20, providing the underlying kinematic support boundary for subsequent calculations of dynamic spatial coordinate mapping.

[0097] Reference Figure 4 Regarding step S200 above, this embodiment of the invention performs specific calibration parameter initialization and spatial offset calculation operations through the physical offset field construction module 20. This step may specifically include the following sub-steps:

[0098] S210, the physical offset field construction module 20 configures and initializes learnable physical calibration parameters inside the deep learning network, and assigns computational graph attributes with inverse differentiability to the calibration parameters.

[0099] In conventional industrial data processing workflows, systems typically treat the electromechanical values ​​read by sensors as representing the ideal physical state. However, under actual operating conditions, due to mechanical wear of the conveyor belt, slippage of the traction wheel, and deformation hysteresis of the equipment support, there is often a systematic time-varying deviation between the theoretical displacement calculated based on the ideal kinematic model and the actual physical displacement. To correct this inherent error, this solution introduces a dynamic calibration mechanism at the network framework level.

[0100] In this embodiment, as a preferred approach, the physical offset field construction module 20 internally defines a set of learnable physical calibration parameters independent of the conventional convolutional kernel weights, including velocity scaling drift coefficients. and vibration phase delay coefficient .

[0101] Specifically, the velocity proportional drift coefficient It is a dimensionless scalar used to characterize the proportional deviation between the actual traction speed of the conveyor belt and the encoder message output speed. Its initial engineering value is set to a constant of 1.0, representing the assumption that the measured speed is the true speed in the initial state. Its value range is usually constrained within the interval [0.9, 1.1] to conform to the conventional mechanical slip phenomenon.

[0102] For vibration phase delay coefficient In basic mechanics, if we assume that a mechanical support is excited by a conveyor belt and undergoes periodic vibration while the system is in a steady-state response phase, its transient structural vibration displacement is inversely proportional to its instantaneous acceleration. This proportionality coefficient is inversely proportional to the square of the support structure's natural frequency. Based on this approximation, this embodiment will... Defined as a calibration parameter used for the equivalent compliance characteristics of an implicitly fitted system, its dimension is the square of a second. During equipment cold start initialization, The initial value, combined with the device's main resonant frequency (e.g., in the 10Hz to 100Hz band), is preset to be at 10. -4 Up to 10 -2 A very small non-zero constant of the order of magnitude.

[0103] At the implementation level, the physical offset field construction module 20 will use the aforementioned velocity proportional drift coefficient. With vibration phase delay coefficient Instantiated as trainable parameter objects in a deep learning framework (such as PyTorch). The system utilizes the framework's underlying interface to configure these parameters as leaf nodes in the network computation graph and activates their automatic differentiation (Autograd) mechanism. This operation establishes the mathematical connectivity of the underlying electromechanical physics formulas in the backpropagation chain of the neural network, enabling subsequent network error gradients to propagate along the computation graph to the physical calibration parameters.

[0104] S220, the physical offset field construction module 20 combines transient electromechanical metadata with learnable physical calibration parameters to calculate and construct basic mechanical displacement variables and generate a global differentiable physical space offset vector.

[0105] After completing parameter initialization, the physical offset field construction module 20 receives the time deviation from the synchronous data acquisition module 10. Transient scalar linear velocity , unit vector of motion direction and high-frequency amplitude vector The physical offset field construction module 20 is based on the physical constraints of uniform linear motion and acceleration. It combines the input deterministic measurement data with the learnable calibration parameters initialized in step S210 through algebraic operations.

[0106] The physical offset field construction module 20 calculates the global instantaneous physical space offset vector of the first modal coordinate system relative to the second modal coordinate system using the following formula. :

[0107] ;

[0108] in, It characterizes the spatial deviation of the actual mechanical position after being affected by the overall system error, and its unified dimension is millimeters. The first part on the right side of the equation (i.e. This is used to calculate the macroscopic translational motion offset compensation along the conveyor belt's direction of movement. To ensure the objective balance of the formula's dimensions, time deviation is considered. Before substituting into the formula, switch to the standard time unit of seconds. The second part on the right side of the equation (i.e. This is used to calculate the compensation for lateral and vertical micro-vibration offsets caused by high-frequency mechanical disturbances. Through the aforementioned constraints, The dimension is millimeters per square second and The dimension of corresponds to the square of a second. Through this strict dimensional consistency constraint, the mathematical structure of this differentiable kinematic equation avoids computational ambiguity arising from the direct addition of variables with different physical meanings.

[0109] Through this calculation process, the physical offset field construction module 20 outputs comprehensive tensor data that integrates prior physical features and parameters of the variables to be optimized. This physical space offset vector will serve as the core baseline data for the deep learning network to adaptively adjust the local receptive field spatial reprojection in subsequent steps.

[0110] Reference Figure 5 Regarding step S300 above, this embodiment of the invention performs adaptive scaling and spatial local window construction operations through the sparse index generation module 30. This step may specifically include the following sub-steps:

[0111] S310, the sparse index generation module 30 determines the physical real-world side length of a single pixel in the feature map based on the receptive field depth of the feature layer of the deep learning network, and converts the global physical space offset vector into a relative pixel offset vector.

[0112] When deep learning networks extract features layer by layer, the spatial resolution of their internal feature tensors decreases continuously with the processing of pooling layers or downsampling modules. This structural characteristic leads to a significant magnification of the physical area represented by a single feature node in a deep network. To ensure that the macroscopic physical biases calculated at the lower layers can be accurately applied to feature maps at different network depths, this embodiment constructs an adaptive scale mapping mechanism based on receptive field characteristics.

[0113] In this embodiment, the first modality sensing data and the second modality sensing data input to the sparse index generation module 30 are extracted by the pre-backbone network and transformed into a three-dimensional feature tensor at a specific network layer. The data structure is represented as a matrix with dimensions equal to the number of channels multiplied by the height multiplied by the width. The cross-modal fusion attention layer currently being processed is set to be located at the [number]th [level] of the network. The sparse index generation module 30 retrieves the system's preset original sensor physical resolution (i.e., the actual physical side length represented by a single pixel in the original image) and extracts the deep learning network from the input layer to the... The cumulative spatial downsampling step size parameter of the layer. The sparse index generation module 30 multiplies the original sensor physical resolution by the cumulative spatial downsampling step size to calculate the physical real-world side length of a single pixel in the theoretical receptive field on the feature map of this specific network layer. The variable The unit of measurement is uniformly set to millimeters per pixel. For the conventional derivation of the receptive field size inside deep learning networks, those skilled in the art can refer to the receptive field calculation rules of standard convolutional neural networks for configuration. The basic derivation process is a well-known technology in this field and will not be elaborated here.

[0114] Based on this, the sparse index generation module 30 utilizes the scale factor of the physical reality size to reduce the dimensionality of the continuous spatial variables output by the physical offset field construction module 20 and map them to a discrete feature space. This is because the feature tensor acquired by the sensor corresponds to a two-dimensional imaging plane, while the global physical space offset vector... Belonging to a vector in a three-dimensional spatial coordinate system, the sparse index generation module 30 first obtains the installation extrinsic parameters of the industrial camera (first mode or second mode sensor) and extracts the global physical space offset vector. The sparse index generation module 30 uses two orthogonal direction components parallel to the two-dimensional imaging plane to project the three-dimensional vector into a two-dimensional physical plane offset vector. Subsequently, the sparse index generation module 30 uses the projected two-dimensional physical plane offset vector... Divide by the physical reality side length of the current layer This transforms and generates a relative pixel offset vector specifically for the current network layer. The conversion calculation formula is as follows:

[0115] ;

[0116] In this operation, the physical offset parameter, measured in millimeters, is converted into a two-dimensional coordinate offset, measured in pixels. This relative pixel offset vector... Having a digital spatial scale that matches the network hierarchy constitutes the core benchmark for guiding the alignment of local features.

[0117] S320, the sparse index generation module 30 determines the pixel tolerance radius of the local sampling window according to the preset mechanical tolerance limit, and generates a dynamic sparse feature index set according to the distance judgment condition.

[0118] In conventional multimodal global attention mechanisms, a single node in the query matrix typically needs to perform a dot product operation with all nodes in the key matrix. This exhaustive computation method often leads to high computational costs when processing high-resolution feature maps, and distant, irrelevant background nodes can easily interfere with the accuracy of feature fusion. To address this, this embodiment introduces dynamic sparse truncation logic based on physical spatial distance constraints through the sparse index generation module 30.

[0119] As a preferred approach, the sparse index generation module 30 pre-acquires the mechanical tolerance limit data of the industrial equipment. This mechanical tolerance limit characterizes the maximum unpredictable physical displacement fluctuation amplitude that the equipment may produce under extreme operating conditions, and its dimension is millimeters. In conventional assembly line inspection scenarios, its value range is typically set from 10 mm to 50 mm depending on the guide rail clearance. The sparse index generation module 30 divides this mechanical tolerance limit by the physical actual side length dimension obtained in step S310. The result is then rounded up to determine the pixel tolerance radius used to limit the range of local attention interactions. .

[0120] For a 3D tensor structure whose feature matrix contains channels, height, and width, spatial feature slices are extracted, and the first modality feature map is set at the [missing information - likely a specific location or region]. The first layer The two-dimensional spatial coordinate vector of each query node is The second modality feature map is at the same level. The two-dimensional spatial coordinate vector of the candidate key nodes is The coordinate vectors mentioned above all use the top-left corner of the feature tensor in both width and height dimensions as the grid origin, and their values ​​are presented as discrete integer indices, with the units corresponding to pixels. The sparse index generation module 30 uses the theoretically expected relative drift position (i.e., the vector summation result) Using the anchor point as the center, a traversal and filtering operation is performed within the second modality feature space. The sparse index generation module 30 filters out feature nodes falling within the effective spatial range based on the Euclidean distance criterion, and combines them to generate a sparse index for the query node. Dynamic sparse feature index set The mathematical boundary conditions for this screening are defined as follows:

[0121] ;

[0122] in, This represents the calculation of the Euclidean distance between two coordinate vectors in a two-dimensional pixel plane rectangular coordinate system.

[0123] Considering the potential algorithmic dead zones caused by edge effects and extreme vibration interference, the sparse index generation module 30 performs an empty set check operation after each filtering step. If a dynamic sparse feature index set is determined to be outside the feature map boundary... If the set is empty, the system will forcibly activate the fallback strategy, adjusting the distance from the expected offset point. The indexes of the most recent valid nodes in the feature map are directly filled into the set to ensure the validity and computational continuity of the tensor dimensions during subsequent network forward propagation.

[0124] The dynamic sparse feature index set generated by the above operations At the algorithmic logic level, local neighborhood nodes in the second modality data that are allowed to participate in cross-modal fusion are explicitly defined. Node indices beyond this Euclidean distance boundary do not participate in subsequent cross-modal attention calculations, thereby limiting the influence of irrelevant distant nodes on the fusion result.

[0125] Reference Figure 6 Regarding step S400 above, this embodiment of the invention performs local feature pairing and physically constrained feature fusion operations through the constrained attention fusion module 40. This step may specifically include the following sub-steps:

[0126] S410, the constrained attention fusion module 40 performs feature pairing between the first modality query node and the second modality key node within the effective range according to the dynamic sparse feature index set, and constructs a specific network layer spatial distance penalty matrix based on the Gaussian decay prior.

[0127] In the first deep learning network During the layer feature interaction process, the first modality feature tensor extracted by the pre-processor (e.g., containing visual texture and shape features of the surface of an industrial inspection object) is linearly projected into a query matrix. Simultaneously, the second modality feature tensor (e.g., the corresponding infrared thermal distribution or depth contour features) is linearly projected into a key matrix and a value matrix, respectively. The channel dimension of the above feature tensors in the hidden layers is set to be uniformly set to... The constrained attention fusion module 40 targets the first modality feature map... For each query node, extract its feature vector, denoted as . Combined with the dynamic sparse feature index set generated in the previous steps The constrained attention fusion module 40 extracts the corresponding feature vector from the key matrix of the second modality, denoted as... , where node index satisfy Through this data slicing operation, the system establishes local pairing relationships at the digital level that conform to the constraints of physical rigid boundaries.

[0128] After establishing effective paired nodes, those skilled in the art will understand that even within the local tolerance window, the confidence level of a feature node containing effective alignment information should decrease as it moves further away from the expected physical drift center. To conform to this common-sense engineering physics, this embodiment introduces a spatial penalty mechanism based on Gaussian distribution characteristics to flexibly suppress edge noise. For nodes falling into the dynamic sparse feature index set... Each candidate node within The constrained attention fusion module 40 calculates the spatial distance penalty value for the specific node pair based on the Gaussian decay prior formula. Its mathematical calculation logic is defined as follows:

[0129] ;

[0130] in, This represents the spatial coordinate vector of the query node within a two-dimensional grid. Represents the spatial coordinate vector of the candidate key node. It is a relative pixel offset vector. The expected drift center point of the query node on the second modality feature map. This represents the square of the Euclidean distance between two coordinate vectors in a two-dimensional plane, with dimensions in pixels squared. This is to ensure the final generated penalty value... For dimensionless pure numerical values, the variance hyperparameter of the denominator position. The variance hyperparameter, used to characterize the tolerance and smoothness of local feature matching, is also measured in squared pixels. As a preferred approach, this variance hyperparameter... The calibration is performed based on the empirical distribution of pixel center offset in multimodal images under historical operating conditions, with a value typically configured within the range of 1.0 to 9.0 pixel squares. Setting this value too small will result in excessive suppression of effective features, while setting it too large will fail to filter out background noise. Through the above calculations, the system generates a negative penalty term variable adapted to the current physical drift state.

[0131] S420, the constrained attention fusion module 40 injects the spatial distance penalty matrix into the dot product operation layer, truncates invalid interaction features through a masking mechanism, and outputs the final cross-modal feature fusion result.

[0132] After obtaining the valid feature vectors and their corresponding spatial distance penalty values, the constrained attention fusion module 40 performs the underlying tensor fusion calculation. The system will then use the feature vectors of the query nodes... The transpose matrix of the corresponding candidate key node eigenvectors Inner product operations are performed to evaluate the semantic correlation between two cross-modal feature nodes. To prevent excessively large feature dimensions from causing variance overflow in the dot product result and thus leading to gradient vanishing, the constrained attention fusion module 40 divides the dot product result by a dimension scaling factor. For the basic scaling dot product similarity operation logic in deep learning networks, those skilled in the art can refer to the standard multi-head attention architecture for implementation. Its basic tensor multiplication and addition calculation method is a well-known technology in this field and will not be elaborated here.

[0133] In this embodiment, the constrained attention fusion module 40 uses the spatial distance penalty value calculated in step S410. As a bias term, it is directly added to the result of the dot product operation. This applies to items not included in the dynamic sparse feature index set. For the remaining irrelevant nodes, the constraint attention fusion module 40 constructs a tensor masking mechanism in the feature computation graph at the bottom layer of the network, assigning the superposition value corresponding to the irrelevant nodes to a preset floating-point minimum value (such as -10). 9 This avoids floating-point overflow and achieves computational shielding in an engineering sense.

[0134] The constrained attention fusion module 40 inputs the superimposed tensor data into the Softmax activation function to perform exponential normalization. Since ineffective paired nodes are filled with the aforementioned floating-point minimum values, their attention weights approach zero within the numerical precision range after exponential mapping. Candidate nodes within the effective range, due to the superimposed negative spatial distance penalty, will have their final assigned relative attention weights smoothly decay as the degree of physical spatial deviation increases.

[0135] The constrained attention fusion module 40 combines the normalized attention weight tensor with the eigenvectors of the value matrix of the second modality. Perform a weighted summation calculation. For the first mode... The final cross-modal fusion output feature vector of each node Its complete calculation formula is expressed as:

[0136] ;

[0137] Through the aforementioned soft masking truncation process incorporating physical prior constraints, cross-modal feature points that do not conform to the current hardware transient kinematic state are subject to a spatial distance penalty term during normalization. The cross-modal feature fusion result output by the constrained attention fusion module 40... The network architecture improves its adaptability to hardware spatiotemporal misalignment in raw multimodal data. The fused feature vector possesses comprehensive multimodal semantic representation capabilities after electromechanical error compensation, and is subsequently input into subsequent classification or regression decoders along the network data flow to perform terminal business tasks such as product surface defect identification or dimensional measurement.

[0138] Reference Figure 7 Regarding step S500 above, this embodiment of the invention performs specific error calculation and joint parameter update operations through the closed-loop reverse optimization module 50. This step may specifically include the following sub-steps:

[0139] S510, the closed-loop reverse optimization module 50 extracts the predicted value of the final output of the deep learning network, compares it with the real target label to calculate the overall error distribution of the network, and constructs the target loss function.

[0140] When training the network model, a training sample set containing multimodal data needs to be pre-constructed. In this embodiment, the training samples are derived from the first modal sensing data, the second modal sensing data, and the synchronously encapsulated transient electromechanical metadata acquired by the synchronous data acquisition module 10 during historical operation. For specific industrial inspection business scenarios (such as product surface defect identification), business personnel or automated labeling systems pre-assign real target labels to the above samples, such as category unique hot codes representing "scratches," "cracks," or "normal."

[0141] To achieve the transformation from features to business meaning, the cross-modal feature fusion result output by the constrained attention fusion module 40 is input along the computation graph to the decoder network deployed at the end of the network. As a preferred approach, this decoder network contains a global average pooling layer and a multilayer perceptron structure. After the feature tensor is reduced to a one-dimensional global feature vector through pooling, it is mapped by the multilayer perceptron to an array of logical values ​​consistent with the preset number of defect categories. The closed-loop reverse optimization module 50 receives the predicted probability distribution generated after the logical values ​​are activated by the Softmax function. The closed-loop reverse optimization module 50 compares this predicted probability distribution with the aforementioned true target labels, calculates the distribution divergence between the predicted and true values ​​using the cross-entropy loss function, and thus determines the scalar form of the target loss function. The calculation logic of the target loss function and the sample label matching mechanism in conventional deep learning tasks can be adaptively replaced by those skilled in the art according to the actual regression or classification business needs. The basic calculation process is a well-known technology in the field and will not be described in detail here.

[0142] S520, the closed-loop backpropagation module 50 executes the backpropagation algorithm based on the target loss function, so that the gradient is propagated along the computation graph, realizing the joint update of deep learning network parameters and learnable physical calibration parameters.

[0143] The closed-loop inverse optimization module 50 obtains the target loss function. After obtaining the scalar value, the automatic differentiation engine of the underlying deep learning framework is triggered. Target loss function The gradient value is propagated back through the dynamically constructed computational graph that begins to propagate forward through the network. This gradient value is then passed to the deep learning network parameters. This set contains the weights of each convolutional kernel, the linear layer projection matrix, and the bias term used for feature extraction.

[0144] Unlike conventional training logic that only updates numerical weights, the cross-modal feature fusion layer in this scheme is mathematically directly controlled by the underlying physical equations. After passing through the conventional network layers, the gradient values ​​continue to backpropagate through the normalized activation function and the spatial distance penalty operator according to the chain rule of calculus. Since the conversion process of the relative pixel offset vector in the aforementioned steps is a linear scalar division, and the construction formula of the global physical space offset vector includes continuously differentiable algebraic addition and multiplication operations, this mathematical characteristic ensures the continuity and connectivity of the backpropagation link. The gradient flow, along the calculation path of the relative pixel offset vector and the physical offset vector, is finally delivered to the preset leaf node in the physical offset field construction module 20, namely the velocity scaling drift coefficient. With vibration phase delay coefficient .

[0145] Considering the significant differences in magnitude and sensitivity between network digital weights and physical calibration parameters, the closed-loop inverse optimization module 50 configures differentiated learning rate strategies for different parameter groups in the optimizer. (Deep learning network parameters) The initial value is 10. -3 Up to 10 -4 Base learning rate of magnitude As for the coefficient that carries physical meaning and The system assigns a small independent learning rate with a decay coefficient of 0.01 to 0.1 to prevent the physical boundary from deviating too much from the expected operating condition.

[0146] Deep learning network parameters Velocity proportional drift coefficient and vibration phase delay coefficient The synchronous gradient descent update formulas are defined as follows:

[0147] ;

[0148] ;

[0149] ;

[0150] in, This represents the independent learning rate specifically configured for the physical calibration parameters. Through the aforementioned closed-loop iterative calculation method, the system fully utilizes the statistical gradient direction inherent in the large-scale training samples during the fitting of multimodal data semantic features. Under continuous iterative optimization, the data-driven gradient not only adjusts the digital weights used for feature representation but also implicitly corrects the time-varying mechanical drift error of the underlying industrial hardware, which is difficult to accurately calibrate using traditional testing instruments. This mechanism establishes a bidirectional coupling and synchronous optimization system between the electromechanical physical hardware error equation and the digital feature extraction weights within the model framework. After the deep learning network has undergone multiple iterations and the target loss function converges to a preset threshold, the closed-loop reverse optimization module 50 freezes and solidifies the current deep learning network parameters θ and the learnable physical calibration parameters. and The solidified deep learning network was officially built and deployed as an inference model on the industrial inspection terminal. During the actual online inference phase, the system only executes the forward propagation process from steps S100 to S400, without performing reverse parameter updates. Thus, relying on the learned prior laws of physical errors, it outputs high-precision multimodal detection results in real time, ensuring that the final deployed model can adapt to the physical and spatiotemporal misalignment conditions of the actual production line.

[0151] To further clarify the collaborative working process of the technical solution described in this invention, the following will illustrate it through a specific working scenario example of surface defect detection in industrial production lines.

[0152] On an industrial production line, a visible light camera is deployed as the first modal sensor, and an infrared thermal imager is deployed as the second modal sensor. Together, they detect the workpieces moving on the conveyor belt. The working process is as follows:

[0153] During system operation, the synchronous data acquisition module 10 acquires visible light images and infrared thermal distribution images. The synchronous data acquisition module 10 reads the bus data of the control system, extracts the time deviation at the moment of image capture, the linear speed of the conveyor belt fed back by the servo motor, and the triaxial values ​​of the accelerometer on the mounting bracket, and encapsulates them into transient electromechanical metadata.

[0154] The physical offset field construction module 20 receives transient electromechanical metadata. Combining the internally initialized velocity proportional drift coefficient and vibration phase delay coefficient, the physical offset field construction module 20 calculates the actual physical space offset vector caused by the conveyor belt motion and equipment vibration at that instant using kinematic formulas. The sparse index generation module 30 obtains the physical reality side length of the current feature layer of the deep learning network, converts the physical space offset vector into a relative pixel offset vector, and determines the local pixel tolerance radius based on the equipment's mechanical tolerance.

[0155] Reference Figure 8 In the figure, the horizontal and vertical axes represent the two-dimensional pixel coordinate system of the deep feature map of the network. Solid circles in the figure represent specific query nodes in the first modality feature map. Hollow squares represent the default theoretically aligned node group of the second modality in a conventional network without physical compensation. The crosshairs in the figure represent the expected drift center point calculated and mapped by the physical offset field construction module 20. Dashed circles represent the local search boundary defined by the sparse index generation module 30 based on the tolerance radius.

[0156] Depend on Figure 8 It can be seen that, due to the influence of the workpiece moving with the conveyor belt and on-site mechanical vibration, there is a coordinate deviation between the actual position and the theoretical alignment position of the second modal feature. The sparse index generation module 30 generates a dynamic sparse feature index set based on the expected drift center point where the crosshair is located. The system only retains feature nodes that fall within the range of the dashed circle for subsequent calculations, directly excluding the data of the peripheral background nodes.

[0157] The constrained attention fusion module 40 acquires a dynamic sparse feature index set and performs feature query and pairing calculations within the defined effective node range. The constrained attention fusion module 40 calculates the spatial distance between each candidate node and the expected drift center point, generating a spatial distance penalty value with Gaussian distribution characteristics. This penalty value is added as a bias term to the feature dot product operation result, performing weighted and normalized processing on the features.

[0158] Reference Figure 9 This figure reflects the weight allocation results of the constrained attention fusion module 40 in the local space. The X and Y axes in the figure represent the partial spatial coordinates of the second modality feature map, and the Z axis represents the attention weight value assigned to the corresponding node.

[0159] like Figure 9 As shown, the peak value of the attention weight surface is concentrated in the coordinate region corresponding to the expected drift center point. As the spatial nodes deviate from this center point, the attention weight values ​​exhibit a smooth decay trend. At the tolerance boundary (i.e.... Figure 8 In the coordinate region outside the dashed circle (in the figure), the weight surface is truncated due to the tensor masking mechanism, and the values ​​are directly reduced to zero. This figure shows that the system not only logically limits the computational range, but also reduces the feature fusion ratio of edge noise based on the degree of physical deviation within the effective range.

[0160] The constrained attention fusion module 40 outputs cross-modal feature fusion results. This result is then processed by a classification network to generate predicted categories of workpiece defects. The closed-loop backpropagation module 50 receives the predicted categories and compares them with the true target labels, calculating the target loss function. The closed-loop backpropagation module 50 performs backpropagation, feeding back the error gradient. Simultaneously updating the convolutional weights of each layer, the network adjusts the velocity scaling drift coefficient and vibration phase delay coefficient within the physical offset field construction module 20. After multiple iterations of data samples, the system automatically corrects the error between the theoretical physical model and the actual mechanical working conditions, making... Figure 8 The expected drift center point is located more accurately, and the final output is a detection result that adapts to the actual physical offset.

[0161] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a deep learning network that integrates attention mechanisms and multimodal data, characterized in that, Includes the following steps: Simultaneously acquire first modal sensing data and second modal sensing data, and extract transient electromechanical metadata at the moment of capture of the first modal sensing data and the second modal sensing data; Based on the transient electromechanical data and the preset learnable physical calibration parameters, a globally differentiable physical space offset vector is constructed to characterize the spatial coordinate deviation. The parameters of the deep learning network are obtained, the global differentiable physical space offset vector is reduced in dimension and mapped to the feature level, the relative pixel offset vector is determined, and a dynamic sparse feature index set is generated based on the relative pixel offset vector and the preset mechanical tolerance limit. Based on the dynamic sparse feature index set, attention weights containing spatial distance penalty terms are generated in the corresponding local space to perform dot product operations on the feature matrices corresponding to the first modality perception data and the second modality perception data, and output cross-modality feature fusion results. The target loss function is calculated based on the cross-modal feature fusion result, and the backpropagation algorithm is executed to jointly update the deep learning network parameters and the learnable physical calibration parameters.

2. The method for constructing a deep learning network that integrates attention mechanisms and multimodal data according to claim 1, characterized in that, The process of simultaneously acquiring first modal sensing data and second modal sensing data, and extracting transient electromechanical data at the moment of acquisition of the first modal sensing data and second modal sensing data, includes: Extract the first timestamp of the first modal sensing data and the second timestamp of the second modal sensing data, and generate the time deviation between the first timestamp and the second timestamp through difference calculation; Read encoder data from the electromechanical control bus to determine transient scalar linear velocity and extract the preset motion direction unit vector; The three-axis analog signal output by the accelerometer at the instant of capturing the first modal sensing data and the second modal sensing data is acquired, and a transient acceleration vector is generated; The time deviation, the transient scalar linear velocity, the unit vector of the direction of motion, and the transient acceleration vector are combined and encapsulated into the transient electromechanical data.

3. The method for constructing a deep learning network that integrates attention mechanisms and multimodal data according to claim 2, characterized in that, The learnable physical calibration parameters include a velocity proportional drift coefficient and a vibration phase delay coefficient; the construction of a globally differentiable physical space offset vector to characterize spatial coordinate deviation based on the transient electromechanical data and the preset learnable physical calibration parameters includes: The macroscopic translational motion offset compensation amount is determined by multiplying the transient scalar linear velocity, the time deviation, the unit vector of the motion direction, and the velocity proportional drift coefficient. Multiply the transient acceleration vector by the vibration phase delay coefficient to determine the micro-vibration offset compensation amount; The macroscopic translational offset compensation amount and the microscopic vibration offset compensation amount are algebraically summed to generate the global differentiable physical space offset vector.

4. The method for constructing a deep learning network that integrates attention mechanisms and multimodal data according to claim 1, characterized in that, The process of obtaining deep learning network parameters, reducing the dimensionality of the globally differentiable physical space offset vector to the feature level, and determining the relative pixel offset vector includes: Extract the cumulative spatial downsampling step size of the deep learning network, and combine it with the preset original sensor physical resolution to determine the physical real side length of a single pixel in the feature map; The global differentiable physical space offset vector is reduced in dimension and projected into a two-dimensional physical plane offset vector; The relative pixel offset vector is generated by dividing the two-dimensional physical plane offset vector by the physical reality side length.

5. The method for constructing a deep learning network that integrates attention mechanisms and multimodal data according to claim 4, characterized in that, The step of generating a dynamic sparse feature index set based on the relative pixel offset vector and a preset mechanical tolerance limit includes: Divide the mechanical tolerance limit by the physical real side length to determine the pixel tolerance radius; The sum of the two-dimensional spatial coordinate vector of the query node on the feature map of the deep learning network and the relative pixel offset vector is used as the expected drift center point; Candidate key nodes whose Euclidean distance between the two-dimensional spatial coordinate vector and the expected drift center point is less than or equal to the pixel tolerance radius are selected and combined to generate the dynamic sparse feature index set.

6. The method for constructing a deep learning network that integrates attention mechanisms and multimodal data according to claim 5, characterized in that, The step of generating attention weights containing spatial distance penalties in the corresponding local space based on the dynamic sparse feature index set includes: Extract the feature vector of the query node as the query feature, and extract the feature vectors corresponding to all candidate key nodes in the dynamic sparse feature index set as candidate key features; Calculate the square of the Euclidean distance between the two-dimensional spatial coordinate vectors of all the candidate key nodes and the expected drift center point; Using a preset variance hyperparameter, a decay calculation is performed on the squared value of the Euclidean distance to generate a negative spatial distance penalty term.

7. The method for constructing a deep learning network that integrates attention mechanisms and multimodal data according to claim 6, characterized in that, The step of performing a dot product operation on the feature matrices corresponding to the first modality sensing data and the second modality sensing data, and outputting the cross-modality feature fusion result, includes: Perform an inner product operation on the transpose matrix of the query feature and the candidate key feature to obtain a dot product result, and then add the spatial distance penalty term to the dot product result; Feature nodes that do not belong to the dynamic sparse feature index set are assigned a preset floating-point minimum value, and the dot product operation result after superimposing the spatial distance penalty term is normalized using an activation function to generate the attention weights. The attention weights are weighted and summed with the feature vectors of the value matrix corresponding to the second modality perception data to generate the cross-modal feature fusion result.

8. The method for constructing a deep learning network that integrates attention mechanisms and multimodal data according to claim 1, characterized in that, The step of calculating the target loss function based on the cross-modal feature fusion result includes: The cross-modal feature fusion result is input into the decoder network for feature dimensionality reduction and mapping processing, and the predicted probability distribution is output. By comparing the predicted probability distribution with the preset real target label, the distribution divergence between the predicted probability distribution and the real target label is calculated using the cross-entropy loss function, and the target loss function is constructed.

9. The method for constructing a deep learning network that integrates attention mechanisms and multimodal data according to claim 1, characterized in that, The execution of the backpropagation algorithm to jointly update the deep learning network parameters and the learnable physical calibration parameters includes: The automatic differentiation mechanism of the network computing framework is triggered, so that the gradient of the target loss function is backpropagated along the computation graph to the parameters of the deep learning network. The gradient of the target loss function is transmitted in reverse through the calculation path of the spatial distance penalty term and the relative pixel offset vector according to the chain rule, and then to the learnable physical calibration parameters. Perform gradient descent calculations and simultaneously update the values ​​of the deep learning network parameters and the learnable physical calibration parameters.

10. The method for constructing a deep learning network that integrates attention mechanisms and multimodal data according to claim 9, characterized in that, The synchronous update of the values ​​of the deep learning network parameters and the learnable physical calibration parameters includes: Assign a base learning rate to the parameters of the deep learning network, and calculate the first parameter update amount based on the base learning rate and the gradient value corresponding to the parameters of the deep learning network. An independent learning rate is assigned to the learnable physical calibration parameter, and a second parameter update amount is calculated based on the gradient value corresponding to the independent learning rate and the learnable physical calibration parameter, while limiting the value of the independent learning rate to be less than the value of the base learning rate; Based on the first parameter update amount and the second parameter update amount, the numerical updates of the deep learning network parameters and the learnable physical calibration parameters are completed synchronously.

Images