A multi-modal perception method based on armored vehicle shooting simulation evaluation

Abstract

The application discloses a kind of multi-modal perception methods based on armored vehicle shooting simulation evaluation, comprising, obtaining the physical perception data of visual image sequence and vibration signal sequence and the digital simulation data of shooting data sequence;Visual image sequence is processed with motion amplification, and vibration signal sequence is processed with time-frequency conversion;The processed physical data and digital data are input into double-flow neural network model, and the measured and theoretical feature coding of binary is output;The hamming distance of feature coding is calculated, when the distance is greater than threshold value, the physical compensation parameter is calculated according to the coding difference using decoder network, and is written into fire control system error compensation unit.The application realizes the quantization comparison of physical world response and digital world instruction in unified space by cross-modal feature fusion and hash mapping, and constructs the correction link from deviation evaluation to parameter inversion, improves the accuracy of shooting simulation evaluation and the intelligent level of fire control system error correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of shooting simulation evaluation, and in particular to a multi-modal perception method based on armored vehicle shooting simulation evaluation. BACKGROUND

[0002] The comprehensive performance evaluation of the fire control system of an armored vehicle is a key link to ensure the operational effectiveness and striking accuracy of weapon equipment. With the rapid evolution of modern optoelectronic detection technology, embedded computing, and digital signal processing algorithms, the testing and evaluation means for live ammunition shooting have gradually shifted from traditional target range point statistics and manual observation to data backtracking and digital simulation analysis based on multi-source sensors. The current mainstream technology usually relies on exterior ballistic measurement, high-speed camera systems, and vehicle-mounted bus recorders to obtain the trajectory of the projectile, dynamic images of the muzzle, and fire control calculation logs, respectively, in an attempt to build a virtual simulation evaluation environment through semi-physical simulation or digital twin technology to achieve quantitative analysis of shooting accuracy.

[0003] However, the existing technology still has significant limitations in processing multi-modal heterogeneous data during live ammunition shooting. Traditional evaluation methods often process physical perception data (such as mechanical vibration waveforms and muzzle visual images) and digital simulation data (such as fire control calculation instructions) separately, making it difficult to bridge the "semantic gap" between unstructured visual / signal data and structured bus parameters, resulting in an inability to accurately quantify the consistency deviation between the physical execution system and the digital calculation logic. In particular, for the slight deformation and high-frequency mechanical vibration of the barrel at the moment of shooting, conventional simulation monitoring methods lack feature enhancement mechanisms based on motion amplification and time-frequency transformation, making it difficult to capture key transient features. In addition, existing systems generally lack a feedback mechanism based on deep hash mapping, which cannot calculate the virtual-real difference in a unified Hamming space and automatically invert physical compensation parameters, limiting the real-time and intelligent level of dynamic error correction for actual fire control systems. SUMMARY

[0004] This section aims to outline some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, abstract, and title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0005] In view of the above existing problems, the present application is proposed. Therefore, the present application provides a multi-modal perception method based on armored vehicle shooting simulation evaluation to solve the problems raised in the background art.

[0006] To solve the above technical problems, the present application provides the following technical solutions: a multi-modal perception method based on armored vehicle shooting simulation evaluation, comprising:

[0007] acquiring physical perception data and digital simulation data of the armored vehicle in a live firing time period, the physical perception data including a sequence of visual images for a muzzle and barrel region and a sequence of vibration signals for a turret mechanical structure, and the digital simulation data including a sequence of firing parameters intercepted from a fire control system bus;

[0008] performing a motion amplification process based on spatial multi-scale decomposition on the sequence of visual images to generate an enhanced sequence of visual images, and performing a time-frequency transform process on the sequence of vibration signals to generate a two-dimensional time-frequency distribution matrix;

[0009] inputting the enhanced sequence of visual images and the two-dimensional time-frequency distribution matrix into a physical perception branch of a pre-constructed dual-stream neural network model to output a binary measured feature code, and inputting the sequence of firing parameters into a digital simulation branch of the dual-stream neural network model to output a binary theoretical feature code;

[0010] performing an exclusive OR operation on the measured feature code and the theoretical feature code, and counting the number of non-zero bits in the operation result to obtain a Hamming distance, when the Hamming distance is greater than a preset threshold, calculating a physical compensation parameter using a decoder network according to a difference vector of the measured feature code and the theoretical feature code, and writing the physical compensation parameter into a calculation unit of the fire control system.

[0011] As a preferred scheme of the multi-modal perception method based on armored vehicle firing simulation evaluation, after acquiring the physical perception data and the digital simulation data, a data time alignment step is further included, which includes:

[0012] taking the moment when the gun firing of the console in the fire control system is triggered as a common time zero point;

[0013] retrieving the moment when the first wave peak with an amplitude exceeding a noise threshold in the sequence of vibration signals as a physical reference point;

[0014] retrieving the moment when the average pixel brightness of the sequence of visual images has a step change as a visual reference point;

[0015] calculating the time difference of the physical reference point and the visual reference point relative to the common time zero point, and performing sliding alignment and interception on the sequence of visual images, the sequence of vibration signals and the sequence of firing parameters according to the time difference.

[0016] As a preferred scheme of the multi-modal perception method based on armored vehicle firing simulation evaluation, the acquired sequence of firing parameters includes:

[0017] Data on azimuth, elevation, propellant temperature, air temperature, crosswind, and turret angular velocity during artillery firing.

[0018] As a preferred embodiment of the multimodal perception method based on armored vehicle firing simulation evaluation described in this invention, the step of performing motion amplification processing based on spatial multi-scale decomposition includes:

[0019] Each frame of the visual image sequence is decomposed using a Laplacian pyramid to obtain sub-band signals with different spatial frequencies;

[0020] Select a sub-band signal corresponding to the inherent frequency range of the tube and perform time-domain bandpass filtering;

[0021] The filtered subband signal is multiplied by an amplification factor and then superimposed on the original Laplace pyramid decomposition signal.

[0022] An inverse pyramid transform is performed on the superimposed signals to reconstruct an enhanced visual image sequence containing magnified displacement features.

[0023] As a preferred embodiment of the multimodal perception method based on armored vehicle firing simulation evaluation described in this invention, the step of performing time-frequency transformation processing includes:

[0024] A complex mother wavelet function is selected to perform continuous wavelet transform on the one-dimensional vibration signal sequence;

[0025] By changing the scale factor and translation factor, the wavelet coefficients of the vibration signal at different time points and frequency scales can be calculated.

[0026] The modulus of the wavelet coefficients is extracted and mapped into a two-dimensional matrix format, which is used as the two-dimensional time-frequency distribution matrix.

[0027] As a preferred embodiment of the multimodal perception method based on armored vehicle firing simulation evaluation described in this invention, the dual-stream neural network model has the following structural configuration:

[0028] The physical perception branch includes a convolutional neural network layer for extracting fusion features of the enhanced visual image sequence and the two-dimensional time-frequency distribution matrix;

[0029] The digital simulation branch includes a multilayer perceptron and a recurrent neural network layer, used to extract the temporal features of the firing parameter sequence;

[0030] Both the physical perception branch and the digital simulation branch are connected to a hash mapping layer at their ends. The hash mapping layer contains a fully connected operation unit and a hyperbolic tangent activation unit, which are used to map features into continuous numerical vectors.

[0031] As a preferred scheme of the multi-modal perception method based on armored vehicle shooting simulation evaluation, wherein: the step of outputting the binary measured feature code and the theoretical feature code includes:

[0032] Obtaining a continuous numerical vector output by the hash mapping layer;

[0033] Discretizing the continuous numerical vector by applying a sign function;

[0034] Mapping numerical values greater than zero to binary bit 1 and mapping numerical values less than or equal to zero to binary bit 0 to generate a fixed-length binary code sequence.

[0035] As a preferred scheme of the multi-modal perception method based on armored vehicle shooting simulation evaluation, wherein: the network parameters of the dual-flow neural network model are obtained by training, and the training process includes:

[0036] Inputting pairs of sample data, the sample data including measured-simulation data pairs labeled as matching and measured-simulation data pairs labeled as not matching;

[0037] Calculating the distance between the feature codes output by the model;

[0038] Minimizing a loss function by a backpropagation algorithm, the loss function constraining the feature code distance of matching measured-simulation data pairs to approach zero and constraining the feature code distance of not matching measured-simulation data pairs to be greater than a set margin value.

[0039] As a preferred scheme of the multi-modal perception method based on armored vehicle shooting simulation evaluation, wherein: the physical compensation parameter is calculated using a decoder network, including:

[0040] Calculating a difference vector obtained by subtracting the theoretical feature code from the measured feature code;

[0041] Inputting the difference vector into a multi-layer regression network;

[0042] Outputting a multi-dimensional vector including azimuth angle, elevation angle value and angular velocity correction value as the physical compensation parameter through nonlinear mapping of the multi-layer regression network.

[0043] As a preferred scheme of the multi-modal perception method based on armored vehicle shooting simulation evaluation, wherein: the calculation unit of the fire control system includes:

[0044] Converting the calculated physical compensation parameter into a floating-point number format specified by the communication protocol of the fire control system;

[0045] The converted parameters are updated to the resolving unit of the fire control computer by sending a write instruction through a vehicle-mounted data bus.

[0046] Compared with the prior art, the scheme has the beneficial effects that:

[0047] 1、 The present application unifies the projection of physical perception unstructured visual / signal data and digital simulation structured parameter sequence to Hamming space to generate binary feature codes that can be directly compared by constructing a double-flow neural network model and hash mapping, solving the problem of heterogeneous data fragmentation processing and semantic gap in the prior art, and realizing the quantization and evaluation of physical to digital simulation.

[0048] 2、 For the slight vibration of the barrel at the moment of firing of the armored vehicle and the transient impact of the mechanical structure, the present application adopts a motion amplification processing based on spatial multi-scale decomposition and a time-frequency analysis method based on continuous wavelet transform to image the key dynamic information that is difficult to detect in the original data, so as to improve the model's ability to capture the core features of the physical process, and solve the problem of insufficient evaluation accuracy caused by insufficient feature extraction of traditional simulation monitoring means.

[0049] 3、 In addition, the present application also constructs a simulation monitoring and evaluation to parameter correction mode, when the Hamming distance representing the virtual-real difference exceeds the threshold value, not only can the deviation be identified, but also the abstract feature difference vector can be inverted into compensation parameters (azimuth angle, elevation angle, angular velocity correction value) with clear physical meaning by using a specially trained decoder network, and output to the fire control system, solving the problem that the prior art cannot automatically generate physical compensation parameters and relies on manual intervention, realizing adaptive learning and dynamic correction of actual fire control system errors, and improving the intelligent level of the fire control system and the shooting accuracy of the armored vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0051] Figure 1 The overall flowchart of the multi-modal perception method based on armored vehicle shooting simulation evaluation according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present application.

[0053] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. The present application, however, can be practiced in a variety of ways beyond the specific details set forth herein without departing from the scope of the present application. It can be appreciated by those skilled in the art that the present application can be practiced without such specific details.

[0054] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. The "in one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an embodiment that is independent of or selected from other embodiments.

[0055] The present application is described in detail in conjunction with the schematic diagram. In the detailed description of the embodiments of the present application, the cross-sectional view of the device structure is partially enlarged without the general proportion for the convenience of description, and the schematic diagram is only an example, which should not limit the scope of protection of the present application. In addition, the three-dimensional spatial dimensions of length, width and depth should be included in actual manufacture.

[0056] Meanwhile, in the description of the present application, it should be noted that the terms "upper, lower, inner and outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first, second or third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0057] In the present application, unless otherwise explicitly specified and limited, the terms "mounting, connection, connection" should be understood broadly, for example: it can be fixed connection, detachable connection or integral connection; it can also be mechanical connection, electrical connection or direct connection, it can also be indirectly connected through intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0058] Embodiment 1

[0059] Reference Figure 1This is the first embodiment of the present invention, which provides a multimodal perception method based on armored vehicle firing simulation evaluation, including:

[0060] S1. Acquire physical perception data and digital simulation data of armored vehicles during live-fire shooting. The physical perception data includes visual image sequences of the muzzle and barrel area and vibration signal sequences of the turret mechanical structure. The digital simulation data includes firing parameter sequences extracted from the fire control system bus.

[0061] It should be noted that, because live-fire exercises of armored vehicles are a complex dynamic process involving high temperature and pressure, strong impact and vibration, and millisecond-level rapid response, traditional single data sources cannot fully characterize the system state. Therefore, the core of this step is to simultaneously collect measured responses from the physical world and theoretical commands from the digital world through a deployed heterogeneous sensor network, thereby establishing a comprehensive and high-fidelity multimodal dataset for subsequent consistency assessment of virtual and real states.

[0062] Specifically, the multimodal dataset consists of physical sensing data and digital simulation data. The physical sensing data is characterized by visual image sequences and vibration signal sequences, while the digital simulation data is characterized by firing parameter sequences.

[0063] Furthermore, using external high-speed cameras deployed at the test site, continuous filming was conducted on the muzzle brake and barrel area of ​​the armored vehicle. The purpose of continuous filming was to directly capture the "barrel whipping" effect caused by the enormous recoil energy and pressure wave transmission within the barrel at the moment of firing, as well as phenomena such as muzzle flash and gas combustion disturbance. It should be explained that the barrel whipping effect is the elastic vibration phenomenon of the barrel caused by recoil during firing. Its core principle is similar to the whipping of a whip; that is, as energy is transferred from the breech (proximal end) to the muzzle (far end), the vibration amplitude is amplified step by step due to the decrease in mass. However, the minute elastic deformation and high-frequency vibration of the barrel are the key physical factors affecting the initial attitude (i.e., initial disturbance) of the projectile after leaving the barrel, which directly relates to firing accuracy. Based on this, we define the sequence of visual images acquired through continuous filming as a set:

[0064]

[0065] in, Indicates at time The captured single-frame image, It is a time variable. and These are the start and end times of the data collection cycle, respectively. It is the sampling time interval, and its reciprocal is the camera's frame rate.

[0066] Further, in order to ensure that the dynamic process of barrel vibration can be captured without distortion, the sampling frame rate of the camera must meet the Nyquist sampling theorem. And considering that the first and higher order natural frequencies of the barrel are usually distributed in the range of hundreds of hertz, in the present embodiment, the sampling frame rate of the camera is preferably higher than twice the main vibration frequency of the barrel, for example, set to 1000 fps or higher, to ensure that the subsequent motion amplification processing can effectively restore the micro displacement.

[0067] Further, high-sensitivity acceleration sensors are installed at key mechanical structure nodes of the turret, such as the trunnion, gun base or stabilizer actuator connection. These sensors are used to record the structural dynamic response of the entire turret system of the armored vehicle under the impact load of firing. The vibration signals can reflect the energy release and conduction path from ignition to projectile ejection, as well as the coupling vibration characteristics between components from the perspective of mechanical transmission, serving as complementary data to the displacement observed by visual image observation.

[0068] Specifically, the vibration signal sequence is represented as a one-dimensional time sequence:

[0069]

[0070] wherein, is the vibration signal sequence under one-dimensional time sequence, represents the vibration signal amplitude (such as acceleration value ) measured at time .

[0071] It should be noted that in order to capture the steep front and rich high-frequency components of the impact signal, the sampling frequency of the sensor needs to be set high enough, usually in the order of tens of kilohertz (kHz) (for example, ≥20 kHz), and the sensor should have a wide enough dynamic range to avoid signal clipping distortion under strong impact simulation.

[0072] Further, by accessing the fire control system data bus (such as CAN bus) of the armored vehicle through a data logger or bus analyzer, the theoretical parameters calculated and issued by the fire control computer to each actuator at the moment before firing are intercepted and recorded in real time. These data represent the "ideal" firing parameters calculated by the fire control system based on sensor input and trajectory model, which is the "theoretical expectation" of this shot in the digital world.

[0073] Specifically, the obtained firing parameter sequence includes:

[0074] Azimuth angle command value and elevation angle command value calculated by the fire control: These command values are the aiming angles that the gun servo system needs to execute, which are the theoretical reference of the firing direction.

[0075] Setting the temperature of the propellant, the temperature of the air value: the temperature of the propellant, which is the key parameter affecting the burning rate and bore pressure, the fire control system will be based on the temperature to correct the theoretical calculation of the initial velocity of the projectile.

[0076] Crosswind sensor reading value: the wind speed and direction data measured by the crosswind sensor, which is the main external environment input for the fire control system to calculate the wind deflection correction.

[0077] The angular velocity value of the turret: the turret rotation speed measured by the angular velocity sensor in the dynamic working conditions such as armored vehicle shooting while moving or rapid turret swinging, which is the key parameter for the fire control system to calculate and compensate the lead angle.

[0078] It should be noted that the sequence of firing data parameters can be formalized as a set of time-synchronized multi-dimensional vectors Each vector is defined as:

[0079]

[0080] Wherein, is the azimuth angle command value. is the elevation angle command value. is the temperature value of the propellant. is the reading value of the crosswind sensor. is the rotation speed of the turret.

[0081] In addition, in order to ensure the accurate correspondence between physical perception data and digital simulation data in time, the collection of all data streams must be based on a unified synchronization reference. The firing trigger pulse time of the fire control system is used as the common time zero point (T0) in the present scheme. The electric pulse signal generated by the firing trigger pulse is the explicit instruction for the fire control system to start the firing circuit, which marks the logical starting point of the entire shooting event, and has very high instantaneity and certainty. Each data acquisition device (high-speed camera, vibration signal acquisition sensor, bus recorder and other sensors) needs to record the arrival time of this trigger pulse, and the subsequent data processing will also be time-aligned based on this, in order to ensure that from the generation of the theoretical instruction to the occurrence of the physical response, the entire causal chain has continuity and comparability in the time dimension.

[0082] Specifically, for the visual reference point (T0) Calibration of the visual fiducial point: search the collected visual image sequence, when the gun is fired, the appearance of the muzzle flame will cause a sharp change in the average brightness of the pixels in the field of view. Therefore, when the frame with the maximum brightness value change rate (i.e. the brightness difference between adjacent frames) is found, it is marked as the visual fiducial point. This method is less sensitive to slow changes in brightness than simple step detection, and can more accurately locate the sudden moment of flame ejection. It needs to be explained that this moment corresponds to the moment when the combustion energy of the gunpowder begins to release outward.

[0083] Specifically, for the calibration of the physical fiducial point (t Calibration of the visual fiducial point: search the collected visual image sequence, when the gun is fired, the appearance of the muzzle flame will cause a sharp change in the average brightness of the pixels in the field of view. Therefore, when the frame with the maximum brightness value change rate (i.e. the brightness difference between adjacent frames) is found, it is marked as the visual fiducial point. This method is less sensitive to slow changes in brightness than simple step detection, and can more accurately locate the sudden moment of flame ejection. It needs to be explained that this moment corresponds to the moment when the combustion energy of the gunpowder begins to release outward.

[0084] In addition, since the sampling frequencies of different modal data are inconsistent (for example, the video frame rate of the visual sensor and the high-frequency analog signal sampling of the vibration sensor), before time alignment, each modal data needs to be resampled. Specifically, by setting a uniform time step, the high-frequency vibration signal can be downsampled or framed feature extraction (such as calculating the root mean square value of vibration in each time window), and the low-frequency data can be interpolated, so that each modal data input into the neural network has a corresponding mapping relationship in the time dimension.

[0085] It needs to be emphasized that for this noise threshold, a certain length of vibration signal can be collected in a non-firing state (for example, only the power of the fire control system is turned on but not fired), the peak value of the signal is calculated, and two to three times of the peak value is taken as the noise threshold, so as to effectively filter out the interference of environmental and electronic noise.

[0086] Further, the time difference (t and ). According to the two time differences, the vibration signal sequence and the visual image sequence are slid along the time axis to align their physical event starting points with the theoretical common time zero point. At the same time, according to the aligned time stamps, all three data sequences (visual image sequence, vibration signal sequence and shooting parameter sequence) are intercepted by a unified time window, for example, a data segment from several milliseconds before the zero point to several milliseconds after the zero point is intercepted.

[0087] S2, motion amplification processing based on spatial multi-scale decomposition is performed on the visual image sequence to generate an enhanced visual image sequence, and time-frequency transformation processing is performed on the vibration signal sequence to generate a two-dimensional time-frequency distribution matrix.

[0088] It should be noted that the purpose of this step is to enhance and represent the original physical perception data of two different modalities (i.e., the visual image sequence and the vibration signal sequence) to solve the problem that the existing simulation monitoring means cannot capture the barrel micro-deformation and high-frequency mechanical vibration transient characteristics. By explicitly amplifying and transforming the weak physical phenomenon, the key dynamic information implicitly contained in the original signal is converted into significant features that can be efficiently learned by a neural network.

[0089] Further, for the visual image sequence, the barrel vibration at the moment of shooting (i.e., the "barrel whipping" effect) is a small and fast motion, which is often difficult to detect with the naked eye in the original high-speed video. If the original image sequence is directly input into the neural network, the model may not be able to effectively learn the key motion features carried by the small pixel displacement. Therefore, the linear motion amplification technology based on spatial multi-scale decomposition is adopted to visually enhance this small motion without introducing significant noise. The processing process is as follows:

[0090] S201.1, the visual image sequence is decomposed into a Laplacian pyramid. It should be explained that this decomposition process is an over-complete image representation method that can decompose the visual image into a series of sub-band signals of different spatial frequencies. Its physical meaning is to simulate the visual system and separate the image information according to the "from coarse to fine" scale. The decomposition process is as follows:

[0091] A Gaussian pyramid is constructed wherein, , is obtained by Gaussian blurring and downsampling . The layer of the Laplacian pyramid is calculated by the difference between the adjacent two layers of the Gaussian pyramid:

[0092]

[0093] wherein, represents the subband image of the layer, which contains the details and texture information at this spatial frequency scale. represents the up-sampling operation.

[0094] It should be noted that through this S201.1 step, the original visual image is decomposed into a set of subband signals and the top layer of the Gaussian pyramid .

[0095] S201.2, select the subband signal of a certain level to perform time domain filtering. The significance is that the vibration of the barrel is mainly concentrated in its inherent frequency, so we are interested in the pixel changes with a certain frequency in the time dimension. Based on this, for a selected level of the Laplacian pyramid , the pixel values at the same spatial position of all frames form a one-dimensional time series . Then, an ideal band-pass filter is applied to the time series, and the passband frequency range of the filter is set to correspond to the inherent frequency range of the barrel. The inherent frequency range can be determined in advance by finite element modal analysis (FEM), by consulting weapon design data, or by preliminary experimental modal analysis (such as hammering method) on the barrel. For example, if the first-order inherent frequency of the barrel is known to be 200Hz, the filter range can be set between 190Hz and 210Hz. The purpose is to accurately extract the intensity change component caused by the periodic vibration of the barrel from all pixel changes, denoted as .

[0096] S201.3, multiply the subband signal obtained after filtering, which only contains the target motion information by an amplification coefficient , and superimpose it with the original Laplacian pyramid decomposition signal:

[0097]

[0098] wherein, is the enhanced subband signal. The amplification coefficient The value of the coefficient needs to be balanced between clear visibility of the movement and no excessive distortion of the image. The value can start from a small value (e.g. 10), and the coefficient is gradually increased, and the enhanced visual image sequence is observed to select a critical value that can visually clearly identify the barrel vibration pattern, but will not cause excessive distortion of the image background or obvious amplification of noise. Alternatively, a quantitative index such as the change in signal-to-noise ratio (SNR) or structural similarity index (SSIM) of the image sequence before and after enhancement can be used to select the amplification coefficient that has the most significant motion amplification effect while maintaining high image quality. In this embodiment, the value is in the range of 10-50.

[0099] It should be emphasized that the core of S201.3 is to amplify the phase change of the target frequency component, and when these amplified phase changes are reconstructed into images, they will exhibit macroscopic displacement of spatial position.

[0100] S201.4, using the enhanced Laplacian pyramid layer and the original Gaussian pyramid top layer performing pyramid inverse transformation to reconstruct the enhanced image of a single frame The process is the inverse operation of decomposition, that is, the up-sampled bottom layer image is added to the detail information of the next higher level layer by layer, and finally the full resolution image is restored. By performing the above process on each frame of visual image in the visual image sequence, the enhanced visual image sequence can be finally generated. In this visual image sequence, the barrel vibration that was previously difficult to detect becomes clearly visible, providing high-quality input for feature extraction by the neural network.

[0101] Further, for the vibration signal sequence, the mechanical vibration generated by the firing of the gun is a typical non-stationary, transient impact signal, and its frequency component changes dramatically over time. Traditional Fourier transform can only obtain the global frequency distribution of the signal and cannot reveal the dynamic characteristics of the frequency evolution over time. In order to solve this problem, the present application adopts a time-frequency analysis method to convert the one-dimensional time sequence vibration signal into a two-dimensional "time-frequency image" so as to be input into the neural network together with the visual image. The processing process is as follows:

[0102] S202.1, selecting a complex mother wavelet function (e.g. complex Morlet wavelet) to perform continuous wavelet transform on the one-dimensional vibration signal sequence It should be explained that a complex mother wavelet is selected because it contains amplitude and phase information, which can more comprehensively describe the signal characteristics. Continuous wavelet transform (CWT) is performed by stretching and translating the mother wavelet and performing inner product operation with the signal to be analyzed, and its mathematical definition is as follows:

[0103]

[0104] wherein, is a complex wavelet coefficient at scale and time shift . is the complex conjugate of the mother wavelet function . is a scale factor inversely proportional to frequency, used to stretch the mother wavelet to match different frequency components. is a time shift factor, used to slide the mother wavelet along the time axis to locate the time of event occurrence.

[0105] S202.2, a series of complex wavelet coefficients are calculated by continuously changing the scale factor and the time shift factor within a pre-set scale range and the entire time domain. It is emphasized that this process is equivalent to scanning the entire signal with a "time-frequency analysis window" to obtain the frequency components and their intensities contained in the signal at each time point.

[0106] S202.3, the modulus (i.e. amplitude) of all calculated complex wavelet coefficients is extracted, denoted as . The modulus represents the energy intensity of the signal at time shift and scale . Subsequently, these modulus values are mapped into a two-dimensional matrix according to the corresponding relationship between time shift and scale. In this two-dimensional matrix, the row index corresponds to the scale (or the converted frequency), the column index corresponds to the time shift (which can be represented as time here), and the value of the element in the matrix is denoted as . It should be explained that this two-dimensional matrix is a two-dimensional time-frequency distribution matrix, also known as a wavelet time-frequency spectrum or a scale diagram, which visually displays the distribution of the vibration signal in the time-frequency plane in the form of an image, successfully converting the one-dimensional non-stationary signal into two-dimensional structured data suitable for image processing.

[0107] It should be noted that the processing steps of the above-mentioned visual image sequence and vibration signal sequence are parallel, and after the processing steps, the two heterogeneous physical perception data (visual image sequence and vibration signal sequence) can be uniformly converted into two image-based, feature-enhanced representations.

[0108] S3, the enhanced visual image sequence and the two-dimensional time-frequency distribution matrix are input into the physical perception branch of the pre-constructed dual-flow neural network model, and the binary measured feature code is output. The firing data sequence is input into the digital simulation branch of the dual-flow neural network model, and the binary theoretical feature code is output.

[0109] It should be noted that the task of this step is to build a dual-flow neural network model for processing heterogeneous data from the physical world and the digital world, and finally projecting these heterogeneous data into the same low-dimensional Hamming space to solve the "semantic gap" problem between different data types. It should be emphasized that in this process, the present solution also adopts deep hashing technology to compress complex system states into efficient and comparable binary codes as the basis for subsequent quantitative evaluation and difference inversion.

[0110] Further, the built dual-flow neural network model contains two specially designed parallel branches, namely the physical perception branch and the digital simulation branch.

[0111] It should be explained that the physical perception branch is mainly responsible for processing physical perception data representing "what actually happened", and its input is the enhanced visual image sequence generated in step S2 and the two-dimensional time-frequency distribution matrix . The processing flow is as follows:

[0112] Since the input of the physical perception branch is all image data, the core of this branch is the convolutional neural network (CNN). Specifically, this physical perception branch contains two parallel CNN backbone networks:

[0113] A 3D-CNN is used to process the enhanced visual image sequence to extract the spatio-temporal features of phenomena such as barrel vibration and muzzle flame.

[0114] A 2D-CNN is used to process the two-dimensional time-frequency distribution matrix to extract key pattern features of vibration impact energy in the time-frequency plane.

[0115] After the high-dimensional features extracted by the two CNN backbone networks are subjected to global average pooling (Global Average Pooling) or flattening (Flatten) operations, they are concatenated together to form a unified, high-dimensional fusion feature vector .

[0116] It should be noted that the physical perception branch allows the network to learn and represent the physical process of a single armored vehicle live firing from two dimensions of visual macro motion and mechanical micro vibration, and the fused feature vector is a highly condensed description of the physical state of this firing event.

[0117] Correspondingly, the digital simulation branch is responsible for processing the digital simulation data representing "what should theoretically happen", whose input is the sequence of firing parameters obtained in the S1 step. Its processing flow is as follows:

[0118] Since the input is structured time series data, the structure design of the digital simulation branch is a combination of multi-layer perception (MLP) and recurrent neural network layer (RNN):

[0119] Each multi-dimensional vector is processed through an MLP network to learn the non-linear combination relationship between different parameters at this moment (such as drug temperature, crosswind, angular velocity, etc.). Then, the feature sequence encoded by the MLP network is sent to a recurrent neural network (such as GRU, whose hidden layer node number can be set to 128 or 256) to capture the time sequence features of the evolution of these parameters over time. The hidden state of the last time step of the RNN is taken as the summary of the entire time series data to form the theoretical feature vector .

[0120] It should be noted that the digital simulation branch enables the model to understand the dynamic evolution process of the fire control system before a series of instructions and parameters, rather than looking at a certain instantaneous value in isolation. The final output feature vector is a deep semantic representation of the "theoretical expectation" of this shot.

[0121] Further, after the high-dimensional feature vectors and are extracted independently by the two parallel branches, they need to be mapped to a unified Hamming space.

[0122] Specifically, a hash mapping layer is connected at the end of the physical perception branch and the digital simulation branch. The hash mapping layer is located at the end of the above two parallel branches and is composed of a fully connected layer and a tanh activation function, which compresses the high-dimensional feature into a continuous feature vector with a length of 32 bits or 64 bits. Its role is to linearly project the high-dimensional feature vector of any dimension to a pre-set dimension (i.e. the length of the hash code, for example bits), and then perform nonlinear activation through the tanh activation function. Among them, for the physical perception branch, the output continuous numerical vector is calculated as follows:

[0123]

[0124] And for the digital simulation branch, the output continuous numerical vector is calculated as follows:

[0125]

[0126] wherein, is a continuous numerical vector with length , and each element of it has a value range of [-1, 1]. is the weight matrix of the fully connected layer. is the bias vector of the fully connected layer. The function serves to introduce nonlinearity and compress the output value to the interval [-1, 1] to stabilize the model training process and prepare for discretization.

[0127] Further, in order to generate the final binary code, it is necessary to perform discretization processing on the above continuous numerical vector;

[0128] Specifically, the numerical vector greater than zero is mapped to binary bit 1, and the numerical vector less than or equal to zero is mapped to binary bit 0 to generate a fixed-length binary code sequence. Wherein, the discretization operation follows a threshold judgment rule: taking the above as an example, each bit of the binary actual feature code can be determined as follows: if the th element of is greater than zero, the th element of the actual feature code is 1; if the th element of is less than or equal to zero, the th element of the actual feature code is 0. It should be emphasized that , and , the binary theoretical feature code also applies to this judgment rule.

[0129] It should be emphasized that the parameters of the double-flow neural network model (such as and all learnable parameters in CNN / RNN) are not randomly set, but obtained through pre-supervised training. The training process aims to let the model learn that "like attracts like". The training process of the model is as follows:

[0130] For the actual-simulation data pairs labeled as matching (i.e. physical perception data and digital simulation data from the same successful shot), the model outputs binary codes with a very close distance (ideally, the Hamming distance is 0). For data pairs labeled as not matching (for example, pairing physical data of A shots with digital data of B shots), the model outputs codes with a relatively far distance (greater than a set margin ). Specifically, in actual training, these mismatched data pairs can be generated in batches through an efficient strategy, for example, in a training batch (batch), for any one actual-simulation data pair, assuming all other measured data within the batch or simulated data can be paired with or to form mismatched pairs (i.e., an "in-batch negative sampling" strategy), thereby efficiently constructing a large number of negative samples for contrastive learning. To achieve this goal, the present scheme employs a contrastive loss function (ContrastiveLoss) for a measured-simulated data pair, whose loss is defined as:

[0131]

[0132] where is the label, when the data pair is matched , and mismatched . represents the Hamming distance between two binary encodings and . is a pre-set margin value, which is a hyperparameter, and the margin value defines the minimum distance that should be maintained between the feature encodings of mismatched data pairs in the Hamming space. Its role is to provide a clear "push away" goal for model learning, to prevent all encodings from collapsing into similar positions. The value of is usually determined by hyperparameter tuning methods such as grid search on the validation set, for example, selecting several candidate values within the range of to conduct experiments, and during the experiment, choosing the value that allows the model to achieve the best discrimination performance (such as the highest average precision mean mAP) on the validation set as the initial value of the margin value.

[0133] Through the backpropagation algorithm, the loss is calculated with respect to the gradients of the parameters in the dual-flow neural network model, and the parameters are updated according to the gradients to minimize the total loss on the entire training data set. This process can force the dual-flow neural network model to learn a mapping relationship, so that semantically similar (matched) multi-modal inputs are also mapped to similar positions in the Hamming space, while semantically unrelated inputs are pushed away.

[0134] It should be noted that by constructing a dual-flow neural network model, the complex observations of the physical world and the theoretical instructions of the digital world can be converted into measured feature encodings and theoretical feature encodings that can be directly quantitatively compared in the same compact space, providing efficient and meaningful feature representations for subsequent virtual-real consistency evaluation.

[0135] S4, performing bitwise XOR operation on the measured feature code and the theoretical feature code, counting the number of non-zero bits in the operation result to obtain the Hamming distance, when the Hamming distance is greater than a preset threshold, calculating the physical compensation parameter according to the difference vector of the measured feature code and the theoretical feature code using a decoder network, and writing the physical compensation parameter into a calculation unit of the fire control system.

[0136] It should be noted that the task of this step is to efficiently quantitatively evaluate the consistency between "theory" and "reality" through Hamming distance, and when significant deviation is found, a specially trained decoder network is used to invert the abstract feature code difference into specific, physically meaningful error compensation values, and these compensation values are written into the fire control system to achieve adaptive correction of future shooting accuracy, thereby solving the problem that traditional methods cannot automatically invert the virtual-real difference into physical compensation parameters, thereby limiting the real-time performance of dynamic error correction.

[0137] Further, since in the S3 step, we have mapped the complex physical measurement process and the digital simulation instruction into two binary codes with lengths of respectively, that is, the measured feature code and the theoretical feature code , then based on this, the consistency of the two codes is quantitatively measured.

[0138] Specifically, bitwise XOR operation is used to compare the two binary codes. The Hamming distance is defined as the number of different characters at corresponding positions in two equal-length code strings. In fact, in the binary (represented by 0 / 1) scenario, it is equivalent to counting the number of "1"s after performing bitwise XOR operation on the two codes, and its mathematical representation is:

[0139]

[0140] wherein is the calculated Hamming distance, which is a non-negative integer. and are the binary values of the two codes at the th position. represents the bitwise XOR operator.

[0141] It should be noted that the physical meaning of the Hamming distance is that it quantifies the execution fidelity of the theoretical instructions of the fire control system. When , it means that the measured feature code and the theoretical feature code are completely consistent, indicating that the entire chain from digital instructions to physical responses is highly consistent, and the system state is "as expected" running. When The deviation of the physical world from the theoretical expectations of the digital world occurs in certain characteristic dimensions. The larger the distance, the more significant and prevalent this deviation is. The Hamming distance is chosen because its calculation involves only bitwise operations, resulting in extremely high hardware efficiency, making it suitable for embedded or real-time systems requiring rapid decision-making.

[0142] Furthermore, after calculating the Hamming distance, it is necessary to determine whether the deviation needs to be corrected.

[0143] Specifically, the calculated Hamming distance With a preset threshold Compare. If If the deviation between real and simulated fire in this armored vehicle firing is considered to be within an acceptable noise range or the system's normal tolerance, no correction is required, and the judgment process terminates. If the system exhibits a significant bias that requires correction, the compensation parameter calculation process is initiated.

[0144] It should be noted that this preset threshold The value of the threshold needs to be balanced between the system's "sensitivity" and "stability." A threshold that is too low will cause the system to overreact to minor noise, leading to frequent corrections and instability. Conversely, a threshold that is too high may affect early deviations in system accuracy. In this embodiment, the specific value can be determined empirically using the following method: A large number of shooting data pairs known to be "normal" (i.e., shooting accuracy within the allowable range) are collected, and their corresponding Hamming distances are calculated to form a normal sample Hamming distance distribution. Simultaneously, a small number of shooting data pairs known to be "abnormal" (e.g., artificially introduced fixed errors) are collected, and their Hamming distances are calculated. Based on this, the preset threshold can be selected as a value slightly higher than the upper bound of the normal sample Hamming distance distribution (e.g., the mean plus three standard deviations) and significantly lower than the Hamming distance of abnormal samples, to ensure effective differentiation between normal fluctuations and significant deviations, avoiding overreaction to normal system noise.

[0145] Furthermore, once it is determined that a correction is needed, the measured feature codes are calculated. With theoretical feature encoding The difference vector between :

[0146]

[0147] It's important to explain that because the Hamming distance is a scalar, it only tells us "how large the deviation is," but not "where the deviation is located or what kind of deviation it is." The difference vector... It is a vector that preserves the "directional" information of the deviation. If the encoding is represented by {-1,1}, then... Each element of the difference vector can be {-2, 0, 2}, i.e., indicates which abstract feature dimension learned by the dual-stream neural network model has flipped from 1 to -1 or from -1 to 1. The difference vector constitutes a kind of "diagnostic fingerprint" of the system bias state.

[0148] Further, an abstract "diagnostic fingerprint" is translated into concrete physical compensation parameters by a decoder network.

[0149] Specifically, in the present embodiment, the decoder network is a multi-layer regression network comprising a number of hidden layers of multi-layer perceptron (MLP). The input of the multi-layer perceptron is the difference vector of dimension . The output of the multi-layer perceptron is a multi-dimensional vector containing azimuth correction, elevation correction and muzzle velocity correction.

[0150]

[0151] wherein are the compensation amounts of azimuth, elevation and muzzle velocity respectively that need to be made to the fire control system.

[0152] It should be noted that the decoder network is paired trained with the dual-stream encoding network in the S3 step. The training target of the decoder network is to establish a mapping from the feature encoding difference to the physical error amount. The training data set is constructed as follows:

[0153] A series of live firings of armored vehicles are conducted under controlled conditions, of which part of the firings are reference firings (without additional error) and the other part artificially and quantitatively introduces known physical errors. For example, by modifying the fire control system parameters, a known azimuth deviation (such as -0.2 mil) or elevation deviation (such as +0.1 mil) is superimposed on the command issued to the servo system, or ammunition with known deviation of muzzle velocity is used. Specifically, a dedicated simulation test software interface can be used to directly superimpose a precise and known bias (for example, with a step size of 0.05 mil) on the digital command issued by the fire control computer to the servo controller, rather than through physical adjustment. In addition to using ammunition with known performance deviation, a key parameter (such as the set temperature of the propellant) used to calculate the muzzle velocity in the ballistic calculation module of the fire control system can be artificially modified, and the corresponding theoretical muzzle velocity change of this modification amount is recorded. At the same time, high-precision measuring equipment such as external ballistic radar is used to measure the real muzzle velocity of the firing to verify the effectiveness of the error simulation.

[0154] For each time a known error (for example, azimuth error Elevation / slope error Initial velocity error The system fires the data, collects its physical perception data, and pairs it with “ideal” digital simulation data under the same conditions (i.e., theoretical parameters without introducing errors).

[0155] The paired data is input into the pre-trained two-stream neural network model to obtain the measured feature encoding. and theoretical feature encoding And calculate its difference vector. .

[0156] The calculated difference vector is used as the input to the training samples, and the artificially introduced known physical error vector is used as the input. This serves as the label for the sample. By learning from a large number of such sample pairs and optimizing using regression loss functions such as mean squared error (MSE), the decoder network can establish a nonlinear mapping between the bias vector in Hamming space and the error quantity in the physical parameter space. This enables it to understand which combination of physical errors most likely corresponds to a particular type of coding difference.

[0157] Furthermore, the calculated physical compensation parameters are written into the fire control system.

[0158] Specifically, firstly, the floating-point compensation value (i.e., the multi-dimensional vector) output by the decoder network is... The data is converted into a floating-point number format specified by the fire control system communication protocol. This floating-point format includes data type (such as float or double), byte order (big-endian or little-endian), etc., to ensure that the fire control computer can correctly parse it. Then, a write command is sent through the vehicle data bus (such as the CAN bus). This command contains the target device address (fire control computer), the target memory address (system error table area), and the formatted compensation parameter data. When the command is received by the fire control computer, the system error table area in its internal non-volatile memory (such as Flash or EEPROM) is updated.

[0159] It should be noted that writing the compensation parameters into non-volatile memory means that the correction effect is persistent. Therefore, in the next armored vehicle firing mission, the fire control computer will automatically load these compensation values ​​when performing ballistic calculations, pre-correcting the theoretical calculation results. In this way, the system learns and compensates for its own system errors through self-perception and evaluation during one (or more) live-fire exercises, thereby dynamically improving the accuracy of subsequent armored vehicle firing.

[0160] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, apparatus, or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code. Embodiments of the present application are also directed to computer program products comprising computer readable program code to carry out the methods of the present application when the program code is executed on one or more computing devices.

[0161] The present application is described in reference to the flowchart illustrations and / or block diagrams according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0162] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0164] While preferred embodiments of the application have been described, modifications and additions can be made without departing from the spirit of the application. Accordingly, the appended claims are intended to cover all such modifications and additions as fall within the true spirit and scope of the application.

[0165] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A multi-modal perception method based on armored vehicle firing simulation evaluation, characterized in that, The method comprises the following steps: acquiring physical perception data and digital simulation data of the armored vehicle within a live firing period, wherein the physical perception data comprises a sequence of visual images for the muzzle and barrel area and a sequence of vibration signals for the turret mechanical structure, and the digital simulation data comprises a sequence of firing data parameters intercepted from the fire control system bus; performing motion amplification processing based on spatial multi-scale decomposition on the sequence of visual images to generate an enhanced visual image sequence, and performing time-frequency transform processing on the sequence of vibration signals to generate a two-dimensional time-frequency distribution matrix; inputting the enhanced visual image sequence and the two-dimensional time-frequency distribution matrix into a physical perception branch of a pre-constructed dual-flow neural network model to output a binary measured feature code, and inputting the sequence of firing data parameters into a digital simulation branch of the dual-flow neural network model to output a binary theoretical feature code; performing an exclusive-OR operation on the measured feature code and the theoretical feature code, and counting the number of non-zero bits in the operation result to obtain a Hamming distance, and when the Hamming distance is greater than a preset threshold, calculating a physical compensation parameter according to a difference vector of the measured feature code and the theoretical feature code by using a decoder network, and writing the physical compensation parameter into a calculation unit of the fire control system.

2. The multi-modal perception method based on armored vehicle firing simulation evaluation of claim 1, wherein, After acquiring the physical perception data and the digital simulation data, a data time alignment step is further included, which comprises the following steps: taking the moment when the gun firing of the console in the fire control system is triggered as a common time zero point; searching for the moment when the first wave crest with an amplitude exceeding a noise threshold in the sequence of vibration signals, and marking it as a physical reference point; searching for the moment when the average pixel brightness of the sequence of visual images has a step change, and marking it as a visual reference point; calculating the time difference between the physical reference point and the visual reference point relative to the common time zero point, and performing sliding alignment and interception on the sequence of visual images, the sequence of vibration signals and the sequence of firing data parameters according to the time difference.

3. The multi-modal perception method based on armored vehicle firing simulation evaluation of claim 2, wherein, The sequence of firing data parameters acquired comprises: azimuth angle data, elevation angle data, charge temperature data, air temperature data, crosswind data and turret angular velocity data when the gun is fired.

4. The multi-modal perception method based on armored vehicle firing simulation evaluation of claim 1, wherein, The motion amplification processing based on spatial multi-scale decomposition comprises the following steps: performing Laplacian pyramid decomposition on each frame of image in the sequence of visual images to obtain sub-band signals of different spatial frequencies; selecting the sub-band signals corresponding to the barrel inherent frequency range to perform time domain band pass filtering; multiplying the filtered sub-band signals by an amplification coefficient, and then superimposing them with the original Laplacian pyramid decomposition signals; performing inverse pyramid transform on the superimposed signals to reconstruct an enhanced visual image sequence containing amplified displacement features.

5. The multi-modal perception method based on armored vehicle firing simulation evaluation of claim 1, wherein, The time-frequency transform processing comprises the following steps: selecting a complex mother wavelet function to perform continuous wavelet transform on the one-dimensional sequence of vibration signals; calculating the wavelet coefficients of the vibration signals at different time points and frequency scales by changing the scale factor and the translation factor; extracting the modulus values of the wavelet coefficients, and mapping the modulus values into a two-dimensional matrix format as the two-dimensional time-frequency distribution matrix.

6. The multi-modal perception method based on armored vehicle firing simulation evaluation of claim 1, wherein, The dual-flow neural network model is configured as The physical perception branch comprises a convolutional neural network layer for extracting fusion features of the enhanced visual image sequence and the two-dimensional time-frequency distribution matrix; The digital simulation branch comprises a multi-layer perception and a recurrent neural network layer for extracting time sequence features of the firing parameter sequence; The end of the physical perception branch and the digital simulation branch are both connected with a hash mapping layer, which comprises a full connection operation unit and a hyperbolic tangent activation unit, for mapping the features into a continuous numerical vector.

7. The multi-modal perception method based on armored vehicle firing simulation evaluation of claim 6, wherein, The step of outputting the binary measured feature code and the theoretical feature code comprises: obtaining the continuous numerical vector output by the hash mapping layer; applying a sign function to the continuous numerical vector for discretization processing; mapping the numerical value greater than zero to binary bit 1 and mapping the numerical value less than or equal to zero to binary bit 0 to generate a fixed-length binary code sequence.

8. The multi-modal perception method based on armored vehicle firing simulation evaluation as claimed in claim 6, wherein, The network parameters of the double-flow neural network model are obtained through training, and the training process comprises: inputting paired sample data, wherein the sample data comprises measured-simulation data pairs labeled as matching and measured-simulation data pairs labeled as not matching; calculating the distance between the feature codes output by the model; minimizing the loss function through a back propagation algorithm, wherein the loss function constrains the feature code distance of the matching measured-simulation data pairs to be close to zero and constrains the feature code distance of the not matching measured-simulation data pairs to be greater than a set margin value.

9. The multi-modal perception method based on armored vehicle firing simulation evaluation of claim 1, wherein, The calculation of the physical compensation parameter by the decoder network comprises: calculating a difference vector obtained by subtracting the theoretical feature code from the measured feature code; inputting the difference vector into a multi-layer regression network; outputting a multi-dimensional vector comprising an azimuth angle, an elevation angle value and an angular velocity correction value as the physical compensation parameter through nonlinear mapping of the multi-layer regression network.

10. The multi-modal perception method based on armored vehicle firing simulation evaluation of claim 1, wherein, The calculation unit of the fire control system comprises: converting the calculated physical compensation parameter into a floating point number format specified by the communication protocol of the fire control system; sending a write instruction through the vehicle-mounted data bus to update the converted parameter to the calculation unit of the fire control computer.

Images