Method, device and equipment for identifying multiple targets around electric shovel by fusing multi-modal data and medium

Abstract

The invention discloses an electric shovel surrounding multi-target identification method and device fusing multi-modal data, equipment and a medium, and relates to the field of target detection. Information data is preprocessed, a semantic mask is determined based on a semantic segmentation network, and cross-modal interaction from an image to a point cloud is carried out; according to the motion increment and the multi-modal data key features, determining a motion track set and a detection frame of each detection target around the electric shovel, and adopting an adaptive weight optimization mechanism to optimize the detection frame and a corresponding motion track based on a total loss function; an LM algorithm is adopted to carry out iterative optimization by taking a minimum total loss function of a multi-task loss system as a target, and secondary optimization detection frame track information and a detection frame after secondary optimization are determined, so that a multi-modal space-time depth fusion detector can be attached to appearance detection frames corresponding to detection targets around the electric shovel; and perception detection identification of a three-dimensional target is realized. According to the invention, multi-modal space-time deep fusion can be realized, and the detection precision is improved.

Description

Technical Field

[0001] This application relates to the field of target detection, and in particular to a method, apparatus, equipment and medium for identifying multiple targets around an electric shovel by fusing multimodal data. Background Technology

[0002] Open-pit mines, with their spacious operating areas and relatively enclosed environments, have become ideal testing and application scenarios for unmanned operation of large-scale electric shovels. The core of unmanned systems lies in the precise perception and rapid response of the unmanned electric shovel to the dynamic environment, requiring high-precision positioning, target recognition, and real-time acquisition of target location, size, and category. Three-dimensional target detection relies on sensors such as visual cameras and LiDAR to achieve three-dimensional environmental perception, directly determining the safety and reliability of the unmanned system. However, open-pit mines generally face harsh environmental challenges such as complex terrain, variable lighting, and dust interference. Therefore, designing a multimodal data adaptive fusion detector driven by an inertial measurement unit (IMU) to improve the accuracy and robustness of three-dimensional target detection around the shovel is crucial for ensuring the safe operation of unmanned electric shovels and achieving intelligent decision-making. This has significant theoretical and practical value for promoting the engineering application of intelligent unmanned mining technology.

[0003] To address the problem of 3D target detection in open-pit mines, current methods rely on visual cameras or LiDAR technology, extracting features through convolutional neural networks and deep point cloud networks, respectively. However, single-modal sensors have inherent limitations. On the one hand, while visual images possess order and density uniformity, they lack depth and reflection intensity information. On the other hand, although LiDAR point clouds contain 3D coordinates and depth information, they are prone to missing information in occluded or distant scenes, leading to missed detections. While cross-modal fusion utilizes mechanisms such as deep attention to optimize detection performance, it suffers from issues such as inconsistent spatiotemporal fusion, fixed feature fusion weights, and insufficient anti-interference generalization ability of the loss function. Summary of the Invention

[0004] The purpose of this application is to provide a method, device, equipment and medium for identifying multiple targets around an electric shovel by fusing multimodal data, which can achieve deep spatiotemporal fusion of multimodal data and improve detection accuracy.

[0005] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a method for identifying multiple targets around an electric shovel by fusing multimodal data. This method is applied to a multimodal spatiotemporal depth fusion detector. The multimodal spatiotemporal depth fusion detector is mounted on an open-pit mine electric shovel. The multimodal spatiotemporal depth fusion detector includes: a control module, a multimodal sensor, an image-point cloud feature extraction module, a spatiotemporal characteristic adaptive weight optimization module, and a multi-scale loss optimization module, all connected sequentially. The multimodal sensor, image-point cloud feature extraction module, spatiotemporal characteristic adaptive weight optimization module, and multi-scale loss optimization module are all connected to the control module. The method for identifying multiple targets around an electric shovel by fusing multimodal data includes: Acquire information data about the area surrounding the electric shovel; the information data includes: image information data collected by the camera included in the multimodal sensor, point cloud data collected by the lidar included in the multimodal sensor, and IMU data collected by the inertial measurement unit included in the multimodal sensor. The control module controls the image-point cloud feature extraction module to preprocess the information data, and the semantic segmentation network determines the semantic mask based on the preprocessed information data, and performs cross-modal interaction from image to point cloud to determine the key features of multimodal data; the key features of multimodal data include the optimal matching pair obtained by searching the similarity matrix of cross-modal features using the Hungarian algorithm; the preprocessed information data includes motion increments; Based on the control module's spatiotemporal characteristics, the adaptive weight optimization module determines the set of motion trajectories of each detection target around the electric shovel according to the motion increment and the key features of the multimodal data, and determines the detection box. Then, the adaptive weight optimization mechanism is used to optimize the detection box and the corresponding movement trajectory based on the total loss function to obtain the optimized detection box trajectory information and the optimized detection box. The control module controls the multi-scale loss optimization module, which employs the LM algorithm to minimize the total loss function of the multi-task loss system. Based on the optimized detection box trajectory information and the optimized detection box, iterative optimization is performed to determine the secondary optimized detection box trajectory information and the secondary optimized detection box. Among them, the multimodal spatiotemporal depth fusion detector uses the secondary optimized detection box. Based on the trajectory information of the secondary optimized detection box, it can fit the appearance detection boxes corresponding to each detection target around the electric shovel to achieve the perception, detection and recognition of three-dimensional targets.

[0006] Secondly, this application provides a multi-target identification device around an electric shovel that integrates multimodal data, comprising: The information data acquisition module is used to acquire information data around the electric shovel; the information data includes: image information data collected by the camera included in the multimodal sensor, point cloud data collected by the lidar included in the multimodal sensor, and IMU data collected by the inertial measurement unit included in the multimodal sensor. The cross-modal interaction module is used to preprocess the information data based on the image-point cloud feature extraction module controlled by the control module, determine the semantic mask based on the preprocessed information data using the semantic segmentation network, and perform cross-modal interaction from image to point cloud to determine key features of multimodal data; the key features of multimodal data include the optimal matching pair obtained by searching the similarity matrix of cross-modal features using the Hungarian algorithm; the preprocessed information data includes motion increments; The optimization module is used to determine the set of motion trajectories of each detection target around the electric shovel based on the motion increment and the key features of the multimodal data, and to determine the detection box, and to optimize the detection box and the corresponding movement trajectory based on the total loss function using an adaptive weight optimization mechanism, so as to obtain the optimized detection box trajectory information and the optimized detection box. The secondary optimization module, based on the control module's control of the multi-scale loss optimization module, employs the LM algorithm with the objective of minimizing the total loss function of the multi-task loss system. It iteratively optimizes the detection box trajectory information and the optimized detection box based on the optimized detection box trajectory information, determining the secondary optimized detection box trajectory information and the secondary optimized detection box. The multimodal spatiotemporal depth fusion detector uses the secondary optimized detection box trajectory information to fit the appearance detection boxes corresponding to each detection target around the electric shovel, thereby achieving the perception, detection, and recognition of three-dimensional targets.

[0007] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for multi-target recognition around an electric shovel by fusing multimodal data.

[0008] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for identifying multiple targets around an electric shovel by fusing multimodal data.

[0009] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, apparatus, device, and medium for multi-target recognition around an electric shovel by fusing multimodal data. Based on the collaborative operation of a control module, a multimodal sensor, an image-point cloud feature extraction module, a spatiotemporal adaptive weight optimization module, and a multi-scale loss optimization module connected sequentially, the method preprocesses the information data, determines semantic masks based on a semantic segmentation network, and performs cross-modal interaction from image to point cloud. Based on motion increments and key features of multimodal data, it determines the set of motion trajectories of each detected target around the electric shovel and defines detection boxes. An adaptive weight optimization mechanism is then used to optimize the detection boxes and their corresponding motion trajectories based on the total loss function. The LM algorithm is employed with the goal of minimizing the total loss function of the multi-task loss system. Iterative optimization is performed based on the optimized detection box trajectory information and the optimized detection boxes to determine the secondary optimized detection box trajectory information and the secondary optimized detection boxes. This allows the multimodal spatiotemporal deep fusion detector to fit the appearance detection boxes corresponding to each detected target around the electric shovel, achieving the perception, detection, and recognition of three-dimensional targets. Throughout the collaborative processing, multimodal spatiotemporal deep fusion is achieved, ultimately improving detection accuracy. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart of a method for identifying multiple targets around an electric shovel by fusing multimodal data; Figure 2 A flowchart illustrating the design of a multi-target identification method around an electric shovel that integrates multimodal data. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] In an exemplary embodiment, a method for identifying multiple targets around an electric shovel by fusing multimodal data is provided. This method is applied to a multimodal spatiotemporal depth fusion detector. The multimodal spatiotemporal depth fusion detector is mounted on an open-pit mine electric shovel. The detector includes a control module, a multimodal sensor, an image-point cloud feature extraction module, a spatiotemporal characteristic adaptive weight optimization module, and a multi-scale loss optimization module, all connected sequentially. The multimodal sensor, image-point cloud feature extraction module, spatiotemporal characteristic adaptive weight optimization module, and multi-scale loss optimization module are all connected to the control module.

[0015] like Figure 1 As shown, the method for identifying multiple targets around an electric shovel by fusing multimodal data includes: Step 100: Acquire information data around the electric shovel. The information data includes: image information data collected by the camera included in the multimodal sensor, point cloud data collected by the lidar included in the multimodal sensor, and IMU data collected by the inertial measurement unit included in the multimodal sensor.

[0016] Step 200: The control module controls the image-point cloud feature extraction module to preprocess the information data, and uses a semantic segmentation network to determine the semantic mask based on the preprocessed information data, and performs cross-modal interaction from image to point cloud to determine the key features of multimodal data. The key features of multimodal data include the optimal matching pair obtained by searching the similarity matrix of cross-modal features using the Hungarian algorithm; the preprocessed information data includes motion increments.

[0017] Step 300: Based on the control module, the spatiotemporal characteristic adaptive weight optimization module determines the set of motion trajectories of each detection target around the electric shovel according to the motion increment and key features of multimodal data, and determines the detection box. Then, the adaptive weight optimization mechanism is adopted to optimize the detection box and the corresponding movement trajectory based on the total loss function, so as to obtain the optimized detection box trajectory information and the optimized detection box.

[0018] Step 400: Based on the control module, the multi-scale loss optimization module uses the LM algorithm with the objective of minimizing the total loss function of the multi-task loss system. Iterative optimization is performed based on the optimized detection box trajectory information and the optimized detection box itself to determine the secondary optimized detection box trajectory information and the secondary optimized detection box. The multimodal spatiotemporal depth fusion detector uses the secondary optimized detection box trajectory information to fit the appearance detection boxes corresponding to each detection target around the electric shovel, thereby achieving the perception, detection, and recognition of three-dimensional targets. Using the method mentioned in this application to locate and identify targets around the electric shovel ensures the safe operation of the electric shovel.

[0019] The expression for the total loss function of the multi-task loss system is: .

[0020] in, The total loss function of the multi-task loss system; The vertical linear alignment loss function; Let IMU motion consistency loss function be used. This is the distance regression decay loss function; These are the weights corresponding to the vertical linear alignment loss function; These are the weights corresponding to the IMU motion consistency loss function; These are the weights corresponding to the distance regression decay loss function.

[0021] In one embodiment, the image-point cloud feature extraction module is controlled by the control module to preprocess the information data, and a semantic mask is determined based on the preprocessed information data using a semantic segmentation network. Cross-modal interaction between the image and point cloud is also performed to determine key features of the multimodal data. Specifically, this includes: Based on the control module, the image-point cloud feature extraction module is controlled, and the quaternion method is used to solve the IMU data to obtain the three-dimensional attitude angles. The three-dimensional attitude angles include: roll angle, pitch angle and yaw angle.

[0022] The z-axis feature vector is determined. The z-axis feature vector is obtained by normalizing and concatenating the triaxial acceleration and angular velocity contained in the IMU data using a multi-scale convolutional network.

[0023] Based on the z-axis feature vector, the yaw angle is enhanced using image information data, and a feedforward-feedback joint architecture is used to correct and optimize the yaw angle, yielding the corrected result. The feedforward-feedback joint architecture includes a feedforward path and a feedback path; the feedforward path uses an extended Kalman filter to update the state estimate; the feedback path is used for the linear mapping from features to attitude correction.

[0024] The corrected results are weighted and fused to obtain attitude estimation output information, and pre-integration is performed in adjacent time intervals within a set time interval to obtain pre-processed information data.

[0025] Based on the semantic segmentation network, semantic masks are determined according to the preprocessed information data, and cross-modal interaction between images and point clouds is performed to determine key features of multimodal data.

[0026] As an optional implementation, a semantic segmentation network is used to determine semantic masks based on preprocessed information data and to perform cross-modal interaction from image to point cloud to determine key features of multimodal data, specifically including: The point cloud data is projected onto the coordinate system corresponding to the camera in the multimodal sensor based on the preprocessed information data, so as to achieve spatial alignment of the multimodal information data.

[0027] A convolutional neural network is used to extract features from spatially aligned image data, and a semantic segmentation network is used to determine the semantic mask of the image. Semantic regions are visualized and rendered using class activation mapping to obtain a semantic region heatmap.

[0028] Based on semantic region heatmaps, spatially aligned point cloud data is projected onto the image plane, and a one-to-one correspondence is established between the point cloud and image pixels. Furthermore, based on the image feature information after feature extraction and the semantic analysis results after determining the image semantic mask, a downsampling strategy is used to filter and determine the semantic information-filtered data.

[0029] Based on semantic information, data is filtered to perform cross-modal interaction from image to point cloud in order to determine key features of multimodal data.

[0030] Based on semantic information, data is filtered to perform cross-modal interaction from image to point cloud to determine key features of multimodal data, specifically including: The point cloud feature information and image feature information after feature extraction are encoded respectively, and a cross-modal interaction mechanism is adopted to filter data based on semantic information to enhance the point cloud features and obtain enhanced feature information.

[0031] The similarity matrix of cross-modal features is calculated based on the enhanced feature information; the Hungarian algorithm is used to search the similarity matrix of cross-modal features to obtain the optimal matching pair, which is then identified as the key feature of the multimodal data.

[0032] In one embodiment, the control module controls the spatiotemporal characteristic adaptive weight optimization module to determine the set of motion trajectories of each detected target around the electric shovel based on the motion increment and key features of multimodal data, and to determine the detection box. Furthermore, an adaptive weight optimization mechanism is used to optimize the detection box and its corresponding movement trajectory based on the total loss function, resulting in optimized detection box trajectory information and optimized detection boxes. Specifically, this includes: Based on the control module, the spatiotemporal characteristic adaptive weight optimization module optimizes the trajectory smoothness by combining the pose transformation matrix with the Lie algebra pose increment according to the motion increment. It also completes cross-frame target two-dimensional association based on the key features of multimodal data to determine the motion trajectory set of each detected target around the electric shovel and determine the detection box.

[0033] An adaptive weight optimization mechanism is employed. Based on the motion trajectory set and detection boxes, the detection boxes and their corresponding movement trajectories are optimized using a total loss function, resulting in optimized detection box trajectory information and optimized detection boxes. The expression for the total loss function is as follows: .

[0034] in, This is the total loss function; The dynamic weights of the motion continuity loss function; This is the motion continuity loss function; The dynamic weights of the consistency loss function; This is a joint loss function based on the consistency loss function.

[0035] The adaptive weight optimization mechanism is a quantization standard mechanism for 3D target detection boxes determined by using a weight voting method and integrating six quantization indicators. The quantization standard mechanism includes the point cloud weight ratio and image weight ratio based on occlusion state and distance segmentation. The six quantization indicators include: point cloud coverage, point cloud density deviation rate, noise point ratio, gray-scale contrast deviation, texture entropy value, and detail discernibility.

[0036] This application proposes a multimodal adaptive fusion 3D target detector (i.e., a multimodal spatiotemporal depth fusion detector) that integrates grayscale images (i.e., image information data), point cloud data, and IMU data. Based on IMU data acquired by an inertial measurement unit, this detector achieves spatiotemporal alignment of motion state constraints, adaptive weight optimization, and multi-task loss weight optimization, effectively addressing the shortcomings of existing fusion schemes. It provides a high-precision 3D target detection solution for the application of unmanned mining shovels in mining scenarios, ensuring the accuracy of equipment operation and scene adaptability. The multimodal spatiotemporal depth fusion detector mentioned in this application consists of an image-point cloud feature extraction module, a spatiotemporal characteristic adaptive weight optimization module, and a multi-scale loss optimization module. Through modular collaborative design, it achieves a comprehensive improvement in detection performance.

[0037] In the image-point cloud feature extraction module, firstly, the inertial measurement unit (IMU) collects acceleration and angular velocity data in real time and calculates the attitude angles. The roll and pitch angles are corrected using acceleration components, while the yaw angle is optimized based on attitude information by designing a Z-axis multi-scale convolution kernel, thus achieving accurate correction of the point cloud attitude. Secondly, the corrected attitude angles are pre-integrated to generate motion increments to strengthen motion constraints. Simultaneously, the IMU is used to achieve time synchronization of multimodal data, and image texture information is combined to guide the feature extraction process of key points in the point cloud. Finally, the collected point cloud data, image information data, and IMU data are encoded and processed separately. A cross-modal interaction mechanism is introduced to construct a two-way interaction channel between point cloud and image features. By calculating the similarity matrix between features, efficient search for optimal matching pairs is achieved.

[0038] In the spatiotemporal adaptive weight optimization module, firstly, based on the motion increment output by the image-point cloud feature extraction module, the smoothness of the target trajectory is optimized by combining it with a preset pose transformation formula. Simultaneously, the optimal matching pair output is used to perform cross-frame target two-dimensional association, and the trajectory set is determined through result filtering and integration. Secondly, a dual constraint mechanism is constructed: on the one hand, a motion continuity loss function is established by defining the motion error in the trajectory's time dimension; on the other hand, reflectivity-semantic consistency constraints are implemented, calculating the consistency loss corresponding to reflectivity features and semantic labels respectively and weighting and fusing them into a joint loss. Based on this, according to the real-time reliability assessment results of IMU and visual data, the weight ratio of the two types of losses in the total loss is dynamically adjusted, constructing a global loss function to iteratively optimize the target trajectory. Finally, based on the actual occlusion level in the mining environment, the occlusion type (no occlusion, few occlusions, many occlusions) of the current scene is pre-assessed; then, based on the assessment results, the fusion weights of point cloud and image are dynamically adjusted during the feature fusion stage, ultimately achieving an adaptive distribution of multimodal fusion weights to match the perception requirements under different occlusion scenarios.

[0039] In the multi-scale loss optimization module, a multi-task loss system (including vertical beam alignment loss, IMU motion consistency loss, and distance decay regression loss) is constructed based on real-time data quality indicators such as point cloud density, image sharpness, and IMU noise level. The weights of each loss are dynamically allocated according to data quality to optimize the total loss function. The total loss function is iteratively minimized through the Levenberg-Marquardt algorithm (LM), ultimately achieving iterative improvement in the accuracy of the detection boxes.

[0040] In practical applications, the design process of the method mentioned in this application is as follows: Figure 2 As shown.

[0041] Multimodal sensor configuration: Multimodal input: On one hand, the visual camera captures the environmental scene through the principle of perspective projection, and the acquired Grayscale image is represented as a three-dimensional tensor G ( In this model, H and W represent the height and width of the image, respectively, and C=1 represents a single Grayscale channel. On the other hand, LiDAR scans the three-dimensional environment by emitting laser beams, generating a three-dimensional spatial representation as a disordered and unevenly dense point cloud. The spatial distribution of the point cloud directly reflects the geometric contour information of targets in the environment. Point cloud data can be represented as a set... Each point contained therein, i.e. All are four-dimensional vectors Where N is the total number of point clouds, x, y, z represent spatial coordinates, and r is the reflection intensity value, whose distribution density changes dynamically with the target distance and surface characteristics.

[0042] Camera-LiDAR-IMU Projection: This demonstrates the scanning results of the camera and LiDAR in the same scene, as well as the coordinate transformation relationship between the two modes after IMU correction. This represents the calibration coordinate transformation matrix of the Lidar Coordinate System (LCS) relative to the IMU coordinate system. This represents the calibration coordinate transformation matrix of the Camera Coordinate System (CCS) relative to the IMU coordinate system. Based on the principle of homogeneous coordinate transformation, this is the calibration coordinate transformation matrix of the LiDAR coordinate system relative to the camera coordinate system. It is possible Perform the calculation.

[0043] In addition, The inherent matrix of the visual camera, which represents the three-dimensional points of the LiDAR. The process of projecting onto the image plane It can be represented as: .

[0044] in, For point Depth in the visual camera coordinate system is used to normalize image coordinates. For point The coordinates of the projected points in the image. Through homogeneous coordinate transformation, the three-dimensional points in the LiDAR coordinate system are mapped to the two-dimensional image plane of the vision camera, achieving spatial alignment of multimodal data.

[0045] Module 1: Image-Point Cloud Feature Extraction (Image-Point Cloud Feature Extraction Module): To address the shortcomings of traditional feature extraction methods, such as low sensitivity to pose changes, insufficient adaptive focusing capability of key point cloud regions, and inadequate fusion of point cloud geometry and image semantic information, this application proposes an image-point cloud feature interaction and extraction module, namely, the image-point cloud feature extraction module. This module first performs computation and pre-integration on IMU data, then achieves pose correction and feature point selection through Z-axis convolution and multi-source data alignment, and finally outputs key point features based on a bidirectional cross-modal interaction mechanism. This module consists of three parts: 1) an IMU pose-aware Z-axis multi-scale convolutional network; 2) IMU pre-integration and image texture-guided weighted key point cloud regions; and 3) IMU-driven cross-modal bidirectional interaction.

[0046] IMU pose-aware Z-axis multi-scale convolutional network: Triaxial acceleration is synchronously acquired via IMU. and angular velocity After Kalman filtering for noise reduction and time alignment preprocessing, the quaternion method is used to calculate the three-dimensional attitude angles (roll angle is...). The pitch angle is yaw angle is ).

[0047] .

[0048] in, Angular velocity in quaternion form All are four-element rotation axis vectors. It is a four-element angle scalar. The roll angle is calculated using the acceleration component. and pitch angle Optimize and correct. ω represents the angular velocity corresponding to the x, y, and z axes, respectively; T is the transpose.

[0049] To address the issue of yaw angle being susceptible to interference, a multi-scale convolutional network along the Z-axis is constructed for enhancement: the Z-axis angular velocity and acceleration data are normalized and concatenated into a feature vector. . This represents the acceleration along the z-axis.

[0050] Input one-dimensional convolution kernels of different scales to extract high-frequency, mid-frequency and low-frequency features respectively, and fuse them to form multi-scale Z-axis features.

[0051] After mean normalization to reduce the interference of extreme values ​​on the data distribution, and then unit variance standardization to preserve the distribution trend of the original data, the data is finally made to approximately follow a normal distribution. These are the mean values ​​of angular velocity and acceleration along the Z-axis, respectively. These are the variances of angular velocity and acceleration along the Z-axis, respectively. For the first The angular velocity of each sample data point on the Z-axis For the first The acceleration of each sample data point along the Z-axis is used to obtain the normalized feature vector. , This represents the normalized angular velocity corresponding to the z-axis. This represents the normalized acceleration along the z-axis. Since images acquired by the visual camera exhibit better features along the z-axis, the z-axis features of the grayscale image are used to enhance the yaw angle output by the IMU, thereby improving its anti-interference capability. The input is fed into three sets of one-dimensional convolutional kernels of different sizes to obtain three sets of features: small-sized convolutional kernels capture high-frequency short-temporal features. Medium-sized convolutional kernels extract mid-frequency motion features Large-size convolutional kernels are used to mine low-frequency trend features. The three types of features are concatenated to form Z-axis multi-scale features. : .

[0052] in, , as well as All of these are convolution kernel weights.

[0053] Subsequently, a feedforward-feedback joint architecture is used to optimize the yaw angle: the feedforward path uses multi-scale features as observations and updates the state estimate through an extended Kalman filter; the feedback path establishes a linear mapping from features to attitude correction, dynamically adjusts the correction amount, and finally weights and fuses the two results to improve the robustness of attitude estimation. The feedforward path directly uses the Z-axis features extracted by multi-scale convolution as state observations and updates the yaw angle estimate through an extended Kalman filter framework. Let the state vector in the extended state-space model be... State transition equation : .

[0054] in, This is the state transition equation from the previous time step; To differentiate the yaw angle, represents the yaw rate, and represents the rate of change of the yaw angle with respect to time.

[0055] In the observation model, the Z-axis feature is used as the observed value. : .

[0056] The final yaw angle is estimated using Kalman filtering and feedforward. : .

[0057] Wherein, the state transition matrix , For process noise, the observation matrix ; To observe noise. The state estimation vector based on observations at time k. The first element in the equation (which is the estimated yaw angle); To set the time interval.

[0058] The feedback path establishes a linear mapping model from the feature space to the attitude correction, dynamically adjusts the correction weights using Kalman gain, and records the yaw angle calculated in the traditional solution. : .

[0059] Linear mapping from features to corrections : .

[0060] Feedback on final yaw angle estimation : .

[0061] in, This is the estimated yaw angle value after Kalman filtering at the previous moment; The three types of features are spliced ​​together to form Z-axis multi-scale features; Zero bias, For correction factor, All parameters are learnable. Finally, the two correction results are weighted and fused to form a pose estimation output with strong anti-interference capabilities.

[0062] IMU pre-integration and image texture-guided weighted point cloud key region: First, pre-integration is performed on the preprocessed data to obtain motion increments such as rotation, velocity, and displacement between adjacent time intervals: Pre-integration is performed on adjacent time intervals. Set time interval : .

[0063] Relative rotation increment : .

[0064] Relative velocity increment : .

[0065] Acceleration in an inertial frame : relative displacement increment : .

[0066] in, for relative time The rotation matrix at time step. It is the angular velocity vector; This is the vector of gravitational acceleration; The acceleration is in the body coordinate system.

[0067] Subsequently, iterative optimization of the IMU pre-integration was performed using subsequent frame data to effectively suppress accumulated errors, strengthen constraints on high-frequency motion states, and improve the stability of attitude and motion parameters. Next, multi-source data alignment and fusion were performed. The spatiotemporal synchronization relationship between the visual camera, LiDAR, and IMU was clarified through extrinsic parameter calibration. First, temporal interpolation was performed on the LiDAR point cloud to compensate for the time difference. Then, based on the IMU pre-integration results, the point cloud was accurately projected onto the current visual camera coordinate system to achieve spatial alignment of multimodal data. Then, image feature extraction and semantic analysis were performed. A Convolutional Neural Network (CNN) was used to extract key points in the image and calculate the corresponding feature descriptors. At the same time, a semantic segmentation network was used to generate an image semantic mask to filter out semantically salient regions such as personnel and mining trucks, reducing irrelevant background interference. Subsequently, Class Activation Mapping (CAM) was used to visualize the semantically salient regions of interest to the CNN. On this basis, a heatmap corresponding to the semantically salient region was generated through three auxiliary operations: image preprocessing, activation map postprocessing, and visualization rendering. After generating the heatmap, we further conduct point cloud feature mapping guided by the heatmap of semantically salient regions: First, we project the preprocessed and aligned point cloud onto the image plane to establish a one-to-one correspondence between the point cloud and image pixels. Then, combining image features and semantic analysis results, we use a uniform downsampling strategy to avoid point cloud feature aggregation and ensure uniform spatial distribution, and select 3D points in the point cloud that correspond to the semantically salient regions of the image. Finally, we calculate 3D descriptors for the selected point cloud features and fuse them with the feature descriptors of the corresponding image regions, which significantly enhances cross-modal matching ability and completes the extraction of multimodal key point features.

[0068] IMU-driven cross-modal bidirectional interaction: When performing cross-modal feature representation based on the aforementioned content, point cloud and image features need to be encoded separately and then fused with IMU features. The specific process is as follows: The encoding of point cloud features is divided into local geometric features and global contextual geometric features: the former extracts local geometric features through the PointNet++ network. The latter, on the other hand, uses Transformer or graph neural networks to aggregate global context to generate... Image feature encoding is divided into semantic features and location encoding: semantic features are obtained by extracting multi-scale feature maps using CNNs. Positional encoding preserves pixel spatial relationships by adding spatial location information, encoding the IMU pre-integration result into a feature vector. The data is mapped to the same dimensional space as the point cloud and image features via a fully connected layer, achieving IMU feature fusion. Based on this, a cross-modal interaction mechanism is applied: on one hand, geometrically guided semantics is achieved through a point cloud-to-image cross-modal interaction mechanism—first, the point cloud is projected onto the image plane to establish a correspondence, and then attention weights are calculated. : .

[0069] in, For the first Point cloud features, For the first Image features, A multilayer perceptron for key networks. It is a multilayer perceptron for querying networks. The activation function is used. Weighted aggregation of point cloud geometric features enhances the geometric consistency of image semantic features. : .

[0070] On the other hand, semantic-guided geometry is achieved through a cross-modal interaction mechanism from image to point cloud—first, semantic features are back-projected into three-dimensional space, and then attention weights are calculated. : .

[0071] Filtering and enhancing point cloud features using semantic information : .

[0072] Improve the semantic discriminative power of geometric features. Then, calculate the correlation between IMU features and point cloud / image features. and : .

[0073] .

[0074] in, The features of the inertial measurement unit are processed using a multilayer perceptron; To process global point cloud features using a multilayer perceptron; The global image features are processed using a multilayer perceptron; This is the activation function.

[0075] Adjust attention weights and : .

[0076] .

[0077] The aggregated features are then updated to complete IMU-guided attention adjustment. Finally, geometric-semantic joint matching is achieved through bidirectional matching and optimization: cross-modal similarity matrix is ​​calculated based on features enhanced by bidirectional attention. : .

[0078] in, This is the guiding feature for the i-th point cloud; Let j be the guiding feature of the image.

[0079] The Hungarian algorithm is used to search for the optimal matching pair, and the key features are finally extracted.

[0080] Module 2: Spatiotemporal Adaptive Weight Optimization, i.e., the Spatiotemporal Adaptive Weight Optimization module: Existing methods for real-time data processing after feature extraction suffer from insufficient spatiotemporal coherence and fixed point cloud-image fusion weights. Therefore, a spatiotemporal adaptive weight optimization module is proposed. This module predicts the target's motion trajectory based on IMU data and enhances the spatiotemporal coherence of detection by constructing a dual constraint of motion continuity and reflectivity-semantic consistency loss. Simultaneously, an innovative adaptive weight optimization mechanism is designed, which adaptively adjusts the fusion weights of the image and point cloud in segments according to the target's distance, taking into account the occlusion state of the working scene, thus achieving real-time data processing. The proposed spatiotemporal adaptive weight optimization module consists of two parts: 1) IMU trajectory prediction and dual constraints to enhance the spatiotemporal coherence of detection; and 2) an adaptive weight optimization mechanism driven by the scene's occlusion state.

[0081] IMU Trajectory Prediction and Double Constraint Enhancement of Spatiotemporal Coherence: To optimize the trajectory and suppress long-term integral drift, firstly, before the large mining electric shovel begins operation, it is controlled to complete one full rotation. This pre-operation acquires the rotational radii data of the lidar installation location during subsequent operations, providing a priori basis for subsequent trajectory optimization. Then, based on the rotation and displacement increments, a recursive formula is used... : .

[0082] pose transformation matrix Combine Li algebra pose increment This indicates that the trajectory smoothness is optimized, whereby... It is a rotation increment matrix; This is the translation increment matrix; simultaneously, relying on the optimal matching pairs output by the image-point cloud feature interaction and extraction module, cross-frame target two-dimensional association is completed to construct a trajectory set. ;in, All are trajectory elements in the trajectory set, where n is the number of elements in the set. Next, motion continuity constraints are applied. First, feature points (including keypoints and descriptors) are extracted from the image sequence, and their motion is tracked using a RAFT network. Cross-frame projection is initialized using IMU trajectory prediction, and mismatched points are removed using RANSAC. Finally, pixel-level motion error is defined. : .

[0083] in, for The coordinates of the point observed on the image plane at that moment. for The coordinates of the point observed on the image plane at that moment. For projection function, The distance is the Euclidean square distance. Let be the pose transformation matrix from time t to time t+1. Also, calculate the continuity loss function. : .

[0084] in, For a moment arrive The velocity variation error was then investigated. Subsequently, reflectivity-semantic consistency constraints were implemented. First, the reflectivity of the LiDAR point cloud was extracted and projected onto the image plane to obtain the reflectivity value of the target point. Constructing a reflectance map And calculate the reflectivity change between adjacent frames. : .

[0085] in, Let be the reflectance of the (t+1)th frame; Let be the reflectance of the t-th frame.

[0086] Then, semantic features are extracted using a DeepLab semantic segmentation network to obtain target category labels and construct semantic consistency metrics. Then, the consistency loss function containing reflectivity and semantic constraints is calculated. and and joint losses : .

[0087] .

[0088] .

[0089] in, This is a semantic consistency metric, representing semantic similarity. Weights related to semantic categories; This represents the number of reflectance variation terms involved in the calculation. The reflectance change of the j-th adjacent frame; For the corresponding Weighting coefficients; Semantic categories are defined; finally, dynamic weight adjustment and joint optimization are performed, first evaluating IMU and visual reliability. and : .

[0090] .

[0091] in, The pre-integrated covariance matrix, For image gradient variance, As an ambiguity index, the dynamic weights of the motion continuity loss function are then calculated. : .

[0092] Dynamic weights of the consistency loss function : .

[0093] Total loss function : .

[0094] This optimizes the movement trajectory.

[0095] 2) Adaptive weight optimization mechanism driven by scene occlusion state: To achieve dynamic weight allocation of point cloud-image features, the strategy process is as follows: First, baseline data collection was conducted: Under clear, unobstructed weather conditions, point cloud and grayscale image data of mine cards, other operating equipment, and personnel at multiple locations within the mining area were collected to establish "unobstructed baseline values," including the number and density of point clouds in the near distance [0, 75m] and the far distance (75, 150m] range, as well as the grayscale contrast, texture entropy value, and total number of identifiable details of the target image within each distance range.

[0096] Then, data preprocessing is performed: point cloud is statistically filtered to remove discrete noise points, and then radius filtering is used to retain valid points around the target; image is grayscale normalized to eliminate illumination interference, and ROI is extracted through target localization, and sharpness index is calculated only for ROI.

[0097] Secondly, segmented index calculation is performed: for the preprocessed point cloud and image, key indexes are calculated in segments according to "near distance" and "long distance", including lidar point cloud indexes and grayscale image indexes.

[0098] Calculation of lidar point cloud metrics: Point cloud coverage : .

[0099] in, This represents the number of point clouds representing the target at the current distance segment. This represents the number of unobstructed point clouds within the same distance segment.

[0100] Point cloud density deviation rate : .

[0101] in, The current distance segment point cloud density, This is the baseline density for the same distance segment.

[0102] Noise point percentage : .

[0103] in, This represents the number of noise points removed after statistical filtering of the current distance segment. This represents the current distance from the total number of point clouds.

[0104] Grayscale image index calculation: Gray-scale contrast deviation : .

[0105] in, , The contrast of the current ROI. As a reference contrast for the same distance segment, , These are the maximum and minimum grayscale values ​​within the ROI.

[0106] Texture entropy : .

[0107] Where L=256 (grayscale level). Let be the probability of an element in the gray-level co-occurrence matrix. The gray-level co-occurrence matrix is ​​set to a distance of 1 and an angle of 0.

[0108] Detail discernibility : .

[0109] in, This represents the number of reference details that can be identified in the current image. The total number of baseline details.

[0110] Then, the occlusion status is determined: a "weighted voting method" is used to integrate 6 quantitative indicators, each with a weight of 1. (Total weight 1), the threshold for judging the occlusion status of a single indicator is shown in Table 1. Based on the calculated value, it is judged as "no occlusion", "little occlusion", and "more occlusion", and is scored as 3 points, 2 points, and 1 point respectively. The total score is... for: .

[0111] in, For the first The scores of each indicator.

[0112] State classification: Unobstructed (Most indicators meet the unobstructed standard) minimal obstruction (Some indicators deviate from the benchmark but are not seriously missing), and there is significant occlusion. (Most indicators are severely missing, and the validity of the data has been significantly reduced.)

[0113] Table 1 Thresholds for Determining Occlusion Status

[0114] Based on the occlusion state and distance segmentation determined above, the weight allocation criteria for point cloud and image are as follows: Unobstructed state (maximizing complementary advantages): Near distance: Image semantic detail extraction has significant advantages, and point cloud provides 3D accuracy assistance for security verification. Weight allocation is 50%-60% for image (dominant semantic recognition) and 40%-50% for point cloud (assisting accuracy verification). Long distance: Point cloud has strong penetration and no visual blur, and 3D contour output is accurate; the semantic supplementary value of image decreases with distance. Weight allocation is 55%-65% for point cloud (dominant localization and recognition) and 35%-45% for image (assisting semantic classification).

[0115] Less obstruction state (balanced and coordinated anti-interference): Near distance: The 3D accuracy of point clouds and the semantic advantages of images are slightly weakened but still effective. A balanced weight distribution is 45%-55% for point clouds and 45%-55% for images. Far distance: Point clouds are more resistant to dust interference, but the detail resolution of images is significantly reduced. A weight distribution is 60%-70% for point clouds and 30%-40% for images (only as supplementary information).

[0116] Multiple occlusion states (point cloud dominant information extraction): Near range: Point clouds, after filtering, can still extract effective 3D information, which is the core guarantee for safe operation; images can only confirm the existence of the target. The weight allocation is 70%-80% for point clouds and 20%-30% for images (redundancy supplementation). Far range: Point clouds can only construct a rough 3D outline of the target, and images completely lose their semantic extraction value. The weight allocation is 90%-95% for point clouds (the only reliable source of information) and 5%-10% for images (minimum retention).

[0117] The weight distribution of point cloud and image is allocated by adaptively adjusting the network within the aforementioned dynamic weight range, as follows: Input layer: The core input is “occlusion status (no occlusion / little occlusion / more occlusion) + distance segment (close distance 0-75m / far distance 75-150m)”. This input does not require additional complex index calculations and can be directly extracted from the occlusion judgment results completed in the early stage and the ranging data output by the lidar in real time, which effectively simplifies the input acquisition process.

[0118] Decision layer: Based on the combination type of "occlusion state + distance segmentation", directly match the preset point cloud and image basic weight intervals; on this basis, for the matched point cloud basic weight interval and image basic weight interval, determine the midpoint of each interval and the point cloud weight midpoint. and image weight midpoint This provides a benchmark parameter for subsequent fine-tuning of the weights.

[0119] A value coefficient K is introduced to achieve fine-tuning of the weights based on the midpoint of the basic weight interval. K is used to quantify the "reliability advantage of the point cloud relative to the image," and its value range is determined by both the occlusion state and the target distance, as follows: Unobstructed close-range: Image semantic extraction has a slight advantage in unobstructed scenes. Taking a negative value for K makes the image weight slightly higher than that of the point cloud after fine-tuning.

[0120] Unobstructed long-distance: The advantages of long-distance point cloud 3D localization are significant in unobstructed scenes. Taking a positive value for K makes the point cloud weight slightly higher than that of the image after fine-tuning.

[0121] Less occlusion at close range: Occlusion interference is slight, point cloud accuracy is close to image semantic value, K fluctuates slightly around zero, and the weights are basically balanced.

[0122] Less occlusion at long distances: Occlusion interference is slight, but the image detail recognition decreases at long distances. Taking a positive value for K makes the point cloud weight slightly higher after fine-tuning.

[0123] Numerous close-range occlusions: Severe occlusion interference means that the point cloud is the core effective information source. Taking a large positive value for K makes the point cloud weight significantly higher than that of the image after fine-tuning.

[0124] Significant occlusion at long distances: Occlusion interference is severe and the point cloud at long distances is only slightly affected, but it is still the main source of information. K is taken as a small positive value to ensure that the point cloud weight still dominates after fine-tuning.

[0125] During the fine-tuning process, it is necessary to ensure that the final weight always falls within the preset basic weight range and avoids exceeding the reasonable range.

[0126] The weight value is calculated based on the midpoint of the basic weight interval of the point cloud and the image, and quantitatively adjusted by the value coefficient K to finally obtain the fusion weight value of the point cloud and the image. as well as .

[0127] .

[0128] Output layer: The final output is the point cloud weight ratio and image weight ratio based on the occlusion state and distance segmentation, which provides a quantitative basis for the subsequent establishment of 3D object detection boxes.

[0129] Module 3: Multi-scale loss optimization, i.e., the multi-scale loss optimization module: To address the limitations of static loss functions in secondary data processing methods, particularly their inability to adaptively adjust based on differences in data modalities or optimization requirements for different loss terms, which restricts the generalization and adaptability of the methods, a multi-scale loss optimization module is proposed. This module constructs a multi-task loss system comprising vertical beam alignment loss, IMU motion consistency loss, and distance decay regression loss, and dynamically adjusts the multi-modal loss weights based on real-time data quality. Finally, secondary optimization is performed on the output of the spatiotemporal adaptive weight optimization module, effectively improving the accuracy of 3D target detection. The proposed multi-scale loss optimization achieves optimization by constructing a dynamic weight multi-task loss system.

[0130] Construction and optimization of dynamic weighted multi-task loss system: First, multimodal data preprocessing and quality assessment are performed. Point cloud data is processed and density is calculated. The three-dimensional coordinates of each point cloud beam are extracted by dividing the laser beams by vertical angle. Point cloud vertical angle discretization and grouping are then performed. : .

[0131] in, This is the original point cloud collection. For the first The center angle of each wire harness For angular intervals, The point cloud in the original point cloud set, for The vertical angle is then used, and the density is calculated by the spatiotemporal characteristic adaptive weight optimization module. Finally, the Laplacian operator is used to evaluate the image sharpness index. : .

[0132] in, Input image (grayscale). The Laplace operator is used to estimate IMU noise using the trace of the pre-integrated covariance matrix. : .

[0133] in, The velocity variation is used as the basis for the calculation. Then, a multi-task loss system is constructed, first constraining the vertical alignment of adjacent frames' vertical bundles to enhance the consistency of the point cloud structure using a vertical linear alignment loss function. : .

[0134] .

[0135] in, For the number of wire harnesses, The number of points in each point cloud. Here is the pose transformation matrix. The transformed point coordinates are the pose transformation matrix. The partial derivatives; and For corresponding points in the same bundle of point clouds in adjacent frames, density weighting : .

[0136] in, For the first Cluster cloud density, The density is the highest among all point cloud densities; the IMU motion consistency loss function is then constrained to ensure consistency between visually estimated pose and IMU pre-integration results. : .

[0137] in, and For IMU pre-integration of pose and velocity, and For visual estimation results, The noise standard deviation is given by Update. This is the proportionality coefficient; This is a bias term.

[0138] Finally, lower weights are assigned to distant point clouds to adapt the distance regression attenuation loss function to the density decay characteristics of point clouds. : .

[0139] in , For the true value, For predicted values, To predict the target number, This is the attenuation coefficient.

[0140] Secondly, dynamic weight generation and adaptive adjustment are performed, starting with the quality indicators. , and Normalization: .

[0141] .

[0142] .

[0143] in, , These are the density values ​​for the near-field region and the density values ​​for the far-field region, respectively. This represents the upper limit of image sharpness. Set the IMU noise tolerance threshold, and then generate the weights corresponding to each loss function. , and : .

[0144] .

[0145] .

[0146] in, , , These are the adjustment parameters for point cloud, image, and IMU, respectively. To find the minimum value (to avoid numerical stability issues caused by a denominator of 0). Finally, calculate the total loss function. : .

[0147] The LM algorithm is used to optimize the state variables. The total loss function is minimized through iteration, and the weight parameters are dynamically adjusted in real time using a feedback mechanism to optimize the accuracy of the detection box.

[0148] In summary, this method not only provides a new 3D target detection technology solution for unmanned operation of large electric shovels in open-pit mines, but also further improves the multimodal fusion theory of visual cameras, lidar, and IMU, supplementing multi-sensor collaborative perception. Future work will focus on optimization in three aspects: first, improving the cross-modal bidirectional interaction mechanism to strengthen the spatiotemporal correlation of multimodal features; second, optimizing the refined adaptive strategy of the spatiotemporal characteristic adaptive weight optimization module and the loss function design of the MIO module to improve the accuracy of weight allocation and loss constraints; and third, improving the detection box correction algorithm under complex terrain to generate detection boxes that better match the actual appearance of the target, further improving the accuracy of 3D target detection. The subsequent work will promote the integration of this multimodal fusion method into the unmanned operation process of large electric shovels in open-pit mines, providing technical support for the safe operation of vehicles on complex unstructured roads and effectively avoiding potential safety risks such as collisions.

[0149] In one exemplary embodiment, a multi-target identification device around an electric shovel that fuses multimodal data is also provided, comprising: The information data acquisition module is used to acquire information data around the electric shovel; the information data includes: image information data collected by the camera included in the multimodal sensor, point cloud data collected by the lidar included in the multimodal sensor, and IMU data collected by the inertial measurement unit included in the multimodal sensor.

[0150] The cross-modal interaction module is used to preprocess information data based on the image-point cloud feature extraction module controlled by the control module, and to determine semantic masks based on the preprocessed information data using a semantic segmentation network, as well as to perform cross-modal interaction between the image and the point cloud to determine key features of multimodal data. Key features of multimodal data include the optimal matching pair obtained by searching the similarity matrix of cross-modal features using the Hungarian algorithm; the preprocessed information data includes motion increments.

[0151] The optimization module is used to determine the set of motion trajectories of each detection target around the electric shovel based on the motion increment and key features of multimodal data, and to determine the detection box. The adaptive weight optimization mechanism is used to optimize the detection box and the corresponding movement trajectory based on the total loss function, so as to obtain the optimized detection box trajectory information and the optimized detection box.

[0152] The secondary optimization module, based on the control module, uses the LM algorithm to minimize the total loss function of the multi-task loss system. It iteratively optimizes the detection box trajectory information and the optimized detection box based on the optimized detection box trajectory information, determining the secondary optimized detection box trajectory information and the secondary optimized detection box. The multimodal spatiotemporal depth fusion detector uses the secondary optimized detection box trajectory information to fit the appearance detection boxes corresponding to various detection targets around the electric shovel, thus achieving the perception, detection, and recognition of three-dimensional targets.

[0153] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0154] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0155] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for identifying multiple targets around an electric shovel by fusing multimodal data, characterized in that, The multi-target recognition method around the electric shovel, which integrates multimodal data, is applied to a multimodal spatiotemporal depth fusion detector. The multimodal spatiotemporal depth fusion detector is installed on an open-pit mine electric shovel. The multimodal spatiotemporal depth fusion detector includes: a control module, a multimodal sensor, an image-point cloud feature extraction module, a spatiotemporal characteristic adaptive weight optimization module, and a multi-scale loss optimization module, all connected in sequence. The multimodal sensor, image-point cloud feature extraction module, spatiotemporal characteristic adaptive weight optimization module, and multi-scale loss optimization module are all connected to the control module. The method for identifying multiple targets around an electric shovel by fusing multimodal data includes: Acquire information data about the area surrounding the electric shovel; the information data includes: image information data collected by the camera included in the multimodal sensor, point cloud data collected by the lidar included in the multimodal sensor, and IMU data collected by the inertial measurement unit included in the multimodal sensor. The control module controls the image-point cloud feature extraction module to preprocess the information data, and the semantic segmentation network determines the semantic mask based on the preprocessed information data, and performs cross-modal interaction from image to point cloud to determine the key features of multimodal data; the key features of multimodal data include the optimal matching pair obtained by searching the similarity matrix of cross-modal features using the Hungarian algorithm; the preprocessed information data includes motion increments; Based on the control module's spatiotemporal characteristics, the adaptive weight optimization module determines the set of motion trajectories of each detection target around the electric shovel according to the motion increment and the key features of the multimodal data, and determines the detection box. Then, the adaptive weight optimization mechanism is used to optimize the detection box and the corresponding movement trajectory based on the total loss function to obtain the optimized detection box trajectory information and the optimized detection box. The control module controls the multi-scale loss optimization module, which employs the LM algorithm to minimize the total loss function of the multi-task loss system. Based on the optimized detection box trajectory information and the optimized detection box, iterative optimization is performed to determine the secondary optimized detection box trajectory information and the secondary optimized detection box. Among them, the multimodal spatiotemporal depth fusion detector uses the secondary optimized detection box. Based on the trajectory information of the secondary optimized detection box, it can fit the appearance detection boxes corresponding to each detection target around the electric shovel to achieve the perception, detection and recognition of three-dimensional targets.

2. The method for identifying multiple targets around an electric shovel by fusing multimodal data according to claim 1, characterized in that, The control module controls the image-point cloud feature extraction module to preprocess the information data, and uses a semantic segmentation network to determine the semantic mask based on the preprocessed information data, as well as perform cross-modal interaction from image to point cloud to determine key features of multimodal data, specifically including: Based on the control module, the image-point cloud feature extraction module is controlled, and the quaternion method is used to solve the IMU data to obtain the three-dimensional attitude angles; the three-dimensional attitude angles include: roll angle, pitch angle and yaw angle; The z-axis feature vector is determined; the z-axis feature vector is obtained by normalizing and concatenating the triaxial acceleration and angular velocity contained in the IMU data using a multi-scale convolutional network. Based on the z-axis feature vector, the yaw angle is enhanced using image information data, and a feedforward-feedback joint architecture is used to correct and optimize the yaw angle to obtain the correction result. The feedforward-feedback joint architecture includes a feedforward path and a feedback path. The feedforward path uses an extended Kalman filter to update the state estimate, and the feedback path is used to perform a linear mapping from features to attitude correction. The correction results are weighted and fused to obtain attitude estimation output information, and pre-integration is performed in adjacent time intervals within a set time interval to obtain pre-processed information data. Based on the semantic segmentation network, semantic masks are determined according to the preprocessed information data, and cross-modal interaction between images and point clouds is performed to determine key features of multimodal data.

3. The method for identifying multiple targets around an electric shovel by fusing multimodal data according to claim 2, characterized in that, Based on a semantic segmentation network, semantic masks are determined from preprocessed information data, and cross-modal interaction between images and point clouds is performed to determine key features of multimodal data, specifically including: Based on the preprocessed information data, the point cloud data is projected onto the coordinate system corresponding to the camera included in the multimodal sensor to achieve spatial alignment of the multimodal information data. Convolutional neural networks are used to extract features from spatially aligned image information data, and semantic masks are determined based on semantic segmentation networks. Semantic regions are visualized and rendered using class activation mapping to obtain semantic region heatmaps. Based on the semantic region heatmap, the spatially aligned point cloud data is projected onto the image plane, and the point cloud is mapped one-to-one with the image pixels. Furthermore, based on the image feature information after feature extraction and the semantic analysis results after determining the image semantic mask, a downsampling strategy is used to filter and determine the semantic information filtering data. Based on the semantic information, data is filtered to perform cross-modal interaction from image to point cloud in order to determine key features of multimodal data.

4. The method for identifying multiple targets around an electric shovel by fusing multimodal data according to claim 3, characterized in that, Based on the semantic information, data is filtered to perform cross-modal interaction from image to point cloud to determine key features of multimodal data, specifically including: The point cloud feature information and image feature information after feature extraction are encoded respectively, and a cross-modal interaction mechanism is adopted to filter data to enhance point cloud features based on the semantic information to obtain enhanced feature information; Calculate the similarity matrix of cross-modal features based on enhanced feature information; The Hungarian algorithm is used to search the similarity matrix of cross-modal features to find the optimal matching pair, which is then identified as the key feature of the multimodal data.

5. The method for identifying multiple targets around an electric shovel by fusing multimodal data according to claim 1, characterized in that, Based on the control module's spatiotemporal characteristics, the adaptive weight optimization module determines the set of motion trajectories of each detected target around the electric shovel according to the motion increment and the key features of the multimodal data, and determines the detection box. Furthermore, using an adaptive weight optimization mechanism, the detection box and its corresponding movement trajectory are optimized based on the total loss function to obtain the optimized detection box trajectory information and the optimized detection box. Specifically, this includes: Based on the control module, the spatiotemporal characteristic adaptive weight optimization module optimizes the trajectory smoothness by combining the pose transformation matrix with the Lie algebra pose increment according to the motion increment, and completes cross-frame target two-dimensional association based on the key features of the multimodal data to determine the motion trajectory set of each detected target around the electric shovel and determine the detection box. An adaptive weight optimization mechanism is employed. Based on the motion trajectory set and detection boxes, the detection boxes and their corresponding movement trajectories are optimized using a total loss function, resulting in optimized detection box trajectory information and optimized detection boxes. The expression for the total loss function is: ； in, This is the total loss function; The dynamic weights of the motion continuity loss function; This is the motion continuity loss function; The dynamic weights of the consistency loss function; This is a joint loss function based on the consistency loss function.

6. The method for identifying multiple targets around an electric shovel by fusing multimodal data according to claim 1, characterized in that, The adaptive weight optimization mechanism is a quantization standard mechanism for 3D target detection boxes determined by using a weight voting method and integrating six quantization indicators. The quantization standard mechanism includes the point cloud weight ratio and image weight ratio based on occlusion state and distance segmentation. The six quantization indicators include: point cloud coverage, point cloud density deviation rate, noise point ratio, grayscale contrast deviation, texture entropy value, and detail discernibility.

7. The method for identifying multiple targets around an electric shovel by fusing multimodal data according to claim 1, characterized in that, The expression for the total loss function of the multi-task loss system is: ； in, The total loss function of the multi-task loss system; The vertical linear alignment loss function; Let IMU motion consistency loss function be used. This is the distance regression decay loss function; These are the weights corresponding to the vertical linear alignment loss function; These are the weights corresponding to the IMU motion consistency loss function. These are the weights corresponding to the distance regression decay loss function.

8. A multi-target identification device around an electric shovel that integrates multimodal data, characterized in that, include: The information data acquisition module is used to acquire information data around the electric shovel; the information data includes: image information data collected by the camera included in the multimodal sensor, point cloud data collected by the lidar included in the multimodal sensor, and IMU data collected by the inertial measurement unit included in the multimodal sensor. The cross-modal interaction module is used to preprocess the information data based on the image-point cloud feature extraction module controlled by the control module, determine the semantic mask based on the preprocessed information data using the semantic segmentation network, and perform cross-modal interaction from image to point cloud to determine key features of multimodal data; the key features of multimodal data include the optimal matching pair obtained by searching the similarity matrix of cross-modal features using the Hungarian algorithm; the preprocessed information data includes motion increments; The optimization module is used to determine the set of motion trajectories of each detection target around the electric shovel based on the motion increment and the key features of the multimodal data, and to determine the detection box, and to optimize the detection box and the corresponding movement trajectory based on the total loss function using an adaptive weight optimization mechanism, so as to obtain the optimized detection box trajectory information and the optimized detection box. The secondary optimization module, based on the control module's control of the multi-scale loss optimization module, employs the LM algorithm with the objective of minimizing the total loss function of the multi-task loss system. It iteratively optimizes the detection box trajectory information and the optimized detection box based on the optimized detection box trajectory information, determining the secondary optimized detection box trajectory information and the secondary optimized detection box. The multimodal spatiotemporal depth fusion detector uses the secondary optimized detection box trajectory information to fit the appearance detection boxes corresponding to each detection target around the electric shovel, thereby achieving the perception, detection, and recognition of three-dimensional targets.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method for multi-target identification around an electric shovel by fusing multimodal data as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for identifying multiple targets around an electric shovel by fusing multimodal data as described in any one of claims 1-7.

Images