Shovel loading quality real-time evaluation method and system based on multi-modal data fusion

Abstract

The application provides a kind of real-time evaluation method and system for electric shovel loading quality based on multi-modal data fusion.The method controls three-dimensional perception, inertia, pressure, image and sound acquisition device set on electric shovel to obtain loading operation data, completes time alignment, preprocessing, modal coding and feature fusion according to unified clock;According to the inertia and pressure time sequence, the operation stage is identified, the three-dimensional point cloud is mapped to the reference coordinate model of the empty standard carriage in the unloading stage and the shared height matrix is formed, and the evaluation data of loading distribution uniformity, material surface flatness and carriage filling rate are obtained, and the action specification and cycle efficiency evaluation data are obtained;The evaluation data is converted into evaluation results and operation guidance information through feedback rule base, and output and storage through man-machine interface and voice. The application is helpful to form executable real-time evaluation and feedback closed loop in the loading process.

Description

Technical Field

[0001] This invention relates to the field of intelligent mining equipment and safety production supervision technology, and in particular to a method and system for real-time evaluation of electric shovel loading quality based on multimodal data fusion. Background Technology

[0002] In open-pit mining, loading electric shovels onto mining trucks is a core production step. Currently, the quality and efficiency of operations rely heavily on the experience and judgment of the electric shovel operators. Different operators have different standards for judging whether the load is full, level, and where to unload, which may lead to problems such as underloading, overloading, repeated shoveling, uneven material distribution, and collisions between the bucket and the truck bed. This also hinders production statistics and operational review.

[0003] Existing weighing systems typically provide the total weight after loading is complete, making it difficult to reflect the spatial distribution of materials within the truck bed during unloading. Traditional two-dimensional vision solutions are easily affected by lighting, dust, and shadows. Single attitude or load sensors cannot simultaneously reflect the truck bed's spatial state, bucket movements, and abnormal impacts. While some monitoring systems can record operational data, the lack of a unified time, coordinate, and feature interface between various heterogeneous data sets makes it difficult to form a processing chain that can be directly used for operational phase identification, spatial quality assessment, and on-site feedback.

[0004] Therefore, there is a need for a method and system that can perform time alignment, modal processing, and feature fusion of 3D point cloud, motion posture, load change, image, and sound data, and perform corresponding quality evaluation and feedback according to the operation stage. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time evaluation method and system for electric shovel loading quality based on multimodal data fusion, which enables multi-source heterogeneous operation data to form a continuous and executable processing chain, and provides spatial quality evaluation and operation prompts during the unloading process using an operation stage-driven evaluation mechanism.

[0006] To achieve the above objectives, this invention provides a real-time evaluation method for electric shovel loading quality based on multimodal data fusion, comprising: The system controls the three-dimensional sensing sensor, inertial measurement unit, pressure sensor, image acquisition device and sound acquisition device installed on the electric shovel to collect three-dimensional point cloud data, motion posture data, load change data, visual image data and sound data in real time to obtain multimodal operation data; Using the unified clock of the edge computing unit as a reference, the multimodal operation data is time-aligned and preprocessed. The corresponding encoders are used to extract the features of each modality. The feature vectors within the same time window are dimension-mapped and spliced ​​together to form a multimodal fusion feature sequence. The current loading operation stage is identified based on the characteristics of motion posture data and load change data, and the operation stage is used as the starting condition for the corresponding quality evaluation task. During the unloading stage, the 3D point cloud data is mapped to the reference coordinate model established by the empty standard car body. A shared height matrix is ​​formed based on the internal space grid of the car body, and evaluation data of loading distribution uniformity, material surface flatness and car body filling rate are obtained based on the shared height matrix. Evaluation data of action standardization are obtained in each operation stage, and cycle efficiency evaluation data are obtained after a complete loading cycle. The system utilizes a feedback rule base to convert quality evaluation data into quality evaluation results and generate operation guidance information. This operation guidance information is then fed back to the electric shovel operator, and multimodal operation data, quality evaluation data, quality evaluation results, and feedback records are stored.

[0007] Furthermore, the three-dimensional perception sensor is a millimeter-wave radar or lidar; the inertial measurement unit is used to acquire the attitude and motion parameters of the bucket or actuator; the pressure sensor is used to acquire pressure time-series data characterizing changes in bucket load; the image acquisition device is used to assist in identifying the boundaries and relative positions of the cargo box; and the sound acquisition device is used to extract the characteristics of collision sound and abnormal impact sound.

[0008] Furthermore, the modal coding includes point cloud coding, temporal coding, image coding, and sound coding. The coding results are mapped by dimension and then spliced ​​together according to the same time window. Finally, they are fused together by a fully connected fusion layer to form a multimodal fusion feature sequence.

[0009] Furthermore, the operation phases include the excavation phase, the full bucket rotation phase, the unloading phase, the empty bucket swing phase, and the positioning phase; the action standardization evaluation task runs continuously in each operation phase, the spatial quality evaluation task runs in the unloading phase, and the cycle efficiency evaluation task runs after the complete loading cycle is completed.

[0010] Furthermore, during system initialization, the empty standard car body is scanned and a reference coordinate model is established. The three-dimensional point cloud of the unloading stage is converted to the reference coordinate model, the points outside the car body and outliers are removed, the interior of the car body is divided into spatial grids, and a shared height matrix is ​​formed based on the average height of the material points in each spatial grid relative to the car body floor.

[0011] Furthermore, loading distribution uniformity evaluation data is obtained based on the regional average height and dispersion of the shared height matrix in the length and width directions of the carriage. Material surface flatness evaluation data is obtained based on the local gradient or surface roughness formed by the height difference between adjacent grids. The current loading volume is calculated based on the shared height matrix to obtain carriage filling rate evaluation data.

[0012] Furthermore, the cycle efficiency evaluation data is obtained after the positioning phase ends and the complete loading cycle is confirmed, based on the normalized results of the total cycle time and the connection interval between each operation phase relative to the benchmark values ​​of the same vehicle type and the same ore rock conditions.

[0013] Furthermore, pressure time-series data, as a characteristic of load variation, is used in the identification of operation phases and the evaluation of action standardization, and is not directly used as a material weight measurement value.

[0014] Furthermore, the training objectives of the machine learning model include spatial quality objectives generated by calculation from the shared height matrix, efficiency objectives generated by loop duration, and action standardization objectives generated by manual annotation from a unified scale; mean squared error weighted sum is used as the training loss, and mean absolute error is used as the validation and testing metric.

[0015] Furthermore, the feedback rule base adopts mutually exclusive scoring intervals and rule priorities; when multiple rules are satisfied simultaneously, emergency risk information takes precedence over severe deviation information, general deviation information, and cyclic summary information.

[0016] Furthermore, the operation guidance information is provided through a human-machine interface (HMI) and voice feedback. The HMI includes a real-time guidance area and a results display area. The real-time guidance area displays a heat map of the carriage filling and directional prompts, while the results display area displays evaluation data and operation suggestions after the cycle is completed.

[0017] This invention also provides a real-time quality assessment system for electric shovel loading based on multimodal data fusion, comprising a data acquisition layer, an edge computing layer, and an application feedback layer; the data acquisition layer collects multimodal operation data, the edge computing layer performs time synchronization, submodal encoding, feature fusion, operation stage identification, shared height matrix construction, and quality evaluation, and the application feedback layer performs rule determination, HMI and voice output, and data storage.

[0018] The present invention has achieved the following beneficial effects: This invention integrates 3D point cloud, motion posture, load change, image and sound data with different sampling frequencies and data formats into a fusion feature sequence according to a unified time window. This creates a clear initiation relationship between the identification results of the operation stage and subsequent evaluation tasks, which helps to reduce the interference of irrelevant stage data on the evaluation.

[0019] This invention maps the three-dimensional point cloud to the reference coordinates of an empty standard car body during the unloading stage, and generates evaluation data on loading distribution uniformity, material surface flatness, and car body filling rate using the same shared height matrix. This reduces repetitive spatial modeling and provides a unified data basis for directional cues for higher or lower areas within the car body.

[0020] This invention combines numerical quality evaluation data with mutually exclusive rule intervals and priorities, enabling process adjustment prompts, emergency alarms, and cycle end summaries to be output according to different operational states. This helps reduce the burden on drivers to understand complex evaluation data during continuous loading operations and reduces feedback conflicts.

[0021] This invention uses an edge computing layer to complete the real-time processing of the current loading cycle, and can upload evaluation results, abnormal data fragments and feedback records to the monitoring platform for subsequent statistics, verification and validated model updates, which helps to improve the system's adaptability to different drivers, vehicle types and mining conditions.

[0022] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the application.

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram illustrating the steps of the method described in this invention. Detailed Implementation

[0025] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0026] like Figure 1 As shown, this embodiment of the invention provides a real-time evaluation method for electric shovel loading quality based on multimodal data fusion, which sequentially completes multimodal operation data acquisition, time synchronization and feature fusion, operation stage identification and quality evaluation, rule determination and operation guidance, as well as feedback and data management.

[0027] Step 1: Control the multimodal sensors installed on the electric shovel to collect loading operation data in real time and obtain multimodal operation data.

[0028] The multimodal sensor includes a 3D sensing sensor, an inertial measurement unit, a pressure sensor, an image acquisition device, and a sound acquisition device. Multimodal operational data includes 3D point cloud data, motion attitude data, load change data, visual image data, and sound data. Each data packet carries a sensor identifier, a sampling timestamp, and a data validity identifier.

[0029] Step 2 involves time synchronization, preprocessing, modal coding, and feature fusion of the multimodal operation data to form a multimodal fusion feature sequence.

[0030] The edge computing unit provides a unified clock to each acquisition device or performs periodic correction on the local clocks of each acquisition device. Using a preset time window as the smallest fusion unit, high-frequency inertial and pressure time-series data are resampled, 3D point clouds are aggregated within the time window, effective frames closest to the center of the time window are selected from image frames, and audio segments corresponding to the time window are extracted from sound signals, so that data within the same time window corresponds to the same operating state.

[0031] In the preprocessing stage, outlier removal, external point filtering, and coordinate transformation are performed on the 3D point cloud; outlier removal, filtering, and normalization are performed on the inertial and pressure time series data; target regions containing the relative positions of the truck and bucket are extracted from the image data; and noise reduction and time-spectral or Mel-spectral features are extracted from the sound data.

[0032] Point cloud encoders, temporal encoders, image encoders, and sound encoders are used to process data of their respective modalities. Point cloud encoders can employ PointNet or 3D convolutional neural networks, temporal encoders can use long short-term memory networks or Transformer models, and image and sound encoders can use lightweight convolutional neural networks. The feature vectors output by each encoder are dimensionally mapped and then concatenated according to time windows. These concatenated vectors are then passed through a fully connected fusion layer to form a multimodal fusion feature vector. The fusion feature vectors from consecutive time windows constitute a multimodal fusion feature sequence.

[0033] Each modality's feature vector corresponds to a valid identifier. When auxiliary modalities such as images or sounds lack valid data within a certain time window, a preset placeholder vector and invalid identifier are used in the stitching process, enabling the model to identify the missing modality. When the 3D point cloud data is invalid, the spatial quality evaluation for the corresponding time window is paused, and sensor status information is output. Image or sound data is not used to replace the generation of the carriage height matrix.

[0034] Step 3: Identify the work stage based on the multimodal fusion feature sequence, and perform corresponding quality evaluation according to the work stage to obtain quality evaluation data.

[0035] The operation phase identification model primarily reads the temporal characteristics generated by the inertial measurement unit and pressure sensors, outputting the state probabilities of the digging phase, full bucket rotation phase, unloading phase, empty bucket swing phase, and positioning phase. The current operation phase is determined based on the duration for which the state probability continuously exceeds the state threshold and the preset phase transition sequence, thus avoiding repeated phase switching caused by instantaneous vibrations.

[0036] When the identification result is inconsistent with the preset stage transfer order, the previous valid stage is retained and confirmation is awaited in the subsequent time window; when multiple consecutive time windows fail to form a valid stage, the current state is marked as a pending confirmation state, the spatial quality evaluation that depends on a clear stage is suspended, but the sensor state record is retained.

[0037] The model adopts a two-level structure of "modal encoder - feature fusion layer - task output head". The output of the feature fusion layer is fed into the operation stage recognition head, motion standardization evaluation head, spatial quality evaluation head and cycle efficiency evaluation head respectively; the operation stage recognition result serves as a task gating signal, which enables the spatial quality evaluation head to start only during the unloading stage and the cycle efficiency evaluation head to start only after a complete loading cycle is completed.

[0038] The quality evaluation data includes evaluation data on loading distribution uniformity, material surface smoothness, carriage filling rate, operational standardization, and cycle efficiency. All evaluation data are uniformly mapped to a scoring range of 0 to 100, with higher scores indicating that the corresponding quality state is closer to the preset target.

[0039] Step 4: Input the quality evaluation data into the feedback rule base to obtain the quality evaluation results, and generate operation guidance information according to the quality evaluation results.

[0040] Step 5: Output operation guidance information to the electric shovel driver through HMI and voice, and store multimodal operation data, quality evaluation data, quality evaluation results and feedback records; when uploading is required, only select evaluation results, abnormal time window data segments and feedback records to upload to the mine area monitoring platform to reduce the on-site communication burden.

[0041] In one sensor deployment example, a 3D perception sensor is positioned on the top of the electric shovel cab or on the boom to cover the truck bed; an inertial measurement unit is positioned on the bucket or boom; a pressure sensor is positioned in the bucket hydraulic circuit or lifting mechanism; an image acquisition device is positioned to observe the truck bed boundary and the relative position of the bucket; and a sound acquisition device is positioned facing the unloading area. An edge computing unit is located locally on the electric shovel and communicates with each acquisition device.

[0042] The 3D sensing sensor can be a millimeter-wave radar or a lidar, and its output is used to characterize the spatial position of the vehicle body boundaries and material surfaces. 3D point cloud data is used to establish a representation of the vehicle body's spatial height, but it is not directly used to claim the vehicle's load center of gravity or rollover probability; the evaluation results of loading distribution uniformity can serve as an operational reference for reducing significant off-center loading.

[0043] The inertial measurement unit acquires pitch angle, roll angle, angular velocity, and acceleration to identify the bucket's motion state and perform attitude compensation for sensor coordinates. Pressure sensors acquire the trend of hydraulic pressure changes to distinguish between digging, loaded rotation, and unloading states, and to assist in identifying abnormal impacts. Without a specific calibration relationship between pressure and material weight, pressure time-series data is not used as the weight of a single bucket or the cumulative loaded weight.

[0044] The image acquisition device is used to assist in extracting the boundaries of the cargo compartment, the type of the cargo compartment, and the relative position of the bucket and the cargo compartment; the image coordinates and the coordinates of the 3D sensing sensor are correlated through the external parameter relationship obtained by joint calibration. The sound acquisition device is used to identify collision sounds, bucket slamming sounds, or abnormal impact sounds. Its function is limited to assisting in the judgment of abnormal operation and equipment status, and does not replace the measurement of the spatial morphology of materials by 3D point cloud.

[0045] During system initialization, multi-sensor joint calibration is performed. The joint calibration includes at least the extrinsic parameters of the 3D sensing sensor relative to the electric shovel's installation coordinate system, the extrinsic parameters of the image acquisition device relative to the 3D sensing sensor, the installation orientation of the inertial measurement unit, and a unified clock offset. Calibration parameters are stored in the edge computing unit for subsequent coordinate transformation and time alignment.

[0046] Therefore, multimodal fusion is not simply storing sensor data in parallel, but rather forming a clear processing relationship: first, matching the same operational state using time windows; then, generating features of the same dimension from each modal encoder; subsequently, forming shared features through stitching and fully connected fusion; and finally, controlling the corresponding evaluation task based on the operational stage identification results. The spatial coordinates of the 3D point cloud are still retained and separately entered into the shared height matrix construction process, while the fused features are used to supplement stage, action, and abnormal state information.

[0047] The point cloud encoder can simultaneously output global point cloud features and the set of valid points inside the carriage; the time sequence encoder outputs the motion and load change features of the current time window; the image encoder outputs boundary and relative position features; and the sound encoder outputs the category features of normal sound, collision sound, and abnormal impact sound.

[0048] The point cloud global features, temporal features, image features, and sound features are linearly mapped to give each feature a preset length; each feature, valid identifier, and current time window number are then concatenated to form a fused input vector.

[0049] Multiple fusion input vectors are input into the operation stage identification head to form the state probability of each operation stage; the fusion input vectors and stage states are input into the action standardization evaluation head to determine whether the rotation is smooth, whether the unloading height or unloading speed exceeds the corresponding working condition threshold, and whether there is any abnormal impact.

[0050] During the unloading stage, the spatial quality evaluation head receives spatial statistical features formed by the shared height matrix, and can also receive boundary confidence and sensor validity identifiers from the fused input vector, so as to pause the output of spatial score when the boundary recognition confidence is insufficient or the point cloud validity is insufficient.

[0051] After the complete loading cycle is completed, the cycle efficiency evaluation head reads the total cycle duration and the interval between each stage determined by a unified timestamp to form cycle efficiency evaluation data. Thus, the modal coding branch and the evaluation task branch correspond to the input data processing and output tasks respectively, and are not used as "multi-branch" at the same level.

[0052] Action standardization evaluation can be achieved by combining model regression scoring with rule-based deduction. The model outputs a basic action standardization score; when a sudden change in slewing angular velocity, a discharge height exceeding the working condition threshold, or an abnormal impact is detected in the sound classification result, the basic score is corrected according to the preset deduction rules to obtain action standardization evaluation data.

[0053] A shared height matrix mechanism for spatial quality assessment. This mechanism constructs a height matrix using only 3D point clouds with 3D spatial coordinates; image, sound, inertial, or pressure data do not directly generate raster heights.

[0054] During the initialization phase, an empty standard carriage is placed in a scannable position to obtain the three-dimensional point cloud of the carriage floor and side walls. A reference coordinate model is established with the carriage length direction as the X-axis, the width direction as the Y-axis, and the height direction as the Z-axis. The reference surface of the carriage floor, the internal boundary of the carriage, and the standard volume are recorded.

[0055] In the subsequent unloading stage, the current point cloud is transformed into the reference coordinate model using the extrinsic parameters between the 3D perception sensor and the reference coordinate model, as well as the attitude compensation parameters provided by the inertial measurement unit. Points outside the car, points on the side walls, and outliers are removed based on the side walls and internal boundaries of the car, resulting in a set of material points inside the car.

[0056] The interior area of ​​the carriage is divided into spatial grids according to preset grid length and grid width. The grid size is set according to the carriage size, sensor point cloud density, and required feedback area resolution, so that an effective grid can contain material points that meet statistical requirements.

[0057] For each spatial grid, calculate the average height of the material points within that grid relative to the reference plane of the car floor to obtain the corresponding grid height; all grid heights are arranged according to the length and width directions of the car to form a shared height matrix.

[0058] When the number of valid material points in a grid is lower than the valid point count threshold, the grid is marked as invalid. If the number of its adjacent valid grids meets the interpolation condition, a temporary height is obtained by interpolation using the heights of the adjacent valid grids; otherwise, the invalid mark is maintained and the grid waits for subsequent point cloud updates to avoid abnormal peaks and valleys forming directly from sparse point clouds.

[0059] The evaluation of loading distribution uniformity includes evaluation in the length direction and evaluation in the width direction. The average height and dispersion of each row and column of the shared height matrix are statistically analyzed. The dispersion is normalized relative to the allowable upper limit of the corresponding working condition to obtain the uniformity sub-scores in the length and width directions. These sub-scores are then combined according to preset weights to form the loading distribution uniformity evaluation data.

[0060] Simultaneously, the carriage is divided into multiple guide zones: left, right, front, and rear, and the average height of each guide zone is calculated. The average height difference between the left and right zones is used to determine the direction of lateral distribution deviation, and the average height difference between the front and rear zones is used to determine the direction of longitudinal distribution deviation. The operating instructions only indicate that adjustments should be prioritized towards the zone with the lower average height, and do not output precise movement distances when bucket movement calibration and material drop models are lacking.

[0061] The evaluation of material surface smoothness calculates the local gradient based on the height difference between adjacent grids in the shared height matrix, and can further calculate the deviation of the effective grid height relative to the local fitted surface. The larger the local gradient and deviation, the more obvious the peaks and valleys or roughness of the material surface; after normalizing them relative to the allowable upper limit, the material surface smoothness evaluation data is obtained.

[0062] The car body filling rate evaluation calculates the corresponding grid volume based on the height and area of ​​each effective grid and sums them to obtain the current loading volume. The current loading volume is then divided by the standard car body volume recorded during the initialization phase to obtain the car body filling rate. Invalid grids are handled in the same way as the height matrix, and the coverage ratio of effective grids is recorded. If the coverage ratio is lower than a set value, no deterministic filling rate result is output.

[0063] The values ​​for uniformity, flatness, and filling rate can be mapped to a scoring range of 0 to 100 according to on-site quality standards. The mapping parameters are configuration parameters, determined by valid historical samples or on-site acceptance standards of the same vehicle type and the same mining and rock conditions, and are recorded together with the model version and rule version.

[0064] A complete loading cycle begins with the first effective entry during the excavation stage, sequentially undergoes full bucket rotation, unloading, empty bucket swing, and positioning, and terminates when the positioning stage ends and the stage sequence is complete. The total cycle duration is the difference between the cycle termination timestamp and the cycle start timestamp, and the interval between each stage is the waiting time between effective switching between adjacent stages.

[0065] The cycle efficiency score is obtained by combining the time efficiency sub-score and the smoothness of connection sub-score. The time efficiency sub-score is normalized based on the total cycle time relative to the target baseline time and the allowable upper limit time for the same vehicle type and the same mining conditions; the smoothness of connection sub-score is normalized based on the proportion of the connection interval of each stage to the total cycle time; the two sub-scores are combined according to preset weights and mapped to 0 to 100.

[0066] The training data for the machine learning model uses complete loading cycles as the sample unit, covering different drivers, truck models, electric shovel models, and mining conditions. During data collection, sensor configuration, calibration parameters, vehicle model parameters, and working condition identifiers are recorded simultaneously to ensure that target values ​​are generated using the same criteria during training and validation.

[0067] The training objectives for uniformity of loading distribution, flatness of material surface, and filling rate of the truck bed are automatically generated by the shared height matrix and its calculation rules; the training objective for cycle efficiency is automatically generated by the normalization rules of total cycle time and connection interval; the training objective for action standardization is scored by multiple annotators with electric shovel operation or training experience according to a unified scale for the complete cycle.

[0068] The action standardization scale uses a score from 0 to 100, and records at least the rotational smoothness, unloading height, unloading speed, and abnormal impact. The median of multiple annotation results is used as the training target; when the difference between the highest and lowest scores exceeds the review threshold, the cycle is reviewed jointly by the annotators, and samples that cannot reach a consistent conclusion are not included in the training set.

[0069] The training, validation, and test sets are divided according to a complete loading cycle. Time windows from the same complete cycle must not be assigned to different sets. The specific division ratio can be selected based on the data size, but it should be ensured that each set covers the main vehicle types and mining conditions, and a division list should be recorded.

[0070] During model training, the weighted sum of the mean squared errors of each evaluation task is used as the training loss, with the weights for each task set according to the scoring scale and sample size. During validation and testing, the mean absolute error is used to evaluate each task separately, and the model's suitability for deployment is determined by combining stage recognition accuracy and on-site processing latency. Mean squared error is used for parameter optimization, while mean absolute error is used for model validation; their purposes differ.

[0071] After the trained candidate model is validated on independent test sets and replay data, a model version number is generated. The edge computing unit only loads the validated model version; after updating the model deployment, the previous stable version is retained, and a rollback can be performed if the new version produces abnormal output or exceeds the processing latency limit.

[0072] The quality evaluation results in the feedback rule base are distinguished from the quality evaluation data: the quality evaluation data are numerical values ​​of uniformity, flatness, filling rate, action standardization, and cycle efficiency; the quality evaluation results are state conclusions such as normal, general deviation, serious deviation, or emergency risk formed based on rule intervals.

[0073] Taking the evaluation of loading distribution uniformity as an example, a first uniformity threshold and a second uniformity threshold below the first uniformity threshold are set. When the uniformity score is not lower than the first uniformity threshold, the evaluation result is normal; when the uniformity score is lower than the first uniformity threshold but not lower than the second uniformity threshold, the evaluation result is general distribution deviation; when the uniformity score is lower than the second uniformity threshold, the evaluation result is severe distribution deviation. The intervals do not overlap.

[0074] The system sets three loading intervals for the fill rate: a continued loading interval, a near-target interval, and an interval exceeding the upper limit. Emergency risk rules are set for abnormal impacts. When multiple evaluation results occur within the same time window, alarms for abnormal impacts or collision risks take priority, followed by alarms for exceeding the fill rate upper limit or severe distribution deviations, and then general position adjustments. A summary of the results after the complete cycle is output last.

[0075] When generating position adjustment information, the direction is determined based on the average height difference between the left and right regions and the average height difference between the front and rear regions of the shared height matrix. The degree of adjustment (minor or priority adjustment) is described based on the deviation level of the difference. If a calibration model between the bucket's motion and the material drop position is not established, precise displacement parameters such as "moved by several meters" are not output.

[0076] HMI is a human-machine interface. During the unloading phase, the real-time guidance area of ​​HMI displays the two-dimensional outline of the cargo compartment, a filled heat map generated by the shared height matrix, the effective status of sensors, and directional prompts pointing to areas with lower average heights. When the spatial evaluation is paused, the real-time guidance area displays the corresponding sensor status without showing unconfirmed scores.

[0077] After a complete loading cycle, the HMI results display area shows evaluation data on loading distribution uniformity, material surface flatness, wagon filling rate, action standardization, and cycle efficiency. It also displays the quality evaluation results and a work suggestion corresponding to the highest priority deviation.

[0078] Voice feedback is used to output short-term actionable prompts and emergency alarms. Regular announcements are used for general position adjustments, enhanced announcements for severe deviations, and priority announcements for abnormal impacts or collision risks, which can be accompanied by prominent visual indicators. A cyclical summary is output after the current unloading and emergency alarm events conclude to avoid information overlap between multiple channels.

[0079] Feedback rules and thresholds can be configured offline by authorized administrators based on vehicle parameters, mining conditions, and on-site quality standards. Once the rules are updated and validated through playback, they are activated during non-critical operating periods. Each ongoing loading cycle continues to use the rule version used when that cycle was started, ensuring consistent scoring criteria.

[0080] The system of this invention includes a data acquisition layer, an edge computing layer, and an application feedback layer. These layers are connected via wired or wireless communication. Real-time on-site evaluation and feedback are performed by the edge computing layer, without relying on a remote platform for continuous online operation.

[0081] The data acquisition layer includes 3D sensing sensors, inertial measurement units, pressure sensors, image acquisition devices, and sound acquisition devices already installed on the electric shovel. The data acquisition layer performs sampling and data transmission but does not perform the physical installation of the sensors.

[0082] The edge computing layer includes a unified clock and time synchronization module, a data preprocessing module, a modal coding module, a feature fusion module, a job phase identification module, a shared height matrix construction module, a quality evaluation module, and a model version management module.

[0083] The application feedback layer includes a feedback rules module, an HMI feedback module, a voice and alarm module, and a storage management module. The feedback rules module converts quality evaluation data into quality evaluation results, the HMI feedback module and the voice and alarm module output operation instructions according to priority, and the storage management module records the corresponding data and version information.

[0084] The system may also include a data communication module, used to upload quality evaluation results, abnormal data fragments, and feedback records to the mine monitoring platform when communication conditions permit. The monitoring platform can be used for statistics, operation playback, and candidate model training, but candidate models need to be verified and version managed before being deployed to the edge computing layer.

[0085] The edge computing layer can employ industrial control computers or edge servers, with priority given to ensuring time synchronization, stage identification, shared height matrix updates, and on-site feedback. Original long-term videos or complete point clouds can be cyclically overwritten according to the storage strategy, while abnormal time window data segments and evaluation results are retained for a longer period.

[0086] In one system product implementation, the HMI feedback module, voice and alarm module, data communication module, and model version management module are all connected to the edge computing layer. When an abnormal impact, severe distribution deviation, or exceeding the fill rate limit occurs, the voice and alarm module outputs alarms according to priority, and the data communication module synchronously records the corresponding abnormal segment index.

[0087] The system described above collects data through pre-installed sensors, and the processor performs time alignment, code fusion, stage identification, spatial quality evaluation, rule determination, and feedback output. Therefore, the executable steps of the system and method topics correspond to each other and do not include the meaning of physical sensor installation completed by computer programs.

[0088] In this embodiment, multimodal operation data, fused feature sequence, quality evaluation data, and quality evaluation result have different meanings. Multimodal operation data refers to raw sensor data or preprocessed data; fused feature sequence is a combination of modal coding features within the same time window; quality evaluation data is a numerical score; and quality evaluation result is a rule-based judgment status.

[0089] By making the above distinctions, the source and purpose of each data object can be clearly identified during storage, transmission, and feedback, avoiding the confusion between model output scores and rule-based judgment conclusions.

[0090] This invention forms a continuous action chain through operation phase gating, shared height matrix, and time-sharing hierarchical feedback, which helps to provide quantitative information on loading space status, action status, and cycle efficiency without increasing the calculation of irrelevant phases.

[0091] It should be noted that the directional indication described in this invention is used to indicate higher or lower compartment areas and is not necessarily equivalent to vehicle load center of gravity calculation, rollover probability prediction, or precise bucket displacement control; if the above functions are required, corresponding sensor calibration and dynamic models should be established separately.

[0092] The model, rule threshold, grid scale, and time window length can be configured according to the electric shovel model, truck model, point cloud density, and on-site processing latency requirements. The configuration values ​​and versions should be associated with and saved with the corresponding evaluation records.

[0093] The system of this invention corresponds to the method of this invention. Each module in the system can be implemented by software, hardware, or a combination of software and hardware. The module division is used to describe the functional relationship and does not limit the specific number of processors or their deployment location.

[0094] Without changing the above data processing relationships, some non-real-time statistical functions of the edge computing layer can be performed by the mining area monitoring platform, but operation phase identification, spatial quality evaluation and on-site operation guidance are preferably completed locally on the electric shovel or near the edge equipment.

[0095] In this invention, terms such as "first" and "second" are used only to distinguish different thresholds, modules, or data objects, and do not indicate order, quantity, or importance.

[0096] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0097] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A real-time evaluation method for the loading quality of electric shovels based on multimodal data fusion, characterized in that, include: The system controls the three-dimensional perception sensor, inertial measurement unit, pressure sensor, image acquisition device and sound acquisition device installed on the electric shovel to collect three-dimensional point cloud data, motion posture data, load change data, visual image data and sound data in real time during the loading process of the electric shovel, so as to obtain multimodal operation data. The multimodal operation data is timestamped based on the unified clock of the edge computing unit. Each modal data is resampled or segment matched according to a preset time window. Point cloud denoising and coordinate transformation, time series data filtering and normalization, image region extraction and sound feature extraction are performed respectively. The feature vectors obtained by modal encoding within the same time window are dimension-mapped and spliced ​​together to form a multimodal fusion feature sequence. The current loading operation stage is identified based on the motion posture data features and load change data features in the multimodal fusion feature sequence, and the operation stage is used as the starting condition for the quality evaluation task. During the unloading stage, the three-dimensional point cloud data is mapped to a reference coordinate model established by an empty standard car body. A shared height matrix is ​​formed based on the spatial grid inside the car body, and loading distribution uniformity evaluation data, material surface flatness evaluation data, and car body filling rate evaluation data are obtained based on the shared height matrix. Action standardization evaluation data are obtained at each stage of the loading operation, and cycle efficiency evaluation data are obtained after a complete loading cycle. The loading distribution uniformity evaluation data, material surface flatness evaluation data, truck bed filling rate evaluation data, action standardization evaluation data, and cycle efficiency evaluation data are used as quality evaluation data. The quality evaluation data are converted into quality evaluation results using a feedback rule base and operation guidance information is generated. The operation guidance information is fed back to the electric shovel driver, and the multimodal operation data, quality evaluation data, quality evaluation results, and feedback records are stored.

2. The method according to claim 1, characterized in that, The three-dimensional sensing sensor is a millimeter-wave radar or lidar, used to acquire three-dimensional point cloud data of the truck bed and the materials inside; the inertial measurement unit is used to acquire the attitude angle, angular velocity, and acceleration of the bucket or electric shovel actuator; the pressure sensor is installed in the bucket hydraulic circuit or lifting mechanism to acquire pressure time-series data characterizing changes in bucket load; the image acquisition device is used to assist in identifying the truck bed boundary and the relative position of the bucket and the truck bed; the sound acquisition device is used to extract the characteristics of collision sound and abnormal impact sound.

3. The method according to claim 1, characterized in that, The multimodal encoding includes: encoding 3D point cloud data using a point cloud encoder, encoding motion posture data and load change data using a temporal encoder, encoding visual image data using an image encoder, and encoding sound data using a sound encoder; mapping the feature vectors output by each encoder to a preset dimension and then concatenating them according to a time window, and forming the multimodal fusion feature sequence through a fully connected fusion layer, wherein the point cloud encoder uses PointNet or a 3D convolutional neural network, the temporal encoder uses a long short-term memory network or a Transformer model, and the image encoder and sound encoder use convolutional neural networks.

4. The method according to claim 1, characterized in that, The operation phases include the digging phase, the full bucket rotation phase, the unloading phase, the empty bucket swing phase, and the positioning phase; the operation phase identification model outputs the state probability of each operation phase, and determines the current operation phase based on the state probability, the preset duration, and the phase transition order; The action standardization evaluation task runs continuously in each operation stage, the spatial quality evaluation task based on the shared height matrix runs only in the unloading stage, and the cycle efficiency evaluation task runs after the positioning stage ends and a complete loading cycle is confirmed.

5. The method according to claim 1, characterized in that, Based on the shared height matrix, evaluation data on loading distribution uniformity, material surface flatness, and car body filling rate are obtained, including: scanning empty standard cars during system initialization, establishing a reference coordinate model with the length, width, and height directions of the car body as coordinate axes, and determining the car body floor, side wall boundaries, and standard volume. Based on the calibration parameters of the three-dimensional perception sensor relative to the electric shovel and the truck body, the three-dimensional point cloud of the unloading stage is converted to the reference coordinate model, the points outside the truck body, outliers and side wall points are removed, and the interior of the truck body is divided into multiple spatial grids. The average height of the material points in each spatial grid relative to the floor of the carriage is used as the grid height to form the shared height matrix; when the number of grid points is lower than the effective number of points threshold, the corresponding grid is marked as an invalid grid, and interpolation is performed based on the adjacent valid grids or waiting for subsequent point cloud updates; The loading distribution uniformity evaluation data is obtained based on the average height and dispersion of the regions in the length and width directions of the carriage according to the shared height matrix, and the distribution deviation direction is determined based on the average height difference between the left and right regions and the average height difference between the front and rear regions. Material surface smoothness evaluation data are obtained based on the local gradient or surface roughness formed by the height difference between adjacent grid cells in the shared height matrix. The current loading volume is calculated based on the shared height matrix, grid area, and car floor reference, and the car body filling rate evaluation data is obtained based on the ratio of the current loading volume to the standard volume.

6. The method according to claim 1, characterized in that, The motion standardization evaluation data is determined based on motion posture data, load change data, and sound data, and is used to characterize rotational stability, unloading height, unloading speed, and whether there is abnormal impact between the bucket and the truck body; the pressure time series data is used as a load change feature to participate in the identification of the operation stage and the motion standardization evaluation, and is not directly used as a material weight measurement value. The cycle efficiency evaluation data is obtained after the complete loading cycle is completed, based on the total cycle time determined by a unified timestamp and the connection interval between adjacent operation stages. The total cycle time and connection interval are normalized relative to the benchmark values ​​of the same vehicle type and the same mining conditions, and the cycle efficiency score is set from 0 to 100 according to the preset weight combination.

7. The method according to claim 3, characterized in that, The training data for the machine learning model uses the complete loading cycle as the sample unit and covers different drivers, different vehicle types, and different mining and rock conditions. The training objectives for loading distribution uniformity, material surface flatness, and truck bed filling rate are generated from the calculation results of the shared height matrix. The training objective for cycle efficiency is generated from the normalized results of the total cycle time and connection interval. The training objective for action standardization is generated by multiple annotators with experience in electric shovel operation scoring according to a unified scale and taking the median. All training objectives are uniformly mapped to a scoring range of 0 to 100. The training set, validation set, and test set are divided into units based on a complete loading cycle to avoid data segments from the same loading cycle being allocated across sets. The weighted sum of the mean squared errors of each evaluation task is used as the training loss, and the mean absolute error is used as the evaluation metric for the validation set and the test set. When the score difference between annotators exceeds the preset review threshold, the corresponding sample is reviewed or removed; the verified model version is loaded and run by the edge computing unit, and the unverified updated model does not directly replace the on-site running model.

8. The method according to claim 1, characterized in that, The feedback rule base converts quality evaluation data into quality evaluation results of normal, general deviation, serious deviation, and emergency risk according to mutually exclusive scoring intervals; when multiple rules are satisfied at the same time, they are output in the order of emergency risk, serious deviation, general deviation, and cyclical summary. The operation guidance information is provided through a human-machine interface (HMI) and voice output. The real-time guidance area of ​​the HMI displays a heat map of the car body filling and directional prompts pointing to areas with lower average heights. The results display area displays various quality evaluation data, quality evaluation results, and operation suggestions after the complete loading cycle is completed.

9. A real-time evaluation system for the loading quality of electric shovels based on multimodal data fusion, characterized in that, include: The data acquisition layer includes a three-dimensional sensing sensor, an inertial measurement unit, a pressure sensor, an image acquisition device, and a sound acquisition device installed on the electric shovel, used to collect multimodal operation data; The edge computing layer includes a time synchronization and preprocessing module, a modal coding and feature fusion module, a job stage identification module, a shared height matrix construction module, and a quality evaluation module, which are used to perform time alignment, feature fusion, job stage identification, and quality evaluation in the method of claim 1. The application feedback layer includes a feedback rule module, an HMI feedback module, a voice feedback module, and a storage management module. It is used to convert quality evaluation data into quality evaluation results and operation guidance information, output the operation guidance information to the electric shovel driver, and store multimodal operation data, quality evaluation data, quality evaluation results, and feedback records.

10. The system according to claim 9, characterized in that, It also includes a data communication module and an alarm module. The data communication module is used to upload quality evaluation results, abnormal data fragments and feedback records to the mining area supervision platform. The alarm module is used to output alarms with voice and eye-catching identification when abnormal impact between the bucket and the carriage, serious distribution deviation or filling rate exceeding the upper limit threshold is detected. The edge computing layer only loads the new model version after verification and retains the current model version for rollback.

Images