Multimodal-based port equipment data fusion method and system

Abstract

The application discloses a port equipment data fusion method and system based on multi-modal, relates to the technical field of data processing, and comprises the following steps: collecting multi-modal heterogeneous data in the running process of port equipment, forming an initial multi-modal equipment data set, preprocessing the initial multi-modal port equipment data, and obtaining a standardized multi-modal data set; extracting features from the standardized multi-modal equipment data, and obtaining state feature data corresponding to different modes; establishing a correlation between state features of different modes through the state feature data corresponding to different modes, and obtaining a multi-modal initial correlation feature set; and performing consistency analysis on the multi-modal data based on a mode conflict perception mechanism, and obtaining a reliable mode feature set after conflict correction. The application improves the reliability of multi-modal data fusion, state representation capability and accuracy of port equipment running state analysis.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for fusion of port equipment data based on multimodality. Background Technology

[0002] With the development of smart ports and automated terminals, port equipment such as quay cranes, yard cranes, automated guided vehicles (AGVs), stacker-reclaimers, tippers, belt conveyors, and bucket elevators generate multimodal heterogeneous data during operation, including visual images, vibration signals, electrical parameters, control commands, and environmental monitoring data. Since different modal data can reflect the operating status of port equipment from different perspectives, existing technologies typically employ multimodal data fusion methods to preprocess, extract features, and perform fusion analysis on data from different sources to improve the accuracy of port equipment condition monitoring and fault diagnosis.

[0003] Existing fusion methods primarily rely on feature similarity or fixed fusion strategies, failing to adequately consider intermodal conflict information and event propagation relationships. This can easily lead to duplicate or conflicting information being included in the fusion process, affecting the ability of the comprehensive state features to represent the actual operational status of port equipment. Therefore, improving the consistency of information across different modalities during multimodal data fusion and establishing a more rational fusion mechanism based on event propagation relationships are problems that current multimodal data fusion technology for port equipment needs to address. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a multimodal port equipment data fusion method to solve the problem that the multimodal data fusion results in the prior art are difficult to accurately reflect the actual operating status of port equipment.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a port equipment data fusion method based on multimodal data fusion, which includes, Collect multimodal heterogeneous data during the operation of port equipment to form an initial multimodal equipment dataset. Preprocess the initial multimodal port equipment data to obtain a standardized multimodal dataset. Feature extraction is performed on standardized multimodal equipment data to obtain state feature data corresponding to different modes; By using state feature data corresponding to different modalities, the correlation between state features of different modalities is established to obtain an initial set of multimodal correlation features; Based on the modal conflict perception mechanism, consistency analysis is performed on multimodal data to obtain a set of reliable modal features after conflict correction. Multimodal association and fusion relationships are established based on event causal chain constraints to obtain event-driven fusion features; Event-driven fusion features are fused to obtain comprehensive state features of port equipment, and the fused state results of port equipment are obtained based on the comprehensive state features.

[0007] As a preferred embodiment of the multimodal port equipment data fusion method described in this invention, the step of forming an initial multimodal equipment data set includes: Collect multimodal heterogeneous data during the operation of port equipment, including visual data, vibration data, electrical data, control data, and environmental data; Visual data, vibration data, electrical data, control data, and environmental data are labeled and processed, and data type information, acquisition time information, and equipment association information are added to them respectively. They are then combined to form an initial multimodal equipment data set.

[0008] As a preferred embodiment of the multimodal port equipment data fusion method described in this invention, the standardized multimodal data set includes: The initial multimodal port equipment dataset is cleaned, formatted, time-corrected, and normalized to remove abnormal, duplicate, and invalid data, thus obtaining the processed initial multimodal port equipment dataset. Based on the processed initial multimodal port equipment data set, the visual data, vibration data, electrical data, control data, and environmental data in the processed initial multimodal port equipment data set are unified in terms of time base and adjusted in terms of data scale to obtain a standardized multimodal data set.

[0009] As a preferred embodiment of the multimodal port equipment data fusion method described in this invention, the state feature data corresponding to the different modalities include: Modal classification processing is performed on visual data, vibration data, electrical data, control data, and environmental data in standardized multimodal equipment data to determine the spatial state information corresponding to visual data, the mechanical state information corresponding to vibration data, the dynamic state information corresponding to electrical data, the operation control information corresponding to control data, and the environmental state information corresponding to environmental data. Feature extraction is performed on the visual data, vibration data, electrical data, control data, and environmental data in the standardized multimodal equipment data after modal classification processing to obtain visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data. Visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data are integrated to form state feature data corresponding to different modes.

[0010] As a preferred embodiment of the multimodal port equipment data fusion method described in this invention, the initial multimodal association feature set includes: Temporal relationship analysis was performed on the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data corresponding to different modes. The state feature data corresponding to different modes were matched according to the acquisition time to determine the temporal change relationship of the state feature data corresponding to different modes during the operation of the same port equipment. By comparing the direction, magnitude, and timing of changes in visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data over a continuous time range, the state response relationship between the state feature data corresponding to different modes is obtained. Based on the state response relationship between the state feature data corresponding to different modes, the response process between the state feature data corresponding to different modes is matched, and the correlation between the state features of different modes is determined according to the degree of correspondence of state change time, the degree of consistency of state change direction, and the degree of overlap of state change process. Based on the correlation between different modal state features, visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data with correlation are associated and combined to form a multimodal initial correlation feature set.

[0011] As a preferred embodiment of the multimodal port equipment data fusion method described in this invention, the set of credible modal features after conflict correction includes: The visual state feature data, vibration state feature data, electrical state feature data, control state feature data and environmental state feature data in the initial association feature set of multimodal mode are calculated in chronological order to continuously change state. The starting position of state change is determined according to the state change between adjacent time nodes, and time nodes with the same continuous change direction are combined into state response intervals to obtain the set of state response intervals corresponding to different modes. Extract the start time, end time, duration, and direction of change of the state response intervals from the set of state response intervals corresponding to different modes. Use the start time, end time, duration, and direction of change of the state response as modal response matching features to obtain the set of response process features corresponding to different modes. Time offset calibration is performed on the response process features in the response process feature set corresponding to different modes, so as to use the state response interval corresponding to the control state feature data as the reference response process; By adjusting the time offset of the state response intervals corresponding to visual state feature data, vibration state feature data, electrical state feature data, and environmental state feature data, the modal response correlation is determined based on the degree of overlap of the start time of change, the degree of overlap of the duration of change, and the degree of consistency of the direction of state change after the time offset adjustment, thus obtaining a set of modal response correlations. By correcting the correlation between visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data, the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data that satisfy the modal response correlation are identified as correlated modal state feature data. Modal state feature data that do not satisfy the modal response correlation relationship are reduced in correlation degree to obtain a set of reliable modal features after conflict correction.

[0012] As a preferred embodiment of the multimodal port equipment data fusion method described in this invention, the event-driven fusion features include: The visual state feature data, vibration state feature data, electrical state feature data, control state feature data and environmental state feature data in the conflict-corrected credible modal feature set are used to identify state change trigger points. The event occurrence time node is determined according to the position where the change in state feature exceeds the state change threshold within a continuous time range. The continuously associated state change nodes are divided into equipment state change events to obtain a multimodal state change event set. The sequence of events in the set of multimodal state change events, including visual state change events, vibration state change events, electrical state change events, control state change events, and environmental state change events, is analyzed. By comparing the occurrence time, end time, and duration of different modal state change events, a set of multimodal event temporal relationships is obtained. Based on the event propagation path analysis of different modal state change events in the multimodal event time sequence relationship set, the event propagation direction is determined according to the state change response relationship between successive events; By determining whether a previous state change event causes a subsequent state change event, the causal relationship between state change events of different modalities is determined, and a set of causal relationships of events is obtained. The association paths of visual state change events, vibration state change events, electrical state change events, control state change events, and environmental state change events in the event causal relationship set are filtered, and the association paths of multimodal state change events that satisfy the event occurrence order, state response relationship, and event propagation direction are retained to obtain the event causal chain set. The fusion relationship is determined by the position of different modal state feature data in the event causal chain. Different modal state feature data in the same event propagation path are combined to obtain event-driven fusion features.

[0013] As a preferred embodiment of the multimodal port equipment data fusion method described in this invention, the comprehensive state characteristics of the port equipment include: Based on the event causal chain set corresponding to the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data in the event-driven fusion features, the different modal state feature data are arranged according to the event occurrence order and state response order to obtain the event-driven fusion features after event association sorting. By analyzing the state change information and state response relationship corresponding to different modal state feature data, the fusion correlation relationship between different modal state feature data is determined, and a fusion correlation feature set is obtained. Visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data are fused together, and repeated state information in the propagation process of the same event is merged to obtain a comprehensive state feature set. The spatial status information, mechanical status information, power status information, control status information, and environmental status information are integrated to form the comprehensive status characteristics of port equipment.

[0014] As a preferred embodiment of the multimodal port equipment data fusion method described in this invention, the port equipment fusion status result includes: Based on the correspondence between spatial state information, mechanical state information, power state information, control state information and environmental state information, the associated states between different state dimensions of port equipment are determined. By comparing the change direction, change degree and change time corresponding to different state information, the overall operating state change of port equipment is determined. State fusion analysis is performed on spatial state information, mechanical state information, dynamic state information, control state information and environmental state information, and the state descriptions corresponding to different state dimensions are comprehensively correlated to form fused state information; The system identifies and classifies the state categories and levels of spatial state information, mechanical state information, power state information, control state information, and environmental state information, and generates a fusion state result for port equipment by combining the correlation between different state information.

[0015] Secondly, this invention provides a port equipment data fusion system based on multimodal data fusion, including: The preprocessing module collects multimodal heterogeneous data during the operation of port equipment to form an initial multimodal equipment data set. It then preprocesses the initial multimodal port equipment data to obtain a standardized multimodal data set. The extraction module performs feature extraction on standardized multimodal equipment data to obtain state feature data corresponding to different modes. The association module establishes the association relationship between the state features of different modes through the state feature data corresponding to different modes, and obtains the initial association feature set of multimodal modes; The analysis module performs consistency analysis on multimodal data based on the modal conflict perception mechanism to obtain a set of reliable modal features after conflict correction. The fusion module establishes multimodal association fusion relationships based on event causal chain constraints to obtain event-driven fusion features; The integration module fuses event-driven fusion features to obtain comprehensive status features of port equipment, and then obtains the fusion status results of port equipment based on these comprehensive status features.

[0016] The beneficial effects of this invention are as follows: By collecting visual data, vibration data, electrical data, control data, and environmental data during the operation of port equipment, a multimodal initial correlation feature set is established after preprocessing and feature extraction; a modal conflict perception mechanism is introduced to perform consistency analysis on the state features of different modalities, correct conflict information, and obtain a credible modal feature set; an event causal chain is constructed by combining the event occurrence sequence, state response relationship, and event propagation direction, an event-driven multimodal correlation fusion relationship is established, and different modal state features are fused according to the event causal chain to eliminate duplicate state information during the propagation of the same event, forming a comprehensive state feature of port equipment, obtaining a fused state result that can accurately reflect the operating state of port equipment, and improving the credibility of multimodal data fusion, state characterization capability, and accuracy of port equipment operating state analysis. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a multimodal port equipment data fusion method.

[0019] Figure 2 This is a schematic diagram of a port equipment data fusion system based on multimodality. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Reference Figures 1-2 As one embodiment of the present invention, this embodiment provides a multimodal port equipment data fusion method, including the following steps: S1. Collect multimodal heterogeneous data during the operation of port equipment to form an initial multimodal equipment data set. Preprocess the initial multimodal port equipment data to obtain a standardized multimodal data set.

[0024] S1.1 Collect multimodal heterogeneous data during the operation of port equipment, including visual data, vibration data, electrical data, control data, and environmental data.

[0025] Furthermore, different types of operational status information are acquired through data acquisition devices installed at various locations on the port equipment. Visual data characterizes the external structural state, operational posture, and spatial state changes of the port equipment; vibration data characterizes the vibration response of the mechanical components during operation; electrical data characterizes the changes in electrical parameters during operation; control data characterizes the changes in control commands during control execution; and environmental data characterizes the changes in environmental conditions during operation. By collecting visual, vibration, electrical, control, and environmental data from different sources and in different formats, the acquired data can reflect the spatial, mechanical, dynamic, operational control, and environmental state information of the port equipment during operation.

[0026] S1.2 Perform data identification processing on visual data, vibration data, electrical data, control data, and environmental data, add data type information, acquisition time information, and equipment association information respectively, and combine them to form an initial multimodal equipment data set.

[0027] Furthermore, based on different data sources, corresponding data type information is added to visual data, vibration data, electrical data, control data, and environmental data to distinguish the status information categories corresponding to different data. Based on the time recording information during different data acquisition processes, acquisition time information is added to visual data, vibration data, electrical data, control data, and environmental data to determine the time correspondence of different data during port equipment operation. Based on the port equipment and acquisition location information corresponding to different data, equipment association information is added to visual data, vibration data, electrical data, control data, and environmental data to determine the association between different data and corresponding port equipment. After completing the addition of data type information, acquisition time information, and equipment association information, the visual data, vibration data, electrical data, control data, and environmental data of the corresponding port equipment are associated and combined according to the equipment association information. This allows data from different sources to form an associated data set according to the correspondence, ensuring that subsequent processing can perform joint analysis on different modal data during the operation of the same port equipment, forming an initial multimodal equipment data set.

[0028] S1.3. Perform data cleaning, format unification, time correction and normalization on the initial multimodal port equipment dataset to remove abnormal data, duplicate data and invalid data in the initial multimodal port equipment dataset, and obtain the processed initial multimodal port equipment dataset.

[0029] Furthermore, abnormal, duplicate, and invalid data generated during the data acquisition process are identified and removed. Abnormal data includes data missing, data deviating from the normal range of variation, and data recording errors that occur during the acquisition process. Duplicate data includes data recorded repeatedly within the same acquisition time. Invalid data includes data that cannot reflect the operating status of port equipment. The data formats of visual, vibration, electrical, control, and environmental data from different sources are standardized to ensure that different types of data can be stored and processed using a unified data representation format. Based on the acquisition time information corresponding to different data, time correction is performed on visual, vibration, electrical, control, and environmental data with time deviations to ensure that the state change processes corresponding to different modal data are under a unified time reference. Normalization processing is performed based on the data scale characteristics of different modal data to obtain the processed initial multimodal port equipment data set.

[0030] S1.4. Based on the processed initial multimodal port equipment data set, the visual data, vibration data, electrical data, control data, and environmental data in the processed initial multimodal port equipment data set are unified in terms of time base and adjusted in terms of data scale to obtain a standardized multimodal data set.

[0031] Furthermore, based on the acquisition time information in the processed initial multimodal port equipment dataset, the visual data, vibration data, electrical data, control data, and environmental data are processed to unify the time reference. This allows the state change processes corresponding to different modal data to be analyzed according to the same time reference, avoiding deviations in the judgment of state change relationships between different modalities caused by different data acquisition frequencies and time offsets. Based on the data characteristics of visual data, vibration data, electrical data, control data, and environmental data, the data scale of different modal data is adjusted so that different modal data have a unified data expression range while maintaining their own state change characteristics, thus obtaining a standardized multimodal dataset.

[0032] S2. Extract features from standardized multimodal equipment data to obtain state feature data corresponding to different modes.

[0033] S2.1 Perform modal classification processing on the visual data, vibration data, electrical data, control data and environmental data in the standardized multimodal equipment data to determine the spatial state information corresponding to the visual data, the mechanical state information corresponding to the vibration data, the dynamic state information corresponding to the electrical data, the operation control information corresponding to the control data and the environmental state information corresponding to the environmental data.

[0034] Furthermore, based on the data type information corresponding to different data sources in the standardized multimodal equipment data, visual data, vibration data, electrical data, control data, and environmental data are classified and processed. Data with image representation is classified as visual data, data reflecting the motion response and structural vibration changes of equipment components is classified as vibration data, data reflecting the power supply status and operating parameter changes of equipment is classified as electrical data, data reflecting equipment control commands and operation adjustment processes is classified as control data, and data reflecting changes in the equipment operating environment is classified as environmental data.

[0035] Specifically, visual data is used to describe the location, shape changes, and spatial state information of the external structure of port equipment; vibration data is used to describe the vibration response and mechanical state information of the mechanical components of port equipment during operation; electrical data is used to describe the changes in electrical parameters and power state information of port equipment during operation; control data is used to describe the control behavior and operation control information of port equipment during the execution of control; and environmental data is used to describe the changes in environmental conditions and environmental state information within the operating area of ​​port equipment. This allows different modal data to be mapped to different state dimensions during the operation of port equipment, determining the spatial state information corresponding to visual data, the mechanical state information corresponding to vibration data, the power state information corresponding to electrical data, the operation control information corresponding to control data, and the environmental state information corresponding to environmental data.

[0036] S2.2. Perform feature extraction on the visual data, vibration data, electrical data, control data, and environmental data in the standardized multimodal equipment data after modal classification processing to obtain visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data.

[0037] Furthermore, based on the spatial state information corresponding to the visual data, feature extraction is performed on the image change information in the visual data to obtain visual state feature data reflecting the external structural state, positional change state, and operational posture change state of the port equipment; based on the mechanical state information corresponding to the vibration data, feature extraction is performed on the vibration response change information in the vibration data to obtain vibration state feature data reflecting the mechanical operating state and vibration change process of the port equipment; based on the power state information corresponding to the electrical data, feature extraction is performed on the operating parameter change information in the electrical data to obtain electrical state feature data reflecting the power output state and electrical operation change process of the port equipment; based on the operation control information corresponding to the control data, feature extraction is performed on the control command change information and execution state change information in the control data to obtain control state feature data reflecting the operation adjustment process of the port equipment; based on the environmental state information corresponding to the environmental data, feature extraction is performed on the environmental parameter change information in the environmental data to obtain environmental state feature data reflecting the changes in the operating environment of the port equipment.

[0038] S2.3 Integrate visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data to form state feature data corresponding to different modes.

[0039] Furthermore, based on the state information categories corresponding to visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data, the state feature data corresponding to different modes are classified and organized. This ensures that visual state feature data corresponds to spatial state change processes, vibration state feature data corresponds to mechanical state change processes, electrical state feature data corresponds to dynamic state change processes, control state feature data corresponds to operational control change processes, and environmental state feature data corresponds to environmental state change processes. Based on the data correlation relationships between the state feature data corresponding to different modes, the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data are integrated accordingly. This ensures that the state feature data of different modes maintains a consistent correlation with the corresponding port equipment operation process, forming state feature data corresponding to different modes.

[0040] S3. By using the state feature data corresponding to different modalities, establish the correlation between the state features of different modalities and obtain the initial correlation feature set of multimodalities.

[0041] S3.1 Perform time relationship analysis on the visual state feature data, vibration state feature data, electrical state feature data, control state feature data and environmental state feature data in the state feature data corresponding to different modes, match the state feature data corresponding to different modes according to the acquisition time, and determine the time change relationship of the state feature data corresponding to different modes during the operation of the same port equipment.

[0042] Furthermore, based on the acquisition time information corresponding to visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data, the state feature data corresponding to different modes are arranged in chronological order, and time matching is performed according to the same acquisition time or adjacent acquisition time, so that the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data form corresponding time series. According to the state changes corresponding to each time node in the time series, the chronological relationship of the changes of visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data during the operation of port equipment is analyzed. When the state feature data corresponding to different modes all undergo state changes within the same time range, it is determined that there is a corresponding time relationship between the state feature data corresponding to different modes. When the state feature data corresponding to different modes undergo sequential changes, a corresponding time relationship is established according to the order of state changes, thus determining the time change relationship of the state feature data corresponding to different modes during the operation of the same port equipment.

[0043] S3.2 By comparing the direction, magnitude, and timing of changes in visual state characteristic data, vibration state characteristic data, electrical state characteristic data, control state characteristic data, and environmental state characteristic data over a continuous time range, the state response relationship between the state characteristic data corresponding to different modes is obtained.

[0044] Furthermore, based on the temporal relationship, state change analysis is performed on visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data over a continuous time range. The direction, amplitude, and occurrence time of change of the state feature data corresponding to different modes within the continuous time range are determined. The direction of change characterizes the trend of state change, the amplitude characterizes the degree of state change, and the occurrence time characterizes the temporal location corresponding to the state change. The direction of change within the corresponding time range is compared among the visual, vibration, electrical, control, and environmental state feature data. When the state feature data corresponding to different modes have the same direction of change, the state change trend is considered consistent. The amplitude of change is compared; when the amplitude of change of the state feature data corresponding to different modes changes synchronously or continuously, the degree of state change is considered to be corresponding. The occurrence time of change is compared; when the state feature data corresponding to different modes changes within the same or continuous time range, the state change is considered to have a time response relationship, thus obtaining the state response relationship between the state feature data corresponding to different modes.

[0045] S3.3. Based on the state response relationship between the state feature data corresponding to different modes, match the response process between the state feature data corresponding to different modes, and determine the correlation between the state features of different modes based on the degree of correspondence of state change time, the degree of consistency of state change direction, and the degree of overlap of state change process.

[0046] Furthermore, based on the start and end times of state changes, the degree of correspondence between the state change time ranges of different modal state feature data is compared. When the state change time ranges of different modal state feature data are the same or within a continuous response time range, a correspondence is determined between the state change times. The direction of state change is used to determine whether the change trends of different modal state feature data are consistent. When the change directions of different modal state feature data are the same, a consistent relationship between the state change directions is determined. The duration of state change is used to determine whether there is a continuous overlap between the state change processes of different modal state feature data. When there is a continuous correspondence between the state change duration processes of different modal state feature data, a correspondence between the state change processes is determined. When different modal state feature data simultaneously satisfy the correspondence between state change time, the consistency of state change direction, and the correspondence between state change processes, a correlation is determined between the different modal state feature data, thus obtaining the correlation between different modal state features.

[0047] S3.4. Based on the correlation between different modal state features, the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data with correlation are associated and combined to form a multimodal initial correlation feature set.

[0048] Furthermore, based on the correlation between different modal state characteristics, visual state characteristic data, vibration state characteristic data, electrical state characteristic data, control state characteristic data, and environmental state characteristic data are screened accordingly. Visual state characteristic data, vibration state characteristic data, electrical state characteristic data, control state characteristic data, and environmental state characteristic data with correlation are combined according to the correspondence, so that different modal state characteristic data with corresponding state response relationships during the operation of the same port equipment form a unified correlation unit. According to the corresponding time change relationship between different correlation units, multiple correlation units are sorted and summarized, so that the correlation units formed in different time ranges are arranged in chronological order, forming a multimodal initial correlation feature set containing the correlation between visual state characteristic data, vibration state characteristic data, electrical state characteristic data, control state characteristic data, and environmental state characteristic data.

[0049] S4. Based on the modal conflict perception mechanism, perform consistency analysis on multimodal data to obtain a set of reliable modal features after conflict correction.

[0050] S4.1 Calculate the continuous state change of the visual state feature data, vibration state feature data, electrical state feature data, control state feature data and environmental state feature data in the initial association feature set of multimodal modes in chronological order. Determine the starting position of state change based on the state change between adjacent time nodes, and combine time nodes with the same continuous change direction into state response intervals to obtain the set of state response intervals corresponding to different modes.

[0051] Furthermore, the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data in the initial multimodal correlation feature set are arranged according to the acquisition time sequence to form time series corresponding to different modal state feature data. For any time series corresponding to modal state feature data, the continuous state change amount is calculated based on the changes in state feature values ​​between adjacent time nodes. The continuous state change amount is used to characterize the degree of state change between adjacent time nodes. Based on the continuous state change amount, it is determined whether there is a continuous state change process between different time nodes. When the continuous state change amount corresponding to continuous time nodes maintains the same direction of change, the continuous time nodes are determined as the same state change process. When the continuous state change amount corresponding to continuous time nodes changes direction or the state change is interrupted, the starting position and ending position of the state change are determined based on the position of the change direction. Time nodes with the same continuous change direction are combined, and the time range corresponding to the starting position to the ending position of the state change is determined as the state response interval, thus obtaining the state response interval sets corresponding to the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data, respectively.

[0052] Define the time series corresponding to the state feature data of any modality in the initial multimodal association feature set: ; in, For the first The time series corresponding to each modal state feature data For the first The state feature values ​​corresponding to each time node. This is an index for modal types, which include visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data. This represents the number of time nodes in the time series.

[0053] Calculate the continuous state changes between adjacent time points in chronological order: ; in, For the first The modal state feature data in the first... The continuous state changes corresponding to each time point. For the first The state feature values ​​corresponding to each time node. For the first The state feature values ​​corresponding to each time node.

[0054] The expression for continuous state change is: ; in, This represents the normalized continuous state change. For the first The maximum change amplitude in the time series corresponding to each modal state feature data. To prevent tiny adjustment parameters with a denominator of zero.

[0055] S4.2 Extract the start time, end time, duration, and direction of change of the state response intervals from the set of state response intervals corresponding to different modes, and use the start time, end time, duration, and direction of change of the change as modal response matching features to obtain the set of response process features corresponding to different modes.

[0056] Furthermore, based on the set of state response intervals corresponding to different modes, features are extracted from the state response intervals corresponding to visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data. The change start time, change end time, change duration, and state change direction corresponding to each state response interval are determined. The change start time indicates the time position when the state change begins, the change end time indicates the time position when the state change ends, the change duration indicates the duration of the state change process, and the state change direction indicates the trend of the state feature data within the state response interval. The change start time, change end time, change duration, and state change direction are combined to form modal response matching features that can describe the change process of state feature data of different modes. This allows the state response intervals corresponding to different modes to be compared through unified features, thereby obtaining the set of response process features corresponding to different modes.

[0057] S4.3 Perform time offset calibration on the response process features in the response process feature set corresponding to different modes, and use the state response interval corresponding to the control state feature data as the reference response process.

[0058] Furthermore, based on the response process feature sets corresponding to different modes, time correspondence analysis is performed on the response process features corresponding to visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data to determine the time offset relationship between the state feature data of different modes. Taking the state response interval corresponding to the control state feature data as the reference response process, the time offset adjustment is performed on the response process features corresponding to visual state feature data, vibration state feature data, electrical state feature data, and environmental state feature data according to the time position of the state change of the control state feature data, so that the state response processes corresponding to the state feature data of different modes can be mapped to a unified time reference, thus completing the time offset calibration of the response process features.

[0059] S4.4 By adjusting the time offset of the state response intervals corresponding to the visual state feature data, vibration state feature data, electrical state feature data and environmental state feature data, the modal response correlation is determined based on the degree of overlap of the start time of change, the degree of overlap of the duration of change and the degree of consistency of the direction of state change after the time offset adjustment, and a set of modal response correlations is obtained.

[0060] Furthermore, a time correspondence analysis was performed on the state response intervals corresponding to visual state feature data, vibration state feature data, electrical state feature data, and environmental state feature data to obtain the time offset relationship between different modal state feature data. Due to differences in acquisition frequency and state response delays in the data acquisition process for different modalities, the state response intervals corresponding to different modal state feature data may have time misalignments. Therefore, based on the state response process corresponding to the control state feature data as a reference, the time offset of the state response intervals corresponding to visual state feature data, vibration state feature data, electrical state feature data, and environmental state feature data was adjusted to ensure that the state response intervals corresponding to different modal state feature data are within a unified time reference range.

[0061] The starting times of changes in the state response intervals corresponding to different modal state characteristic data are compared. When the starting times of changes in two state response intervals are within the corresponding time range after time offset adjustment, it is determined that the starting times of changes are coincident. The duration of changes in the state response intervals corresponding to different modal state characteristic data is compared. When the continuous change processes of two state response intervals are corresponding, it is determined that the duration of changes is coincident. The direction of state change within the state response intervals corresponding to different modal state characteristic data is compared. When the direction of state change corresponding to different modal state characteristic data is consistent, it is determined that the direction of state change is consistent.

[0062] Based on the degree of overlap in the start time, duration, and direction of change, a comprehensive judgment is made regarding the relationships between visual state feature data, vibration state feature data, electrical state feature data, and environmental state feature data. When different modal state feature data simultaneously satisfy the correspondences of start time, duration, and direction of change, a modal response correlation is established. When different modal state feature data only satisfy some of the judgment conditions, the reliability of the corresponding modal response correlation is reduced, and a valid modal response correlation is not determined. Based on the above judgment results, a set of modal response correlations corresponding to visual state feature data, vibration state feature data, electrical state feature data, and environmental state feature data is obtained.

[0063] S4.5. By correcting the correlation between visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data, the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data that satisfy the modal response correlation relationship are determined as the correlated modal state feature data.

[0064] Furthermore, based on the modal response correlation set, the existing correlations between visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data in the initial multimodal correlation feature set are reassessed. Visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data that meet the conditions of corresponding change start time, corresponding change duration, and consistent state change direction in the state response process are retained as correlated modal state feature data. For state feature data that originally had correlations but did not meet the modal response correlations, the credibility of the corresponding correlations is reduced, so that the correlations between different modal state feature data can be corrected according to the actual state response process, and the correlated modal state feature data are determined.

[0065] S4.6 Reduce the correlation degree of modal state feature data that do not satisfy the modal response correlation relationship to obtain a set of reliable modal features after conflict correction.

[0066] Let the first The association confidence value of each modal state feature data is: ; in, The degree of overlap in the start times of the changes The degree of overlap in the duration of the changes The degree to which the direction of state change is consistent.

[0067] The weighting coefficients represent the degree of overlap in the start times of the changes. The weighting coefficients represent the degree of overlap in the duration of the changes. The weighting coefficient represents the degree of consistency in the direction of state changes.

[0068] when: Then reduce the association weight of the modality state feature data: ; in, Original modal association weights, The modal association weights are after conflict correction.

[0069] By correcting the correlation between visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data, the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data that satisfy the modal response correlation are identified as correlated modal state feature data. For modal state feature data that do not satisfy the modal response correlation relationship, calculate the correlation confidence value, adjust the correlation weight of the corresponding modal state feature data according to the correlation confidence value, reduce its contribution ratio in multimodal fusion calculation, reduce its correlation with other modal state feature data, and combine the modal state feature data with the correlated modal state feature data to obtain the conflict-corrected confidence modal feature set.

[0070] Specifically, based on the modal response correlation set, the correlation status between visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data is determined. When the degree of overlap in the start time, duration, or direction of change of different modal state feature data does not meet the modal response correlation requirements, a response conflict is determined to exist between the corresponding modal state feature data. When only some response conditions are mismatched between modal state feature data, the corresponding modal state feature data is determined to be weakly correlated, and the correlation priority of the modal state feature data corresponding to the weakly correlated state is reduced in the correlation combination process. When there is no correspondence in change time, duration, or direction of change between modal state feature data, the correlation degree between the corresponding modal state feature data is reduced, so that the corresponding modal state feature data is not used as the main correlation basis in subsequent correlation combinations.

[0071] For visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data that satisfy the modal response correlation relationship, the corresponding correlation relationship is maintained; for modal state feature data that do not satisfy the modal response correlation relationship, the correlation degree is reduced, so that the influence of modal state feature data with response conflicts on the multimodal state feature correlation process is reduced, while modal state feature data with auxiliary state expression function is retained, and a set of reliable modal features after conflict correction is obtained.

[0072] S5. Establish multimodal association and fusion relationships based on event causal chain constraints to obtain event-driven fusion features.

[0073] S5.1. Identify the state change trigger points of the visual state feature data, vibration state feature data, electrical state feature data, control state feature data and environmental state feature data in the conflict-corrected credible modal feature set. Determine the event occurrence time node based on the position where the state feature change exceeds the state change threshold within a continuous time range. Divide the continuously associated state change nodes into equipment state change events to obtain a multimodal state change event set.

[0074] Furthermore, based on the time change processes corresponding to the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data in the conflict-corrected credible modal feature set, the state change amount between adjacent state feature data within a continuous time range is calculated in chronological order. The state change amount is then compared with a state change threshold. When the state feature change amount within a continuous time range exceeds the state change threshold, the corresponding time position is determined as the state change trigger position, and the time corresponding to the state change trigger position is determined as the event occurrence time node. The state change threshold is used to determine whether the state feature data has undergone a change process with actual state change significance. When the state feature change amount does not exceed the state change threshold, the state feature data is determined to be within the normal change range. When the state feature change amount exceeds the state change threshold, the state feature data is determined to have undergone a state change.

[0075] Specifically, based on the state changes corresponding to different time points, state change nodes with related changes within a continuous time range are combined. When visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data all have corresponding state changes within a continuous time range, the corresponding state change nodes are divided into the same equipment state change event, thus transforming the continuous state change process into equipment state change events with a time range, and obtaining a multimodal state change event set.

[0076] S5.2 Analyze the sequence of occurrence of visual state change events, vibration state change events, electrical state change events, control state change events, and environmental state change events in the multimodal state change event set. By comparing the occurrence time, end time, and duration of different modal state change events, obtain the set of multimodal event time sequence relationships.

[0077] Furthermore, the occurrence time, end time, and duration of different modal state change events are obtained, and the visual state change events, vibration state change events, electrical state change events, control state change events, and environmental state change events are sorted according to their occurrence time to determine the initial occurrence order of different modal state change events. By comparing the occurrence time, end time, and duration of different modal state change events, the temporal relationship between different modal state change events is analyzed. When the occurrence and end times of two modal state change events coincide, a time synchronization relationship is determined between the two modal state change events. When the end time of one modal state change event precedes the occurrence time of another modal state change event, the event propagation order is determined based on the corresponding time sequence of the two modal state change events. When the durations of two modal state change events have a continuous correspondence, a continuous process correspondence is determined between the two modal state change events. Based on the time synchronization, event sequence, and continuous process correspondence between different modal state change events, the temporal relationships between different modal state change events are combined to form the event occurrence sequence relationship between different modal state change events, thus obtaining a set of multimodal event temporal relationships.

[0078] S5.3. Based on the event propagation path analysis of different modal state change events in the multimodal event time sequence relationship set, the event propagation direction is determined according to the state change response relationship between the preceding and following events.

[0079] Furthermore, based on the chronological order of occurrence of state change events corresponding to different modalities, adjacent state change events are treated as pairs of events to be analyzed, and the changes in the state feature data corresponding to the state change event after the occurrence of the preceding state change event are compared. When a subsequent state change event occurs after the occurrence of the preceding state change event, and the direction of the state change is consistent with the state response process, an event propagation association is determined between the preceding and subsequent state change events. Based on the event propagation association, the event propagation direction between different modalities is determined. State change events that occur earlier and can trigger subsequent state changes are identified as preceding propagation events, and state change events that occur later and produce corresponding state responses are identified as subsequent response events. Event propagation connections are established based on the time direction from preceding propagation events to subsequent response events. Multiple related event propagation connections are combined. When multiple state change events satisfy the conditions of continuous occurrence order, state response relationship, and consistent event propagation direction, the multiple event propagation connections are connected to form a continuous event propagation path, thus obtaining the event propagation path between different modalities of state change events.

[0080] S5.4 By determining whether a previous state change event causes a subsequent state change event, the causal relationship between different modal state change events is determined, and a set of event causal relationships is obtained.

[0081] Furthermore, the state change information corresponding to two adjacent state change events in the event propagation path is obtained. The state change event that occurs earlier is determined as the previous state change event, and the state change event that occurs later is determined as the next state change event. The occurrence time, state change direction, and state change process corresponding to the previous state change event and the next state change event are obtained respectively.

[0082] The following criteria are used to determine the order of state changes: First, the occurrence of the subsequent state change event is determined by its temporal sequence. If the subsequent state change event occurs after the preceding state change event, and the state change process of the subsequent state change event follows the preceding state change event, then the two state change events satisfy the order of occurrence. Second, the response is determined by the changes in the corresponding state characteristic data of the subsequent state change event. If the state characteristic data of the subsequent state change event shows a state change related to the preceding state change event, and the direction of change is consistent, then the two state change events satisfy the response relationship. Third, the state change trends of the preceding and subsequent state change events are compared. If the direction of state change of the two state change events is consistent, or the state change processes have a continuous correspondence, then the two state change events satisfy the relationship of consistent state change direction.

[0083] The correlation between a preceding state change event and a subsequent state change event is determined based on the order of events, the response to state changes, and the consistency of the direction of state changes. When all three conditions are met, a state propagation influence relationship is established between the preceding and subsequent state change events, and this correlation is defined as a causal relationship. When only some conditions are met, a causal relationship is not determined. A set of causal relationships is obtained based on this process.

[0084] S5.5. Perform path filtering on visual state change events, vibration state change events, electrical state change events, control state change events, and environmental state change events in the event causal relationship set, retain the multimodal state change event association paths that satisfy the event occurrence order, state response relationship, and event propagation direction, and obtain the event causal chain set.

[0085] Furthermore, based on the set of causal relationships between events, the association paths formed between visual state change events, vibration state change events, electrical state change events, control state change events, and environmental state change events are analyzed to obtain the propagation paths between different modal state change events. The order of event occurrence, state response relationship, and event propagation direction in different association paths are judged. When the state change events in the association path occur in chronological order, and there is a corresponding state response relationship between adjacent state change events, and the event propagation direction remains continuous and consistent, the corresponding multimodal state change event association path is retained. When there are cases where the order of event occurrence is inconsistent, the state response relationship is mismatched, or the event propagation direction is discontinuous in the association path, the credibility of the corresponding association path is reduced. By filtering the multimodal state change event association paths that meet the conditions, a set of event causal chains is obtained.

[0086] S5.6 Determine the fusion correlation based on the position of different modal state feature data in the event causal chain, combine different modal state feature data in the same event propagation path, and obtain event-driven fusion features.

[0087] Furthermore, based on the order of state change events in the event causal chain set, the positional relationship of the corresponding state change events in the event causal chain is obtained for visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data. Different modal state feature data that are on the same event propagation path and have a preceding / following response relationship are identified as associated modal state feature data. When different modal state feature data are located at consecutive event positions on the same event propagation path, and there is an event propagation correlation between preceding and following state change events, a fusion correlation is determined between the different modal state feature data. When different modal state feature data are on different event propagation paths, or there is no event propagation correlation, a fusion correlation is not determined between the different modal state feature data, and the fusion correlation between the different modal state feature data is obtained.

[0088] According to the order of events in the event causal chain, the modal state feature data are arranged, and the modal state feature data located at the beginning of event propagation, the intermediate response position of event propagation, and the result position of event propagation are combined accordingly to maintain the positional relationship of different modal state feature data in the event propagation process. For different modal state feature data in the same event propagation path, visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data with state response relationship are combined into the same event association unit, so that different modal state feature data can jointly describe the state change process of the same equipment. Event-driven fusion features are formed based on multiple event association units.

[0089] S6. Fuse the event-driven fusion features to obtain the comprehensive state features of port equipment, and obtain the fusion state results of port equipment based on the comprehensive state features.

[0090] S6.1 Based on the event causal chain set corresponding to the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data in the event-driven fusion features, arrange the different modal state feature data according to the event occurrence order and state response order to obtain the event-driven fusion features after event association sorting.

[0091] Furthermore, based on the event causal chain set corresponding to different modal state feature data in the event-driven fusion features, the corresponding positions of visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data in the event causal chain are obtained. The different modal state feature data are then sorted according to the chronological order of event occurrence. Based on the state response relationship between different modal state feature data in the event causal chain set, the response order of different modal state feature data within the same event propagation process is adjusted, placing modal state feature data whose state changes occur first at the beginning and those whose state changes occur later at the end. When different modal state feature data exist simultaneously within the same event time range, the arrangement order is determined according to the state response relationship, enabling the different modal state feature data to be organized according to the event propagation process, thus obtaining the event-driven fusion features after event association and sorting.

[0092] S6.2 By analyzing the state change information and state response relationship corresponding to different modal state feature data, the fusion correlation relationship between different modal state feature data is determined, and the fusion correlation feature set is obtained.

[0093] Furthermore, based on the event-driven fusion features after event association and sorting, the state change information corresponding to visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data is analyzed. The direction, degree, and time of state change of different modal state feature data during event propagation are obtained. Based on the state response relationship between different modal state feature data, it is determined whether different modal state feature data correspond to the same state change process. When there is a corresponding state change time, the same change direction, or a continuous response process among different modal state feature data, a fusion association relationship is determined to exist between them; when there is no corresponding state change process among different modal state feature data, a fusion association relationship is not determined. Based on the fusion association relationship between different modal state feature data, the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data with association relationships are linked and organized to obtain a fusion association feature set.

[0094] S6.3. The visual state feature data, vibration state feature data, electrical state feature data, control state feature data and environmental state feature data are fused together, and the repeated state information in the same event propagation process is merged to obtain a comprehensive state feature set.

[0095] Furthermore, based on the correlation between visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data in the fusion and correlation feature set, the state feature data of different modes in the same event propagation process are fused and processed, and the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data are converted into corresponding state feature vectors. By comparing the degree of information similarity between the state feature vectors corresponding to different modal state feature data, it is determined whether there is duplicate state information between different modal state feature data.

[0096] When the state feature vectors corresponding to different modal state feature data have a high degree of similarity, it is determined that there is duplicate state information between different modal state feature data, and the impact of duplicate state information on the fusion process is reduced. When the state feature vectors corresponding to different modal state feature data have a low degree of similarity, the differentiated state change information corresponding to different modal state feature data is retained. Based on the event position relationship corresponding to different modal state feature data in the event causal chain set, the event contribution relationship corresponding to different modal state feature data is determined, so that the modal state feature data that is in the key position of event propagation and can reflect the state change process retains the corresponding state information.

[0097] Based on the influence relationship corresponding to the repetitive state information and the event contribution relationship corresponding to the state feature data of different modalities, visual state feature data, vibration state feature data, electrical state feature data, control state feature data and environmental state feature data are fused. The repetitive state information in the propagation process of the same event is merged, and the effective state change information corresponding to the state feature data of different modalities is retained, so that the state feature data of different modalities form a unified state expression and obtain a comprehensive state feature set.

[0098] S6.4 Integrate spatial status information, mechanical status information, power status information, control status information, and environmental status information to form comprehensive status characteristics of port equipment.

[0099] Furthermore, based on the state information categories corresponding to different modal state feature data in the comprehensive state feature set, the spatial state information corresponding to visual state feature data, the mechanical state information corresponding to vibration state feature data, the dynamic state information corresponding to electrical state feature data, the control state information corresponding to control state feature data, and the environmental state information corresponding to environmental state feature data are integrated accordingly. Based on the correlation between different state information, the spatial state information, mechanical state information, dynamic state information, control state information, and environmental state information are combined according to the port equipment operation state change process, so that different state dimension information can jointly characterize the port equipment operation state and form the comprehensive state features of port equipment.

[0100] S6.5. Based on the correspondence between spatial state information, mechanical state information, power state information, control state information and environmental state information, determine the associated states between different state dimensions of port equipment. By comparing the change direction, change degree and change time corresponding to different state information, determine the overall operating state change of port equipment.

[0101] Furthermore, based on the corresponding changes in spatial, mechanical, power, control, and environmental state information within the comprehensive state characteristics of port equipment, the correspondence between different state dimensions is analyzed. When different state information changes within the same time frame and in the same direction, a synchronous change relationship is established between the different state dimensions. When different state information changes sequentially according to the event propagation order, a responsive change relationship is established between the different state dimensions. When different state information changes in opposite directions or there is no corresponding relationship in their change times, the correlation between the different state dimensions is determined to be low. Based on the change direction, degree of change, and change time corresponding to different state information, the overall operational state change process of the port equipment is judged, and the overall operational state change status of the port equipment is determined.

[0102] S6.6 Perform state fusion analysis on spatial state information, mechanical state information, dynamic state information, control state information and environmental state information, and comprehensively associate the state descriptions corresponding to different state dimensions to form fused state information.

[0103] Furthermore, based on the correlation between spatial state information, mechanical state information, power state information, control state information, and environmental state information, a comprehensive analysis is conducted on the state descriptions corresponding to different state dimensions. Different state information reflecting the operation process of the same port equipment is associated and combined. Based on the changing relationships between different state dimensions, spatial state information, mechanical state information, power state information, control state information, and environmental state information are converted into fused state information describing the overall operating state change process of the port equipment, thus forming fused state information.

[0104] S6.7. Identify the state categories and classify the state levels of spatial state information, mechanical state information, power state information, control state information and environmental state information, and generate port equipment fusion state results by combining the correlation between different state information.

[0105] Furthermore, based on the changes in spatial, mechanical, power, control, and environmental state information, the current operating state of the port equipment is determined to belong to a specific state category, and a state level is assigned according to the degree of change corresponding to different state information. When all different state information remains stable, the port equipment operating state is determined to be a stable state; when some state information undergoes abnormal changes but no continuous response relationship is formed, the port equipment operating state is determined to be a changing state; when multiple state information undergoes abnormal changes simultaneously, and there is a corresponding correlation between different state information, the port equipment operating state is determined to be an abnormal state. Combining the correlation between different state information, the state category identification results, and the state level assignment results, a fused state result for the port equipment is generated.

[0106] This embodiment also provides a port equipment data fusion system based on multimodality, including: The preprocessing module collects multimodal heterogeneous data during the operation of port equipment to form an initial multimodal equipment data set. It then preprocesses the initial multimodal port equipment data to obtain a standardized multimodal data set. The extraction module performs feature extraction on standardized multimodal equipment data to obtain state feature data corresponding to different modes. The association module establishes the association relationship between the state features of different modes through the state feature data corresponding to different modes, and obtains the initial association feature set of multimodal modes; The analysis module performs consistency analysis on multimodal data based on the modal conflict perception mechanism to obtain a set of reliable modal features after conflict correction. The fusion module establishes multimodal association fusion relationships based on event causal chain constraints to obtain event-driven fusion features; The integration module fuses event-driven fusion features to obtain comprehensive status features of port equipment, and then obtains the fusion status results of port equipment based on these comprehensive status features.

[0107] In summary, this invention collects visual, vibration, electrical, control, and environmental data during the operation of port equipment. After preprocessing and feature extraction, it establishes an initial multimodal correlation feature set. A modal conflict perception mechanism is introduced to analyze the consistency of different modal state features, correct conflict information, and obtain a reliable modal feature set. An event causal chain is constructed by combining the event occurrence sequence, state response relationship, and event propagation direction, establishing an event-driven multimodal correlation fusion relationship. Based on the event causal chain, different modal state features are fused to eliminate duplicate state information during the propagation of the same event, forming a comprehensive state feature of the port equipment. This yields a fused state result that accurately reflects the operating state of the port equipment, improving the reliability of multimodal data fusion, state representation capability, and accuracy of port equipment operating state analysis.

[0108] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal port equipment data fusion method, characterized by: include, Collect multimodal heterogeneous data during the operation of port equipment to form an initial multimodal equipment dataset. Preprocess the initial multimodal port equipment data to obtain a standardized multimodal dataset. Feature extraction is performed on standardized multimodal equipment data to obtain state feature data corresponding to different modes; By using state feature data corresponding to different modalities, the correlation between state features of different modalities is established to obtain an initial set of multimodal correlation features; Based on the modal conflict perception mechanism, consistency analysis is performed on multimodal data to obtain a set of reliable modal features after conflict correction. Multimodal association and fusion relationships are established based on event causal chain constraints to obtain event-driven fusion features; Event-driven fusion features are fused to obtain comprehensive state features of port equipment, and the fused state results of port equipment are obtained based on the comprehensive state features.

2. The port equipment data fusion method based on multimodal operation as described in claim 1, characterized in that: The formation of the initial multimodal equipment dataset includes, Collect multimodal heterogeneous data during the operation of port equipment, including visual data, vibration data, electrical data, control data, and environmental data; Visual data, vibration data, electrical data, control data, and environmental data are labeled and processed, and data type information, acquisition time information, and equipment association information are added to them respectively. They are then combined to form an initial multimodal equipment data set.

3. The port equipment data fusion method based on multimodal operation as described in claim 2, characterized in that: The standardized multimodal data set includes, The initial multimodal port equipment dataset is cleaned, formatted, time-corrected, and normalized to remove abnormal, duplicate, and invalid data, thus obtaining the processed initial multimodal port equipment dataset. Based on the processed initial multimodal port equipment data set, the visual data, vibration data, electrical data, control data, and environmental data in the processed initial multimodal port equipment data set are unified in terms of time base and adjusted in terms of data scale to obtain a standardized multimodal data set.

4. The port equipment data fusion method based on multimodal operation as described in claim 3, characterized in that: The state feature data corresponding to the different modes include Modal classification processing is performed on visual data, vibration data, electrical data, control data, and environmental data in standardized multimodal equipment data to determine the spatial state information corresponding to visual data, the mechanical state information corresponding to vibration data, the dynamic state information corresponding to electrical data, the operation control information corresponding to control data, and the environmental state information corresponding to environmental data. Feature extraction is performed on the visual data, vibration data, electrical data, control data, and environmental data in the standardized multimodal equipment data after modal classification processing to obtain visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data. Visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data are integrated to form state feature data corresponding to different modes.

5. The port equipment data fusion method based on multimodal data as described in claim 4, characterized in that: The initial multimodal association feature set includes, Temporal relationship analysis was performed on the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data corresponding to different modes. The state feature data corresponding to different modes were matched according to the acquisition time to determine the temporal change relationship of the state feature data corresponding to different modes during the operation of the same port equipment. By comparing the direction, magnitude, and timing of changes in visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data over a continuous time range, the state response relationship between the state feature data corresponding to different modes is obtained. Based on the state response relationship between the state feature data corresponding to different modes, the response process between the state feature data corresponding to different modes is matched, and the correlation between the state features of different modes is determined according to the degree of correspondence of state change time, the degree of consistency of state change direction, and the degree of overlap of state change process. Based on the correlation between different modal state features, visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data with correlation are associated and combined to form a multimodal initial correlation feature set.

6. The port equipment data fusion method based on multimodal operation as described in claim 5, characterized in that: The conflict-corrected set of trusted modal features includes: The visual state feature data, vibration state feature data, electrical state feature data, control state feature data and environmental state feature data in the initial association feature set of multimodal mode are calculated in chronological order to continuously change state. The starting position of state change is determined according to the state change between adjacent time nodes, and time nodes with the same continuous change direction are combined into state response intervals to obtain the set of state response intervals corresponding to different modes. Extract the start time, end time, duration, and direction of change of the state response intervals from the set of state response intervals corresponding to different modes. Use the start time, end time, duration, and direction of change of the state response as modal response matching features to obtain the set of response process features corresponding to different modes. Time offset calibration is performed on the response process features in the response process feature set corresponding to different modes, so as to use the state response interval corresponding to the control state feature data as the reference response process; By adjusting the time offset of the state response intervals corresponding to visual state feature data, vibration state feature data, electrical state feature data, and environmental state feature data, the modal response correlation is determined based on the degree of overlap of the start time of change, the degree of overlap of the duration of change, and the degree of consistency of the direction of state change after the time offset adjustment, thus obtaining a set of modal response correlations. By correcting the correlation between visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data, the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data that satisfy the modal response correlation are identified as correlated modal state feature data. Modal state feature data that do not satisfy the modal response correlation relationship are reduced in correlation degree to obtain a set of reliable modal features after conflict correction.

7. The port equipment data fusion method based on multimodal operation as described in claim 6, characterized in that: The event-driven fusion features include: The visual state feature data, vibration state feature data, electrical state feature data, control state feature data and environmental state feature data in the conflict-corrected credible modal feature set are used to identify state change trigger points. The event occurrence time node is determined according to the position where the change in state feature exceeds the state change threshold within a continuous time range. The continuously associated state change nodes are divided into equipment state change events to obtain a multimodal state change event set. The sequence of events in the set of multimodal state change events, including visual state change events, vibration state change events, electrical state change events, control state change events, and environmental state change events, is analyzed. By comparing the occurrence time, end time, and duration of different modal state change events, a set of multimodal event temporal relationships is obtained. Based on the event propagation path analysis of different modal state change events in the multimodal event time sequence relationship set, the event propagation direction is determined according to the state change response relationship between successive events; By determining whether a previous state change event causes a subsequent state change event, the causal relationship between state change events of different modalities is determined, and a set of causal relationships of events is obtained. The association paths of visual state change events, vibration state change events, electrical state change events, control state change events, and environmental state change events in the event causal relationship set are filtered, and the association paths of multimodal state change events that satisfy the event occurrence order, state response relationship, and event propagation direction are retained to obtain the event causal chain set. The fusion relationship is determined by the position of different modal state feature data in the event causal chain. Different modal state feature data in the same event propagation path are combined to obtain event-driven fusion features.

8. The port equipment data fusion method based on multimodal operation as described in claim 7, characterized in that: The comprehensive status characteristics of the port equipment include: Based on the event causal chain set corresponding to the visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data in the event-driven fusion features, the different modal state feature data are arranged according to the event occurrence order and state response order to obtain the event-driven fusion features after event association sorting. By analyzing the state change information and state response relationship corresponding to different modal state feature data, the fusion correlation relationship between different modal state feature data is determined, and a fusion correlation feature set is obtained. Visual state feature data, vibration state feature data, electrical state feature data, control state feature data, and environmental state feature data are fused together, and repeated state information in the propagation process of the same event is merged to obtain a comprehensive state feature set. The spatial status information, mechanical status information, power status information, control status information, and environmental status information are integrated to form the comprehensive status characteristics of port equipment.

9. The port equipment data fusion method based on multimodal operation as described in claim 8, characterized in that: The port equipment integration status results include: Based on the correspondence between spatial state information, mechanical state information, power state information, control state information and environmental state information, the associated states between different state dimensions of port equipment are determined. By comparing the change direction, change degree and change time corresponding to different state information, the overall operating state change of port equipment is determined. State fusion analysis is performed on spatial state information, mechanical state information, dynamic state information, control state information and environmental state information, and the state descriptions corresponding to different state dimensions are comprehensively correlated to form fused state information; The system identifies and classifies the state categories and levels of spatial state information, mechanical state information, power state information, control state information, and environmental state information, and generates a fusion state result for port equipment by combining the correlation between different state information.

10. A port equipment data fusion system based on multimodal operation, based on the port equipment data fusion method based on any one of claims 1 to 9, characterized in that: This includes a preprocessing module, which collects multimodal heterogeneous data during the operation of port equipment to form an initial multimodal equipment data set, and preprocesses the initial multimodal port equipment data to obtain a standardized multimodal data set. The extraction module performs feature extraction on standardized multimodal equipment data to obtain state feature data corresponding to different modes. The association module establishes the association relationship between the state features of different modes through the state feature data corresponding to different modes, and obtains the initial association feature set of multimodal modes; The analysis module performs consistency analysis on multimodal data based on the modal conflict perception mechanism to obtain a set of reliable modal features after conflict correction. The fusion module establishes multimodal association fusion relationships based on event causal chain constraints to obtain event-driven fusion features; The integration module fuses event-driven fusion features to obtain comprehensive status features of port equipment, and then obtains the fusion status results of port equipment based on these comprehensive status features.

Images