Multi-modal data processing method and device, equipment and storage medium

Abstract

The invention provides a multi-modal data processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining a current collection data set of target equipment according to the operation state of the target equipment, and carrying out the elimination processing of the current collection data set according to the check configuration information and the timestamp information of the current collection data set, and removing the data set to obtain a removed data set, adding the removed data set into a data pool, and screening the data pool according to the equipment operation feature information to obtain a screened data pool. According to the data cleaning method and device, full-automatic data cleaning is achieved, dependence on manual experience and a statistical method is avoided, a large amount of manpower and time cost is saved, the data processing efficiency is improved, operation of the equipment can be associated with the data, and subsequent data analysis and other processing are facilitated.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multimodal data processing method, device, equipment and storage medium. Background Art

[0002] Excavators generate a large amount of multimodal data during operation, which is of great value in machine optimization, fault diagnosis, predictive maintenance, etc. However, data is often subject to various noises and interferences during collection and transmission, resulting in a decline in data quality and seriously affecting data analysis and utilization. Therefore, how to efficiently clean these multimodal data and improve data accuracy and reliability has become an urgent problem to be solved.

[0003] Data cleaning methods in the prior art include manual data cleaning methods and model-based data cleaning methods. Manual data cleaning methods rely on manual labeling and cleaning of data, and identify and process outliers through calculated basic statistics such as mean, median and standard deviation. Model-based data cleaning methods use models to automatically clean data, which requires training a model based on labeled data, and then completing automatic data cleaning based on the model.

[0004] However, manual data cleaning methods are inefficient, cannot process large-scale data, and rely on manual experience, so the processing effect is unstable. Model-based data cleaning methods require a large amount of training data, the model training and adjustment time is long, and the requirements for hardware resources are also relatively high. Summary of the invention

[0005] The purpose of this application is to provide a multimodal data processing method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problems that the manual data cleaning methods in the prior art are inefficient, cannot process large-scale data, and rely on manual experience, so the processing effect is unstable. The model-based data cleaning method requires a large amount of training data, the model training and adjustment time is long, and the requirements for hardware resources are also relatively high.

[0006] To achieve the above objectives, the technical solutions adopted in this application are as follows:

[0007] In a first aspect, the present application provides a multimodal data processing method, the method comprising:

[0008] Acquire a current collection data set of the target device according to the running state of the target device, wherein the current collection data set includes at least one of the following: device posture data, device load data, device image data, and device positioning data;

[0009] Performing elimination processing on the current collected data set according to the inspection configuration information and the timestamp information of the current collected data set to obtain a eliminated data set, and adding the eliminated data set to the data pool;

[0010] The data pool is filtered according to the device operation characteristic information to obtain a filtered data pool, wherein the filtered data pool includes at least one multimodal data set, each of the multimodal data sets is a data set generated when the target device performs a target operation, and the device operation characteristic information is used to describe the data regularity when the target device performs the target operation.

[0011] In a second aspect, the present application provides a multimodal data processing device, the device comprising:

[0012] An acquisition module, used to acquire a current acquisition data set of the target device according to the running state of the target device, wherein the current acquisition data set includes at least one of the following: device posture data, device load data, device image data, and device positioning data;

[0013] A removal module, used for removing the currently collected data set according to the inspection configuration information and the timestamp information of the currently collected data set to obtain a removed data set, and adding the removed data set to the data pool;

[0014] A screening module is used to screen the data pool according to the device operation characteristic information to obtain a screened data pool, wherein the screened data pool includes at least one multimodal data set, each of which is a data set generated when the target device performs a target operation, and the device operation characteristic information is used to describe the data regularity when the target device performs the target operation.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of a multimodal data processing method as described in any one of the first aspects.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a multimodal data processing method as described in any one of the first aspects are executed.

[0017] The beneficial effect of the present application is that by automatically cleaning the collected real-time data, the accuracy and completeness of the multimodal data of the excavator equipment can be significantly improved, ensuring the reliability of the data. In addition, the present application does not need to rely on manual experience and statistical methods, and realizes fully automatic data cleaning, saving a lot of manpower and time costs, and improving data processing efficiency. By screening the data using the data change rules of the equipment operation, a multimodal data set of different operations can be obtained, so that subsequent data analysis and model training can be performed based on high-quality multimodal data sets.

[0018] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram showing the setting of a collection device on a target device provided in an embodiment of the present application is shown;

[0021] Figure 2 A flowchart of a multimodal data processing method provided in an embodiment of the present application is shown;

[0022] Figure 3 A flow chart of collecting data according to the operating status of a target device provided by an embodiment of the present application is shown;

[0023] Figure 4 A flow chart of performing data removal processing on a current collected data set provided by an embodiment of the present application is shown;

[0024] Figure 5 A flow chart of performing data elimination processing provided by an embodiment of the present application is shown;

[0025] Figure 6 Another flow chart for performing data elimination processing provided by an embodiment of the present application is shown;

[0026] Figure 7 A flow chart of performing data alignment processing provided by an embodiment of the present application is shown;

[0027] Figure 8 A flowchart of a process for removing an aligned data set provided by an embodiment of the present application is shown;

[0028] Fig. 9 A flow chart of performing frame skipping check on device image data provided by an embodiment of the present application is shown;

[0029] Fig.10 A flowchart for determining device fault information provided by an embodiment of the present application is shown;

[0030] Fig.11 A flow chart for performing data screening provided by an embodiment of the present application is shown;

[0031] Fig.12 A schematic diagram of the structure of a multimodal data processing device provided in an embodiment of the present application is shown;

[0032] Fig.13 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0033] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0034] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0035] During the operation, the excavator generates a large amount of multimodal data, including sensor data, image data, radar data, and operation logs. These data are of great value in equipment optimization, fault diagnosis, and predictive maintenance.

[0036] Existing data cleaning technologies include data cleaning based on manual experience and simple statistical methods and data cleaning based on large models. However, when cleaning excavator data based on manual experience and simple statistical methods, it relies on manual experience, has problems of low efficiency and poor processing stability. Before data cleaning based on large models, it is necessary to train and adjust the model through a large amount of training data, which will consume a lot of computing resources and have high requirements on hardware resources.

[0037] Based on this, the present application proposes a multimodal data processing method, which collects data according to the operating status of the equipment, and automatically cleans the collected multimodal data of the equipment based on the inspection configuration information and timestamp information to ensure the validity and integrity of the data. Finally, the data is screened based on the rules of the data itself, and the multimodal data is associated with the operation of the excavator equipment to obtain a multimodal data set corresponding to each operation of the excavator equipment. Compared with the prior art, the present application does not rely on manual experience, and can complete the automated processing of data from collection, cleaning to screening, thereby improving the efficiency and stability of data cleaning. Compared with the method of data cleaning based on large models, the present application does not require a large amount of training data, which reduces the consumption of computing resources.

[0038] First, the application scenario of this application is described. The method of this application can be applied to the data cleaning scenario of excavator equipment. Figure 1 The figure shows a schematic diagram of a data acquisition device installed in an excavator. The data acquisition device includes a sensor, a camera, a laser radar and a positioning device. Figure 1 , an inclination sensor is set on the excavator's arm, forearm, bucket, and cabin to obtain the excavator's posture information, and the load information of the excavator is obtained by setting a pressure sensor; real-time 3D reconstruction data is obtained by setting a binocular camera and a laser radar; and real-time transmission of communication data is ensured by setting a communication module, such as 5GCPE. In addition, a positioning device is set in the control box of the excavator, such as RTK (Real-time kinematic) carrier phase differential technology, to obtain the equipment positioning data of the excavator.

[0039] After the collection device of the excavator equipment collects the equipment data, the collected equipment data is transmitted to the electronic device for processing the data through the communication module. The electronic device processes the data received in real time based on the method of the present application, and outputs and displays the processed data to the user.

[0040] Next, combine Figure 2 , the multimodal data processing method of the present application is further described, the execution subject of the method can be an electronic device with processing capabilities, the electronic device can be connected to the target device for communication, and the electronic device can also be deployed on the target device, such as Figure 2 As shown, the method includes:

[0041] S201. Acquire a current collection data set of the target device according to the running state of the target device, where the current collection data set includes at least one of the following: device posture data, device load data, device image data, and device positioning data.

[0042] The target device can be Figure 1 The excavator equipment shown. When the excavator is working, the collection device installed on the excavator equipment can collect the data generated during the working process of the excavator, and transmit the generated data to the electronic device through the communication module. Among them, the electronic device and the excavator equipment can communicate in real time based on the rostopic communication mechanism, the excavator equipment acts as a publisher, and the electronic device acts as a subscriber. After the excavator equipment and the electronic device complete the registration and authentication, the electronic device can obtain the equipment data collected by the excavator equipment in real time.

[0043] The operating status of the target device includes: normal operation and abnormal operation. In the embodiment of the present application, the operating status of the target device can be detected by a data acquisition program in the target device, and the operating status of the target device can be sent to the electronic device. The electronic device can determine whether to send a data acquisition instruction to the target device based on the operating status of the target device. In another implementation, the data acquisition program can determine whether to perform data acquisition based on the operating status of the target device.

[0044] If the target device is operating normally, the data acquisition program can obtain the current data set of the target device through the acquisition device set on the target device. If the target device is operating abnormally, such as when the target device fails or stops moving, the data acquisition program can stop collecting the current data. The current data set includes Figure 1 The real-time data collected by each collection device shown includes the device identification of each collection device.

[0045] The data collection program can send the data collected at the current moment to the electronic device in real time. During the data collection process, the electronic device performs real-time inspection and cleaning of the data at the current moment.

[0046] Device posture data can be Figure 1 The inclination sensor installed on the excavator equipment collects the posture information of the excavator equipment, including the joint angles and joint angular velocities of the excavator equipment's boom, arm, bucket, and cabin joints. The equipment load data can be Figure 1 The load data collected by the pressure sensor installed on the excavator equipment. The equipment image data can be Figure 1 The equipment positioning data may be image data collected by a camera and a laser radar installed on the excavator equipment, and the equipment positioning data may be real-time positioning data collected by a positioning device of the excavator equipment.

[0047] S202: Perform elimination processing on the current collected data set according to the inspection configuration information and the timestamp information of the current collected data set to obtain a eliminated data set, and add the eliminated data set to the data pool.

[0048] Optionally, the inspection configuration information is used to indicate data features of invalid data and to indicate data items that need to be eliminated. Invalid data is abnormal data that is determined not to be used and needs to be eliminated. The inspection configuration information includes: abnormal data thresholds, data items to be eliminated, and abnormal image information. The abnormal data thresholds include: a number threshold, a change threshold, and a data threshold. By calculating the current acquired data, and comparing the calculation result with the number threshold and the change threshold in the inspection configuration information, and comparing the value of the current acquired data with the data threshold, the abnormal data that needs to be eliminated in the current acquired data set can be determined. Abnormal image information is used to indicate image features when an abnormality exists in the image. By comparing the device image data in the current acquired data set with the image features indicated by the abnormal image information, the device image data with abnormalities can be determined.

[0049] The data items to be removed in the check configuration information can indicate the data items in the current collection data set that need to be removed and the data items that need to be retained when the current collection data has an abnormality. By configuring the data to be removed, it can be determined which data items in the current collection data need to be removed at the same time and which data items need to be retained when the current collection data has an abnormality.

[0050] Exemplarily, assuming that the data to be eliminated indicates that device posture data, device image data and device load data are removable data items, and device positioning data is a data item that needs to be retained, after determining that there is an abnormality in the device load data at time t0, the device posture data, device image data and device load data at time t0 can all be eliminated, and the device positioning data at time t0 can be retained.

[0051] According to the timestamp information of the current acquisition data set, the data continuity and integrity in the current acquisition data set can be checked. If the data at the same time is missing, the data items in the current acquisition data set can be removed or retained according to the configuration of the data to be removed in the inspection configuration information. Continuing with the above example, assuming that the device posture data at time t1 is missing, the device image data and device load data at time t1 can be removed according to the instructions of the inspection configuration information, and the device positioning data at time t1 can be retained according to the instructions of the inspection configuration information.

[0052] In an embodiment of the present application, abnormal data in the current collected data set can be checked and eliminated based on the inspection configuration information, and the integrity of the data in the current collected data set can be checked based on the timestamp information to ensure the data integrity and validity of the data items that need to be retained at the same time.

[0053] S203. Filter the data pool according to the device operation characteristic information to obtain a filtered data pool, wherein the filtered data pool includes at least one multimodal data set, each multimodal data set is a data set generated when the target device performs the target operation, and the device operation characteristic information is used to describe the data regularity when the target device performs the target operation.

[0054] The target operation may be a fixed action flow executed by the target device when completing a specific operation during operation, including multiple continuous actions arranged in a fixed order. For example, when the target operation is a loading action, the action flow of the excavator device may be: moving the mechanical arm to the loading position, lowering the bucket, retracting the bucket, raising the bucket, moving the mechanical arm to the unloading position, and placing the bucket. When the target operation is a digging action, the action flow of the excavator device may be: raising the mechanical arm, extending the mechanical arm, lowering the mechanical arm, extending the bucket, lowering the bucket, retracting the bucket, and raising the bucket.

[0055] Optionally, the device operation characteristic information may represent the change pattern of device data when the target device performs a certain operation. The operation performed by the target device includes multiple continuous actions, and the data generated when the target device performs each action in the operation in sequence has a specific business pattern.

[0056] By calculating the data in the data pool, the changing pattern of the data in the data pool is obtained, and the pattern of the data in the data pool is compared with the device operation characteristic information. If the difference between the changing pattern of the data in the data pool and the device operation characteristic information is within a preset range, the data can be associated with the operation corresponding to the device operation characteristic information to obtain a multimodal data set corresponding to the operation.

[0057] Exemplarily, by calculating the data in the data pool, it is found that the data from time t0 to time t3 in the data pool is consistent with the equipment operation characteristic information of the target equipment loading operation. Then, the data from time t0 to time t3 in the data pool can be associated with the target equipment loading operation to obtain a multimodal data set of the target equipment loading operation.

[0058] In an embodiment of the present application, a current acquisition data set of the target device is obtained according to the operating state of the target device, and the current acquisition data set includes at least one of the following: device posture data, device load data, device image data, and device positioning data. The current acquisition data set is eliminated according to the inspection configuration information and the timestamp information of the current acquisition data set to obtain a eliminated data set, and the eliminated data set is added to the data pool. The data pool is filtered according to the device operation characteristic information to obtain a filtered data pool. The filtered data pool includes at least one multimodal data set, and each multimodal data set is a data set generated when the target device performs a target operation. The device operation characteristic information is used to describe the data regularity when the target device performs the target operation.

[0059] By automatically cleaning the collected real-time data, the accuracy and completeness of the multimodal data of the excavator equipment can be significantly improved, ensuring the reliability of the data. In addition, the present application does not need to rely on manual experience and statistical methods, and realizes fully automatic data cleaning, saving a lot of manpower and time costs, and improving data processing efficiency. By screening the data using the data change rules of the equipment operation, a multimodal data set of different operations can be obtained, so that subsequent data analysis and model training can be performed based on high-quality multimodal data sets.

[0060] The following is a further explanation of the above method of obtaining the current collection data set of the target device according to the running status of the target device. Figure 3 As shown, including:

[0061] S301. Obtain the current operating status of the target device.

[0062] Optionally, the current operating state of the target device includes: normal operation and abnormal operation. When the target device fails or stops operating, the current operating state of the target device may be abnormal operation.

[0063] S302: If the current operating state is normal, start the acquisition device of the target device, and use the acquisition device to acquire the current acquisition data set of the target device.

[0064] S303: If the current operating state is abnormal, the acquisition device of the target device is not started.

[0065] As an optional implementation, when the data acquisition program in the target device determines that the current operating status of the target device is normal, the data acquisition program can issue data acquisition instructions to each acquisition device. After receiving the data acquisition instructions, each acquisition device starts to collect real-time acquisition data of the target device. The data acquisition program sends the current acquisition data set containing the real-time acquisition data of each acquisition device to the electronic device for data cleaning processing.

[0066] When the data acquisition program in the target device determines that the current operating state of the target device is abnormal, the data acquisition program does not send a data acquisition instruction to each acquisition device, or sends a stop acquisition instruction to each acquisition device.

[0067] The following is a further description of the above-mentioned elimination of the current collected data according to the inspection configuration information and the timestamp information of the current collected data set to obtain the eliminated data set, such as Figure 4 As shown, the above step S201 includes:

[0068] S401 : performing elimination processing on the current acquisition data set according to the inspection configuration information to obtain a first acquisition data set.

[0069] The inspection configuration information is used to indicate the data characteristics of invalid data and the data items that need to be eliminated. Invalid data is abnormal data that is determined to be unusable and needs to be eliminated. The data characteristics of abnormal data include: data that remains unchanged for a long time, data with irregular jumps, and data that exceeds the standard range. If it is determined that the current data meets the data characteristics of abnormal data, the current data can be regarded as an invalid data item.

[0070] The current collected data set includes multiple data items, and the data in each data item is collected by a collection device. The abnormal data in the current collected data set can be determined according to the inspection configuration information. If the inspection configuration information indicates that the data item corresponding to the abnormal data is a data item that needs to be eliminated when the data is abnormal, the abnormal data can be eliminated, and the data corresponding to the time in other data items that need to be eliminated can also be eliminated based on the inspection configuration information.

[0071] For example Figure 1 If the angle data collected by the angle sensor does not change for a long time, it can be considered that there is an abnormality in the angle data and the angle data is treated as invalid data. Check that the configuration information indicates that the angle data, the pressure data collected by the pressure sensor, and the inclination data collected by the inclination sensor are data items that can be eliminated when an abnormality occurs. The pressure data and inclination data during the period when the angle data does not change can be treated as invalid data.

[0072] After invalid data is removed from the current collection data set, a first collection data set can be obtained. It should be noted that when performing data removal processing, the data items to be removed can be determined according to the inspection configuration information, and the data corresponding to the data items can be removed from the current collection data set.

[0073] Optionally, the inspection configuration information may also indicate abnormal features of the device image. Based on the inspection configuration information, the device image data may be judged to determine whether the abnormal device image data is invalid, and the invalid device image data and the data at the corresponding moment may be removed from the device image data.

[0074] S402: Eliminate the first collected data set according to the timestamp information to obtain a second collected data set.

[0075] The timestamp information may be the timestamp information of each data in the first collected data set. By aligning each data according to the timestamp information of each data, it is possible to determine whether there is missing data in the aligned data, and remove the missing data from the first collected data set to obtain the second collected data set.

[0076] S403: Perform frame skipping check processing on the device image data in the second acquired data set to obtain a data set after elimination.

[0077] During the data collection process, due to data transmission delay or other reasons, image data and video data may skip frames, resulting in data discontinuity. The present application can perform frame skipping check on the device image data at adjacent moments in the second collection data set to determine the presence of frame skipping data and ensure data continuity.

[0078] After performing a frame skipping check on the device image data in the second acquired data set and obtaining the image data with frame skipping, the image data with frame skipping can be removed from the second acquired data set to obtain a data set after removal. When removing the image data with frame skipping, it can also be determined based on the inspection configuration information whether to remove the device posture data, device load data, and device positioning data at the same time as the image data with frame skipping.

[0079] It should be noted that first performing elimination processing based on the inspection configuration information, then performing elimination processing based on the timestamp information, and performing frame skipping check processing on the device image data is only one possible implementation method. It should be understood that it is also possible to first perform elimination processing based on the timestamp information to obtain a first acquisition data set, and then perform elimination processing on the first acquisition data set based on the inspection configuration information to obtain a second acquisition data set, and finally perform frame skipping check processing on the second acquisition data set. The above steps S401-S403 have no strict order and can be executed in parallel or in any order, and the present application does not impose any restrictions on this.

[0080] The following is a further description of the above-mentioned elimination process of the current acquisition data set according to the inspection configuration information to obtain the first acquisition data set, such as Figure 5 As shown, the above step S401 includes:

[0081] S501: Determine data change values ​​of various data in the current collection data set according to the current collection data set and the historical collection data set.

[0082] The current collection data set and the historical collection data set both include collection data at multiple times, and the collection data at each time includes multiple data items. The data change value may be the difference between adjacent times of the same data item.

[0083] S502: Determine whether there is target data to be removed in the current collection data set according to the inspection configuration information, the current collection data set, and the data change value of each item of data in the current collection data set.

[0084] The target data to be removed may be data with abnormalities and data related to the abnormal data. The data related to the abnormal data may be data at the same time as the abnormal data in the data items to be removed indicated by the inspection configuration information.

[0085] The data change value can indicate whether there is an anomaly in the collected data. By judging the data change value according to the inspection configuration information, the data with anomalies in the current collected data set can be obtained, and the data with anomalies can be used as the target data to be eliminated.

[0086] S503: If yes, remove the target data in the current collected data set to obtain a first collected data set.

[0087] For example, assuming that there is an anomaly in the device posture data at time t1 in the current collected data set, the inspection configuration information indicates that the data items that need to be eliminated include: device posture data, device load information, and device image data, then the target data can be the device posture data at time t1, the device load information at time t1, and the device image data at time t1.

[0088] The target data may indicate an abnormal fault point in the target device. In an embodiment of the present application, the target data may also be recorded as a log file and the log file may be sent to the user so that the user can locate the fault point in the target device based on the target data recorded in the log file.

[0089] The following is a further explanation of determining whether there is target data to be removed in the current collection data set according to the inspection configuration information, the current collection data set and the data change value of each data in the current collection data set. The above step S502 includes:

[0090] Traverse each data item in the current collection data set. For the current data item traversed, if the data change value of the current data item is equal to 0 and the number of unchanged times corresponding to the current data item is greater than the preset number threshold corresponding to the current data item in the inspection configuration information, or the data change value of the current data item is greater than the preset change threshold corresponding to the current data item in the inspection configuration information, or the value of the current data item is greater than the preset data threshold corresponding to the current data item in the inspection configuration information, then take the current data item as a target data item; otherwise, increase the number of unchanged times corresponding to the current data item by 1.

[0091] The inspection configuration information may set a number threshold, a change threshold, and a data threshold for each data item, and may inspect each data item based on each inspection configuration information to determine the target data in each data item, and according to the data to be removed indicated by the inspection configuration information, when removing the target data, remove the data to be removed indicated by the inspection configuration information corresponding to the target data. The data corresponding to the target data may be data at the same time as the target data.

[0092] When the data change value of the current item data is equal to 0 and the number of unchanged times corresponding to the current item data is greater than the preset number threshold corresponding to the current item data in the check configuration information, it means that when the data change value has not changed for a long time, the target device may have a fault. Therefore, the data between the time when the data change value starts to be equal to 0 and the time when the data change value is equal to 0 for the last time can be used as a target data.

[0093] It should be noted that determining the number of unchanged times of the data change value is only one possible implementation method given in this application. The starting time and ending time of the data change value can also be determined. If the time between the starting time and the ending time is greater than the preset duration, the data between the starting time and the ending time can be used as the target data.

[0094] If the data change value of the current item of data is greater than the preset change threshold, it means that the data has an irregular sudden jump, and the data before and after the jump corresponding to the data change value can be used as a target data. If the data value of the current item of data is greater than the preset data threshold, it means that the size of the data value exceeds the standard range, and the current item of data can be used as a target data.

[0095] In the embodiment of the present application, by cleaning each item of data in the current collected data set according to the inspection configuration information, the integrity and validity of the data can be guaranteed, and the present application does not rely on manual experience and statistical methods, and can realize automated data cleaning.

[0096] The following is a further description of the above-mentioned elimination of the first collected data set according to the timestamp information to obtain the second collected data set. Figure 6 As shown, the above step S402 includes:

[0097] S601 , aligning first collected data in a first collected data set according to timestamp information and a preset collection frequency to obtain an aligned data set corresponding to the collection frequency.

[0098] It should be noted that the data collection frequencies of various sensors may be different, so the data frequencies of various data may be inconsistent. Therefore, the timestamp information of various data can be sorted based on the preset collection frequency, and the timestamps of various data can be aligned to obtain an aligned data set.

[0099] Optionally, based on the preset acquisition frequency and the current moment, the time range corresponding to the current moment can be determined, and the first acquisition data with timestamp information within the time range can be aligned according to the timestamps to obtain aligned data sets corresponding to the acquisition frequency at each moment.

[0100] As an example, assuming that the acquisition frequency is 10 Hz, the data within a time period of ±0.1 seconds from the current moment can be used as a set of aligned data sets at the current moment. The acquisition frequency can be determined according to the acquisition frequency of each sensor. For example, the acquisition frequency can be the least common multiple of the acquisition frequency of each sensor to ensure that each moment can include various data collected by each sensor.

[0101] S602: Perform elimination processing on each aligned data set to obtain a second collected data set.

[0102] After completing the data alignment process, for each aligned data set, you can check whether the data items therein are complete. If there are missing data items in the aligned data set, and the data items are indispensable data items indicated by the check configuration information, the aligned data set can be eliminated. After completing the elimination process for all aligned data sets, the remaining aligned data sets are used as the second acquisition data set.

[0103] The following is a further description of performing alignment processing on the first collected data in the first collected data set according to the timestamp information and the preset collection frequency to obtain the aligned data set corresponding to the collection frequency, such as Figure 7 As shown, the above step S601 includes:

[0104] S701. Determine a collection timestamp corresponding to a preset collection frequency.

[0105] As a possible implementation, multiple moments to be aligned can be predetermined. For example, when processing the data received at moment t1, moment t1 can be used as a moment to be aligned. The time range corresponding to each moment to be aligned can be determined according to the acquisition frequency. The time range of the moment to be aligned ± the inverse of the acquisition frequency can be used as the time range corresponding to the moment to be aligned.

[0106] S702: Determine at least one first collected data corresponding to the collection timestamp according to the collection timestamp and the timestamp information of each first collected data, and use the at least one first collected data as an item of data in the aligned data set corresponding to the collection frequency.

[0107] After determining the time range, you can filter the data within the time range according to the timestamp information of each data item, align the data items according to the timestamp information, and use the aligned data within the time range as the aligned data set with the collection frequency corresponding to the timestamp to be aligned.

[0108] For example, assuming that the time to be aligned is the 2nd second and the acquisition frequency is 10 Hz, the data between 1.9s and 2.1s can be used as the data corresponding to the time to be aligned, and the sensor data within the time range of 1.9s to 2.1s can be used as the aligned data set corresponding to the 2nd second.

[0109] After determining the aligned data set, each aligned data set can be eliminated to obtain a second acquired data set, such as Figure 8 As shown, specifically including:

[0110] S801, if the first collected data in the aligned data set has missing values, remove the aligned data set;

[0111] If there are missing data items in the aligned data set, the aligned data set may be removed according to the data items that need to be removed as indicated by the check configuration information.

[0112] Continuing with the above example, if the time to be aligned is the 2nd second, the acquisition frequency is 10Hz, and the device load data is missing within the time range of 1.9s to 2.1s, and the check configuration information indicates that the device load data is a data item that must exist, then the data in the second second can be determined as an invalid data segment and deleted.

[0113] S802: If there is no missing value in the first collected data in the aligned data set, retain the aligned data set.

[0114] If the first acquired data in the aligned data set is consistent with the data item that needs to be retained indicated by the inspection configuration information, the aligned data set can be retained.

[0115] For example, if the time to be aligned is the second second, the acquisition frequency is 10 Hz, and there is missing equipment load data within the time range of 1.9s to 2.1s, and the check configuration information indicates that the equipment load data is a data item that can be missing, then the data of the second second can be retained and used as an aligned data set.

[0116] In the embodiment of the present application, by aligning the data according to the timestamp information, the missing data fragments can be deleted to ensure the validity and integrity of the final retained data.

[0117] The following is a further description of the above-mentioned frame skipping check processing on the device image data in the second acquisition data set to obtain the data set after elimination, such as Fig. 9 As shown, the above step S403 includes:

[0118] S901. Binarize the first device image data and the second device image data to obtain a first character string corresponding to the first device image data and a second character string corresponding to the second device image data, wherein the first device image data and the second device image data are device image data at adjacent moments in a second collected data set.

[0119] The image can be binarized by converting the image into a black and white image, setting the pixels above the threshold to 1 and the pixels below the threshold to 0 through a preset threshold, and finally reading the pixel values ​​in the image in a preset order to obtain a binary string corresponding to the device image data.

[0120] Optionally, all images in the device image data may be binarized to obtain a binary string of each image. This application is described by taking images at adjacent moments as an example, and the application should not be limited thereto.

[0121] S902: Determine a Hamming distance between the first device image data and the second device image data according to the first character string and the second character string.

[0122] Hamming distance is an indicator that measures the difference between two binary strings and can be effectively used to detect inter-frame differences. By calculating the first string and the second string, the Hamming distance between the first string and the second string can be obtained. The Hamming distance can represent the similarity between the first device image data represented by the first string and the second device image data represented by the second string.

[0123] S903: Determine whether to perform data elimination processing according to the Hamming distance and a preset distance threshold.

[0124] It should be understood that image data at adjacent moments generally have a high degree of similarity, so by comparing the Hamming distance with a preset distance threshold, it can be determined whether there is frame skipping between the image data of the first device and the image data of the second device.

[0125] If the Hamming distance between the two is greater than the preset distance threshold, it indicates that there may be frame skipping, and the first device image data and the second device image data can be deleted. If the Hamming distance between the two is less than or equal to the preset distance threshold, it indicates that there is no frame skipping, and the first device image data and the second device image data can be retained.

[0126] S904: If yes, remove the second collected data corresponding to the image data of the first device and the second collected data corresponding to the image data of the second device to obtain a data set after removal.

[0127] Optionally, if the Hamming distance between the two is greater than a preset distance threshold, the data items in the second acquired data corresponding to the first device image data and the data items in the second acquired data corresponding to the second device image data can be eliminated according to the data items that need to be eliminated as indicated by the inspection configuration information to obtain a data set after elimination.

[0128] In the embodiment of the present application, by performing frame skipping detection on the device image data, discontinuous data can be eliminated to ensure data continuity and improve data quality.

[0129] In the above step S502, when the target data to be eliminated is determined according to the data change value, the invalid data information in the current collected data set can be determined based on the target data at the same time, such as Fig.10 As shown, the method of the present application also includes:

[0130] S1001. Determine invalid data information in a current collected data set according to the inspection configuration information.

[0131] S1002. Determine and output fault information of the target device based on the invalid data information.

[0132] The invalid data information includes the collection device identification and the abnormal data. The collection device identification can be Figure 1 The acquisition device in the data includes, for example, a sensor identifier, a camera identifier, a laser radar identifier, etc. The acquisition device identifier is used to indicate the data item to which the data belongs. Each acquisition device identifier corresponds to a data item, and the data in the data item is acquired by the acquisition device corresponding to the acquisition device identifier. The abnormal data includes the target data with abnormalities determined based on the data change value and the inspection configuration information in the above step S502, and the device image data with green screen failure, etc.

[0133] As a possible implementation method, invalid data information can be recorded in a log in the form of a log, so that users can quickly locate hardware failure points based on the log and maintain or replace the faulty device in a timely manner.

[0134] In the embodiment of the present application, by recording invalid data information, maintenance personnel can quickly locate the fault point in the equipment based on the invalid data information, and maintain and replace the fault point in time.

[0135] Optionally, the device operation characteristic information includes posture change information and load change information of multiple operations. In the embodiment of the present application, the data pool can be filtered according to the device operation characteristic information to obtain a filtered data pool, which includes at least one multimodal data set, such as Fig.11 As shown, the above step S203 includes:

[0136] S1101. Determine the posture change information of the target device according to the device posture data of the target time period in the data pool.

[0137] The posture change information of the target device may represent the posture change of each joint of the target device in the target time period.

[0138] It should be understood that when the excavator equipment performs the same operation, the position and posture changes of its joints have certain regularities. As a possible implementation method, a change curve can be drawn for the equipment posture data of each joint of the excavator equipment, and the change curve is used as the posture change information of the target equipment.

[0139] When the target device performs multiple operations, there is generally an interval between the multiple operations, and the target period can be the period between the intervals. In another implementation, the target period can also be a period selected by the user.

[0140] S1102: Determine load change information of a target device according to device load data of a target period in a data pool.

[0141] The load change information can represent the load change of the target device within the target period. A change curve can be drawn for the equipment load data of the excavator equipment, and the drawn curve is used as the load change information of the target device.

[0142] S1103. If the difference between the posture change information and the posture change information of the target operation in the device operation characteristic information is within the first difference interval, and the difference between the load change information and the load change information of the target operation in the device operation characteristic information is within the second difference interval, then the data of the target time period in the data pool is used as the multimodal data set of the target operation.

[0143] In the present application, the baseline posture change curve and the baseline load change curve of each joint can be drawn in advance for various operations of the target device, and the baseline change curve can be used as the posture change information of the device operation characteristic information, and the baseline load change curve can be used as the load change information of the device operation characteristic information.

[0144] When matching device operations with data, the posture change curves of each joint of the target device in the target period can be compared with the baseline posture change curve, and the load change curve of the target device in the target period can be compared with the baseline load change curve. If the differences between the corresponding points at each moment on the curve are within the preset first difference interval, the data of the target period in the data pool can be added to the multimodal data set of the target operation.

[0145] In an embodiment of the present application, by matching the data in the data pool with the rules of each operation, the operation and the data can be associated to form a multimodal data set for each operation, thereby providing reliable data support for subsequent deep learning and large model learning based on multimodal data.

[0146] Based on the same inventive concept, a multimodal data processing device corresponding to the multimodal data processing method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned multimodal data processing method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0147] Fig.12 A schematic diagram of the structure of a multimodal data processing device provided in an embodiment of the present application is shown, and the device includes an acquisition module 1201, a rejection module 1202 and a screening module 1203.

[0148] An acquisition module 1201 is used to acquire a current acquisition data set of a target device according to the running state of the target device, wherein the current acquisition data set includes at least one of the following: device posture data, device load data, device image data, and device positioning data;

[0149] The elimination module 1202 is used to eliminate the current collected data set according to the inspection configuration information and the timestamp information of the current collected data set to obtain a eliminated data set, and add the eliminated data set to the data pool;

[0150] The screening module 1203 is used to screen the data pool according to the device operation characteristic information to obtain a screened data pool, which includes at least one multimodal data set, each multimodal data set is a data set generated when the target device performs the target operation, and the device operation characteristic information is used to describe the data regularity when the target device performs the target operation.

[0151] In a feasible implementation scheme, the acquisition module 1201 is specifically used for:

[0152] Get the current running status of the target device;

[0153] If the current operating state is normal, the acquisition device of the target device is started, and the current acquisition data set of the target device is acquired through the acquisition device;

[0154] If the current operating status is abnormal, the acquisition device of the target device will not be started.

[0155] In a feasible implementation scheme, the elimination module 1202 is specifically used for:

[0156] Eliminate the current acquisition data set according to the inspection configuration information to obtain a first acquisition data set;

[0157] Eliminate the first collected data set according to the timestamp information to obtain a second collected data set;

[0158] A frame skipping check process is performed on the device image data in the second acquired data set to obtain a data set after elimination.

[0159] In a feasible implementation scheme, the elimination module 1202 is specifically used for:

[0160] Determine the data change value of each data in the current collection data set according to the current collection data set and the historical collection data set;

[0161] Determine whether there is target data to be eliminated in the current collection data set according to the inspection configuration information, the current collection data set and the data change value of each data in the current collection data set;

[0162] If so, the target data in the current collected data set is removed to obtain a first collected data set.

[0163] In a feasible implementation scheme, the elimination module 1202 is specifically used for:

[0164] Traverse each data item in the current collection data set. For the current data item traversed, if the data change value of the current data item is equal to 0 and the number of unchanged times corresponding to the current data item is greater than the preset number threshold corresponding to the current data item in the inspection configuration information, or the data change value of the current data item is greater than the preset change threshold corresponding to the current data item in the inspection configuration information, or the value of the current data item is greater than the preset data threshold corresponding to the current data item in the inspection configuration information, then take the current data item as a target data item; otherwise, increase the number of unchanged times corresponding to the current data item by 1.

[0165] In a feasible implementation scheme, the elimination module 1202 is specifically used for:

[0166] Performing alignment processing on the first collected data in the first collected data set according to the timestamp information and the preset collection frequency to obtain an aligned data set corresponding to the collection frequency;

[0167] Each aligned data set is eliminated to obtain a second acquired data set.

[0168] In a feasible implementation scheme, the elimination module 1202 is specifically used for:

[0169] If the first collected data in the aligned data set has missing values, the aligned data set is discarded;

[0170] If there is no missing value in the first collected data in the aligned data set, the aligned data set is retained.

[0171] In a feasible implementation scheme, the elimination module 1202 is specifically used for:

[0172] Binarization is performed on the first device image data and the second device image data to obtain a first character string corresponding to the first device image data and a second character string corresponding to the second device image data, wherein the first device image data and the second device image data are device image data at adjacent moments in the second acquisition data set;

[0173] Determine the Hamming distance between the first device image data and the second device image data according to the first character string and the second character string;

[0174] Determine whether to perform data elimination based on the Hamming distance and the preset distance threshold;

[0175] If so, the second collected data corresponding to the image data of the first device and the second collected data corresponding to the image data of the second device are eliminated to obtain a data set after elimination.

[0176] In one feasible implementation, the rejection module 1202 is further configured to:

[0177] According to the inspection configuration information, determine the invalid data information in the current collection data set;

[0178] Failure information of the target device is determined and output based on the invalid data information.

[0179] In a feasible implementation scheme, the device operation characteristic information includes posture change information and load change information of multiple operations;

[0180] The screening module 1203 is specifically used for:

[0181] Determine the posture change information of the target device according to the device posture data of the target period in the data pool;

[0182] Determine the load change information of the target device according to the device load data of the target period in the data pool;

[0183] If the difference between the posture change information and the posture change information of the target operation in the device operation characteristic information is within the first difference interval, and the difference between the load change information and the load change information of the target operation in the device operation characteristic information is within the second difference interval, then the data of the target time period in the data pool is used as the multimodal data set of the target operation.

[0184] By automatically cleaning the collected real-time data, the accuracy and completeness of the multimodal data of the excavator equipment can be significantly improved, ensuring the reliability of the data. In addition, the present application does not need to rely on manual experience and statistical methods, and realizes fully automatic data cleaning, saving a lot of manpower and time costs, and improving data processing efficiency. By screening the data using the data change rules of the equipment operation, a multimodal data set of different operations can be obtained, so that subsequent data analysis and model training can be performed based on high-quality multimodal data sets.

[0185] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown, including: a processor 1301, a storage medium 1302 and a bus 1303, wherein the storage medium 1302 stores machine-readable instructions executable by the processor 1301. When the electronic device runs a multimodal data processing method in the embodiment, the processor 1301 communicates with the storage medium 1302 via the bus 1303, and the processor 1301 executes the machine-readable instructions. The processor 1301 performs the following steps in the preamble of the method item:

[0186] Acquire a current collection data set of the target device according to the running state of the target device, wherein the current collection data set includes at least one of the following: device posture data, device load data, device image data, and device positioning data;

[0187] Eliminate the current collected data set according to the inspection configuration information and the timestamp information of the current collected data set to obtain a data set after elimination, and add the data set after elimination to the data pool;

[0188] The data pool is filtered and processed according to the device operation characteristic information to obtain a filtered data pool, which includes at least one multimodal data set, each multimodal data set is a data set generated when the target device performs the target operation, and the device operation characteristic information is used to describe the data regularity when the target device performs the target operation.

[0189] In a feasible implementation manner, when the processor 1301 executes the acquisition of the current collected data set of the target device according to the running state of the target device, it is specifically used to:

[0190] Get the current running status of the target device;

[0191] If the current operating state is normal, the acquisition device of the target device is started, and the current acquisition data set of the target device is acquired through the acquisition device;

[0192] If the current operating status is abnormal, the acquisition device of the target device will not be started.

[0193] In a feasible implementation manner, when the processor 1301 performs a removal process on the current collected data according to the inspection configuration information and the timestamp information of the current collected data set to obtain the removed data set, it is specifically used to:

[0194] Eliminate the current acquisition data set according to the inspection configuration information to obtain a first acquisition data set;

[0195] Eliminate the first collected data set according to the timestamp information to obtain a second collected data set;

[0196] A frame skipping check process is performed on the device image data in the second acquired data set to obtain a data set after elimination.

[0197] In a feasible implementation manner, when the processor 1301 performs the elimination process on the current acquisition data set according to the inspection configuration information to obtain the first acquisition data set, it is specifically configured to:

[0198] Determine the data change value of each data in the current collection data set according to the current collection data set and the historical collection data set;

[0199] Determine whether there is target data to be eliminated in the current collection data set according to the inspection configuration information, the current collection data set and the data change value of each data in the current collection data set;

[0200] If so, the target data in the current collected data set is removed to obtain a first collected data set.

[0201] In a feasible implementation manner, when the processor 1301 determines whether there is target data to be removed in the current collection data set according to the inspection configuration information, the current collection data set, and the data change value of each data in the current collection data set, it is specifically used to:

[0202] Traverse each data item in the current collection data set. For the current data item traversed, if the data change value of the current data item is equal to 0 and the number of unchanged times corresponding to the current data item is greater than the preset number threshold corresponding to the current data item in the inspection configuration information, or the data change value of the current data item is greater than the preset change threshold corresponding to the current data item in the inspection configuration information, or the value of the current data item is greater than the preset data threshold corresponding to the current data item in the inspection configuration information, then take the current data item as a target data item; otherwise, increase the number of unchanged times corresponding to the current data item by 1.

[0203] In a feasible implementation manner, when the processor 1301 performs the elimination process on the first collected data set according to the timestamp information to obtain the second collected data set, it is specifically configured to:

[0204] Performing alignment processing on the first collected data in the first collected data set according to the timestamp information and the preset collection frequency to obtain an aligned data set corresponding to the collection frequency;

[0205] Each aligned data set is eliminated to obtain a second acquired data set.

[0206] In a feasible implementation manner, when the processor 1301 performs the elimination process on each aligned data set to obtain the second acquired data set, it is specifically configured to:

[0207] If the first collected data in the aligned data set has missing values, the aligned data set is discarded;

[0208] If there is no missing value in the first collected data in the aligned data set, the aligned data set is retained.

[0209] In a feasible implementation manner, when the processor 1301 performs frame skipping check processing on the device image data in the second acquired data set to obtain the eliminated data set, it is specifically used to:

[0210] Binarization is performed on the first device image data and the second device image data to obtain a first character string corresponding to the first device image data and a second character string corresponding to the second device image data, wherein the first device image data and the second device image data are device image data at adjacent moments in the second acquisition data set;

[0211] Determine the Hamming distance between the first device image data and the second device image data according to the first character string and the second character string;

[0212] Determine whether to perform data elimination based on the Hamming distance and the preset distance threshold;

[0213] If so, the second collected data corresponding to the image data of the first device and the second collected data corresponding to the image data of the second device are eliminated to obtain a data set after elimination.

[0214] In a feasible implementation manner, the processor 1301 is further configured to:

[0215] According to the inspection configuration information, determine the invalid data information in the current collection data set;

[0216] Failure information of the target device is determined and output based on the invalid data information.

[0217] In a feasible implementation manner, the device operation characteristic information includes posture change information and load change information of multiple operations;

[0218] When the processor 1301 performs screening processing on the data pool according to the device operation characteristic information to obtain a screened data pool, wherein the screened data pool includes at least one multimodal data set, the processor 1301 is specifically configured to:

[0219] Determine the posture change information of the target device according to the device posture data of the target period in the data pool;

[0220] Determine the load change information of the target device according to the device load data of the target period in the data pool;

[0221] If the difference between the posture change information and the posture change information of the target operation in the device operation characteristic information is within the first difference interval, and the difference between the load change information and the load change information of the target operation in the device operation characteristic information is within the second difference interval, then the data of the target time period in the data pool is used as the multimodal data set of the target operation.

[0222] By automatically cleaning the collected real-time data, the accuracy and completeness of the multimodal data of the excavator equipment can be significantly improved, ensuring the reliability of the data. In addition, the present application does not need to rely on manual experience and statistical methods, and realizes fully automatic data cleaning, saving a lot of manpower and time costs, and improving data processing efficiency. By screening the data using the data change rules of the equipment operation, a multimodal data set of different operations can be obtained, so that subsequent data analysis and model training can be performed based on high-quality multimodal data sets.

[0223] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor performs the following steps:

[0224] Acquire a current collection data set of the target device according to the running state of the target device, wherein the current collection data set includes at least one of the following: device posture data, device load data, device image data, and device positioning data;

[0225] Eliminate the current collected data set according to the inspection configuration information and the timestamp information of the current collected data set to obtain a data set after elimination, and add the data set after elimination to the data pool;

[0226] The data pool is filtered and processed according to the device operation characteristic information to obtain a filtered data pool, which includes at least one multimodal data set, each multimodal data set is a data set generated when the target device performs the target operation, and the device operation characteristic information is used to describe the data regularity when the target device performs the target operation.

[0227] In a feasible implementation manner, when the processor executes the acquisition of a current collection data set of the target device according to the running state of the target device, the processor is specifically configured to:

[0228] Get the current running status of the target device;

[0229] If the current operating state is normal, the acquisition device of the target device is started, and the current acquisition data set of the target device is acquired through the acquisition device;

[0230] If the current operating status is abnormal, the acquisition device of the target device will not be started.

[0231] In a feasible implementation manner, when the processor performs a removal process on the current collected data according to the inspection configuration information and the timestamp information of the current collected data set to obtain the removed data set, it is specifically used to:

[0232] Eliminate the current acquisition data set according to the inspection configuration information to obtain a first acquisition data set;

[0233] Eliminate the first collected data set according to the timestamp information to obtain a second collected data set;

[0234] A frame skipping check process is performed on the device image data in the second acquired data set to obtain a data set after elimination.

[0235] In a feasible implementation manner, when the processor performs a culling process on the current acquisition data set according to the inspection configuration information to obtain the first acquisition data set, the processor is specifically configured to:

[0236] Determine the data change value of each data in the current collection data set according to the current collection data set and the historical collection data set;

[0237] Determine whether there is target data to be eliminated in the current collection data set according to the inspection configuration information, the current collection data set and the data change value of each data in the current collection data set;

[0238] If so, the target data in the current collected data set is removed to obtain a first collected data set.

[0239] In a feasible implementation manner, when the processor determines whether there is target data to be removed in the current collection data set according to the inspection configuration information, the current collection data set, and the data change value of each data in the current collection data set, the processor is specifically used to:

[0240] Traverse each data item in the current collection data set. For the current data item traversed, if the data change value of the current data item is equal to 0 and the number of unchanged times corresponding to the current data item is greater than the preset number threshold corresponding to the current data item in the inspection configuration information, or the data change value of the current data item is greater than the preset change threshold corresponding to the current data item in the inspection configuration information, or the value of the current data item is greater than the preset data threshold corresponding to the current data item in the inspection configuration information, then take the current data item as a target data item; otherwise, increase the number of unchanged times corresponding to the current data item by 1.

[0241] In a feasible implementation manner, when the processor performs the elimination process on the first collected data set according to the timestamp information to obtain the second collected data set, it is specifically used to:

[0242] Performing alignment processing on the first collected data in the first collected data set according to the timestamp information and the preset collection frequency to obtain an aligned data set corresponding to the collection frequency;

[0243] Each aligned data set is eliminated to obtain a second acquired data set.

[0244] In a feasible implementation manner, when the processor performs the elimination processing on each aligned data set to obtain the second acquired data set, it is specifically used to:

[0245] If the first collected data in the aligned data set has missing values, the aligned data set is discarded;

[0246] If there is no missing value in the first collected data in the aligned data set, the aligned data set is retained.

[0247] In a feasible implementation manner, when the processor performs frame skipping check processing on the device image data in the second acquired data set to obtain the eliminated data set, it is specifically used to:

[0248] Binarization is performed on the first device image data and the second device image data to obtain a first character string corresponding to the first device image data and a second character string corresponding to the second device image data, wherein the first device image data and the second device image data are device image data at adjacent moments in the second acquisition data set;

[0249] Determine the Hamming distance between the first device image data and the second device image data according to the first character string and the second character string;

[0250] Determine whether to perform data elimination based on the Hamming distance and the preset distance threshold;

[0251] If so, the second collected data corresponding to the image data of the first device and the second collected data corresponding to the image data of the second device are eliminated to obtain a data set after elimination.

[0252] In a feasible implementation manner, the processor is further configured to:

[0253] According to the inspection configuration information, determine the invalid data information in the current collection data set;

[0254] Failure information of the target device is determined and output based on the invalid data information.

[0255] In a feasible implementation manner, the device operation characteristic information includes posture change information and load change information of multiple operations;

[0256] When the processor performs screening processing on the data pool according to the device operation characteristic information to obtain a screened data pool, wherein the screened data pool includes at least one multimodal data set, the processor is specifically used to:

[0257] Determine the posture change information of the target device according to the device posture data of the target period in the data pool;

[0258] Determine the load change information of the target device according to the device load data of the target period in the data pool;

[0259] If the difference between the posture change information and the posture change information of the target operation in the device operation characteristic information is within the first difference interval, and the difference between the load change information and the load change information of the target operation in the device operation characteristic information is within the second difference interval, then the data of the target time period in the data pool is used as the multimodal data set of the target operation.

[0260] The embodiment of the present application can significantly improve the accuracy and completeness of the multimodal data of the excavator equipment and ensure the reliability of the data by automatically cleaning the collected real-time data. In addition, the present application does not need to rely on manual experience and statistical methods, and realizes fully automatic data cleaning, saving a lot of manpower and time costs and improving data processing efficiency. By screening the data using the data change rules of the equipment operation, a multimodal data set of different operations can be obtained, so that subsequent data analysis and model training can be performed based on the high-quality multimodal data set.

[0261] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0262] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0263] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0264] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0265] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0266] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0267] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A multimodal data processing method, characterized in that: include: Acquire a current collection data set of the target device according to the running state of the target device, wherein the current collection data set includes at least one of the following: device posture data, device load data, device image data, and device positioning data; Performing elimination processing on the current collected data set according to the inspection configuration information and the timestamp information of the current collected data set to obtain a eliminated data set, and adding the eliminated data set to the data pool; The data pool is filtered according to the device operation characteristic information to obtain a filtered data pool, wherein the filtered data pool includes at least one multimodal data set, each of the multimodal data sets is a data set generated when the target device performs a target operation, and the device operation characteristic information is used to describe the data regularity when the target device performs the target operation.

2. The method according to claim 1, characterized in that The step of obtaining a current collection data set of the target device according to the running state of the target device includes: Obtaining the current operating status of the target device; If the current operating state is normal, starting the acquisition device of the target device, and acquiring the current acquisition data set of the target device through the acquisition device; If the current operating state is abnormal, the acquisition device of the target device is not started.

3. The method according to claim 1, characterized in that The step of performing elimination processing on the currently collected data according to the inspection configuration information and the timestamp information of the currently collected data set to obtain a data set after elimination includes: Performing elimination processing on the current acquisition data set according to the inspection configuration information to obtain a first acquisition data set; Performing a elimination process on the first collected data set according to the timestamp information to obtain a second collected data set; A frame skipping check process is performed on the device image data in the second acquired data set to obtain the eliminated data set.

4. The method according to claim 3, characterized in that The step of performing elimination processing on the current acquisition data set according to the inspection configuration information to obtain a first acquisition data set includes: Determining a data change value of each data in the current collection data set according to the current collection data set and the historical collection data set; Determining whether there is target data to be eliminated in the current collection data set according to the inspection configuration information, the current collection data set, and data change values ​​of each item of data in the current collection data set; If so, the target data in the current collected data set is removed to obtain the first collected data set.

5. The method according to claim 4, characterized in that The determining whether there is target data to be eliminated in the current collection data set according to the inspection configuration information, the current collection data set, and the data change value of each data in the current collection data set includes: Traverse each data item in the current collection data set, and for the current item of data traversed, if the data change value of the current item of data is equal to 0 and the number of unchanged times corresponding to the current item of data is greater than the preset number threshold corresponding to the current item of data in the inspection configuration information, or the data change value of the current item of data is greater than the preset change threshold corresponding to the current item of data in the inspection configuration information, or the value of the current item of data is greater than the preset data threshold corresponding to the current item of data in the inspection configuration information, then take the current item of data as one of the target data; otherwise, increase the number of unchanged times corresponding to the current item of data by 1.

6. The method according to claim 3, characterized in that: The step of performing elimination processing on the first collected data set according to the timestamp information to obtain a second collected data set includes: Performing alignment processing on the first collected data in the first collected data set according to the timestamp information and the preset collection frequency to obtain an aligned data set corresponding to the collection frequency; Each of the aligned data sets is eliminated to obtain the second acquired data set.

7. The method according to claim 6, characterized in that The step of performing elimination processing on each of the aligned data sets to obtain the second acquired data set includes: If there are missing values ​​in the first collected data in the aligned data set, the aligned data set is discarded; If there is no missing value in the first collected data in the aligned data set, the aligned data set is retained.

8. The method according to claim 3, characterized in that The performing frame skipping check processing on the device image data in the second acquired data set to obtain the eliminated data set includes: Binarization is performed on the first device image data and the second device image data to obtain a first character string corresponding to the first device image data and a second character string corresponding to the second device image data, wherein the first device image data and the second device image data are device image data at adjacent moments in the second acquired data set; Determine a Hamming distance between the first device image data and the second device image data according to the first character string and the second character string; Determine whether to perform data elimination processing according to the Hamming distance and a preset distance threshold; If so, the second collected data corresponding to the first device image data and the second collected data corresponding to the second device image data are eliminated to obtain the eliminated data set.

9. The method according to claim 1, characterized in that: The method further comprises: Determine invalid data information in the current collected data set according to the inspection configuration information; Failure information of the target device is determined and output based on the invalid data information.

10. The method according to claim 1, characterized in that The device operation characteristic information includes posture change information and load change information of multiple operations; The data pool is screened according to the device operation characteristic information to obtain a screened data pool, wherein the screened data pool includes at least one multimodal data set, including: Determining the posture change information of the target device according to the device posture data of the target time period in the data pool; Determining load change information of the target device according to the device load data of the target period in the data pool; If the difference between the posture change information and the posture change information of the target operation in the device operation characteristic information is within a first difference interval, and the difference between the load change information and the load change information of the target operation in the device operation characteristic information is within a second difference interval, then the data of the target time period in the data pool is used as the multimodal data set of the target operation.

11. A multimodal data processing device, characterized in that: include: An acquisition module, used to acquire a current acquisition data set of the target device according to the running state of the target device, wherein the current acquisition data set includes at least one of the following: device posture data, device load data, device image data, and device positioning data; A removal module, used for removing the currently collected data set according to the inspection configuration information and the timestamp information of the currently collected data set to obtain a removed data set, and adding the removed data set to the data pool; A screening module is used to screen the data pool according to the device operation characteristic information to obtain a screened data pool, wherein the screened data pool includes at least one multimodal data set, each of which is a data set generated when the target device performs a target operation, and the device operation characteristic information is used to describe the data regularity when the target device performs the target operation.

12. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of a multimodal data processing method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal data processing method as claimed in any one of claims 1 to 10 are executed.

Images