Industrial data fusion analysis method
By combining vision sensors and wearable sensors, and using neural networks to extract and fuse feature data, the problems of inefficient monitoring workers connecting notebook hard disk data lines and limited information acquisition in the prior art are solved, achieving a more accurate and comprehensive monitoring effect.
Patent Information
- Application Number
- CN202411703685.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-04-15
AI Technical Summary
The prior art monitors workers to connect to laptop hard disk data cables, which are inefficient and cannot monitor and analyze workers' activities in real time. The information obtained by a single modal sensor is limited, which is easily blocked by production equipment, resulting in some visual information loss.
Image feature data is collected through factory vision sensors and wearable sensors to collect time series data, recursive neural networks and convolutional neural networks are used to extract action features and spatial features, and fuse them through multimodal sensor data fusion model, ultimately achieving security warning for workers to connect to laptop hard disk data lines.
It realizes more comprehensive and accurate monitoring of the data cables of workers connecting to laptop hard disks, reduces excessive dependence on visual sensor information, can better deal with production equipment occlusion, and improves monitoring accuracy.
Smart Images
Figure CN119829933A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data fusion, and more specifically, to a method for industrial data fusion analysis. Background Art
[0002] The production of laptops plays an important role in the national economy. However, when connecting the laptop hard drive data cable, workers' incorrect operations affect production safety and efficiency. At present, the monitoring methods for workers connecting the laptop hard drive data cable include manual activity analysis and target detection methods based on the visual system YOLOv5 model. Manual activity analysis mainly relies on manual observation and recording, which is inefficient and cannot monitor and analyze workers' activities in real time. The target detection method based on the visual system YOLOv5 model uses visual sensors to obtain image information and detects and identifies targets in the image through deep learning algorithms. It has the advantages of low cost and high efficiency. However, the single modality sensor obtains limited information about workers connecting the laptop hard drive data cable, and the visual sensor is easily blocked by production equipment, resulting in the loss of some visual information. In order to solve the above problems, a technical solution is provided. Summary of the invention
[0003] In order to overcome the above-mentioned defects of the prior art, the present invention provides a method for industrial data fusion analysis, which collects image feature data of workers connecting to laptop hard disk data cables through factory visual sensors, collects time series data of workers connecting to laptop hard disk data cables through wearable sensors, extracts motion features in the time series data through recursive neural networks, extracts spatial features in the image feature data through convolutional neural networks, and performs sampling rate unification processing on spatial features and motion features through linear interpolation, fuses the spatial features extracted by the convolutional neural network and the motion features extracted by the recursive neural network, realizes multimodal data fusion, and finally uses the multimodal sensor data fusion model to issue safety warnings to workers when they connect to laptop hard disk data cables.
[0004] To achieve the above object, the present invention provides the following technical solution: a method for industrial data fusion analysis, which comprises the following specific steps: Step S1, collecting image feature data of a worker connecting a laptop hard drive data cable through a visual sensor; Step S2, collecting action timing data of workers connecting to the laptop hard drive data cable through wearable sensors; Step S3, fusing the action time series data and the image feature data to build a multimodal sensor data fusion model; extracting action features from the action time series data through a recursive neural network, and extracting spatial features from the image feature data through a convolutional neural network. The formula of the multimodal sensor data fusion model is: , Where: is the model output after the fusion of spatial features and motion features. is the weight matrix, is the bias term, It is the feature after the fusion of spatial features and motion features. The feature formula for the fusion of spatial features and motion features is: , Where: is the spatial feature of the output of the first image frame, is the spatial feature of the output of the second image frame, For the The spatial features of the output of image frames, for The set of spatial features output by each image frame, is the action feature of the first time step, is the action feature of the second time step, For the The action features of time steps, for A collection of action features at time steps; Step S4: The multimodal sensor data fusion model issues a safety warning to the worker when connecting the laptop hard drive data cable.
[0005] As a further solution of the present invention, the image feature data collected by the visual sensor when the worker connects the laptop hard disk data cable includes the following specific contents: The camera obtains the video data of the worker connecting the laptop hard drive data cable. The visual sensor processes the video data into multiple frames of image feature data of the worker connecting the laptop hard drive data cable. The frame rate is an important parameter in the image feature data. It indicates the number of image frames collected per second. A higher frame rate can provide a smoother image sequence. Real-time observation of the worker's movements and the connection status of the laptop hard drive data cable, and timely detection of any unsafe behavior are crucial for monitoring the worker to connect the laptop hard drive data cable as required. The frame rate formula of the image sequence is: , Where: is the frame rate of the image sequence, are all constants, used to adjust the weight and influence of each parameter. Data transmission bandwidth The power of reflects the nonlinear effect of bandwidth on frame rate. is the sensor noise The square and exponential negative function indicates that the greater the noise, the greater the negative impact on the frame rate. is the resolution, For temperature The power of For lighting conditions.
[0006] As a further solution of the present invention, the wearable sensor collects the time series data of workers connecting to the laptop hard drive data cable, including the following specific contents: Wearable sensors are installed on workers' hands to accurately sense the status of workers' hands connecting to the laptop hard drive data cable. The sensor can focus on and efficiently collect the body movement data of workers when connecting to the laptop hard drive data cable. The sensor is designed to cover the worker's hand area. When the worker does not connect the hard drive data cable, the sensor can quickly capture this abnormal movement. When the worker's action of connecting the hard drive data cable is incorrect, the sensor can also collect abnormal data. Wearable sensors are arranged at key positions such as the base of the finger, the middle of the finger, the fingertip, the palm, the back of the hand and the wrist. The palm is the main part of the hand, and its movement can often reflect the operation intention and movement trend of the whole hand. The opening and closing of the palm, the left and right swing, and the up and down flipping can all be sensed by the sensor. For the monitoring of the base, middle and fingertip parts, the movement information of each joint can be fully obtained. Therefore, according to this layout, the overall movement and rotation of the palm can be monitored to capture a larger range of movements, and the movement information of each joint can be fully obtained. Then the data from sensors at different positions are integrated to reconstruct the complete movement of the hand. Data integration can give full play to the advantages of each sensor and make up for the limitations of a single sensor. Six wearable sensors are embedded in the glove. This high-density layout can accurately reproduce the complex movements of the hand. The wearable sensor has built-in accelerometers, gyroscopes and geomagnetic sensors. The accelerometer is used to detect the linear movement of the hand in space, including up and down, left and right, front and back acceleration. The accelerometer can accurately detect the translational movement of the worker's hand, including lifting up, lowering, moving to the left, swinging back, pushing forward and withdrawing backward. The value output by the accelerometer is the three-axis acceleration value. , these three axes represent three different directions in space, and these values describe the acceleration of the worker's hand in each direction; the gyroscope is used to measure the angular velocity of the hand. It can identify the rotational movement of the worker's hand around each axis, such as the rotation of the fingers, the flipping of the palm, and other subtle movements. In the work scene, the rotation of the hand can reflect the worker's operating habits and working status. Similarly, the gyroscope also outputs three-axis data. These three axes represent different rotation directions and can accurately capture the rotation speed of the hand around each axis; the geomagnetic sensor is mainly used to provide absolute orientation information. It provides a stable reference system for the system by measuring the strength and direction of the earth's magnetic field, helping the system to calibrate the direction and avoid the cumulative error caused by gyroscope drift. The gyroscope will drift during long-term use, resulting in inaccurate measurement results. The geomagnetic sensor can calibrate the measurement results of the gyroscope by continuously detecting information about the earth's magnetic field to ensure the measurement accuracy and stability of the entire system.
[0007] As a further solution of the present invention, a multimodal sensor data fusion model is constructed to fuse time series data and image feature data, including the following specific contents: The recursive neural network model is used to extract the motion information of wearable sensors. In the analysis of a worker connecting a laptop hard drive data cable, the key point detection of the wearable sensor on the hand obtains the time series motion data and captures the motion features. The convolutional neural network model is used to extract the visual information in the image, and the pre-trained deep learning model is used to extract the spatial features at the frame level. The accelerometer and gyroscope on the wearable sensor are used to record the hand motion data. The formula for the time series data obtained by the wearable sensor is: , Where: Time series data obtained from wearable sensors, is the set of real numbers, is the number of time steps, that is, the number of moments collected by the wearable sensor, is the three-axis acceleration, is the three-axis rotation speed, For absolute orientation information, the magnitude of data from different sensors varies greatly, so the data needs to be scaled to the range of [0,1] or [-1,1] to reduce the magnitude differences between different feature dimensions; The time series data is input into the recurrent neural network model. The recurrent neural network processes the time series data through its loop structure, gradually capturing the dependencies between time steps, thereby extracting hand movement features. Input data for time steps is a 3D vector, and the action characteristic formula of the worker’s hand at this moment is: , Where: is the motion feature of the wearable sensor, is the set of real numbers, is the input data for the first time step, is the input data for the second time step, for Time step The input data, is the three-axis acceleration, is the three-axis rotation speed, For absolute position information, the recurrent neural network calculates The formula for updating the hidden state is: , Where: For the The hidden state at the time step, For the The input data of the time step, is the hidden state of the previous time step, is the input weight matrix, is the hidden state weight matrix, is the bias term, As activation function, nonlinear features are added, and the recurrent neural network extracts The action characteristics of the maintenance worker in the time step are: ; The image feature data is obtained by using the visual system. The formula of the image feature data is: , Where: is the image feature data, is the set of real numbers, is the length of the image sequence, is the image channel, is the width of the image, is the height of the image, for each frame in the sequence , the spatial feature formula extracted by the convolutional neural network is: , Where: For the The spatial characteristics of the frame image output, is the index in the height direction of the feature map, is the index in the width direction of the feature map, is the channel index in the feature map, which is used to distinguish the feature maps output by different convolution kernels. is the position index of the convolution kernel in the height direction, is the position index of the convolution kernel in the width direction, is the weight of the convolution kernel, is the offset, is the activation function, For the input image Frame at position No. The pixel value of each channel; The spatial features and motion features have different sampling rates. Linear interpolation is used to synchronize the two in the time dimension. The interpolated motion feature values The formula is: , Where: is the action feature value after interpolation, and is the sampling time point of the action feature, and , It's time point The sampled action feature value, It's time point The sampled action feature value, For the intermediate time points, linearly interpolate the action features so that their timestamps match the timestamps of the spatial features; The recursive neural network extracts motion features from the action time series data, and the convolutional neural network extracts spatial features from the image feature data. The model formula after the motion features and spatial features are fused is: , Where: is the model output after the fusion of spatial features and motion features. is the weight matrix, is the bias term, It is the feature after the fusion of spatial features and motion features. The feature formula for the fusion of spatial features and motion features is: , Where: is the spatial feature of the output of the first image frame, is the spatial feature of the output of the second image frame, For the The spatial features of the output of image frames, for The set of spatial features output by each image frame, is the action feature of the first time step, is the action feature of the second time step, For the The action features of time steps, for A collection of action features at time steps; The technical effects and advantages of the method for industrial data fusion analysis of the present invention are as follows: The present invention collects image feature data of workers connecting laptop hard disk data cables through factory visual sensors, and collects action timing data of workers connecting laptop hard disk data cables through wearable sensors installed on workers' hands. Multi-sensors are used to sense the situation of workers connecting laptop hard disk data cables, so that the behavior of workers not connecting laptop hard disk data cables according to specifications can be discovered more comprehensively and accurately. Multi-sensors are used to reduce excessive reliance on image information collected by visual sensors, so that the factory can better cope with the occlusion of production equipment. Convolutional neural networks are used to extract spatial features from image feature data, and recursive neural networks are used to extract action features from action timing data. Spatial features and action features are fused to construct a multimodal sensor data fusion model. Different types of data are used in a comprehensive manner to improve the accuracy of monitoring whether workers are connecting laptop hard disk data cables according to specifications. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 The present invention is a flow chart of a method for industrial data fusion analysis. DETAILED DESCRIPTION
[0009] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0010] Example 1 The present invention provides a method for industrial data fusion analysis, which comprises the following steps: Step S1, collecting image feature data of a worker connecting a laptop hard drive data cable through a visual sensor; Step S2, collecting action timing data of workers connecting to the laptop hard drive data cable through wearable sensors; Step S3, fusing the action time series data and the image feature data to build a multimodal sensor data fusion model; extracting action features from the action time series data through a recursive neural network, and extracting spatial features from the image feature data through a convolutional neural network. The formula of the multimodal sensor data fusion model is: , Where: is the model output after the fusion of spatial features and motion features. is the weight matrix, is the bias term, It is the feature after the fusion of spatial features and motion features. The feature formula for the fusion of spatial features and motion features is: , Where: is the spatial feature of the output of the first image frame, is the spatial feature of the output of the second image frame, For the The spatial features of the output of image frames, for The set of spatial features output by each image frame, is the action feature of the first time step, is the action feature of the second time step, For the The action features of time steps, for A collection of action features at time steps; Step S4: The multimodal sensor data fusion model issues a safety warning to the worker when connecting the laptop hard drive data cable.
[0011] The image feature data collected by the visual sensor when the worker connects the laptop hard drive data cable includes the following specific contents: The camera obtains the video data of the worker connecting the laptop hard drive data cable. The visual sensor processes the video data into multiple frames of image feature data of the worker connecting the laptop hard drive data cable. The frame rate is an important parameter in the image feature data. It indicates the number of image frames collected per second. A higher frame rate can provide a smoother image sequence. It can observe the worker connecting the laptop hard drive data cable in real time and detect any unsafe behavior in time. It is very important to monitor the worker connecting the laptop hard drive data cable according to regulations. The frame rate formula of the image sequence is: , Where: is the frame rate of the image sequence, are all constants, used to adjust the weight and influence of each parameter. Data transmission bandwidth The power of reflects the nonlinear effect of bandwidth on frame rate. is the sensor noise The square and exponential negative function indicates that the greater the noise, the greater the negative impact on the frame rate. is the resolution, For temperature The power of The lighting conditions in the frame rate formula of the image sequence will be affected by the occlusion of the production equipment. If the production equipment in the factory is large or the layout is unreasonable, it may block the propagation of light, resulting in weakened light or shadows in some areas, which will make the image of the worker connecting the laptop hard drive data cable collected by the visual sensor unclear, making the sensor unable to detect the existence of the target or unable to obtain the complete characteristics of the target. The lighting conditions will also be affected by bad weather. Bad weather will cause the uniformity of light to deteriorate and uneven darkness to appear, and will also change the color temperature of the light, making the light look colder or warmer, which will affect the color presentation of the object.
[0012] The wearable sensor collects the worker's action time series data when connecting the laptop hard drive data cable, including the following specific contents: Wearable sensors are installed on workers' hands to accurately sense the status of workers' hands connecting to the laptop hard drive data cable. The sensor can focus on and efficiently collect body movement data when connecting the laptop hard drive data cable. The sensor is designed to mainly cover the worker's hand area. When the worker does not connect the laptop hard drive data cable, the sensor can quickly capture this abnormal movement. When the worker's action of connecting the laptop hard drive data cable is incorrect, the sensor can also collect abnormal data; Wearable sensors are designed to be flexible and wearable and integrated into gloves. This design has many advantages. On the one hand, the flexible material can better fit the shape of the hand and deform naturally with the movement of the hand without causing any hindrance to the movement of the hand. At the same time, the flexible sensor can adapt to hands of different sizes and shapes, which improves the versatility of the device. On the other hand, the design integrated into the glove can provide higher capture accuracy and improve the wearing comfort of workers. In order to accurately capture every subtle movement of the hand, wearable sensors are arranged at key positions such as the base, middle, fingertips, palm, back of hand and wrist. As the main part of the hand, the movement of the palm can often reflect the overall operation intention and movement trend of the hand. The opening and closing, left and right swinging, and up and down flipping of the palm can all be sensed by the sensor. For the monitoring of the base, middle and fingertips, the movement information of each joint can be fully obtained. Therefore, according to this layout, the overall movement and rotation of the palm can be monitored to capture a larger range of movements, and the movement information of each joint can be fully obtained. Then the data from sensors at different positions are integrated to reconstruct the complete movement of the hand. Data integration can give full play to the advantages of each sensor and make up for the limitations of a single sensor. Six wearable sensors are embedded in the glove. This high-density layout can accurately restore the complex movements of the hand; The wearable sensor has built-in accelerometers, gyroscopes and geomagnetic sensors. The accelerometer is used to detect the linear movement of the hand in space, including up and down, left and right, front and back acceleration. The accelerometer can accurately detect the translational movement of the worker's hand, including lifting up, lowering, moving to the left, swinging back, pushing forward and withdrawing backward. The value output by the accelerometer is the three-axis acceleration value. , these three axes represent three different directions in space, and these values describe the acceleration of the worker's hand in each direction; the gyroscope is used to measure the angular velocity of the hand. It can identify the rotational movement of the worker's hand around each axis, such as the rotation of the fingers, the flipping of the palm, and other subtle movements. In the work scene, the rotation of the hand can reflect the worker's operating habits and working status. Similarly, the gyroscope also outputs three-axis data. These three axes represent different rotation directions and can accurately capture the rotation speed of the hand around each axis; the geomagnetic sensor is mainly used to provide absolute orientation information. It provides a stable reference system for the system by measuring the strength and direction of the earth's magnetic field, helping the system to calibrate the direction and avoid the cumulative error caused by gyroscope drift. The gyroscope will drift during long-term use, resulting in inaccurate measurement results. The geomagnetic sensor can calibrate the measurement results of the gyroscope by continuously detecting information about the earth's magnetic field to ensure the measurement accuracy and stability of the entire system.
[0013] Constructing a multimodal sensor data fusion model to fuse time series data and image feature data includes the following specific contents: The recursive neural network model is used to extract the motion information of wearable sensors. In the analysis of a worker connecting a laptop hard drive data cable, the key point detection of the wearable sensor on the hand obtains the time series motion data and captures the motion features. The convolutional neural network model is used to extract the visual information in the image, and the pre-trained deep learning model is used to extract the spatial features at the frame level. The accelerometer, gyroscope and geomagnetic sensor on the wearable sensor are used to record the hand motion data. The formula for the time series data obtained by the wearable sensor is: , Where: Time series data obtained from wearable sensors, is the set of real numbers, is the number of time steps, that is, the number of moments collected by the wearable sensor, is the three-axis acceleration, is the three-axis rotation speed, For absolute orientation information, the magnitude of data from different sensors varies greatly, so the data needs to be scaled to the range of [0,1] or [-1,1] to reduce the magnitude differences between different feature dimensions; The time series data is input into the recurrent neural network model. The recurrent neural network processes the time series data through its loop structure, gradually capturing the dependencies between time steps, thereby extracting the hand movement features. Input data is a 3D vector, and the action characteristic formula of the maintenance worker’s hand at this moment is: , Where: is the motion feature of the wearable sensor, is the set of real numbers, For each time step The input data, is the three-axis acceleration, is the three-axis rotation speed, For absolute position information, the recurrent neural network calculates The formula for updating the hidden state is: , Where: For the The hidden state at the time step, For the The input data of the time step, is the hidden state of the previous time step, is the input weight matrix, is the hidden state weight matrix, is the bias term, As activation function, nonlinear features are added, and the recurrent neural network extracts The action characteristics of the maintenance worker in the time step are: ; The image feature data is obtained by using the visual system. The formula of the image feature data is: , Where: is the image feature data, is the set of real numbers, is the length of the image sequence, is the image channel, is the width of the image, is the height of the image, for each frame in the sequence , the spatial feature formula extracted by the convolutional neural network is: , Where: For the The spatial characteristics of the frame image output, is the index in the height direction of the feature map, is the index in the width direction of the feature map, is the channel index in the feature map, which is used to distinguish the feature maps output by different convolution kernels. is the position index of the convolution kernel in the height direction, is the position index of the convolution kernel in the width direction, is the weight of the convolution kernel, is the offset, is the activation function, For the input image Frame at position No. The pixel value of each channel; The spatial features and motion features have different sampling rates. Linear interpolation is used to synchronize the two in the time dimension. The interpolated motion feature values The formula is: , Where: is the action feature value after interpolation, and is the sampling time point of the action feature, and , It's time point The sampled action feature value, It's time point The sampled action feature value, For the intermediate time points, linearly interpolate the action features so that their timestamps match the timestamps of the spatial features; The recursive neural network extracts motion features from the action time series data, and the convolutional neural network extracts spatial features from the image feature data. The model formula after the motion features and spatial features are fused is: , Where: is the model output after the fusion of spatial features and motion features. is the weight matrix, is the bias term, It is the feature after the fusion of spatial features and motion features. The feature formula for the fusion of spatial features and motion features is: , Where: is the spatial feature of the output of the first image frame, is the spatial feature of the output of the first image frame, For the The spatial features of the output of image frames, for The set of spatial features output by each image frame, is the action feature of the first time step, is the action feature of the second time step, For the The action features of time steps, for A collection of action features at each time step.
[0014] The present invention collects image sequence data of workers connecting the laptop hard drive data cable through factory visual sensors. The YOLOv5 algorithm is used for target detection to capture the action of whether the worker correctly connects the laptop hard drive data cable. The image sequence data has the characteristics of time series length and can show the entire wearing process in chronological order. The frame rate is a key parameter affecting image quality. It is affected by factors such as lighting conditions, equipment occlusion and weather. The sensor installed on the worker's hand collects the action data of connecting the laptop hard drive data cable. The sensor is embedded in the glove and uses accelerometers, gyroscopes and geomagnetic sensors to capture the linear motion, angular velocity and absolute orientation information of the hand, thereby realizing accurate monitoring of complex hand movements. The hand movement features are extracted through a recursive neural network. The input data includes three-axis acceleration, three-axis angular velocity and orientation information. The recursive neural network uses its cyclic structure to capture the dependency between time steps, extracts the action features of the worker when connecting the laptop hard drive data cable, and uses a convolutional neural network to extract the hand movement features. The network extracts spatial features in the image. The model extracts spatial features by processing each frame of data in the image sequence for further fusion. Due to the different sampling rates of the motion features of the time series data and the spatial features of the image sequence data, linear interpolation is required for time alignment. After interpolation, the motion features and spatial features maintain consistent timestamps. The convolutional neural network extracts spatial features from the image feature data, and the recursive neural network extracts the motion features given in the motion timing data. The spatial features and motion features are fused, and the multimodal sensor data fusion model is fused to comprehensively utilize different types of data to improve the accuracy of detecting whether workers connect the laptop hard drive data cable according to specifications.
[0015] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0016] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for industrial data fusion analysis, characterized in that: The steps include: Step S1, collecting image feature data of a worker connecting a laptop hard drive data cable through a visual sensor; Step S2, collecting action timing data of workers connecting to the laptop hard drive data cable through wearable sensors; Step S3, fusing the action time series data and the image feature data to build a multimodal sensor data fusion model; extracting action features from the action time series data through a recursive neural network, and extracting spatial features from the image feature data through a convolutional neural network. The formula of the multimodal sensor data fusion model is: ， Where: is the model output after the fusion of spatial features and motion features. is the weight matrix, is the bias term, It is the feature after the fusion of spatial features and motion features. The feature formula for the fusion of spatial features and motion features is: ， Where: is the spatial feature of the output of the first image frame, is the spatial feature of the output of the second image frame, For the The spatial features of the output of image frames, for The set of spatial features output by each image frame, is the action feature of the first time step, is the action feature of the second time step, For the The action features of time steps, for A collection of action features at time steps; Step S4: The multimodal sensor data fusion model issues a safety warning to the worker when connecting the laptop hard drive data cable.
2. The method for industrial data fusion analysis according to claim 1 is characterized in that ,In step S1, the image sequence data of workers connecting the ,hard disk data cable of their laptops collected by the visual sensor includes the ,following specific contents: The camera obtains the video data of workers connecting the laptop hard drive data cable. The visual sensor processes the video data into multiple frames of image feature data of workers connecting the laptop hard drive data cable. The image feature data formula is: , Where: is the image feature data, is the set of real numbers, is the length of the image sequence, is the image channel, is the width of the image, The height of the image.
3. The method for industrial data fusion analysis according to claim 1 is characterized in that ,In step S2, the action timing data of workers connecting to the laptop hard disk data cable are collected through wearable sensors, including the following specific contents: The wearable sensor is used to collect the action timing data of workers when they connect the laptop hard drive data cable. The wearable sensor has a built-in accelerometer, gyroscope and geomagnetic sensor. The formula for obtaining the action timing data is: ， Where: is the action timing data, is the weight parameter used to adjust the acceleration, is the weight parameter used to adjust the angular velocity, is a nonlinear combination, The output of the accelerometer The acceleration values of the three axes, Output of the gyroscope The angular velocity values of the three axes.
4. The method for industrial data fusion analysis according to claim 1, characterized in that: In step S3, the action time series data and the image feature data are fused to construct a multimodal sensor data fusion model, including the following specific contents: The recursive neural network model extracts motion features from motion time series data, and the convolutional neural network model extracts spatial features from image feature data. The sampling rates of spatial features and motion features are unified through linear interpolation to construct a multimodal sensor data model.
5. The method for industrial data fusion analysis according to claim 1, characterized in that: In step S4, the multimodal sensor data fusion model security warning includes the following specific contents: The standard image feature data of the worker connecting the laptop hard drive data cable according to the specification is obtained through the visual sensor, and the standard action time series data of the worker connecting the laptop hard drive data cable according to the specification is obtained through the wearable sensor. The standard spatial features of the image feature data are extracted through the convolutional neural network model, and the standard action features of the action time series data are extracted by the recursive neural network. The standard spatial features and the standard action features are input into the multimodal sensor data model of the worker connecting the laptop hard drive data cable according to the specification to obtain the output of the multimodal sensor data model of the worker connecting the laptop hard drive data cable according to the specification. If the real-time obtained multimodal sensor data model output is outside the preset allowable deviation value range of the multimodal sensor data model output of the worker connecting the laptop hard drive data cable according to the specification, a safety warning is triggered.
6. The method for industrial data fusion analysis according to claim 4, characterized in that: The action features extracted by the recurrent neural network model for action time series data include the following specific contents: After the action time series data is formatted, it is input into the recursive neural network model. The action time series data is processed through the recursive neural network loop structure to extract action features. The formula for updating the hidden state of the recursive neural network at each time step is: ， Where: For the The hidden state at the time step, For the The input data of the time step, is the hidden state of the previous time step, is the input weight matrix, is the hidden state weight matrix, is the bias term, As activation function, nonlinear features are added, and the recurrent neural network extracts The action characteristics of the maintenance worker in the time step are: .
7. The method for industrial data fusion analysis according to claim 4, characterized in that: The convolutional neural network model extracts the spatial features of image sequence data, including the following specific contents: For each frame in the image sequence , is the number of image frames, and the spatial feature formula extracted by the convolutional neural network is: ， Where: For the The spatial characteristics of the frame image output, is the index in the height direction of the feature map, is the index in the width direction of the feature map, is the channel index in the feature map, which is used to distinguish the feature maps output by different convolution kernels. is the position index of the convolution kernel in the height direction, is the position index of the convolution kernel in the width direction, is the weight of the convolution kernel, is the offset, is the activation function, For the input image Frame at position No. The pixel value of each channel.