Labor scene multi-modal capturing method and system based on edge computing

Abstract

This invention discloses a method and system for multimodal capture of labor scenarios based on edge computing, belonging to the field of data processing networks and distributed computing technology. It includes acquiring network link quality parameters and node distribution density parameters and performing bandwidth prediction, generating an available transmission capacity assessment value, adjusting the acquisition frequency and data compression ratio, generating an adapted data stream, processing it, generating a multimodal fusion feature vector, performing classification and detection, generating real-time status labels for the labor scenario, making decisions based on the available transmission capacity assessment value, generating a data upload decision command and performing conditional judgments, and sending the matching results to a cloud server. This invention employs a collaborative processing mechanism combining network state-aware driven data source adaptive adjustment and scenario semantic-driven upload decision, enabling the filtering and transmission of labor scenario information in dynamic network environments, improving the response capability to critical events and the efficiency of network resource utilization.

Description

Technical Field

[0001] This invention relates to the fields of data processing networks and distributed computing technology, and in particular to a method and system for capturing multimodal labor scenarios based on edge computing. Background Technology

[0002] Currently, edge computing-based multimodal capture of work scenarios refers to a technical architecture that deploys various types of sensors, such as cameras, microphones, and inertial measurement units, on edge computing nodes close to the work site to collect multimodal data, including video, audio, and physical motion. The edge nodes then perform preliminary data processing, and the results are uploaded to a cloud server for unified storage, analysis, and presentation. This architecture is widely used in smart construction sites, smart factories, and other scenarios to achieve real-time monitoring, behavior analysis, and safety early warning of the work process.

[0003] In related technologies, Chinese invention patent CN116866253B discloses a remote switching system for network links based on edge computing, comprising: a data acquisition module for acquiring network link information data; a data processing module for preprocessing the acquired network link information data to obtain edge node data; a calculation and analysis module for performing security status analysis and calculation of the network link; and a remote switching module for adaptive remote switching of the network link based on the analysis and calculation results.

[0004] However, the aforementioned technical solutions suffer from passive data allocation and limited analytical dimensions. They can only passively switch links, but cannot proactively control the amount of data collected to adapt to network changes, which can easily lead to congestion. Furthermore, their analysis is limited to network link data, completely ignoring business content such as action categories and the semantics of abnormal events. Therefore, they cannot identify key information and prioritize transmission accordingly, lacking truly intelligent resource allocation capabilities. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a multimodal capture method and system for labor scenarios based on edge computing. It employs a collaborative processing mechanism that combines network state-aware data source adaptive adjustment with scenario semantic-driven upload decision-making. This mechanism enables intelligent filtering and efficient transmission of labor scenario information in dynamic network environments, improving the system's responsiveness to critical events and the efficiency of network resource utilization.

[0006] The above objectives can be achieved through the following approach:

[0007] A multimodal capture method and system for labor scenarios based on edge computing includes: acquiring the current network link quality parameters and node distribution density parameters of edge nodes at the labor site, and extracting features to generate a network state feature vector; performing adaptive bandwidth prediction based on the network state feature vector to generate an available transmission capacity assessment value; dynamically adjusting the acquisition frequency and data compression ratio of multimodal sensors based on the available transmission capacity assessment value to generate an adapted data stream; performing cross-attention fusion processing on the adapted data stream to extract video, audio, and sensor features, aggregating them after time alignment and semantic enhancement to generate a multimodal fusion feature vector; performing behavior classification and outlier detection based on the multimodal fusion feature vector, jointly determining the labor action category and anomaly probability to generate a real-time status label for the labor scenario; making an upload decision based on the real-time status label of the labor scenario and the available transmission capacity assessment value to generate a data upload decision instruction; and sending the multimodal fusion feature vector and the real-time status label of the labor scenario that meet preset conditions to a cloud server according to the data upload decision instruction.

[0008] Optionally, generating the network state feature vector includes: obtaining real-time signal quality parameters of the satellite communication link and the ground ad hoc network link, using them as network link quality parameters and performing feature extraction to generate a first network parameter; obtaining spatial distribution density parameters of edge nodes as node distribution density parameters, and combining them with current load state parameters to perform feature extraction to generate a second network parameter; weighting and fusing the first network parameter and the second network parameter and performing feature dimensionality reduction to generate a network state feature vector.

[0009] Optionally, generating the available transmission capacity assessment value includes: performing time-series feature modeling based on the network state feature vector to generate a future bandwidth trend vector; obtaining the current actual remaining bandwidth value, and correcting the future bandwidth trend vector to generate the available transmission capacity assessment value.

[0010] Optionally, generating the future bandwidth trend vector includes: obtaining network state feature vectors from consecutive moments prior to the current moment, performing differential operations to generate a bandwidth change rate sequence; determining the fluctuation stage of the network state based on the frequency and extreme values ​​of sign changes in the bandwidth change rate sequence, and generating a fluctuation stage identifier; and weighting and superimposing the fluctuation stage identifier with the network state feature vectors to generate the future bandwidth trend vector.

[0011] Optionally, generating the adapted data stream includes: calculating the allowable acquisition frequency that meets the transmission constraints based on the available transmission capacity assessment value, and generating an acquisition frequency control command; driving video, audio, and sensor synchronous acquisition according to the acquisition frequency control command to obtain synchronous multimodal raw data; and performing adaptive compression encoding on the synchronous multimodal raw data to generate the adapted data stream.

[0012] Optionally, generating the multimodal fusion feature vector includes: inputting the adapted data stream into the feature extraction network of the corresponding modality to generate video behavior features, audio semantic features, and sensor physical features; using a cross-attention mechanism to perform temporal alignment and semantic enhancement on the video behavior features, the audio semantic features, and the sensor physical features to generate alignment-enhanced features; and aggregating the alignment-enhanced features across modal information to generate the multimodal fusion feature vector.

[0013] Optionally, generating real-time status labels for the labor scene includes: classifying and discriminating the multimodal fusion feature vector to obtain a probability distribution of labor action categories; detecting outliers in the multimodal fusion feature vector to obtain the probability of abnormal events; and making a joint decision based on the probability distribution of labor action categories and the probability of abnormal events to generate real-time status labels for the labor scene.

[0014] Optionally, the data upload decision instruction includes: when the real-time status label of the work scene indicates an abnormal event, increasing the upload priority and relaxing the transmission capacity threshold to generate a first decision sub-instruction; when the real-time status label of the work scene indicates a regular work action, determining whether to upload only the key feature summary based on the available transmission capacity assessment value to generate a second decision sub-instruction; and combining the first upload decision sub-instruction and the second upload decision sub-instruction for priority arbitration to generate a data upload decision instruction.

[0015] Optionally, sending the multimodal fusion feature vector and the real-time status label of the labor scene that meet the conditions to the cloud server includes: according to the data upload decision instruction, performing link quality-aware encoding on the multimodal fusion feature vector and the real-time status label of the labor scene, selecting a communication link, and generating an optimized transmission data packet; and sending the optimized transmission data packet to the cloud server through the communication link.

[0016] Based on the same inventive concept, this invention also provides a multimodal capture system for labor scenarios based on edge computing. The system includes: a link awareness module, used to acquire current network link quality parameters and node distribution density parameters of edge nodes at the labor site, and perform feature extraction to generate a network state feature vector; a bandwidth prediction module, used to perform adaptive bandwidth prediction based on the network state feature vector to generate an available transmission capacity assessment value; an acquisition adaptation module, used to dynamically adjust the acquisition frequency and data compression ratio of the multimodal sensors based on the available transmission capacity assessment value to generate an adapted data stream; and a feature fusion module, used to cross-inject the adapted data stream. The system employs a multimodal fusion processing module to extract features from video, audio, and sensors. These features are then aggregated after time alignment and semantic enhancement to generate a multimodal fusion feature vector. A state discrimination module performs behavior classification and outlier detection based on the multimodal fusion feature vector, jointly determining the type of labor action and the probability of anomalies to generate a real-time status label for the labor scene. An upload decision module makes an upload decision based on the real-time status label of the labor scene and the available transmission capacity assessment value, generating a data upload decision instruction. A cloud sending module sends the multimodal fusion feature vector and the real-time status label of the labor scene, meeting preset conditions, to a cloud server according to the data upload decision instruction.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] 1. This invention achieves closed-loop adaptive control of multimodal sensor data acquisition and compression strategies by sensing network status and predicting available transmission capacity at edge nodes. This method enables the data generation rate and data stream volume to be matched in real-time with dynamically changing network capacity. This proactively reduces data volume to avoid congestion and data loss when network conditions are poor, while improving data quality to obtain more details when network conditions are good. This enhances the system's adaptability to heterogeneous and unstable network environments and optimizes bandwidth resource utilization efficiency.

[0019] 2. This invention achieves deep fusion and parsing of multimodal data streams and high-level semantic features at the edge, replacing the traditional method of directly uploading massive amounts of raw or simply encoded data to the cloud. By performing behavior classification and anomaly detection directly at the edge, high-value structured information can be extracted locally, reducing the total amount of data that needs to be transmitted over the network, thereby reducing the long-term occupation of backbone network bandwidth and the storage and computing load on cloud servers.

[0020] 3. This invention establishes an intelligent upload decision-making mechanism driven by both scene semantic importance and real-time network capacity. This mechanism can distinguish between routine events and abnormal emergency events and match different upload strategies accordingly, ensuring the highest transmission priority and lowest reporting latency for critical information such as security incidents. This content-aware differentiated service strategy guarantees that the system can prioritize the reliable transmission of the most important information under limited network resources, improving the real-time performance, reliability, and intelligence of the entire monitoring and early warning service.

[0021] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the multimodal capture method for labor scenarios based on edge computing, according to an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the spatial distribution of edge nodes according to an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of dynamic transmission link selection and quality-aware coding according to an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the structure of the edge computing-based multimodal capture system for labor scenarios according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Reference Figure 1One embodiment of the present invention proposes a multimodal capture method for labor scenarios based on edge computing. It adopts a collaborative processing mechanism that combines network state-aware data source adaptive adjustment with scenario semantic-driven upload decision-making. This enables intelligent filtering and efficient transmission of labor scenario information in a dynamic network environment, improving the system's responsiveness to critical events and the efficiency of network resource utilization.

[0029] The method described in this embodiment specifically includes:

[0030] Obtain the current network link quality parameters and node distribution density parameters of the edge nodes at the work site, and extract features to generate network state feature vectors;

[0031] Optionally, the generated network state feature vector includes:

[0032] The real-time signal quality parameters of the satellite communication link and the ground ad hoc network link are obtained and used as network link quality parameters. Feature extraction is performed to generate the first network parameters.

[0033] The spatial distribution density parameters of the edge nodes are obtained as node distribution density parameters, and combined with the current load state parameters for feature extraction to generate the second network parameters;

[0034] The first network parameters and the second network parameters are weighted and fused, and feature dimensionality reduction is performed to generate a network state feature vector.

[0035] Specifically, when generating the first network parameters, the real-time signal quality parameters of all available communication links for the current node are periodically acquired, for example, every 100 to 500 milliseconds. These parameters include indicators such as the signal-to-noise ratio of satellite communication links and the reference signal received power of ground ad hoc network links. These raw indicators are combined into a vector as the first network parameter characterizing the instantaneous communication channel status. Simultaneously, through beacon message exchange between nodes, the number of neighboring edge nodes within an effective communication radius (e.g., 50 meters) is statistically analyzed, and this number is used as the spatial distribution density parameter of the edge nodes, i.e., the node distribution density parameter. Figure 2 As shown in the diagram, each point represents an edge computing node. The current node is located at the center, and its effective communication radius is represented by a dashed circle. The six nodes within the circle are identified as neighboring nodes; this number, six, is the spatial distribution density parameter of the current node. This parameter, along with acquired current load status parameters such as CPU utilization, memory usage, and network interface packet queuing length, undergoes feature extraction to form a second network parameter that comprehensively reflects the node's own load and network topology pressure. To integrate link quality and node status into a unified feature representation, the first and second network parameters are weighted and fused. The resulting fused feature vector is then calculated. ,have:

[0036] ;

[0037] in, The fusion weight coefficient for the first network parameter is set based on prior knowledge of the communication environment, such as 0.6-0.8 in variable scenarios; The fusion weight coefficients of the second network parameters satisfy... ; The vector composed of the first network parameters originates from the instantaneous communication channel quality indicators such as signal-to-noise ratio and reference signal received power obtained periodically by the link-aware module. The vector formed by the second network parameters is derived from the feature extraction of neighbor node density and internal system load status such as CPU utilization, memory usage, and queue length. For the minimum-maximum normalization function, specifically: Where x is the original parameter value to be normalized. , These represent the minimum and maximum values ​​of the parameter within the observation period or in historical data, respectively. Since the original feature vector after fusion may have a high dimensionality and contain redundant information, principal component analysis is used to perform feature dimensionality reduction. This dimensionality reduction process aims to retain the main variance of the original information; for example, by setting a variance contribution rate threshold of 95%, the original feature vector is projected into a low-dimensional space that retains most of the effective information, ultimately generating a network state feature vector that is compact in dimension and highly expressive.

[0038] For example, the operator configures sensing to be performed every 200 milliseconds. In the current cycle, the signal-to-noise ratio (SNR) of the satellite communication link is measured to be 18 dB, and the reference signal received power of the ground ad hoc network link is -85 dBmW. These two values ​​constitute the original first network parameter vector. The node discovers six neighboring edge nodes within its 50-meter communication radius by broadcasting beacon messages and listening for responses; this is the spatial distribution density parameter. Simultaneously, the current node's CPU utilization is 50%, memory usage is 60%, and the network interface packet queue length is 30 packets. These load parameters, along with the node density parameter, are extracted as the second network parameter. For fusion, these parameters need to be normalized. Assume that, based on historical statistics, the SNR ranges from 5 to 25 dB, the reference signal received power ranges from -100 to -70 dBmW, the number of neighboring nodes ranges from 0 to 15, and the CPU utilization ranges from 10% to 90%. Normalizing the signal-to-noise ratio (SNR) using the minimum-max normalization function yields the normalized SNR value. Similarly, normalizing the remaining parameters yields a vector of normalized first network parameters, including the signal-to-noise ratio and the received power of the reference signal. A vector consisting of a second network parameter including the number of neighboring nodes, CPU utilization, memory usage, and packet queue length. Set the fusion weight coefficients for the first network parameters. The fusion weight coefficient of the second network parameter is 0.7. The value is set to 0.3, resulting in the original fused feature vector. Principal component analysis is performed on this high-dimensional original eigenvector. Setting the variance contribution rate threshold to 95% will automatically generate a transformation matrix and project the original eigenvector to a lower dimension. For example, with four samples, the transformation matrix will be a 6×3 matrix. For the above samples Multiplying by this transformation matrix yields the dimensionality-reduced network state feature vector. This involves generating the final network state feature vector. By normalizing and weighting the original network parameters from different layers and with varying properties, such as signal quality at the physical layer, node density at the topology layer, and load status at the system layer, a feature vector reflecting the current overall network state is successfully generated.

[0039] Based on the network state feature vector, adaptive bandwidth prediction is performed to generate an available transmission capacity assessment value.

[0040] Optionally, the generation of the available transmission capacity assessment value includes:

[0041] Based on the network state feature vector, time-series feature modeling is performed to generate a future bandwidth trend vector;

[0042] Obtain the current actual remaining bandwidth value, and correct the future bandwidth trend vector to generate an assessment value of available transmission capacity.

[0043] Specifically, a time window of length N is maintained, for example, N ranges from 10 to 30, continuously collecting network state feature vectors from the most recent N time steps to form a time series. This sequence is input into a pre-trained Long Short-Term Memory (LSTM) network model. This model learns the non-linear temporal dependencies between historical network state vectors and outputs a future bandwidth trend vector representing the bandwidth change trend over the next M time steps, for example, M ranges from 3 to 5. The input to the LSTM model is the time series composed of network state feature vectors from 10 to 30 consecutive time steps, and the output is the bandwidth change trend vector over the next M time steps. The model adopts a standard LSTM structure, including a forget gate, input gate, output gate, and memory unit. The training set consists of historical network state feature vectors and corresponding future actual bandwidth change rates, trained through supervised learning to minimize the error between the predicted and actual bandwidth change rates. Each element of this vector represents the expected rate of change of bandwidth at the corresponding future time step relative to the current time step. Simultaneously, the actual outbound traffic at the current time step is obtained through the network interface controller, and combined with the theoretical maximum bandwidth of the link, the current actual remaining bandwidth value is calculated. For calculating this current actual remaining bandwidth value ,have:

[0044] ;

[0045] in, This is the theoretical maximum bandwidth of the link, derived from the physical transmission limit reported by the network interface controller or link layer. The actual outbound traffic at the current moment is obtained in real-time from the transmission rate or traffic count acquired by the network interface controller. This step ensures that the prediction baseline is accurate in real time. To obtain the final available transmission capacity assessment, the future bandwidth trend vector is corrected, applying the predicted relative trend to the current absolute bandwidth value. This is used to calculate the available transmission capacity assessment. ,have:

[0046] ;

[0047] in, The predicted rate of change of bandwidth for the next time step is derived from the first element selected from the future bandwidth trend vector output by the LSTM model. For safety factors, the value is typically between 0.8 and 0.95. This formula combines the absolute value based on the current actual situation with future trends predicted by a time-series model to generate an assessment value of available transmission capacity that is both realistic and forward-looking, serving as a basis for guiding subsequent data flow adjustments.

[0048] For example, in a scenario where a Long Short-Term Memory (LSTM) network model is used for bandwidth prediction, network state feature vectors generated in the previous stage are continuously collected, and a time window of length 20 is maintained. Once the window is filled with feature vectors from 20 consecutive time points, this time-series sequence matrix, with a size of 20 times the feature vector dimension, is input as a sample into a pre-trained LSM network model using a large amount of historical data. This model, through its internal gating unit, learns and captures the nonlinear dynamic pattern of network state evolution over time and outputs a future bandwidth trend vector containing predicted values ​​for the next 5 time steps. If the vector output by the model is... Each element represents the expected rate of change of bandwidth at a corresponding future time step relative to the current time step. The first element, the predicted rate of change of bandwidth for the next time step, is selected as -0.1. Simultaneously, the node's network interface controller reports the theoretical maximum bandwidth of the current link as 100 megabits per second, and obtains the actual outbound traffic at the current moment as 60 megabits per second by monitoring the outbound traffic counter in real time. According to the formula, the current actual remaining bandwidth value... (megabits per second). To obtain the final prediction result, a safety factor of 0.9 is set. According to the formula, the usable transmission capacity assessment value can be obtained. (Mbits per second). This 32.4 megabits per second is the final generated available transmission capacity assessment value used to guide subsequent data flow adjustments. This method does not use the prediction results of the timing model in isolation, but combines the relative rate of change representing future trends with the current absolute remaining bandwidth value obtained from real-time measurements of the physical link, thereby generating the available bandwidth assessment result, providing a basis for subsequent adaptive data flow control.

[0049] Optionally, generating the future bandwidth trend vector includes:

[0050] Obtain the network state feature vectors from consecutive time points prior to the current time, perform difference operations, and generate a bandwidth change rate sequence.

[0051] Based on the frequency and extreme values ​​of sign changes in the bandwidth change rate sequence, the fluctuation stage of the network state is determined and a fluctuation stage identifier is generated.

[0052] The fluctuation stage identifier is weighted and superimposed with the network state feature vector to generate a future bandwidth trend vector.

[0053] Specifically, the network state feature vectors from the K consecutive time steps preceding the current time step are obtained, for example, K is between 10 and 20, forming a historical state sequence. A first-order difference operation is performed on this sequence, calculating the difference between the feature vectors of every two adjacent time steps, generating a bandwidth change rate sequence composed of K minus 1 difference vectors. For ease of analysis, the values ​​of the dimensions most relevant to bandwidth capacity in this sequence are focused on, or the L2 norm of the difference vectors is calculated to obtain a scalarized rate change sequence. Two key indicators in this rate change sequence are used to determine the fluctuation stage of the network state. The first indicator is the sign change frequency, i.e., the number of times the rate change value alternates between positive and negative signs. A high frequency, such as more than 5 alternations within a unit window, usually indicates that the network is in a state of oscillation or congestion recovery. The second indicator is the amplitude extreme value, i.e., the maximum absolute value of the rate change. An extreme value far exceeding the statistical average, such as exceeding three standard deviations of the statistical average, usually indicates that a sudden outage or recovery event has occurred in the network. Based on these indicators, using preset rules or a simple decision tree model, the current network dynamics are categorized into several fluctuation stages, such as stable, slightly fluctuating, continuously deteriorating, or oscillating. Examples of preset rules are as follows: if more than 80% of the values ​​in the rate change sequence are of the same sign (all positive or all negative), and the sign change frequency is ≤2, it is judged as continuously deteriorating (all negative) or continuously improving (all positive); fluctuations within ±10% of the mean are judged as stable; if the sign change frequency exceeds 5 times, and the extreme value amplitude does not exceed twice the standard deviation, it is judged as slightly fluctuating; if the sign change frequency exceeds 8 times, and there is a pattern of alternating positive and negative sudden increases and decreases, it is judged as oscillating. This fluctuation stage identifier is then weighted and superimposed with the latest network state feature vector to generate the final future bandwidth trend vector. This superposition process is not a simple mathematical addition; the calculation of this final generated future bandwidth trend vector... ,have:

[0054] ;

[0055] in, The current network state feature vector is obtained by transforming and projecting the original feature vector after fusion. This is the adjustment coefficient for the fluctuation phase, determined according to the rules: 0 when the network is in a stable phase, 0.3 when there are slight fluctuations, 0.8 when the network continues to deteriorate, and 0.5 when the network is oscillating. It is the mean vector of the historical first-order difference vector sequence; , The weights are preset, and their sum is 1. This generates a future bandwidth trend vector that contains rich information about the current network state, as well as dynamic judgments about network change trends.

[0056] For example, network state feature vectors from 15 consecutive time steps prior to the current time step are obtained, forming a historical state sequence. A first-order difference operation is performed on this sequence, that is, the feature vector from the previous time step is subtracted from the feature vector from the feature vector of the next time step, resulting in a bandwidth change rate sequence consisting of 14 difference vectors. Analyzing this sequence, we find that the proportion of negative values ​​is... The sign change occurred only once, with an extreme amplitude of -0.12. Based on historical statistics, the mean of this sequence is -0.067, the standard deviation is 0.035, and three times the standard deviation is 0.105. The extreme value of 0.12 exceeds three times the standard deviation of 0.105, indicating a sudden deterioration event. Therefore, the current network state is determined to be in a continuous deterioration phase. Consequently, a corresponding fluctuation phase identifier is generated, and the fluctuation phase adjustment coefficient is obtained from the rule table. It equals 0.8. At this point, if we obtain the latest network state feature vector... Then, the mean of the 14 historical difference vectors is calculated to obtain the mean vector of the historical first-order difference vector sequence. Let the preset weighting coefficients be set. It is 0.7. The value is 0.3. Based on the formula, the final calculated future bandwidth trend vector is obtained. The direction and magnitude of the future bandwidth trend vector not only reflect the current network state but also include a vector component pointing towards a deteriorating trend. By quantifying this trend judgment into an adjustment coefficient and weighting it with the historical average and the current state, a future bandwidth trend vector is generated that incorporates both current static snapshot information and recent dynamic trends.

[0057] Based on the available transmission capacity assessment value, the acquisition frequency and data compression ratio of the multimodal sensor are dynamically adjusted to generate an adapted data stream.

[0058] Optionally, generating the adapted data stream includes:

[0059] Calculate the allowable acquisition frequency that satisfies the transmission constraints based on the available transmission capacity assessment value, and generate acquisition frequency control instructions;

[0060] The video, audio, and sensor data are synchronously acquired according to the acquisition frequency control command to obtain synchronous multimodal raw data.

[0061] Adaptive compression encoding is performed on the synchronous multimodal raw data to generate an adapted data stream.

[0062] Specifically, following a preset strategy model, the available transmission capacity assessment value is mapped to a discrete acquisition frequency with multiple operating levels. For example, when the available transmission capacity assessment value is higher than 10 Mbps, the acquisition frequency is set to a high level, such as 25 to 30 Hz; when the assessment value is between 5 and 10 Mbps, it is set to a medium level, such as 15 Hz; and when it is lower than 5 Mbps, it is set to a low level, such as 5 Hz. This generates an acquisition frequency control command containing a specific frequency value. This acquisition frequency control command is sent to each sensor driver program. This command acts as a synchronization trigger signal, driving physical sensors such as video sensors (e.g., industrial cameras), audio sensors, and inertial measurement units to synchronously acquire data according to the frequency specified in the command. Synchronization is guaranteed by a unified hardware clock source or a high-precision network time protocol, assigning a unified timestamp to each acquired video frame, audio segment, and sensor reading, thus obtaining synchronous multimodal raw data. To achieve finer control over the data stream, adaptive compression encoding is performed on this synchronous multimodal raw data. For video data, an H.264 or H.265 encoder is invoked, and the quantization parameters in the H.264 or H.265 encoder are dynamically adjusted based on the available transmission capacity assessment. The quantization parameter values ​​typically range from 22 to 38; a lower quantization parameter value corresponds to a lower compression rate and higher image quality. For audio data, the target bitrate of the AAC or Opus encoder is adjusted. The goal of this adaptive process is to make the total bitrate after compression of all modal data approach but not exceed the available transmission capacity assessment. Its objective function can be expressed as:

[0063] ;

[0064] in, The target total bitrate is the sum of the bitrates allocated from video, audio, and sensor data streams, which must satisfy the constraint of not exceeding the available transmission capacity; Assign bitrate to the video data stream; Assigning bitrate to the audio data stream; The encoder allocates a bitrate to the sensor data stream. Based on the allocated bitrate, the encoder dynamically adjusts the compression parameters. For example, when the available transmission capacity assessment value is only 20% of the full feature volume, the encoder automatically enables principal component retention, compressing the original 256-dimensional fused feature into a 32-dimensional summary, so that the output data packet size matches the current tight bandwidth. Finally, the compressed data packets together constitute the adapted data stream, which is now matched to the current network capacity in terms of size.

[0065] For example, if the available transmission capacity assessment value generated in the previous stage is 8 megabits per second, upon receiving this assessment value, a judgment is made according to a preset strategy model. This model stipulates that when the available transmission capacity assessment value is between 5 and 10 megabits per second, the acquisition frequency should be set to a medium level. Therefore, an acquisition frequency control command is generated containing the content "Set all sensor acquisition frequencies to 15 Hz". This command is sent to the device driver layer. An industrial camera connected to the system, a microphone array, and an inertial measurement unit worn on the worker's wrist, upon receiving this command, are triggered by a unified hardware clock to begin synchronous data acquisition at a frequency of 15 times per second, and each set of data is timestamped with the same timestamp, forming synchronous multimodal raw data. This raw data is sent to the adaptive compression encoding stage, with the goal of ensuring that the total bitrate of the compressed video, audio, and sensor data does not exceed 8 megabits per second. A fixed bitrate is allocated to the smaller audio and sensor data, for example, a bitrate of 128 kilobits per second for audio and 64 kilobits per second for sensors. The remaining bitrate available for video is then... (Mbits per second). Accordingly, the target bitrate of the H.265 video encoder is dynamically adjusted to 7.8 megabits per second. The encoder automatically selects a suitable quantization parameter, such as 30, to achieve this target bitrate. The video, audio, and sensor compressed data packets output by the encoder together constitute the adapted data stream. By progressively decomposing and mapping the abstract available network bandwidth assessment value into specific executable sensor acquisition frequencies and data compression parameters, a flow control management approach from the data source is achieved.

[0066] The adapted data stream is subjected to cross-attention fusion processing to extract video, audio and sensor features, which are then aggregated after time alignment and semantic enhancement to generate a multimodal fusion feature vector.

[0067] The generation of the multimodal fusion feature vector includes:

[0068] The adapted data streams are input into the feature extraction networks of the corresponding modalities to generate video behavior features, audio semantic features, and sensor physical features.

[0069] The video behavioral features, the audio semantic features, and the sensor physical features are temporally aligned and semantically enhanced using a cross-attention mechanism to generate aligned and enhanced features.

[0070] The alignment enhancement features are aggregated across modal information to generate a multimodal fusion feature vector.

[0071] Specifically, the received adapted data stream is distributed to three parallel feature extraction networks. The video data stream is fed into a pre-trained 3D convolutional neural network, such as the I3D network, to capture the spatiotemporal dynamics of actions. The input is a continuous sequence of video frames, and the output is a video behavior feature vector. Its structure includes 3D convolutional layers and fully connected layers, and it is pre-trained on large action recognition datasets such as the Kinetics dynamics dataset to capture spatiotemporal dynamic information in labor scenes. The audio data stream is fed into an audio event recognition model based on a convolutional recurrent network structure to extract the category and pattern of sound events and generate audio semantic features. The audio event recognition model takes an audio spectrogram sequence as input and outputs audio semantic features. The model employs a Convolutional Neural Network (CNN) to extract local time-frequency patterns, combined with a Recurrent Neural Network (RNN) such as LSTM to model time dependencies. Trained on an environmental sound event dataset, it is used to identify sound categories and patterns in the workplace. Simultaneously, sensor data streams from wearable devices are processed through a one-dimensional convolutional network to extract sensor physical features reflecting limb movement posture and force. The input is a multi-channel time series from the wearable device, and the output is the sensor physical features. This network consists of multiple one-dimensional convolutional layers and is trained on a collected limb movement dataset to extract posture and force features implicit in signals such as acceleration and angular velocity. After obtaining these three independent feature sequences, a cross-attention mechanism is used to address potential temporal misalignment and semantic gaps between them. The cross-attention mechanism is an information interaction paradigm that allows a feature sequence from one modality to act as a query, actively retrieving and weighting the most relevant information from feature sequences from another modality. For example, by using video behavioral features as queries and audio semantic features and sensor physical features as keys and values, the association weights between each frame in the video and the sound and sensor signals are obtained, thereby enhancing the features of video frames accompanied by sound events or violent physical motion, and vice versa. This process is performed alternately across all modalities, outputting a set of aligned and enhanced features that have undergone mutual calibration and information complementarity. To generate the final multimodal fusion feature vector, this set of aligned and enhanced features is aggregated across modalities. This aggregation is achieved through a simple concatenation operation, and the concatenated high-dimensional long vector is input into a multilayer perceptron network for dimensionality reduction and deep fusion. The final multimodal fusion feature vector is calculated for this process. ,have:

[0072] ;

[0073] in. , , These represent the aligned and enhanced video, audio, and sensor features, respectively. This is the long vector formed by concatenating the first and last parts of the three feature vectors. This is the weight matrix of the second layer, i.e., the output layer, with dimension 1. , The dimension of the final output feature, such as 512, is derived from the dimension of the multilayer perceptron output space preset in the engineering. For the hidden layer dimension; This is the weight matrix of the first layer, i.e., the hidden layer, with dimension 1. , The dimension of the video behavior feature vector. The dimension of the audio semantic feature vector. is the dimension of the sensor's physical feature vector; The bias vector for the first layer, with dimension h, is derived from the learnable offsets provided to the hidden layer neurons in the multilayer perceptron. The bias vector for the second layer, dimension This originates from the learnable offset provided to the output layer neurons in a multilayer perceptron; This is a commonly used hyperbolic tangent function. The final output vector is a multimodal fusion feature vector that contains spatiotemporal and semantic information of the entire scene.

[0074] For example, assume the adapted data stream contains video, audio, and sensor data acquired at 15 Hz. These three data streams are distributed for processing. The video stream is fed into an I3D network, which analyzes the video frame sequence, identifies the action of "waving a welding torch," and outputs a 1024-dimensional video behavioral feature. The audio stream enters a convolutional recurrent network model, identifies the "welding arc sound," and generates a 128-dimensional audio semantic feature. Simultaneously, the sensor data stream from the wristband is processed through a one-dimensional convolutional network to extract patterns reflecting rapid arm movement and wrist rotation, generating a 64-dimensional sensor physical feature. These three feature sequences are then aligned and enhanced. For example, using video features as the query, during time alignment, the time of "welding torch raised to its highest point" in the video features, the time of "peak arc sound" in the audio features, and the time of "maximum wrist angular velocity" in the sensor features are offset by no more than 50 milliseconds on the time axis, and the Pearson correlation coefficient of the three trends exceeds 0.85, indicating extremely high correlation. Therefore, video frames of "waving the welding torch" are used as the query, and the strongly correlated audio features of "welding arc sound" and sensor features of "rapid arm movement" are assigned attention weights of 0.9 or higher, while the weights of irrelevant background noise frames are reduced to below 0.1. After multiple rounds of interactive enhancement, the aligned and enhanced feature vector is obtained. For final fusion, these three vectors are concatenated to form a 1216-dimensional long vector, i.e., the sum of 1024, 128, and 64, denoted as . This long vector is fed into a two-layer multilayer perceptron for deep fusion, with a hidden layer dimension of 256 and an output layer dimension of 512. It is a weight matrix with 256 rows and 1216 columns. It is a 256-dimensional bias vector, which is obtained by the network pre-training to map the input from a 1216-dimensional nonlinearity to a 256-dimensional hidden space; It is a weight matrix with 512 rows and 256 columns. This is a 512-dimensional bias vector, also a pre-trained parameter, used to further map the hidden representations to the final 512-dimensional fused feature space. After this series of calculations, a 512-dimensional multimodal fused feature vector is finally generated, such as... This method utilizes a cross-attention mechanism to solve the problems of temporal alignment and semantic association between multimodal data, laying the foundation for subsequent behavior recognition and anomaly detection.

[0075] Based on the multimodal fusion feature vector, behavior classification and outlier detection are performed, and the labor action category and anomaly probability are jointly determined to generate real-time status labels for the labor scene.

[0076] Optionally, the generation of real-time status tags for the labor scene includes:

[0077] The multimodal fused feature vectors are used for behavior classification to obtain the probability distribution of labor action categories;

[0078] The multimodal fused feature vector is used for outlier detection to obtain the probability of abnormal events.

[0079] A joint decision is made based on the probability distribution of the labor action categories and the probability of the occurrence of the abnormal events to generate real-time status labels for the labor scene.

[0080] Specifically, the received multimodal fused feature vector is simultaneously fed into two parallel discriminative submodules. The first submodule is the behavior classification discriminative module, which internally deploys a multilayer perceptron classifier based on a normalized exponential function. This classifier receives the multimodal fused feature vector and outputs a probability distribution of labor action categories with a dimension equal to the preset number of labor action categories, such as 50 categories. This distribution is a vector whose sum of elements is 1, representing the probability that the current scene belongs to each predefined action. The second submodule is the outlier detection module, whose core is an autoencoder network pre-trained on a large amount of regular labor action data. When the multimodal fused feature vector is input into this network, the network attempts to reconstruct it. The reconstruction loss is quantified by calculating the mean squared error between the input vector and the reconstructed output vector. A higher reconstruction loss means that the current feature vector is dissimilar to all known regular action patterns. This loss value is converted into the probability of anomaly occurrence using the following formula, such as for calculating the probability of anomaly occurrence. ,have:

[0081] ;

[0082] in, The gain coefficient, which controls the steepness of the probability curve, is usually between 5 and 10 and is derived from the empirical adjustment parameter of the sensitivity to the anomaly probability response in the experiment. The real-time reconstruction error originates from the mean square error calculated by the autoencoder network after reconstructing the input multimodal fusion feature vector. The baseline threshold for normal error is derived from the error boundary value used to distinguish between normal and abnormal samples, statistically derived from the validation dataset. A joint decision is made based on the probability distribution of labor action categories and the probability of abnormal events. This decision logic follows a preset priority rule: when the probability of an abnormal event exceeds a high threshold, such as 0.8, the result of behavior classification is ignored, and the real-time status label of the labor scene is directly judged as an "abnormal event"; if... If the value is below this threshold, the probability distribution of labor action categories will be examined. When the maximum probability value exceeds a confidence threshold, such as 0.9, the label will be determined as the corresponding labor action category, such as "routine welding". If neither condition is met, the label will be set to "state uncertain" to indicate that the current scene is in a fuzzy or transitional state, thereby generating the final real-time state label of the labor scene.

[0083] For example, suppose a 512-dimensional multimodal fusion feature vector was generated in the previous stage. This vector is simultaneously sent to two sub-modules. In the behavior classification and discrimination module, a multilayer perceptron classifier processes this vector and outputs a probability distribution vector containing 50 predefined labor action categories. Suppose that in the output, the probability of the "routine welding" category is 0.92, the highest among all categories, and exceeds the confidence threshold of 0.9. At the same time, this multimodal fusion feature vector is also input into the autoencoder network of the outlier detection module. Since "routine welding" is a very typical normal action, the autoencoder can reconstruct the input vector well, with a real-time reconstruction error of only 0.08. If the baseline threshold for normal error calculated based on historical normal data is 0.25, the current error is far below this threshold. Using gain coefficients... The value is 8, which is used to calculate the probability of an abnormal event occurring. This anomaly probability is far below the high threshold of 0.8. The joint decision-making logic judges the situation; since the probability of the anomaly event (0.187) is very low, it adopts the behavior classification result, namely "routine welding" corresponding to the maximum probability value of 0.92, and outputs this as the final real-time status label for the work scenario. By executing classification and anomaly detection in parallel, it can not only identify known routine actions within predefined categories, but also, through outlier detection, quantitatively assess and alert on unseen, unclassifiable, but potentially dangerous or erroneous anomaly patterns.

[0084] Based on the real-time status label of the work scenario and the available transmission capacity assessment value, an upload decision is made, and a data upload decision instruction is generated.

[0085] Optionally, the data upload decision instruction includes:

[0086] When the real-time status label of the labor scene indicates the presence of an abnormal event, the upload priority is increased and the transmission capacity threshold is relaxed to generate a first decision sub-instruction.

[0087] When the real-time status label of the labor scene indicates a routine labor action, determine whether to upload only the key feature summary based on the available transmission capacity assessment value, and generate a second decision sub-instruction.

[0088] The first upload decision sub-instruction and the second upload decision sub-instruction are combined for priority arbitration to generate a data upload decision instruction.

[0089] Specifically, the core decision-making logic is based on a two-branch conditional judgment structure. When the real-time status label of the work scenario clearly indicates the existence of an abnormal event, the generation of the first decision sub-instruction is triggered. This instruction marks the upload priority of the current data packet as the highest level, while relaxing the network bandwidth requirements when making transmission decisions. Specifically, it temporarily ignores or uses a very low, close to zero, transmission capacity threshold, which means that as long as the network link is available, regardless of how low the available transmission capacity assessment value is, an upload must be attempted. This ensures that abnormal data that is crucial to safe production can reach the cloud server for alarms and in-depth analysis as quickly as possible. Therefore, the first decision sub-instruction includes the identifiers of "upload immediately" and "complete features". At the same time, or in non-abnormal situations, the generation logic of the second decision sub-instruction is executed. At this time, the real-time status label of the work scenario indicates a normal work action, and the size of the complete multimodal fusion feature vector to be uploaded is compared with the provided available transmission capacity assessment value. If the size of the complete multimodal fusion feature vector is not greater than the available transmission capacity assessment value, it indicates that the current network is sufficient to carry the complete feature, and the instruction will specify uploading the complete multimodal fusion feature vector. Conversely, if network capacity is insufficient, the instruction will specify uploading only a key feature summary. This key feature summary is a highly condensed version obtained by filtering or reducing the dimensionality of the complete multimodal fusion feature vector. For example, it may be generated through principal component retention, vector quantization, or by selecting only specific modal features. Its data volume is typically only 10% to 30% of the complete vector. The generated first and second decision sub-instructions are then subject to priority arbitration. This arbitration mechanism grants the first decision sub-instruction absolute priority; that is, if an abnormal event is determined, the first decision sub-instruction is unconditionally adopted. Only in the case of a non-abnormal event is the decision result of the second decision sub-instruction adopted. Through this arbitration, a unique data upload decision instruction is finally generated, containing the upload priority, the data type to be uploaded (e.g., complete vector or summary), and a pointer to the target data packet, to guide subsequent cloud transmission operations.

[0090] For example, if two different real-time status tags and available transmission capacity assessment values ​​for different work scenarios are generated at two different times. Case one: In... At any given moment, the real-time status label of the work scene is determined to be an "abnormal event," such as a worker slipping. At this time, regardless of the available transmission capacity assessment value, a first decision sub-instruction will be immediately generated. This instruction states, "Set upload priority to highest, ignore bandwidth limitations, and immediately upload the complete multimodal fusion feature vector and associated original data fragments." Since the first decision sub-instruction has the highest arbitration authority, the final generated data upload decision instruction is this instruction. Scenario Two, in... At a given moment, the real-time status label of the work scene is determined to be "routine work action," such as "parts grinding," and the available transmission capacity is assessed at 1.5 megabits per second. At this point, the generation logic of the second decision sub-instruction is initiated. The size of the complete multimodal fusion feature vector to be uploaded is obtained, assumed to be 2.5 megabytes, or 20 megabits. Since 20 megabits is far greater than the data volume that 1.5 megabits per second can carry, the network capacity is deemed insufficient. Therefore, the content of the second decision sub-instruction is determined to be "set the upload priority to normal, and only upload the key feature summary." This key feature summary is generated by principal component retention of the complete vector, and the data volume is only 15% of the original, i.e., 0.375 megabytes, or 3 megabits, which can be uploaded within a few seconds. In priority arbitration, because... If the event is not abnormal, the first decision sub-instruction is not triggered, and the second decision sub-instruction is adopted. The final data upload decision instruction is "normal priority, upload key feature summary". By establishing a dual decision matrix based on event criticality and network real-time performance, the intelligence and differentiation of data upload are enhanced.

[0091] According to the data upload decision instruction, the multimodal fusion feature vector that meets the preset conditions and the real-time status label of the labor scene are sent to the cloud server.

[0092] Optionally, sending the multimodal fusion feature vector that meets the conditions and the real-time status label of the labor scene to the cloud server includes:

[0093] According to the data upload decision instruction, the multimodal fusion feature vector and the real-time status label of the labor scene are encoded with link quality awareness, and a communication link is selected to generate an optimized transmission data packet;

[0094] The optimized data packet is sent to the cloud server through the communication link.

[0095] Specifically, once a data upload decision instruction generated by the upload decision module is received, the system encapsulates the multimodal fusion feature vector or its summary, along with the real-time status label of the work scenario, according to the instruction. Next, link quality-aware coding is executed. This coding is not data compression, but a forward error correction channel coding technique with dynamically adjustable strength. The system continuously queries the signal quality parameters of available communication links obtained by the link awareness module, such as the bit error rate or signal-to-noise ratio, for satellite communication links and terrestrial ad hoc network links. Based on these parameters, a forward error correction code rate is dynamically selected. For example, when the link quality is poor, a lower code rate, such as half, is used to increase data redundancy to combat transmission errors; when the link quality is good, a higher code rate, such as seven-eighths, is used to save bandwidth. Simultaneously, communication links are selected based on the priority in the data upload decision instruction and the real-time status of each link. For example, when the instruction indicates a high-priority abnormal event, the link with the lowest current latency and the most stable connection is selected first, even if that link is a high-cost satellite link. For low-priority data from routine tasks, the most cost-effective terrestrial ad hoc network link will be selected first, provided its quality meets basic transmission requirements. After encoding and link selection, the encoded data payload, along with necessary metadata such as timestamps and device IDs, is assembled into an optimized transmission data packet. A Differential Service Code Point (DSCP) value is set in the IP header to ensure it receives forwarding treatment commensurate with its data priority during network transmission. This optimized transmission data packet is then sent to the preset cloud server address via the physical network interface of the selected communication link. Figure 3 As shown in the figure, the real-time quality parameters of two available communication links are compared: the satellite communication link has high latency but low packet loss rate, i.e., the link is stable, while the terrestrial ad hoc network link has low latency but high packet loss rate, i.e., the link is unstable. The link will be selected according to the priority in the data upload decision command.

[0096] For example, suppose that based on the "highest priority, upload complete features" data upload decision instruction generated by the upload decision module, an optimized transmission data packet related to an "abnormal event" is prepared to be sent. The complete multimodal fusion feature vector pointed to by the instruction and the real-time status label of the "abnormal event" work scenario are encapsulated. The process then proceeds to the link quality-aware encoding and link selection stage. The link awareness module is queried and finds two available links: the satellite communication link has a high signal-to-noise ratio and is stable, but has a latency of 600 milliseconds; the terrestrial ad hoc network link has a latency of only 50 milliseconds, but its current packet loss rate is as high as 10%, indicating poor signal quality. Based on the "highest priority" in the data upload decision instruction, the link selection strategy prioritizes the reliability and timeliness of data transmission. Although the satellite link has high latency, its stability is more important at this moment, so the satellite communication link is selected. Simultaneously, due to the good quality of the selected link, the link quality-aware encoding module decides to use a high-rate forward error correction code, such as 7 / 8 LDPC code, to reduce redundant data and save bandwidth. The LDPC-encoded data payload, along with metadata such as timestamps and device IDs, is assembled into an optimized transmission data packet. To ensure priority processing of this data packet during network transmission, its IP header has the Differential Service Code Point (DSCP) value set to "EF" (Expedited Forwarding), the highest priority guaranteed forwarding service. This optimized transmission data packet, granted the highest network privileges, is sent via the physical layer satellite communication interface to the pre-defined IP address and port of the cloud server. Through real-time link awareness, the transmission path and channel coding strategy are dynamically selected, enhancing the adaptability of transmission behavior to the network environment.

[0097] Based on the same inventive concept, such as Figure 4 As shown, the present invention also provides a multimodal capture system for labor scenarios based on edge computing, the system comprising:

[0098] The link awareness module is used to obtain the current network link quality parameters and node distribution density parameters of edge nodes at the work site, and to extract features to generate network state feature vectors.

[0099] The bandwidth prediction module is used to perform adaptive bandwidth prediction based on the network state feature vector and generate an available transmission capacity assessment value.

[0100] The acquisition and adaptation module is used to dynamically adjust the acquisition frequency and data compression ratio of the multimodal sensor according to the available transmission capacity assessment value, and generate an adapted data stream.

[0101] The feature fusion module is used to perform cross-attention fusion processing on the adapted data stream, extract video, audio and sensor features, and aggregate them after time alignment and semantic enhancement to generate a multimodal fusion feature vector;

[0102] The state discrimination module is used to classify behaviors and detect outliers based on the multimodal fusion feature vector, jointly discriminate labor action categories and anomaly probabilities, and generate real-time state labels for labor scenarios.

[0103] The upload decision module is used to make upload decisions based on the real-time status tag of the labor scenario and the available transmission capacity assessment value, and generate data upload decision instructions.

[0104] The cloud sending module is used to send the multimodal fusion feature vector that meets the preset conditions and the real-time status label of the labor scene to the cloud server according to the data upload decision instruction.

[0105] It should be noted that the electrical connections between the various units described above do not necessarily represent direct or indirect connections. Any indirect connection method can be applied to the embodiments of the present invention as long as it achieves the purpose of the present invention. The above descriptions are merely exemplary embodiments of the present invention and should not be construed as limiting the scope of the present invention.

[0106] All equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. A multimodal capture method for labor scenarios based on edge computing, characterized in that, The method includes: Obtain the current network link quality parameters and node distribution density parameters of the edge nodes at the work site, and extract features to generate network state feature vectors; Based on the network state feature vector, adaptive bandwidth prediction is performed to generate an available transmission capacity assessment value. Based on the available transmission capacity assessment value, the acquisition frequency and data compression ratio of the multimodal sensor are dynamically adjusted to generate an adapted data stream. The adapted data stream is subjected to cross-attention fusion processing to extract video, audio and sensor features, which are then aggregated after time alignment and semantic enhancement to generate a multimodal fusion feature vector. Based on the multimodal fusion feature vector, behavior classification and outlier detection are performed, and the labor action category and anomaly probability are jointly determined to generate real-time status labels for the labor scene. Based on the real-time status label of the work scenario and the available transmission capacity assessment value, an upload decision is made, and a data upload decision instruction is generated. According to the data upload decision instruction, the multimodal fusion feature vector that meets the preset conditions and the real-time status label of the labor scene are sent to the cloud server.

2. The multimodal capture method for labor scenarios based on edge computing according to claim 1, characterized in that, The generated network state feature vector includes: The real-time signal quality parameters of the satellite communication link and the ground ad hoc network link are obtained and used as network link quality parameters. Feature extraction is performed to generate the first network parameters. The spatial distribution density parameters of the edge nodes are obtained as node distribution density parameters, and combined with the current load state parameters for feature extraction to generate the second network parameters; The first network parameters and the second network parameters are weighted and fused, and feature dimensionality reduction is performed to generate a network state feature vector.

3. The multimodal capture method for labor scenarios based on edge computing according to claim 1, characterized in that, The generated available transmission capacity assessment value includes: Based on the network state feature vector, time-series feature modeling is performed to generate a future bandwidth trend vector; Obtain the current actual remaining bandwidth value, and correct the future bandwidth trend vector to generate an assessment value of available transmission capacity.

4. The multimodal capture method for labor scenarios based on edge computing according to claim 3, characterized in that, The generation of the future bandwidth trend vector includes: Obtain the network state feature vectors from consecutive time points prior to the current time, perform difference operations, and generate a bandwidth change rate sequence. Based on the frequency and extreme values ​​of sign changes in the bandwidth change rate sequence, the fluctuation stage of the network state is determined and a fluctuation stage identifier is generated. The fluctuation stage identifier is weighted and superimposed with the network state feature vector to generate a future bandwidth trend vector.

5. The multimodal capture method for labor scenarios based on edge computing according to claim 1, characterized in that, The generated adapted data stream includes: Calculate the allowable acquisition frequency that satisfies the transmission constraints based on the available transmission capacity assessment value, and generate acquisition frequency control instructions; The video, audio, and sensor data are synchronously acquired according to the acquisition frequency control command to obtain synchronous multimodal raw data. Adaptive compression encoding is performed on the synchronous multimodal raw data to generate an adapted data stream.

6. The multimodal capture method for labor scenarios based on edge computing according to claim 1, characterized in that, The generation of the multimodal fusion feature vector includes: The adapted data streams are input into the feature extraction networks of the corresponding modalities to generate video behavior features, audio semantic features, and sensor physical features. The video behavioral features, the audio semantic features, and the sensor physical features are temporally aligned and semantically enhanced using a cross-attention mechanism to generate aligned and enhanced features. The alignment enhancement features are aggregated across modal information to generate a multimodal fusion feature vector.

7. The multimodal capture method for labor scenarios based on edge computing according to claim 1, characterized in that, The generated real-time status tags for the labor scene include: The multimodal fused feature vectors are used for behavior classification to obtain the probability distribution of labor action categories; Outlier detection is performed on the multimodal fused feature vector to obtain the probability of abnormal events occurring; A joint decision is made based on the probability distribution of the labor action categories and the probability of the occurrence of the abnormal events to generate real-time status labels for the labor scene.

8. The multimodal capture method for labor scenarios based on edge computing according to claim 1, characterized in that, The generated data upload decision instruction includes: When the real-time status label of the labor scene indicates the presence of an abnormal event, the upload priority is increased and the transmission capacity threshold is relaxed to generate a first decision sub-instruction. When the real-time status label of the labor scene indicates a routine labor action, determine whether to upload only the key feature summary based on the available transmission capacity assessment value, and generate a second decision sub-instruction. The first upload decision sub-instruction and the second upload decision sub-instruction are combined for priority arbitration to generate a data upload decision instruction.

9. The multimodal capture method for labor scenarios based on edge computing according to claim 1, characterized in that, Sending the multimodal fusion feature vector and the real-time status label of the labor scene that meet the conditions to the cloud server includes: According to the data upload decision instruction, the multimodal fusion feature vector and the real-time status label of the labor scene are encoded with link quality awareness, and a communication link is selected to generate an optimized transmission data packet; The optimized data packet is sent to the cloud server through the communication link.

10. A multimodal capture system for labor scenarios based on edge computing, applied to the multimodal capture method for labor scenarios based on edge computing as described in any one of claims 1-9, characterized in that, The system includes: The link awareness module is used to obtain the current network link quality parameters and node distribution density parameters of the edge nodes at the work site, and to extract features to generate network state feature vectors. The bandwidth prediction module is used to perform adaptive bandwidth prediction based on the network state feature vector and generate an available transmission capacity assessment value. The acquisition and adaptation module is used to dynamically adjust the acquisition frequency and data compression ratio of the multimodal sensor according to the available transmission capacity assessment value, and generate an adapted data stream. The feature fusion module is used to perform cross-attention fusion processing on the adapted data stream, extract video, audio and sensor features, and aggregate them after time alignment and semantic enhancement to generate a multimodal fusion feature vector; The state discrimination module is used to classify behaviors and detect outliers based on the multimodal fusion feature vector, jointly discriminate labor action categories and anomaly probabilities, and generate real-time state labels for labor scenarios. The upload decision module is used to make upload decisions based on the real-time status tag of the labor scenario and the available transmission capacity assessment value, and generate data upload decision instructions. The cloud sending module is used to send the multimodal fusion feature vector that meets the preset conditions and the real-time status label of the labor scene to the cloud server according to the data upload decision instruction.

Images