A multimodal collaborative industrial voice control method

Abstract

The application provides a kind of multimodal synergistic industrial voice control method.Application to intelligent manufacturing technology field, the method includes collecting industrial field original voice stream, generates noise spectrum entropy value by frequency domain feature analysis, according to the noise interference level determined by dynamic matching array beam forming weight and adaptive filter coefficient, and gets pure voice instruction by noise reduction;Extract its voiceprint features and semantic text, generate instruction semantic structure tree combined with industrial knowledge graph, calculate identity matching degree and semantic analysis confidence respectively, and get instruction comprehensive effectiveness coefficient by weighting;Satisfy the benchmark time to map it to sub-instruction, evaluate process conflict and equipment risk through digital twin pre-play, determine execution strategy;Real-time acquisition of running data in execution, whether to trigger abnormal rollback is determined according to trajectory deviation rate.This improves the efficiency of industrial voice control.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, and in particular to a multimodal collaborative industrial voice control method. Background Technology

[0002] With the rapid development of intelligent manufacturing, voice interaction, as a core method of non-contact human-machine interaction, has been widely applied in equipment control and production scheduling scenarios in industrial settings, becoming an important technical means to improve the convenience of industrial operations. However, existing industrial voice control technologies still have significant technical shortcomings in their application in complex industrial scenarios.

[0003] Industrial environments are typically characterized by various types of dynamic interference noise, including equipment operation, mechanical shock, and fluid noise. Existing speech preprocessing solutions often employ fixed-parameter noise reduction models, failing to dynamically adapt beamforming and filtering strategies to the real-time noise environment. This results in insufficient accuracy in extracting clean speech commands, directly impacting the accuracy of subsequent command recognition. Furthermore, current technologies often focus solely on the semantic parsing of speech commands, lacking a collaborative mechanism for verifying voiceprint legitimacy and semantic parsing confidence. This makes them prone to issues such as illegal operations and command misinterpretations, failing to balance the security and accuracy of command execution.

[0004] In addition, the existing solution lacks the ability to perform virtual pre-execution verification before instruction execution, full-process risk control and real-time anomaly handling during execution, and cannot identify process logic conflicts and equipment safety risks in advance, which can easily lead to production failures and safety accidents, significantly increase on-site operation and maintenance costs, and result in low overall operating efficiency of voice control in industrial sites. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a multimodal collaborative industrial voice control method. The method includes: The system collects raw speech streams from an industrial environment and performs frequency domain feature analysis on the raw speech streams to generate noise spectrum entropy values. Based on the noise spectrum entropy values, it determines the current environmental noise interference level. According to the environmental noise interference level, it dynamically matches array beamforming weights and adaptive filtering coefficients to perform noise reduction processing on the raw speech streams to generate clean speech commands. Based on the clean speech commands, it extracts the corresponding voiceprint feature vectors and semantic text information. Based on the voiceprint feature vectors, it calculates the operator's identity matching degree. Based on the semantic text information and a pre-constructed industrial knowledge graph, it generates a command semantic structure tree. Based on the completeness of each node in the instruction semantic structure tree, the semantic parsing confidence is calculated; based on the weighted calculation result of the operator identity matching degree and the semantic parsing confidence, the instruction comprehensive validity coefficient is determined; when the instruction comprehensive validity coefficient is determined to meet the preset execution benchmark, the instruction semantic structure tree is mapped to a sequence of sub-instructions to be executed; the sequence of sub-instructions to be executed is input into the digital twin system for virtual pre-playing, and the equipment status response curve during the pre-playing process is obtained; based on the equipment status response curve, the process logic conflict index and the equipment safety risk value are calculated respectively. Based on the process logic conflict index and the equipment safety risk value, a multimodal collaborative interaction strategy is determined. If the multimodal collaborative interaction strategy is determined to be a direct execution mode, a control command is issued to the equipment. If the multimodal collaborative interaction strategy is determined to be an interactive confirmation mode, a combined verification signal of visual highlighting and voice inquiry is generated. If the multimodal collaborative interaction strategy is determined to be a circuit breaker blocking mode, an equipment safety lock operation is performed, and the cause of the abnormality is fed back to the operator. During the execution of control commands on the device, real-time operating data is collected; the execution trajectory deviation rate is calculated based on the operating data; and the abnormal rollback process is determined based on the execution trajectory deviation rate.

[0006] Further, the step of determining the current environmental noise interference level based on the noise spectrum entropy value, and dynamically matching array beamforming weights according to the environmental noise interference level, includes: The noise spectrum entropy value is compared with the preset steady-state noise threshold and sudden interference threshold. If the noise spectrum entropy value is less than the steady-state noise threshold, the current environmental noise interference level is determined to be low interference level, and the preset omnidirectional acquisition weight template is called as the array beamforming weight. If the noise spectrum entropy value is greater than or equal to the steady-state noise threshold and less than the sudden interference threshold, then the current environmental noise interference level is determined to be medium interference level, and the directional enhancement weight is calculated based on the sound source localization result, and the directional enhancement weight is used as the array beamforming weight. If the noise spectrum entropy value is greater than or equal to the sudden interference threshold, the current environmental noise interference level is determined to be high interference level, and a narrowband filter weight is generated based on the main frequency band preservation strategy. The array beamforming weight is then corrected by the narrowband filter weight. The main frequency band preservation strategy is used to retain the spectral components within the preset main voice frequency band while filtering out low-frequency mechanical noise and high-frequency electromagnetic interference.

[0007] Further, the determination of the comprehensive validity coefficient of the instruction based on the weighted calculation result of the matching degree of the operator terminal and the confidence degree of the semantic parsing includes: Obtain the permission level factor corresponding to the identity matching degree of the operation terminal, and the intent clarity factor corresponding to the semantic parsing confidence degree; Calculate the product of the permission level factor and the intent clarity factor, and determine the product result as the comprehensive effectiveness coefficient of the instruction; If the overall validity coefficient of the instruction is less than the critical operation threshold, and the sequence of sub-instructions to be executed involves core device control, then it is determined that a secondary identity verification will be initiated. When the secondary identity verification is initiated, a voiceprint verification request is generated, the identity matching degree of the operating terminal is updated according to the verified voiceprint features, and the overall validity coefficient of the instruction is recalculated based on the updated identity matching degree of the operating terminal.

[0008] Further, mapping the instruction semantic structure tree into a sequence of sub-instructions to be executed includes: Based on the equipment and process relationships in the industrial knowledge graph, the logical dependency depth of each operation node in the instruction semantic structure tree is analyzed. A directed acyclic graph model is constructed based on the logical dependency depth, and the execution priority coefficients corresponding to each node are identified in the directed acyclic graph model. Based on the execution priority coefficient, the instruction semantic structure tree is serialized and decomposed to generate the sequence of sub-instructions to be executed; for each sub-instruction in the sequence of sub-instructions to be executed, based on the process specification library, the parameter constraint boundary and default execution timeout duration corresponding to the sub-instruction are matched, and the matched parameter constraint boundary and default execution timeout duration are encapsulated into the attribute corresponding to the sub-instruction.

[0009] Furthermore, the calculation process logic conflict index includes: Extract the rate of change and peak hold time of key parameters from the device status response curve; The difference between the rate of change of the key parameter and the preset process safety rate of change threshold is calculated to obtain the limit of dynamic parameter exceedance. Detect the number of resource usage conflicts in the sequence of sub-instructions to be executed during the virtual rehearsal process; The process logic conflict index is determined based on the weighted sum of the dynamic parameter limit exceedance and the number of resource occupation conflicts. If the process logic conflict index is greater than the pre-simulation blocking threshold, the multimodal collaborative interaction strategy is determined to be a circuit breaker blocking mode. At the same time, the sub-instruction node that caused the conflict is extracted as the conflict source object, and the conflict source object is broadcast a voice message.

[0010] Furthermore, if the multimodal collaborative interaction strategy is determined to be an interaction confirmation mode, then a combined verification signal of visual highlighting and voice inquiry is generated, including: Based on the spatial coordinates of the conflict tracing object in the digital twin system, the visual focus area is determined; Generate a highlight rendering command and a viewpoint rotation trajectory for the visual focus area, and use them as the signal for visual highlighting; Based on the limit exceedance of the dynamic parameters, a risk warning text is generated, and the risk warning text is converted into a speech stream, which is used as the signal for the speech inquiry; After outputting the combined verification signal consisting of the visually highlighted signal and the voice inquiry signal, the interactive response timer is started; If a touch confirmation signal or voice affirmation command is received from the operating terminal within the preset interaction window period, the process logic conflict index will be forcibly set to zero, and the multimodal collaborative interaction strategy will be changed to the direct execution mode.

[0011] Furthermore, the step of determining the multimodal collaborative interaction strategy based on the process logic conflict index and the equipment safety risk value also includes: Obtain the voiceprint permission tag of the current operating terminal, as well as the importance level coefficient of the device terminal; Based on the voiceprint permission tag and the importance level coefficient, a permission device matching matrix is ​​constructed; If the device security risk value is in the medium risk range, and the matching result of the permission device matching matrix shows that the current operating terminal has advanced management permissions, then the interaction confirmation mode is automatically exempted, and the multimodal collaborative interaction strategy is directly adjusted to the direct execution mode. If the device security risk value is in the medium risk range, and the matching result of the permission device matching matrix shows that the current operating terminal only has ordinary operating permissions, then the interactive confirmation mode is maintained, and a third-party authorization signal is required to complete the instruction confirmation.

[0012] Furthermore, the sending of control commands to the device includes: Identify the types of industrial communication protocols supported by the device; Based on the industrial communication protocol type, the corresponding protocol adapter driver is invoked to encapsulate the sequence of sub-instructions to be executed into a heterogeneous control message; Add tamper-proof timestamps and digital signatures to the heterogeneous control messages, and send the messages to the device through a preset encrypted channel; After the heterogeneous control message is sent, the instruction response listening thread is started; if no ACK confirmation frame is received from the device within the default execution timeout period, it is determined that the communication link is abnormal and the communication retry mechanism is triggered; wherein, the communication retry mechanism includes: calculating the retry interval based on the exponential backoff algorithm, and generating a link failure alarm when the number of retries reaches a preset upper limit.

[0013] Furthermore, the step of determining whether to trigger an abnormal rollback process based on the execution trajectory deviation rate includes: The instantaneous deviation value is obtained by comparing the actual operating parameters of the device acquired in real time with the standard parameter values ​​at the corresponding moment in the device status response curve. The cumulative total deviation is obtained by integrating several instantaneous deviation values ​​within a preset time window. The total accumulated deviation is compared numerically with a preset safety tolerance threshold. If the total cumulative deviation is less than or equal to the safety tolerance threshold, the device is deemed to be executing normally, and the execution log feature vector corresponding to this execution is recorded. If the total cumulative deviation exceeds the safety tolerance threshold, then the device is determined to be abnormal and the abnormal rollback process is immediately triggered. The abnormal rollback process includes: immediately generating a reverse compensation instruction, which is used to restore the device state to the initial snapshot state before execution, and synchronously marking the abnormal area on the visualization interface with flashing red icons.

[0014] Furthermore, the method also includes: The execution log feature vector for each instance that is determined to be executed normally is stored in a preset positive sample database; Based on the data distribution trend in the positive sample database, the entity association weights in the industrial knowledge graph are updated periodically. Calculate the historical average accuracy of the semantic parsing confidence; If the historical average accuracy shows a continuous downward trend, then the semantic parsing model is incrementally fine-tuned based on the voice instruction mapping pairs in the execution log feature vector.

[0015] This invention generates noise spectrum entropy values ​​by performing frequency domain feature analysis on raw speech streams from industrial sites. Noise reduction is achieved by dynamically matching array beamforming weights and adaptive filtering coefficients based on environmental noise interference levels. This enables accurate and efficient extraction of clean speech commands in highly interference-prone industrial scenarios, improving the efficiency of front-end preprocessing and the accuracy of subsequent semantic recognition and parsing. Simultaneously, voiceprint feature vectors and semantic text information are extracted from the clean speech commands. A semantic structure tree for the commands is generated by combining a pre-constructed industrial knowledge graph. Weighted calculations of the operator's identity matching degree and semantic parsing confidence degree determine the comprehensive validity coefficient of the commands, enabling the interception of illegal commands and the filtering of ambiguous semantics. By structurally deconstructing and converting voice commands, the efficiency of identity verification, semantic parsing, and compliance management of industrial control commands is improved. The sequence of sub-commands to be executed is input into the digital twin system for virtual pre-simulation. Based on the equipment status response curve, the process logic conflict index and equipment safety risk value are calculated and matched with a hierarchical multimodal collaborative interaction strategy. At the same time, during the command execution process, equipment operation data is collected in real time to calculate the execution trajectory deviation rate and determine the abnormal rollback trigger conditions. This realizes risk pre-verification, hierarchical precise handling, and real-time closed-loop management of the entire control command process, improving the efficiency of risk handling, safety management, and closed-loop management of command execution in industrial equipment control.

[0016] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0017] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of the invention. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A flowchart illustrating a multimodal collaborative industrial voice control method according to an embodiment of the present invention is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0020] Figure 1 A flowchart of a multimodal collaborative industrial voice control method according to an embodiment of the present invention is shown, the method comprising: S101: Collect the raw speech stream from the industrial environment and perform frequency domain feature analysis on the raw speech stream to generate a noise spectrum entropy value; determine the current environmental noise interference level based on the noise spectrum entropy value; dynamically match array beamforming weights and adaptive filtering coefficients according to the environmental noise interference level to perform noise reduction processing on the raw speech stream and generate a clean speech command; extract the corresponding voiceprint feature vector and semantic text information based on the clean speech command; calculate the operator's identity matching degree based on the voiceprint feature vector; generate a command semantic structure tree based on the semantic text information and a pre-constructed industrial knowledge graph.

[0021] In some embodiments, the original speech stream is acquired by a MEMS microphone array deployed in an industrial field, with a sampling rate preset to 16kHz-48kHz and a single frame duration of 20ms-50ms; the frequency domain feature analysis involves performing a fast Fourier transform on a single frame of the original speech stream to obtain a spectral sequence, normalizing the spectral sequence, and then calculating the Shannon entropy value to obtain the noise spectral entropy value.

[0022] In some embodiments, the environmental noise interference level is divided into at least three levels: low, medium, and high. The array beamforming weights and adaptive filtering coefficients are stored in a preset noise adaptation parameter library in a one-to-one correspondence. The adaptive filtering coefficients are iteratively updated using the least mean square algorithm or the recursive least squares algorithm.

[0023] In some embodiments, the voiceprint feature vector is a feature vector obtained by concatenating 12-24 dimensional static features extracted based on Mel frequency cepstral coefficients with corresponding first-order and second-order difference features; the semantic text information is generated by converting clean speech instructions into text through a pre-trained industrial scene speech recognition model, wherein the industrial scene speech recognition model is pre-loaded with a customized vocabulary of industrial equipment and process terms.

[0024] In some embodiments, the matching degree of the operator's identity is the cosine similarity between the extracted voiceprint feature vector and each template in the pre-stored authorized operator voiceprint template library, and the matching degree value ranges from 0 to 1.

[0025] In some embodiments, the pre-built industrial knowledge graph uses industrial equipment, process actions, control parameters, and safety rules as entity nodes, and equipment subordination, process sequence, and parameter constraint as associated edges; the instruction semantic structure tree uses operation intent as the root node, target equipment, execution actions, and control parameters as child nodes, and logical relationships as connecting edges between nodes.

[0026] S102, calculate the semantic parsing confidence score based on the completeness of each node in the instruction semantic structure tree; determine the instruction comprehensive validity coefficient based on the weighted calculation result of the operator identity matching degree and the semantic parsing confidence score; when the instruction comprehensive validity coefficient is determined to meet the preset execution benchmark, map the instruction semantic structure tree into a sequence of sub-instructions to be executed; input the sequence of sub-instructions to be executed into the digital twin system for virtual pre-playing, and obtain the equipment status response curve during the pre-playing process; calculate the process logic conflict index and the equipment safety risk value based on the equipment status response curve. In some embodiments, the semantic parsing confidence is the ratio of the number of validly filled nodes in the instruction semantic structure tree to the total number of required nodes. The required nodes include at least the target device node and the execution action node, and the value range is 0-1.

[0027] In some embodiments, the weighted calculation presets the operator identity matching degree weight as 0.4-0.6, the semantic parsing confidence degree weight as 0.4-0.6, the sum of the weights as 1, and the instruction comprehensive validity coefficient as 0-1.

[0028] In some embodiments, the preset execution benchmark is an instruction comprehensive validity coefficient ≥ 0.85. If the value is lower than this benchmark, the instruction is directly determined to be invalid, and the reason for the invalid instruction is fed back to the operator.

[0029] In some embodiments, the digital twin system is a virtual simulation system that maps one-to-one with physical equipment in the industrial field and synchronizes its status in real time. The equipment status response curve is a continuous curve showing the changes in the equipment's operating parameters, position coordinates, and action sequence over time during the simulation process.

[0030] In some embodiments, the process logic conflict index is used to characterize the degree of conflict between instruction execution and preset process procedures, and the value range is 0-1; the equipment safety risk value is used to characterize the degree of safety threat posed by instruction execution to the equipment body and on-site personnel, and the value range is 0-1.

[0031] S103, based on the process logic conflict index and the equipment safety risk value, determine a multimodal collaborative interaction strategy; if the multimodal collaborative interaction strategy is determined to be a direct execution mode, then issue a control command to the equipment; if the multimodal collaborative interaction strategy is determined to be an interactive confirmation mode, then generate a combined verification signal of visual highlighting and voice inquiry; if the multimodal collaborative interaction strategy is determined to be a circuit breaker blocking mode, then perform an equipment safety lock operation and report the reason for the abnormality to the operation terminal. In some embodiments, the multimodal collaborative interaction strategy is pre-classified into three categories based on the threshold range of the process logic conflict index and the equipment safety risk value: direct execution mode, interactive confirmation mode, and circuit breaker blocking mode.

[0032] In some embodiments, the direct execution mode corresponds to the interval where the process logic conflict index is ≤0.2 and the equipment safety risk value is ≤0.2; the interactive confirmation mode corresponds to the interval where 0.2 < process logic conflict index ≤0.5 or 0.2 < equipment safety risk value ≤0.5; and the circuit breaker blocking mode corresponds to the interval where the process logic conflict index is >0.5 or the equipment safety risk value is >0.5.

[0033] S104: During the execution of control commands on the device side, real-time data of device operation is collected; the execution trajectory deviation rate is calculated based on the operation data; and the abnormal rollback process is determined based on the execution trajectory deviation rate.

[0034] In some embodiments, the device-side operating data includes at least the device's position coordinates, operating speed, load current, action sequence, and process parameters, with the acquisition frequency consistent with the device control cycle.

[0035] In some embodiments, the execution trajectory deviation rate is the ratio of the deviation between the actual running trajectory of the device and the standard trajectory generated by the virtual pre-simulation to the total travel of the standard trajectory, and the value range is 0-1.

[0036] In some embodiments, the triggering benchmark for the abnormal rollback process is an execution trajectory deviation rate greater than a preset deviation tolerance threshold, wherein the deviation tolerance threshold ranges from 0.05 to 0.15.

[0037] In some embodiments, determining the current environmental noise interference level based on the noise spectrum entropy value and dynamically matching array beamforming weights according to the environmental noise interference level includes: comparing the noise spectrum entropy value with a preset steady-state noise threshold and a sudden interference threshold; if the noise spectrum entropy value < the steady-state noise threshold, the current environmental noise interference level is determined to be low interference level, and a preset omnidirectional acquisition weight template is called as the array beamforming weight; if the noise spectrum entropy value ≥ the steady-state noise threshold and < the sudden interference threshold, the current environmental noise interference level is determined to be medium interference level, and a directional enhancement weight is calculated based on the sound source localization result, and the directional enhancement weight is used as the array beamforming weight; if the noise spectrum entropy value ≥ the sudden interference threshold, the current environmental noise interference level is determined to be high interference level, and a narrowband filtering weight is generated based on the main frequency band preservation strategy, and the array beamforming weight is corrected by the narrowband filtering weight; wherein, the main frequency band preservation strategy is used to retain the spectral components within a preset main voice frequency band while filtering out low-frequency mechanical noise and high-frequency electromagnetic interference. According to embodiments of the present invention, different levels of environmental noise interference are classified by comparing the noise spectrum entropy value with the values ​​of dual thresholds, thereby improving the targeting of noise scene adaptation; optimal speech enhancement processing for corresponding interference levels is achieved by hierarchically matching differentiated array beamforming weights, thereby improving the extraction efficiency of clean speech commands; and invalid interference spectrum is filtered out by correcting the weights through a main frequency band preservation strategy, thereby improving the integrity and usability of the target speech signal.

[0038] In some embodiments, the steady-state noise threshold is preset to 0.3-0.4, the sudden interference threshold is preset to 0.6-0.7, and the noise spectrum entropy value ranges from 0 to 1. The higher the entropy value, the stronger the noise interference.

[0039] In some embodiments, the omnidirectional acquisition weight template is a set of equal weighting coefficients for each element of the microphone array, used for indiscriminate acquisition of omnidirectional speech signals.

[0040] In some embodiments, the sound source localization result is calculated using a time difference of arrival algorithm, and the directional enhancement weight is used to enhance the speech signal in the direction of the target sound source while suppressing noise signals in non-target directions.

[0041] In some embodiments, the preset main voice frequency band is the effective frequency band of industrial voice in the range of 300Hz-3400Hz, and the narrowband filter weight sets the weighting coefficient of 1 for the spectral components within the preset main voice frequency band and sets the weighting coefficient of 0-0.2 for the spectral components outside the preset main voice frequency band.

[0042] In some embodiments, determining the comprehensive validity coefficient of an instruction based on the weighted calculation result of the operator identity matching degree and the semantic parsing confidence degree includes: obtaining the permission level factor corresponding to the operator identity matching degree and the intent clarity factor corresponding to the semantic parsing confidence degree; calculating the product of the permission level factor and the intent clarity factor, and determining the product result as the comprehensive validity coefficient of the instruction; if the comprehensive validity coefficient of the instruction is less than the critical operation threshold, and the sequence of sub-instructions to be executed involves core device control, then a secondary identity verification is initiated; wherein, when the secondary identity verification is initiated, a voiceprint review request is generated, the operator identity matching degree is updated according to the reviewed voiceprint features, and the comprehensive validity coefficient of the instruction is recalculated based on the updated operator identity matching degree. According to embodiments of the present invention, a two-dimensional quantitative assessment of instruction validity is achieved by multiplying the permission level factor and the intent clarity factor, thereby improving the rigor of instruction access determination; permission review for high-risk operations is achieved by triggering secondary identity verification in low-coefficient scenarios of core operations, thereby improving the security of industrial control; and dynamic correction of instruction validity is achieved by reviewing voiceprint features to update the matching degree and recalculating the coefficients, thereby improving the accuracy of permission verification.

[0043] In some embodiments, the permission level factor is preset to 0.4-1.0 based on the authorization level of the operator's identity, where administrator level corresponds to 1.0, senior operator corresponds to 0.8, ordinary operator corresponds to 0.6, and visitor corresponds to 0.4; the intent clarity factor is equal to the semantic parsing confidence level, and its value ranges from 0 to 1.

[0044] In some embodiments, the comprehensive validity coefficient of the instruction ranges from 0 to 1, with a higher value indicating stronger legality and executability of the instruction.

[0045] In some embodiments, the critical operation threshold is preset to 0.7, and the core equipment is a preset industrial equipment involved in the main process of the production line and high-risk operations, including but not limited to industrial robots, stamping equipment, and lifting equipment.

[0046] In some embodiments, mapping the instruction semantic structure tree to a sequence of sub-instructions to be executed includes: parsing the logical dependency depth of each operation node in the instruction semantic structure tree based on the equipment process association in the industrial knowledge graph; constructing a directed acyclic graph model based on the logical dependency depth, and identifying the execution priority coefficient corresponding to each node in the directed acyclic graph model; performing serialization decomposition on the instruction semantic structure tree according to the execution priority coefficient to generate the sequence of sub-instructions to be executed; for each sub-instruction in the sequence of sub-instructions to be executed, matching the parameter constraint boundary and default execution timeout duration corresponding to the sub-instruction based on the process specification library, and encapsulating the matched parameter constraint boundary and default execution timeout duration into the attribute corresponding to the sub-instruction. According to embodiments of the present invention, the logical dependency depth of operation nodes is analyzed by industrial knowledge graphs to sort out the instruction execution logic and improve the rationality of instruction decomposition; the execution priority is identified by a directed acyclic graph model to achieve ordered serialization decomposition of instructions and improve the execution efficiency of sub-instruction sequences; the parameter constraints and timeout duration of sub-instructions are matched by a process specification library to achieve standardized limitation of sub-instruction execution boundaries and improve the controllability of equipment operation.

[0047] In some embodiments, the logical dependency depth is the number of preceding dependent nodes corresponding to the operation node. The logical dependency depth of a node without preceding dependencies is 0, and the depth value increases by 1 for each additional level of preceding dependency.

[0048] In some embodiments, the execution priority coefficient is negatively correlated with the logical dependency depth. The smaller the logical dependency depth, the higher the execution priority coefficient. The priority coefficient ranges from 1 to 10, where 1 is the lowest priority and 10 is the highest priority.

[0049] In some embodiments, the serialization decomposition sorts the operation nodes in descending order of execution priority coefficient, and nodes of the same priority are sorted in the order of process sequence.

[0050] In some embodiments, the process specification library pre-stores the upper and lower limits of process parameters, safe operation specifications, and standard execution time for each industrial equipment; the parameter constraint boundary includes the upper limit value and the lower limit value of the parameter, and the default execution timeout is 1.2-1.5 times the standard execution time.

[0051] In some embodiments, calculating the process logic conflict index includes: extracting the rate of change of key parameters and the peak holding time from the equipment status response curve; calculating the difference between the rate of change of key parameters and a preset process safety rate of change threshold to obtain the dynamic parameter limit violation; detecting the number of resource occupation conflicts in the sequence of sub-instructions to be executed during the virtual rehearsal process; determining the process logic conflict index based on the weighted sum of the dynamic parameter limit violation and the number of resource occupation conflicts; if the process logic conflict index > the rehearsal blocking threshold, then the multimodal collaborative interaction strategy is determined to be a circuit breaker blocking mode, and the sub-instruction node causing the conflict is extracted as the conflict tracing object, and the conflict tracing object is broadcast aloud. According to the embodiments of the present invention, by extracting key parameters and calculating the difference from the equipment status response curve, the degree of deviation of process parameters is quantitatively assessed, improving the degree of conflict identification; by weighting the number of resource occupation conflicts and the parameter limit violation, the process logic conflict is comprehensively quantified, improving the accuracy of risk assessment; by circuit breaker blocking and conflict tracing broadcast for threshold-exceeding scenarios, high-risk instructions are intercepted in advance, improving the safety of industrial production.

[0052] In some embodiments, the key parameters are equipment operating parameters directly related to process requirements, including but not limited to operating speed, acceleration, output torque, temperature, and pressure; the key parameter change rate is the amount of change of the key parameter per unit time, and the peak holding time is the duration after the key parameter reaches its peak value.

[0053] In some embodiments, the dynamic parameter limit exceedance is the ratio of the absolute value of the difference between the rate of change of the key parameter and the process safety rate of change threshold to the process safety rate of change threshold, and the value range is 0-1.

[0054] In some embodiments, the resource occupation conflict includes the same device being called by multiple sub-instructions at the same time, the same public resource being occupied by multiple sub-instructions at the same time, and the execution sequence of the sub-instructions conflicting with the preset process sequence.

[0055] In some embodiments, the preset weight for the dynamic parameter exceeding the limit is 0.7, the preset weight for the number of resource occupation conflicts is 0.3, and the process logic conflict index is normalized to the 0-1 range.

[0056] In some embodiments, the pre-simulation blocking threshold is preset to 0.5, and the content of the voice broadcast includes at least the target device, the action to be performed, and the reason for the conflict corresponding to the conflict sub-instruction.

[0057] In some embodiments, if the multimodal collaborative interaction strategy is determined to be an interaction confirmation mode, generating a combined verification signal of visual highlighting and voice inquiry includes: determining a visual focus area based on the spatial coordinates of the conflict tracing object in the digital twin system; generating a highlight rendering instruction and a viewpoint rotation trajectory for the visual focus area, and using them as the visual highlighting signal; generating a risk warning text based on the dynamic parameter limit exceedance, and converting the risk warning text into a voice stream, and using it as the voice inquiry signal; after outputting the combined verification signal composed of the visual highlighting signal and the voice inquiry signal, starting an interaction response timer; if a touch confirmation signal or voice affirmation instruction is received from the operation terminal within a preset interaction window period, the process logic conflict index is forcibly set to zero, and the multimodal collaborative interaction strategy is changed to the direct execution mode. According to embodiments of the present invention, the visual focus area is determined by the spatial coordinates of the conflict tracing object, thereby locating the risk point and improving the targeting of visual prompts; multimodal synchronous prompts of risk information are achieved by combining visual highlighting and voice inquiry signals, thereby improving the effectiveness of interactive confirmation; and dynamic adjustment of the instruction execution strategy is achieved by processing the confirmation signal response during the interaction window, thereby improving the flexibility of industrial control.

[0058] In some embodiments, the visual focus area is the smallest rectangular display area centered on the spatial coordinates of the digital twin system corresponding to the conflict tracing object, covering the entire model.

[0059] In some embodiments, the highlight rendering instruction is used to highlight the edges and deepen the color of the model within the visual focus area, and the viewpoint rotation trajectory is used to control the virtual camera of the digital twin system to rotate around the visual focus area to the optimal observation viewpoint.

[0060] In some embodiments, the risk warning text includes at least the risk level, conflict parameters, risk consequences, and a question asking whether to confirm execution. The voice stream is generated through a preset industrial scenario TTS model.

[0061] In some embodiments, the duration of the interaction response timer is consistent with the duration of a preset interaction window period.

[0062] In some embodiments, the preset interaction window period is preset to 5s-10s; if no confirmation signal is received within the preset interaction window period, it is directly determined that the instruction is abandoned and the circuit breaker blocking mode is triggered.

[0063] In some embodiments, determining the multimodal collaborative interaction strategy based on the process logic conflict index and the equipment security risk value further includes: obtaining the voiceprint permission tag of the current operating terminal and the importance level coefficient of the device; constructing a permission device matching matrix based on the voiceprint permission tag and the importance level coefficient; if the equipment security risk value is in the medium risk range and the matching result of the permission device matching matrix shows that the current operating terminal has advanced management permissions, then the interaction confirmation mode is automatically exempted, and the multimodal collaborative interaction strategy is directly adjusted to the direct execution mode; if the equipment security risk value is in the medium risk range and the matching result of the permission device matching matrix shows that the current operating terminal only has ordinary operation permissions, then the interaction confirmation mode is maintained, and a third-party authorization signal is required to complete the instruction confirmation. According to the embodiments of the present invention, by constructing a matching matrix through the voiceprint permission tag and the equipment importance level coefficient, the matching of operation permissions and equipment levels is realized, improving the rationality of strategy determination; by automatically exempting the medium risk scenario of advanced management permissions, the operation process of high-privilege users is simplified, improving the execution efficiency of industrial control; by requiring third-party authorization for ordinary operation permissions, multiple verifications of medium-risk operations are realized, improving the security of equipment operation.

[0064] In some embodiments, the voiceprint permission tag is bound to the authorization level corresponding to the identity matching degree of the operating terminal, and is divided into four categories: administrator, senior operator, ordinary operator, and visitor; the importance level coefficient of the device terminal is preset to 1-3 according to the coreness of the device, with core devices corresponding to 3, important devices corresponding to 2, and ordinary devices corresponding to 3.

[0065] In some embodiments, in the permission device matching matrix, the administrator tag can match devices of all importance levels, the senior operator tag can match devices of importance level 2 and below, the ordinary operator tag can only match devices of importance level 1, and visitors have no matching permissions.

[0066] In some embodiments, the medium-risk range is the range where the device security risk value is greater than 0.2 and less than or equal to 0.5, and the advanced management permission is the permission corresponding to the administrator tag.

[0067] In some embodiments, the third-party authorization signal is a voiceprint confirmation signal or touch authorization signal from an operator with advanced management privileges. Without obtaining a third-party authorization signal, the system cannot switch to direct execution mode.

[0068] In some embodiments, issuing control commands to the device includes: identifying the industrial communication protocol types supported by the device; based on the industrial communication protocol type, calling the corresponding protocol adapter driver to encapsulate the sequence of sub-instructions to be executed into a heterogeneous control message; adding an anti-tampering timestamp and digital signature to the heterogeneous control message, and sending the message to the device through a preset encrypted channel; after the heterogeneous control message is sent, starting an instruction response listening thread; if no ACK confirmation frame is received from the device within the default execution timeout period, it is determined that the communication link is abnormal, and a communication retry mechanism is triggered; wherein, the communication retry mechanism includes: calculating the retry interval based on the exponential backoff algorithm, and generating a link failure alarm when the number of retries reaches a preset upper limit. According to embodiments of the present invention, by identifying the communication protocol type of the device and calling the corresponding driver, protocol adaptation and compatibility of different devices can be achieved, thereby improving the adaptability of control command issuance; by adding anti-tampering timestamps, digital signatures and encrypted channel transmission to the messages, end-to-end security protection of command transmission can be achieved, thereby improving the anti-tampering capability of control messages; by using response monitoring and exponential backoff retry mechanisms, timely handling of communication anomalies can be achieved, thereby improving the reliability of command transmission.

[0069] In some embodiments, the industrial communication protocol types include, but are not limited to, Modbus, Profinet, EtherCAT, and OPC UA protocols.

[0070] In some embodiments, the protocol adapter driver is pre-integrated into the industrial control gateway, and the format of the heterogeneous control message is fully compatible with the industrial communication protocols supported by the device.

[0071] In some embodiments, the tamper-proof timestamp is the timestamp of the message transmission time with an accuracy of not less than 1ms; the digital signature is generated based on the authorized private key of the operating terminal; and the preset encrypted channel is a secure communication channel established using the TLS 1.3 encryption protocol.

[0072] In some embodiments, the preset upper limit is 3 times, the initial retry interval of the exponential backoff algorithm is 100ms, and the retry interval doubles each time.

[0073] In some embodiments, determining whether to trigger an abnormal rollback process based on the execution trajectory deviation rate includes: obtaining an instantaneous deviation value based on the difference between the actual operating parameters of the device acquired in real time and the standard parameter values ​​at the corresponding time in the device status response curve; performing an integral operation on several instantaneous deviation values ​​within a preset time window to obtain a cumulative total deviation; comparing the cumulative total deviation with a preset safety tolerance threshold; if the cumulative total deviation is ≤ the safety tolerance threshold, determining that the device is executing normally and recording the execution log feature vector corresponding to this execution; if the cumulative total deviation is > the safety tolerance threshold, determining that the device is executing abnormally and immediately triggering the abnormal rollback process; wherein, the abnormal rollback process includes: immediately generating a reverse compensation instruction, the reverse compensation instruction being used to restore the device status to the initial snapshot state before execution, and synchronously marking the abnormal area on the visualization interface with red flashing elements. According to embodiments of the present invention, the quantification of equipment execution deviation is achieved by calculating and integrating the difference between actual operating parameters and standard parameters, thereby improving the accuracy of anomaly identification; the real-time monitoring of equipment execution status is achieved by comparing the total accumulated deviation with the safety threshold, thereby improving the timeliness of anomaly handling; and the rapid recovery of equipment status and the prompting of anomaly points are achieved through reverse compensation rollback of abnormal scenarios and visualization of abnormal areas, thereby improving the fault tolerance and operation and maintenance efficiency of industrial control.

[0074] In some embodiments, the actual operating parameters and standard parameter values ​​are the same type and dimension of equipment operating parameters, including position, speed, torque, pressure, and temperature parameters.

[0075] In some embodiments, the preset time window is 100ms-500ms, and the number of sampling points within the window is consistent with the running data acquisition frequency.

[0076] In some embodiments, the safety tolerance threshold is preset according to the equipment type and process accuracy requirements, and is a non-negative value.

[0077] In some embodiments, the execution log feature vector includes characteristic data such as instruction number, operator information, execution time, device operating parameters, deviation value, and execution result.

[0078] In some embodiments, the initial snapshot state before execution is a complete backup of the device's real-time operating status, location coordinates, and parameter configuration before the control command is issued, and the execution priority of the reverse compensation command is higher than all currently executing control commands.

[0079] In some embodiments, the method further includes: storing the execution log feature vector determined to be executed normally each time into a preset positive sample database; periodically updating the entity association weights in the industrial knowledge graph based on the data distribution trend in the positive sample database; calculating the historical average accuracy of the semantic parsing confidence; and if the historical average accuracy shows a continuous downward trend, incrementally fine-tuning the semantic parsing model based on the voice command mapping pairs in the execution log feature vector. According to embodiments of the present invention, storing the log feature vectors of normally executed operations into a positive sample database achieves the standardized accumulation of high-quality execution data, providing reliable data support for model optimization; updating the entity association weights of the industrial knowledge graph through positive sample data achieves dynamic optimization of the knowledge graph, improving the accuracy of semantic parsing; and incremental fine-tuning after a decrease in semantic parsing accuracy achieves continuous iteration of model capabilities, improving the long-term adaptability of the voice control method.

[0080] In some embodiments, the positive sample database is a sample repository constructed using a time-series database, and each sample is labeled with the corresponding execution result, process effect, and equipment status tag.

[0081] In some embodiments, the periodic update cycle is preset to 24 hours, and the entity association weight is adjusted according to the execution success rate of the corresponding process instruction and the process effect score. The higher the execution success rate, the greater the association weight.

[0082] In some embodiments, the historical average accuracy rate is the ratio of the number of instructions with a semantic parsing confidence level ≥ 0.85 to the total number of instructions within a preset statistical period, and the preset statistical period is 7 days.

[0083] In some embodiments, the continuous downward trend is defined as a month-on-month decrease of more than 5% in the historical average accuracy over three consecutive statistical periods; the voice command mapping pair is a pair of samples of pure voice commands and their corresponding standard command semantic structure trees.

[0084] It should be understood that the various processes described above can be used to rearrange, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0085] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multimodal collaborative industrial voice control method, characterized in that, include: The raw speech stream in an industrial environment is collected, and frequency domain feature analysis is performed on the raw speech stream to generate noise spectrum entropy values. The current environmental noise interference level is determined based on the noise spectrum entropy value. The array beamforming weights and adaptive filtering coefficients are dynamically matched according to the environmental noise interference level to generate a clean voice command after noise reduction processing of the original speech stream. Based on the clean voice command, the corresponding voiceprint feature vector and semantic text information are extracted respectively. The identity matching degree of the operator terminal is calculated based on the voiceprint feature vector. Generate an instruction semantic structure tree based on the semantic text information and the pre-constructed industrial knowledge graph; Based on the completeness of each node in the instruction semantic structure tree, the semantic parsing confidence is calculated; based on the weighted calculation result of the operator identity matching degree and the semantic parsing confidence, the instruction comprehensive validity coefficient is determined; when the instruction comprehensive validity coefficient is determined to meet the preset execution benchmark, the instruction semantic structure tree is mapped to a sequence of sub-instructions to be executed; the sequence of sub-instructions to be executed is input into the digital twin system for virtual pre-playing, and the device status response curve during the pre-playing process is obtained; Based on the equipment status response curve, the process logic conflict index and the equipment safety risk value are calculated respectively. Based on the process logic conflict index and the equipment safety risk value, a multimodal collaborative interaction strategy is determined. If the multimodal collaborative interaction strategy is determined to be in direct execution mode, a control command is sent to the device; if the multimodal collaborative interaction strategy is determined to be in interactive confirmation mode, a combined verification signal of visual highlighting and voice inquiry is generated. If the multimodal collaborative interaction strategy is determined to be a circuit breaker blocking mode, then a device security lock operation is performed, and the cause of the abnormality is reported back to the operator. During the execution of control commands on the device side, real-time data collection of device operation is performed. The execution trajectory deviation rate is calculated based on the runtime data; the abnormal rollback process is then determined based on the execution trajectory deviation rate.

2. The multimodal collaborative industrial voice control method according to claim 1, characterized in that, The step of determining the current environmental noise interference level based on the noise spectrum entropy value and dynamically matching array beamforming weights according to the environmental noise interference level includes: The noise spectrum entropy value is compared with the preset steady-state noise threshold and sudden interference threshold. If the noise spectrum entropy value is less than the steady-state noise threshold, the current environmental noise interference level is determined to be low interference level, and the preset omnidirectional acquisition weight template is called as the array beamforming weight. If the noise spectrum entropy value is greater than or equal to the steady-state noise threshold and less than the sudden interference threshold, then the current environmental noise interference level is determined to be medium interference level, and the directional enhancement weight is calculated based on the sound source localization result, and the directional enhancement weight is used as the array beamforming weight. If the noise spectrum entropy value is greater than or equal to the sudden interference threshold, the current environmental noise interference level is determined to be high interference level, and a narrowband filter weight is generated based on the main frequency band preservation strategy. The array beamforming weight is then corrected by the narrowband filter weight. The main frequency band preservation strategy is used to retain the spectral components within the preset main voice frequency band while filtering out low-frequency mechanical noise and high-frequency electromagnetic interference.

3. The multimodal collaborative industrial voice control method according to claim 2, characterized in that, The determination of the comprehensive validity coefficient of the instruction based on the weighted calculation result of the identity matching degree of the operating terminal and the semantic parsing confidence degree includes: Obtain the permission level factor corresponding to the identity matching degree of the operation terminal, and the intent clarity factor corresponding to the semantic parsing confidence degree; Calculate the product of the permission level factor and the intent clarity factor, and determine the product result as the comprehensive effectiveness coefficient of the instruction; If the overall validity coefficient of the instruction is less than the critical operation threshold, and the sequence of sub-instructions to be executed involves core device control, then it is determined that a secondary identity verification will be initiated. When the secondary identity verification is initiated, a voiceprint verification request is generated, the identity matching degree of the operating terminal is updated according to the verified voiceprint features, and the overall validity coefficient of the instruction is recalculated based on the updated identity matching degree of the operating terminal.

4. The multimodal collaborative industrial voice control method according to claim 3, characterized in that, Mapping the instruction semantic structure tree into a sequence of sub-instructions to be executed includes: Based on the equipment and process relationships in the industrial knowledge graph, the logical dependency depth of each operation node in the instruction semantic structure tree is analyzed. A directed acyclic graph model is constructed based on the logical dependency depth, and the execution priority coefficients corresponding to each node are identified in the directed acyclic graph model. Based on the execution priority coefficient, the instruction semantic structure tree is serialized and decomposed to generate the sequence of sub-instructions to be executed; for each sub-instruction in the sequence of sub-instructions to be executed, based on the process specification library, the parameter constraint boundary and default execution timeout duration corresponding to the sub-instruction are matched, and the matched parameter constraint boundary and default execution timeout duration are encapsulated into the attribute corresponding to the sub-instruction.

5. The multimodal collaborative industrial voice control method according to claim 4, characterized in that, The calculation process logic conflict index includes: Extract the rate of change and peak hold time of key parameters from the device status response curve; The difference between the rate of change of the key parameter and the preset process safety rate of change threshold is calculated to obtain the limit of dynamic parameter exceedance. Detect the number of resource usage conflicts in the sequence of sub-instructions to be executed during the virtual rehearsal process; The process logic conflict index is determined based on the weighted sum of the dynamic parameter limit exceedance and the number of resource occupation conflicts. If the process logic conflict index is greater than the pre-simulation blocking threshold, the multimodal collaborative interaction strategy is determined to be a circuit breaker blocking mode. At the same time, the sub-instruction node that caused the conflict is extracted as the conflict source object, and the conflict source object is broadcast a voice message.

6. The multimodal collaborative industrial voice control method according to claim 5, characterized in that, If the multimodal collaborative interaction strategy is determined to be an interaction confirmation mode, a combined verification signal of visual highlighting and voice inquiry is generated, including: Based on the spatial coordinates of the conflict tracing object in the digital twin system, the visual focus area is determined; Generate a highlight rendering command and a viewpoint rotation trajectory for the visual focus area, and use them as the signal for visual highlighting; Based on the limit exceedance of the dynamic parameters, a risk warning text is generated, and the risk warning text is converted into a speech stream, which is used as the signal for the speech inquiry; After outputting the combined verification signal consisting of the visually highlighted signal and the voice inquiry signal, the interactive response timer is started; If a touch confirmation signal or voice affirmation command is received from the operating terminal within the preset interaction window period, the process logic conflict index will be forcibly set to zero, and the multimodal collaborative interaction strategy will be changed to the direct execution mode.

7. The multimodal collaborative industrial voice control method according to claim 6, characterized in that, The step of determining the multimodal collaborative interaction strategy based on the process logic conflict index and the equipment safety risk value also includes: Obtain the voiceprint permission tag of the current operating terminal, as well as the importance level coefficient of the device terminal; Based on the voiceprint permission tag and the importance level coefficient, a permission device matching matrix is ​​constructed; If the device security risk value is in the medium risk range, and the matching result of the permission device matching matrix shows that the current operating terminal has advanced management permissions, then the interaction confirmation mode is automatically exempted, and the multimodal collaborative interaction strategy is directly adjusted to the direct execution mode. If the device security risk value is in the medium risk range, and the matching result of the permission device matching matrix shows that the current operating terminal only has ordinary operating permissions, then the interactive confirmation mode is maintained, and a third-party authorization signal is required to complete the instruction confirmation.

8. The multimodal collaborative industrial voice control method according to claim 7, characterized in that, The sending of control commands to the device includes: Identify the types of industrial communication protocols supported by the device; Based on the industrial communication protocol type, the corresponding protocol adapter driver is invoked to encapsulate the sequence of sub-instructions to be executed into a heterogeneous control message; Add tamper-proof timestamps and digital signatures to the heterogeneous control messages, and send the messages to the device through a preset encrypted channel; After the heterogeneous control message is sent, the instruction response listening thread is started; if no ACK confirmation frame is received from the device within the default execution timeout period, it is determined that the communication link is abnormal and the communication retry mechanism is triggered; wherein, the communication retry mechanism includes: calculating the retry interval based on the exponential backoff algorithm, and generating a link failure alarm when the number of retries reaches a preset upper limit.

9. The multimodal collaborative industrial voice control method according to claim 8, characterized in that, The step of determining whether to trigger an abnormal rollback process based on the execution trajectory deviation rate includes: The instantaneous deviation value is obtained by comparing the actual operating parameters of the device acquired in real time with the standard parameter values ​​at the corresponding moment in the device status response curve. The cumulative total deviation is obtained by integrating several instantaneous deviation values ​​within a preset time window. The total accumulated deviation is compared numerically with a preset safety tolerance threshold. If the total cumulative deviation is less than or equal to the safety tolerance threshold, the device is deemed to be executing normally, and the execution log feature vector corresponding to this execution is recorded. If the total cumulative deviation exceeds the safety tolerance threshold, then the device is determined to be abnormal and the abnormal rollback process is immediately triggered. The abnormal rollback process includes: immediately generating a reverse compensation instruction, which is used to restore the device state to the initial snapshot state before execution, and synchronously marking the abnormal area on the visualization interface with flashing red icons.

10. The multimodal collaborative industrial voice control method according to claim 9, characterized in that, The method further includes: The execution log feature vector for each instance that is determined to be executed normally is stored in a preset positive sample database; Based on the data distribution trend in the positive sample database, the entity association weights in the industrial knowledge graph are updated periodically. Calculate the historical average accuracy of the semantic parsing confidence; If the historical average accuracy shows a continuous downward trend, then the semantic parsing model is incrementally fine-tuned based on the voice instruction mapping pairs in the execution log feature vector.

Images