Intelligent laboratory human-computer interaction control method and system

Abstract

The present invention provides a human-computer interaction control method and system for an intelligent laboratory, which relates to the field of control technology, including collecting images of experimenters and equipment through a visual acquisition device, and a processing server identifying equipment operation status parameters based on a deep learning model, including switch status, indicator light status, display screen status and fault indication status, and comparing the recognition results with preset parameters to generate an evaluation result. The processing server receives user voice commands, extracts the content of control commands, and determines the target device and control action. The legality of the command is verified and combined with the equipment status evaluation results to determine whether the execution conditions are met. A collaborative control strategy is generated based on the control command and the evaluation results, including control timing, parameters and data interaction methods, and control commands are sent through a field bus network to achieve multi-device collaborative control. The present invention realizes intelligent management and control of laboratory equipment, and improves experimental efficiency and safety.

Description

Technical Field

[0001] The present invention relates to control technology, and in particular to a human-computer interaction control method and system for an intelligent laboratory. Background Art

[0002] Laboratory automation and intelligence are important development directions of modern scientific research. Traditional laboratory equipment control usually relies on manual operation or simple automation procedures, requiring researchers to perform a lot of on-site operation and monitoring. With the increasing complexity of experiments and the growing requirements for experimental efficiency, traditional control methods are difficult to meet the needs of modern laboratories.

[0003] The existing laboratory equipment control technology mainly has the following defects and shortcomings:

[0004] Lack of intelligent control: Most existing control systems are based on pre-set programs and lack the ability to perceive and adaptively adjust the experimental environment and equipment status, making it difficult to cope with complex experimental scenarios and emergencies.

[0005] Difficulty in achieving coordinated control of multiple devices: Modern experiments often require multiple devices to work together, but existing control systems find it difficult to achieve coordinated control of multiple devices, resulting in low experimental efficiency and even experimental errors.

[0006] The human-computer interaction method is not friendly enough: Existing control systems usually use complex graphical interfaces or command line operations, which require high professional skills from users, are inconvenient to use, and make it difficult to achieve efficient human-computer interaction. Summary of the invention

[0007] The embodiments of the present invention provide a human-computer interaction control method and system for an intelligent laboratory, which can solve the problems in the prior art.

[0008] According to a first aspect of the embodiments of the present invention,

[0009] The image information of multiple experimental equipment in the laboratory is collected by multiple visual acquisition devices arranged in the laboratory, and the image information is transmitted to a processing server; the processing server analyzes and processes the image information based on a preset equipment status deep learning model, and identifies the operating status parameters of the multiple experimental equipment, wherein the operating status parameters include the equipment switch status, the equipment operating indicator status, the equipment display status, and the equipment fault indication status; the processing server compares the identified operating status parameters with the preset equipment normal operating parameters to generate an equipment status evaluation result;

[0010] The voice command information of the user is collected by a voice collection device set up in the laboratory, and the voice command information is transmitted to the processing server; the processing server performs semantic analysis on the voice command information, extracts the control command content, and determines the target experimental equipment to be controlled and the corresponding control action according to the control command content, wherein the control action includes equipment power on and off, equipment parameter adjustment, and equipment data collection; after determining the legitimacy of the control command content, the processing server judges whether the target experimental equipment meets the conditions for executing the control action in combination with the equipment status evaluation result;

[0011] The processing server generates a collaborative control strategy for the experimental equipment based on the control instruction content and the equipment status evaluation results, and the collaborative control strategy includes the control timing, control parameters, and data interaction methods of multiple experimental equipment; the processing server sends control instructions to the multiple experimental equipment through the laboratory field bus network to achieve collaborative control of the multiple experimental equipment.

[0012] The processing server analyzes and processes the image information based on a preset device status deep learning model, and identifies the operating status parameters of the multiple experimental devices, including:

[0013] The image information is input into a YOLOv5 target detection network of a device state deep learning model, wherein a spatial attention mechanism module is provided in the YOLOv5 target detection network, and the device area in the image information is feature enhanced by the spatial attention mechanism module, wherein the spatial attention mechanism module performs average pooling and maximum pooling on the input image information to obtain pooling features, and the pooling features are input into a seven-by-seven convolutional layer for convolution operation, and a spatial attention weight matrix is ​​generated by a sigmoid activation function;

[0014] The spatial attention weight matrix is ​​fused with the feature map corresponding to the image information to obtain a fused feature, the fused feature is input into a ResNet backbone network provided with a feature pyramid network, and multi-scale feature extraction is performed on the fused feature through the feature pyramid network, wherein the feature pyramid network performs upsampling on the fused feature, and performs feature splicing with the lateral connection features of the same layer, and outputs a multi-scale feature map through a convolution layer;

[0015] The multi-scale feature map is input into a multi-task learning framework, which includes a device switch state recognition branch, an operating indicator light state recognition branch, and a display state recognition branch, and the losses of the three recognition branches are jointly optimized through a weighted loss function; the optimized state recognition result is input into a bidirectional long short-term memory network to establish a time series state model, and the bidirectional long short-term memory network predicts the device state at the current moment based on the historical state sequence characteristics;

[0016] The state feature vector output by the bidirectional long short-term memory network is input into a Gaussian mixture model, wherein the Gaussian mixture model includes multiple Gaussian components, each of which has a corresponding mixing coefficient, a mean vector and a covariance matrix. By calculating the probability density distribution of the state feature vector, it is determined whether the equipment is in an abnormal state and the operating state parameters of the multiple experimental equipment are identified.

[0017] The method further comprises training the Gaussian mixture model:

[0018] The K-means clustering algorithm is used to pre-group the experimental equipment state training data acquired in advance, and the initial mean vector of each Gaussian component is calculated according to the pre-grouping result, and the initial mean vector is obtained by calculating the mean of all samples in each group; the initial covariance matrix of each Gaussian component is calculated, and the initial covariance matrix is ​​obtained by calculating the difference product of the samples in each group and the corresponding initial mean vector; the initial mixing coefficient of each Gaussian component is calculated, and the initial mixing coefficient is obtained by calculating the ratio of the number of samples in each group to the total number of samples;

[0019] The initial mean vector, the initial covariance matrix, and the initial mixing coefficient are input into the expectation maximization algorithm, and the posterior probability of each sample belonging to each Gaussian component is calculated in the expectation step, and the posterior probability is obtained by calculating the product of the mixing coefficient of each Gaussian component and the probability density and normalizing them; in the maximization step, the mean vector, covariance matrix, and mixing coefficient of the Gaussian component are updated based on the posterior probability, and the updating process is iteratively performed until the model parameters converge;

[0020] The converged Gaussian mixture model is applied to abnormal state detection, by calculating the logarithmic likelihood and Mahalanobis distance of the sample to be detected, wherein the logarithmic likelihood is obtained by calculating the logarithmic value of the weighted probability density of the sample on all Gaussian components, and the Mahalanobis distance is obtained by calculating the minimum value of the standardized distance from the sample to the center of each Gaussian component; the logarithmic likelihood and the Mahalanobis distance are weightedly combined by a weight coefficient to obtain an abnormal detection score;

[0021] A dynamic threshold is calculated based on a sliding time window, and the dynamic threshold is obtained by calculating the weighted sum of the mean and standard deviation of the historical detection scores; the temporal feature anomaly detection score and the spatial feature anomaly detection score are multimodally fused, and the multimodal fusion is achieved by assigning fusion weights to the temporal anomaly detection score and the spatial anomaly detection score respectively and summing them. When the anomaly detection score after the multimodal fusion exceeds the dynamic threshold, it is determined to be an abnormal state.

[0022] The processing server performs semantic analysis on the voice command information, extracts the control command content, and determines the target experimental equipment to be controlled and the corresponding control action according to the control command content, including:

[0023] Extracting Mel-frequency cepstral coefficient features from the input voice command information to obtain Mel-frequency cepstral coefficient features; inputting the Mel-frequency cepstral coefficient features into a Transformer encoder-decoder network, wherein the Transformer encoder-decoder network implements feature encoding by calculating attention weights between a query matrix, a key-value matrix, and a value matrix to obtain encoding features;

[0024] The coding feature is input into a bidirectional intention-slot joint learning model, the bidirectional intention-slot joint learning model calculates the intention recognition probability based on the coding feature, the intention recognition probability and the coding feature are input into a slot recognition module to obtain a slot sequence probability, and the intention recognition probability is multiplied by the slot sequence probability to obtain a semantic understanding probability distribution;

[0025] Construct a semantic dependency analysis graph, map the semantic understanding probability distribution to the node features of the semantic dependency analysis graph, iteratively update the node features based on the graph neural network, and obtain updated node features by aggregating weighted features of adjacent nodes and performing nonlinear transformation in each iteration. After multiple rounds of iterations, a graph representation containing complete semantic dependency relationships is obtained;

[0026] Matching the graph representation with a preset control instruction template library, calculating the similarity between the semantic parameters in the graph representation and the parameters of each template to obtain a matching score, and selecting the template with the highest matching score as the target control instruction template; constructing a state transfer matrix based on the historical dialogue state record, performing conflict detection between the target control instruction template and the state transfer matrix, and eliminating instruction conflicts;

[0027] The control instruction content is extracted from the target control instruction template, and the target experimental equipment to be controlled and the corresponding control action are determined according to the control instruction content.

[0028] The bidirectional intention-slot joint learning model calculates the intention recognition probability based on the coding feature, inputs the intention recognition probability and the coding feature into the slot recognition module to obtain the slot sequence probability, and multiplies the intention recognition probability by the slot sequence probability to obtain the semantic understanding probability distribution, including:

[0029] Constructing a bidirectional long short-term memory network, the bidirectional long short-term memory network includes a forward long short-term memory network unit and a backward long short-term memory network unit, inputting the encoding features into the forward long short-term memory network unit and the backward long short-term memory network unit respectively, obtaining a forward intention feature vector and a backward intention feature vector, and concatenating the forward intention feature vector with the backward intention feature vector to obtain a bidirectional intention feature vector;

[0030] Constructing an intention attention mechanism based on the bidirectional intention feature vector, performing a nonlinear transformation on the bidirectional intention feature vector through a trainable weight matrix and a bias vector, calculating an attention weight distribution, performing a weighted summation of the attention weight distribution and the bidirectional intention feature vector to obtain an intention context vector, and calculating the intention recognition probability based on the intention context vector through a softmax function;

[0031] The intention context vector is concatenated with the hidden feature sequence corresponding to the coding feature, and the concatenated result is input into a gated fusion unit. The gated fusion unit calculates a gated vector through a trainable gated parameter matrix, and adaptively fuses the gated vector with the coding feature to obtain a fused slot feature representation.

[0032] A conditional random field model is constructed, and the fused slot feature representation is input into the conditional random field model. The transfer scores between adjacent slot labels are calculated based on the state transition matrix, and the slot emission score of each time step is calculated. The transfer score and the emission score are added to obtain the overall score of the sequence, and the conditional probability of the slot labeling sequence is calculated by normalization.

[0033] The processing server generates a collaborative control strategy for the experimental equipment according to the control instruction content and the equipment status evaluation result. The collaborative control strategy includes the control timing, control parameters, and data interaction mode of multiple experimental equipment.

[0034] Constructing a dependency graph of experimental equipment, the dependency graph includes a set of equipment nodes and a set of inter-equipment dependency edges, calculating dependency weights based on inter-equipment data dependency, control coupling, and timing constraints, and assigning the dependency weights to each edge in the set of dependency edges;

[0035] Calculate the device control timing based on the dependency graph, for each device node, calculate the earliest start time of the device by the earliest completion time of its predecessor node and the device's own execution time, calculate the latest start time of the device by the latest completion time of its successor node and the device's own execution time, and calculate the device execution time margin based on the earliest start time and the latest start time;

[0036] The execution time margin of the equipment is input as a timing constraint into a multi-objective optimization model, the objective function of the multi-objective optimization model includes equipment operation efficiency, energy consumption and task completion time, the constraint conditions of the multi-objective optimization model include equipment operation parameter range constraints and control conflict constraints between equipment, and the multi-objective optimization model is solved by a genetic algorithm to obtain the optimal control parameters of the equipment;

[0037] A communication message structure between devices is constructed based on the optimal control parameters, and the communication message structure includes device identification, message type, data content, timestamp, and priority information. A multi-device collaborative control strategy is generated based on the device execution time margin, the optimal control parameters, and the communication message structure, and the collaborative control strategy includes control timing, control parameters, and data interaction methods for multiple experimental devices.

[0038] A second aspect of an embodiment of the present invention provides a smart laboratory human-computer interaction control system, including:

[0039] The first unit is used to collect image information of multiple experimental equipment in the laboratory through multiple visual acquisition devices set up in the laboratory, and transmit the image information to a processing server; the processing server analyzes and processes the image information based on a preset equipment status deep learning model, and identifies the operating status parameters of the multiple experimental equipment, wherein the operating status parameters include equipment switch status, equipment operating indicator status, equipment display status, and equipment fault indication status; the processing server compares the identified operating status parameters with preset equipment normal operating parameters to generate an equipment status evaluation result;

[0040] The second unit is used to collect the user's voice command information through a voice collection device set up in the laboratory, and transmit the voice command information to the processing server; the processing server performs semantic analysis on the voice command information, extracts the control command content, and determines the target experimental equipment to be controlled and the corresponding control action according to the control command content, wherein the control action includes equipment power on and off, equipment parameter adjustment, and equipment data collection; after determining the legitimacy of the control command content, the processing server combines the equipment status evaluation result to determine whether the target experimental equipment meets the conditions for executing the control action;

[0041] The third unit is used for the processing server to generate a collaborative control strategy for the experimental equipment according to the control instruction content and the equipment status evaluation results, and the collaborative control strategy includes the control timing, control parameters, and data interaction mode of multiple experimental equipment; the processing server sends control instructions to the multiple experimental equipment through the laboratory field bus network to realize collaborative control of the multiple experimental equipment.

[0042] A third aspect of the embodiments of the present invention

[0043] An electronic device is provided, comprising:

[0044] processor;

[0045] a memory for storing processor-executable instructions;

[0046] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0047] A fourth aspect of the embodiments of the present invention is:

[0048] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0049] The beneficial effects of this application are as follows:

[0050] Intelligent control: Use deep learning models to automatically identify the status of experimental equipment and control it according to voice commands to achieve automation and intelligence of the experimental process.

[0051] Safe and reliable: Through equipment status evaluation and control instruction legitimacy judgment, the safe operation of experimental equipment is ensured to avoid risks caused by misoperation.

[0052] Collaborative and efficient: Supports collaborative control of multiple experimental devices, optimizes control timing and parameters, improves experimental efficiency, and realizes data interaction between devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flow chart of a human-computer interaction control method for a smart laboratory according to an embodiment of the present invention;

[0054] Figure 2 It is a structural schematic diagram of the intelligent laboratory human-computer interaction control system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0057] Figure 1 FIG. 1 is a flow chart of a human-computer interaction control method for a smart laboratory according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0058] S101. Collect image information of multiple experimental equipment in the laboratory through multiple visual acquisition devices set up in the laboratory, and transmit the image information to the processing server; the processing server analyzes and processes the image information based on a preset equipment status deep learning model, and identifies the operating status parameters of the multiple experimental equipment, the operating status parameters including the equipment switch status, the equipment operating indicator status, the equipment display status, and the equipment fault indication status; the processing server compares the identified operating status parameters with the preset equipment normal operating parameters to generate an equipment status evaluation result;

[0059] S102. The voice command information of the user is collected by a voice collection device arranged in the laboratory, and the voice command information is transmitted to the processing server; the processing server performs semantic analysis on the voice command information, extracts the control command content, and determines the target experimental equipment to be controlled and the corresponding control action according to the control command content, wherein the control action includes equipment power on / off, equipment parameter adjustment, and equipment data collection; after determining the legitimacy of the control command content, the processing server judges whether the target experimental equipment meets the conditions for executing the control action in combination with the equipment status evaluation result;

[0060] S103. The processing server generates a collaborative control strategy for the experimental equipment based on the control instruction content and the equipment status evaluation results, and the collaborative control strategy includes the control timing, control parameters, and data interaction methods of multiple experimental equipment; the processing server sends control instructions to the multiple experimental equipment through the laboratory field bus network to achieve collaborative control of the multiple experimental equipment.

[0061] For example, the laboratory intelligent control system first collects personnel behavior data through multimodal sensors, including depth cameras collecting operation postures, spatial positioning devices tracking personnel positions, and equipment interaction interfaces recording operation sequences. The system constructs behavior feature vectors based on the collected data and analyzes the operator's behavior norms in real time.

[0062] Based on the analysis of personnel behavior, the system performs multi-level safety protection detection. First, it conducts a protective equipment wearing check to ensure that the operator correctly wears necessary protective equipment such as gloves, masks, and goggles; at the same time, it monitors whether the personnel are active in the designated safe operating area, such as fume hoods, clean workbenches, and other specified areas; in addition, it also evaluates the degree of match between the operating actions and standard operating specifications.

[0063] Based on the detection results, the system implements graded warning and interlocking control. When minor violations are found, the system prompts the operator through sound and light signals; for more serious violations, the system will restrict the use of related equipment; when major safety hazards are found, the system will trigger an emergency shutdown and notify the management personnel. The warning classification ensures that the system response matches the risk level.

[0064] At the human-machine collaborative control level, the system proactively provides normative guidance based on the current operation stage. It helps operators standardize their operation behaviors by displaying standard operation steps, providing error correction suggestions, and displaying safety precautions. At the same time, the system dynamically adjusts equipment operation parameters based on personnel behavior characteristics, establishes an operation confirmation mechanism to prevent misoperation, and activates emergency plans in abnormal situations.

[0065] To ensure the reliable operation of the system, a complete performance optimization mechanism has been established. The accuracy of behavior recognition is improved by optimizing the feature extraction algorithm, multiple backups are used to ensure data security, and a system self-check mechanism is set up to detect potential problems in a timely manner. At the same time, a fault-tolerant mechanism is established to maintain the normal operation of core functions when some functions fail.

[0066] The system realizes intelligent and standardized laboratory safety management through the organic combination of personnel behavior recognition, safety protection detection, early warning interlocking control, human-machine collaborative control and other links. The system can effectively prevent illegal operations, discover safety hazards in a timely manner, significantly improve the level of laboratory safety management, and provide all-round safety protection for experimental personnel.

[0067] In an optional implementation, the processing server analyzes and processes the image information based on a preset device status deep learning model, and identifying the operating status parameters of the multiple experimental devices includes:

[0068] The image information is input into a YOLOv5 target detection network of a device state deep learning model, wherein a spatial attention mechanism module is provided in the YOLOv5 target detection network, and the device area in the image information is feature enhanced by the spatial attention mechanism module, wherein the spatial attention mechanism module performs average pooling and maximum pooling on the input image information to obtain pooling features, and the pooling features are input into a seven-by-seven convolutional layer for convolution operation, and a spatial attention weight matrix is ​​generated by a sigmoid activation function;

[0069] The spatial attention weight matrix is ​​fused with the feature map corresponding to the image information to obtain a fused feature, the fused feature is input into a ResNet backbone network provided with a feature pyramid network, and multi-scale feature extraction is performed on the fused feature through the feature pyramid network, wherein the feature pyramid network performs upsampling on the fused feature, and performs feature splicing with the lateral connection features of the same layer, and outputs a multi-scale feature map through a convolution layer;

[0070] The multi-scale feature map is input into a multi-task learning framework, which includes a device switch state recognition branch, an operating indicator light state recognition branch, and a display state recognition branch, and the losses of the three recognition branches are jointly optimized through a weighted loss function; the optimized state recognition result is input into a bidirectional long short-term memory network to establish a time series state model, and the bidirectional long short-term memory network predicts the device state at the current moment based on the historical state sequence characteristics;

[0071] The state feature vector output by the bidirectional long short-term memory network is input into a Gaussian mixture model, wherein the Gaussian mixture model includes multiple Gaussian components, each of which has a corresponding mixing coefficient, a mean vector and a covariance matrix. By calculating the probability density distribution of the state feature vector, it is determined whether the equipment is in an abnormal state and the operating state parameters of the multiple experimental equipment are identified.

[0072] The experimental equipment operating status parameter recognition method based on deep learning automatically identifies the operating status parameters of multiple experimental equipment through image analysis technology, including equipment switch status, operating indicator light status, display screen status, etc., and determines whether the equipment is in an abnormal state.

[0073] First, obtain the image information of the experimental equipment. For example, use a camera to collect images of multiple experimental equipment in the laboratory, set the image resolution to 1920*1080 pixels, and the image format to JPEG.

[0074] Then, the acquired image information is input into the YOLOv5 target detection network of the device status deep learning model. A spatial attention mechanism module is provided in the network to enhance the features of the device area in the image information. Specifically, the spatial attention mechanism module performs average pooling and maximum pooling on the input image information to obtain pooled features. For example, a convolution kernel of size 3x3 is used for average pooling and maximum pooling operations. The obtained pooled features are input into a seven-by-seven convolution layer for convolution operation, for example, a convolution layer with a convolution kernel size of 7x7 and a step size of 1 is used. The spatial attention weight matrix is ​​then generated through the sigmoid activation function. For example, the sigmoid activation function maps the output value of the convolution layer to between 0 and 1 to generate a spatial attention weight matrix.

[0075] Next, the spatial attention weight matrix is ​​fused with the feature map corresponding to the image information to obtain the fused features. For example, the spatial attention weight matrix is ​​element-wise multiplied with the feature map extracted by the YOLOv5 network. Then, the fused features are input into the ResNet backbone network equipped with a feature pyramid network (FPN). FPN performs multi-scale feature extraction on the fused features. Specifically, FPN upsamples the fused features, for example, using bilinear interpolation for upsampling. And the upsampled features are feature spliced ​​with the lateral connection features of the same layer. For example, the upsampled features of FPN are spliced ​​with the feature maps of the same resolution in the ResNet network. Finally, the multi-scale feature map is output through the convolution layer. For example, a 1x1 convolution layer is used to process the spliced ​​features and output a multi-scale feature map.

[0076] After that, the multi-scale feature map is input into the multi-task learning framework. The framework includes the device switch state recognition branch, the operation indicator state recognition branch, and the display state recognition branch. The losses of the three recognition branches are jointly optimized through the weighted loss function. For example, the cross entropy loss function is used as the loss function of each branch, and different weights are set according to the importance of each branch. The final loss function is the weighted sum of the loss functions of the three branches. The optimized state recognition results are input into the bidirectional long short-term memory network (BiLSTM) to establish a time series state model. BiLSTM predicts the device state at the current moment based on the historical state sequence features. For example, the device state recognition results of the past 10 moments are used as the input sequence and input into the BiLSTM network to predict the device state at the current moment.

[0077] Finally, the state feature vector output by BiLSTM is input into the Gaussian mixture model (GMM). GMM includes multiple Gaussian components, each of which has a corresponding mixing coefficient, mean vector and covariance matrix. By calculating the probability density distribution of the state feature vector, it is determined whether the device is in an abnormal state and the operating state parameters of multiple experimental devices are identified. For example, if the probability density of the state feature vector in GMM is lower than the preset threshold, the device is judged to be in an abnormal state. Assuming that the switch state recognition result of a certain experimental device is "on", the operation indicator state recognition result is "green", and the display state recognition result is "normal display", the operating state parameters of the device are {switch state: "on", operation indicator state: "green", display state: "normal display"}.

[0078] The solution of this application can:

[0079] Improve recognition accuracy: The spatial attention mechanism module enhances the characteristics of the equipment area, FPN extracts multi-scale features, the multi-task learning framework jointly optimizes multiple recognition tasks, BiLSTM takes into account historical status information, and GMM can identify abnormal states. These technical means jointly improve the accuracy of equipment operation status parameter recognition. Realize automatic recognition: This method automatically identifies equipment operation status parameters through image analysis technology without manual intervention, realizes automatic recognition, and reduces labor costs. Improve equipment management efficiency: This method can monitor the equipment operation status in real time, detect abnormal conditions in a timely manner, provide data support for equipment maintenance and management, and improve equipment management efficiency.

[0080] In an optional implementation, the method further includes training the Gaussian mixture model:

[0081] The K-means clustering algorithm is used to pre-group the experimental equipment state training data acquired in advance, and the initial mean vector of each Gaussian component is calculated according to the pre-grouping result, and the initial mean vector is obtained by calculating the mean of all samples in each group; the initial covariance matrix of each Gaussian component is calculated, and the initial covariance matrix is ​​obtained by calculating the difference product of the samples in each group and the corresponding initial mean vector; the initial mixing coefficient of each Gaussian component is calculated, and the initial mixing coefficient is obtained by calculating the ratio of the number of samples in each group to the total number of samples;

[0082] The initial mean vector, the initial covariance matrix, and the initial mixing coefficient are input into the expectation maximization algorithm, and the posterior probability of each sample belonging to each Gaussian component is calculated in the expectation step, and the posterior probability is obtained by calculating the product of the mixing coefficient of each Gaussian component and the probability density and normalizing them; in the maximization step, the mean vector, covariance matrix, and mixing coefficient of the Gaussian component are updated based on the posterior probability, and the updating process is iteratively performed until the model parameters converge;

[0083] The converged Gaussian mixture model is applied to abnormal state detection, by calculating the logarithmic likelihood and Mahalanobis distance of the sample to be detected, wherein the logarithmic likelihood is obtained by calculating the logarithmic value of the weighted probability density of the sample on all Gaussian components, and the Mahalanobis distance is obtained by calculating the minimum value of the standardized distance from the sample to the center of each Gaussian component; the logarithmic likelihood and the Mahalanobis distance are weightedly combined by a weight coefficient to obtain an abnormal detection score;

[0084] A dynamic threshold is calculated based on a sliding time window, and the dynamic threshold is obtained by calculating the weighted sum of the mean and standard deviation of the historical detection scores; the temporal feature anomaly detection score and the spatial feature anomaly detection score are multimodally fused, and the multimodal fusion is achieved by assigning fusion weights to the temporal anomaly detection score and the spatial anomaly detection score respectively and summing them. When the anomaly detection score after the multimodal fusion exceeds the dynamic threshold, it is determined to be an abnormal state.

[0085] The abnormal state detection method of experimental equipment based on Gaussian mixture model and multimodal feature fusion is as follows:

[0086] First, obtain the experimental equipment status training data. This data can include various operating parameters of the equipment, such as temperature, pressure, current, voltage, vibration frequency, etc. For example, we collected 1,000 sample data of an experimental equipment in normal operation, and each sample contained 5 parameters: temperature, pressure, current, voltage, and vibration frequency.

[0087] Next, the acquired experimental equipment status training data is preprocessed. The purpose of preprocessing is to remove noise and outliers in the data and normalize the data for subsequent model training. For example, we can use mean filtering to remove noise, use the 3-Sigma criterion to remove outliers, and use the Min-Max normalization method to normalize the data to the [0,1] interval.

[0088] Then, use the K-means clustering algorithm to pre-cluster the preprocessed data. The goal of the K-means algorithm is to divide the data into K clusters so that the data within each cluster is as similar as possible and the data between clusters are as different as possible. The choice of K value needs to be adjusted according to the specific situation. For example, we divide the data into 3 clusters, representing different operating states of the equipment.

[0089] According to the results of pre-clustering, calculate the initial parameters of each Gaussian component. For each cluster, calculate the mean of all samples in the cluster as the initial mean vector of the Gaussian component. Calculate the difference product of the samples in the cluster and the corresponding initial mean vector as the initial covariance matrix of the Gaussian component. Calculate the ratio of the number of samples in the cluster to the total number of samples as the initial mixing coefficient of the Gaussian component. For example, the first cluster contains 300 samples, so its initial mixing coefficient is 300 / 1000=0.3.

[0090] The Gaussian mixture model is trained using the expectation-maximization (EM) algorithm. The EM algorithm is an iterative algorithm for estimating the parameters of a Gaussian mixture model. In the expectation step, the posterior probability of each sample belonging to each Gaussian component is calculated. The posterior probability represents the probability that the sample belongs to a certain Gaussian component given the sample value. In the maximization step, the mean vector, covariance matrix, and mixing coefficients of the Gaussian component are updated based on the posterior probability. The expectation step and the maximization step are iterated until the model parameters converge.

[0091] The trained Gaussian mixture model is applied to abnormal state detection. For the sample to be detected, its log likelihood and Mahalanobis distance are calculated. The log likelihood represents the logarithmic value of the weighted probability density of the sample on all Gaussian components. The Mahalanobis distance represents the minimum value of the standardized distance from the sample to the center of each Gaussian component. The log likelihood and Mahalanobis distance are weighted and combined by the weight coefficient to obtain the abnormal detection score. For example, the weight coefficient of the log likelihood is 0.7, and the weight coefficient of the Mahalanobis distance is 0.3.

[0092] The dynamic threshold is calculated based on a sliding time window. The sliding time window refers to the data within a period of time, such as the data in the past hour. The dynamic threshold is obtained by calculating the weighted sum of the mean and standard deviation of the historical detection scores. For example, the dynamic threshold is set to the mean of the historical detection scores plus 1 times the standard deviation.

[0093] Multimodal fusion is performed on the anomaly detection scores of temporal features and spatial features. Temporal features can be time series data, such as detection scores over a period of time in the past. Spatial features can be detection scores at different locations of a device. Multimodal fusion is achieved by assigning fusion weights to the temporal anomaly detection scores and spatial anomaly detection scores respectively and summing them. For example, the fusion weight of the temporal features is 0.6, and the fusion weight of the spatial features is 0.4.

[0094] When the anomaly detection score after multimodal fusion exceeds the dynamic threshold, it is determined to be an abnormal state.

[0095] The solution of this application can:

[0096] Improve detection accuracy: This method combines Gaussian mixture model and multimodal feature fusion, which can more accurately identify abnormal status of experimental equipment and reduce false positives and false negatives. Strong adaptability: This method uses dynamic thresholds and multimodal fusion, which can adapt to different experimental environments and equipment types and has strong generalization capabilities. Good real-time performance: This method calculates dynamic thresholds based on sliding time windows, which can monitor the operating status of experimental equipment in real time and detect abnormal situations in a timely manner.

[0097] In an optional implementation, the processing server performs semantic analysis on the voice instruction information, extracts the control instruction content, and determines the target experimental device to be controlled and the corresponding control action according to the control instruction content, including:

[0098] Extracting Mel-frequency cepstral coefficient features from the input voice command information to obtain Mel-frequency cepstral coefficient features; inputting the Mel-frequency cepstral coefficient features into a Transformer encoder-decoder network, wherein the Transformer encoder-decoder network implements feature encoding by calculating attention weights between a query matrix, a key-value matrix, and a value matrix to obtain encoding features;

[0099] The coding feature is input into a bidirectional intention-slot joint learning model, the bidirectional intention-slot joint learning model calculates the intention recognition probability based on the coding feature, the intention recognition probability and the coding feature are input into a slot recognition module to obtain a slot sequence probability, and the intention recognition probability is multiplied by the slot sequence probability to obtain a semantic understanding probability distribution;

[0100] Construct a semantic dependency analysis graph, map the semantic understanding probability distribution to the node features of the semantic dependency analysis graph, iteratively update the node features based on the graph neural network, and obtain updated node features by aggregating weighted features of adjacent nodes and performing nonlinear transformation in each iteration. After multiple rounds of iterations, a graph representation containing complete semantic dependency relationships is obtained;

[0101] Matching the graph representation with a preset control instruction template library, calculating the similarity between the semantic parameters in the graph representation and the parameters of each template to obtain a matching score, and selecting the template with the highest matching score as the target control instruction template; constructing a state transfer matrix based on the historical dialogue state record, performing conflict detection between the target control instruction template and the state transfer matrix, and eliminating instruction conflicts;

[0102] The control instruction content is extracted from the target control instruction template, and the target experimental equipment to be controlled and the corresponding control action are determined according to the control instruction content.

[0103] The method and system for voice control of experimental equipment are used to accurately control experimental equipment and improve experimental efficiency. The core of this method is to combine speech recognition, natural language understanding and graph neural network technology, which can accurately understand the user's voice instructions and convert them into control actions for experimental equipment.

[0104] First, the received voice command information is preprocessed, including noise reduction, endpoint detection and other operations to improve the accuracy of speech recognition. For example, a noisy speech "Turn on... uh... the heater... to... 50 degrees" becomes a clear speech "Turn on the heater to 50 degrees" after preprocessing. Then, the Mel-frequency cepstral coefficient (MFCC) features of the speech signal are extracted. MFCC features can well reflect the spectral characteristics of speech. For example, the MFCC features of the speech "Turn on the heater to 50 degrees" can be represented as a 13-dimensional vector sequence, where each frame of speech corresponds to a 13-dimensional vector.

[0105] The extracted MFCC features are input into the Transformer encoder-decoder network. The Transformer network captures long-range dependencies in the speech signal through the self-attention mechanism and encodes the speech features into a higher-level representation. For example, if the MFCC feature sequence of "turn on the heater to 50 degrees" is input, the Transformer network outputs a feature sequence containing semantic information.

[0106] The output features of the Transformer network are input into the bidirectional intent-slot joint learning model. The model performs intent recognition and slot filling simultaneously to understand the intent and key parameters of the user's command. For example, for the voice command "turn on the heater to 50 degrees", the intent recognition result is "control device" and the slot filling result is "device: heater, temperature: 50 degrees". Assuming the intent recognition probability is 0.95 and the slot sequence probability is 0.92, the semantic understanding probability distribution is 0.95*0.92=0.874.

[0107] Next, build a semantic dependency analysis graph and map the semantic understanding probability distribution to the node features of the graph. For example, for the intent of "control device", you can build a graph containing nodes such as "control", "device", "heater", "temperature", and "50 degrees". Map the calculated semantic understanding probability distribution of 0.874 to the features of the corresponding node.

[0108] Use graph neural networks to iteratively update node features. Graph neural networks update node features by aggregating information from adjacent nodes to capture the semantic dependencies between nodes. For example, the features of the "heater" node are affected by the features of the "control" and "temperature" nodes. After multiple rounds of iterative updates, the features of each node in the graph contain more complete semantic information.

[0109] Match the updated graph representation with the preset control instruction template library. For example, the preset control instruction template library contains a template such as "Turn on {device} to {temperature}". Calculate the similarity between the graph representation and each template, and select the template with the highest similarity as the target control instruction template. Assume that the matching score of the "Turn on {device} to {temperature}" template is 0.9, and the scores of other templates are all lower than 0.9, then select this template as the target control instruction template.

[0110] A state transition matrix is ​​constructed based on the historical dialogue state records to detect and eliminate command conflicts. For example, if the previous command is "turn off the heater", then the current command "turn on the heater to 50 degrees" has no conflict. If the previous command is "set the heater temperature to 30 degrees", then the current command "turn on the heater to 50 degrees" may have a conflict, which needs to be judged and processed according to the state transition matrix.

[0111] Extract the control instruction content from the target control instruction template. For example, extract "Device: Heater, Temperature: 50 degrees" from the template "Turn on {device} to {temperature}". Determine the target experimental device to be controlled and the corresponding control action based on the extracted control instruction content. For example, send the control instruction to the heater control system to set the temperature to 50 degrees.

[0112] The solution of this application can:

[0113] Improved the accuracy and efficiency of experimental control. By accurately understanding the semantics of voice commands, human errors can be avoided and automated control of experimental equipment can be achieved, thereby improving experimental efficiency. Enhanced the flexibility of experimental control. Users can use natural language voice commands to control experimental equipment without having to remember complex command formats, thereby improving the convenience of experimental operation. Reduced the learning cost of experimental operation. Users can control experimental equipment by voice without having to learn special control software or programming languages, reducing the training cost of experimental personnel.

[0114] In an optional implementation, the bidirectional intent-slot joint learning model calculates the intent recognition probability based on the coding feature, inputs the intent recognition probability and the coding feature into the slot recognition module to obtain the slot sequence probability, and multiplies the intent recognition probability by the slot sequence probability to obtain the semantic understanding probability distribution, including:

[0115] Constructing a bidirectional long short-term memory network, the bidirectional long short-term memory network includes a forward long short-term memory network unit and a backward long short-term memory network unit, inputting the encoding features into the forward long short-term memory network unit and the backward long short-term memory network unit respectively, obtaining a forward intention feature vector and a backward intention feature vector, and concatenating the forward intention feature vector with the backward intention feature vector to obtain a bidirectional intention feature vector;

[0116] Constructing an intention attention mechanism based on the bidirectional intention feature vector, performing a nonlinear transformation on the bidirectional intention feature vector through a trainable weight matrix and a bias vector, calculating an attention weight distribution, performing a weighted summation of the attention weight distribution and the bidirectional intention feature vector to obtain an intention context vector, and calculating the intention recognition probability based on the intention context vector through a softmax function;

[0117] The intention context vector is concatenated with the hidden feature sequence corresponding to the coding feature, and the concatenated result is input into a gated fusion unit. The gated fusion unit calculates a gated vector through a trainable gated parameter matrix, and adaptively fuses the gated vector with the coding feature to obtain a fused slot feature representation.

[0118] A conditional random field model is constructed, and the fused slot feature representation is input into the conditional random field model. The transfer scores between adjacent slot labels are calculated based on the state transition matrix, and the slot emission score of each time step is calculated. The transfer score and the emission score are added to obtain the overall score of the sequence, and the conditional probability of the slot labeling sequence is calculated by normalization.

[0119] An improved bidirectional intent-slot joint learning method is used to understand user intent and extract slot information from user sentences. The core of this method is to jointly learn the two tasks of intent recognition and slot filling, and improve the model performance through attention mechanism and gated fusion unit.

[0120] The encoded features are processed using a bidirectional long short-term memory network (BiLSTM). The encoded features are input into the forward LSTM and the backward LSTM respectively to obtain the forward and backward hidden state sequences. For example, the output of the forward LSTM is a 10x50 matrix, and the output of the backward LSTM is also a 10x50 matrix. The forward and backward hidden states are concatenated at each time step to obtain a 10x100 bidirectional intent feature vector.

[0121] Then, the intent attention mechanism is used to capture important intent information. The bidirectional intent feature vector is transformed nonlinearly, such as using the tanh activation function, and the attention weight is calculated for each time step. The attention weight is weighted summed with the bidirectional intent feature vector to obtain the intent context vector, which is a 100-dimensional vector. Finally, the softmax function is used to convert the intent context vector into an intent recognition probability distribution.

[0122] At the same time, the intent context vector is concatenated with the hidden feature sequence corresponding to the encoded feature (such as the output of BiLSTM). For example, the 100-dimensional intent context vector is concatenated with the 100-dimensional hidden feature of each time step to obtain a 10x200 matrix. The concatenated features are input into the gated fusion unit (GRU). GRU adaptively fuses intent information and slot information through a gating mechanism to obtain a fused slot feature representation, such as a 10x150 matrix.

[0123] Finally, the conditional random field (CRF) model is used to annotate the slots. The fused slot feature representation is input into the CRF model. The CRF model calculates the score of each slot label at each time step and considers the transition probability between adjacent slot labels. The probability distribution of the slot label sequence is obtained by calculating the scores of all possible slot label sequences and normalizing them using the softmax function.

[0124] The solution of this application can:

[0125] Improved the accuracy of intent recognition and slot filling. Through joint learning and information sharing, the model can better capture the correlation between intent and slots, thereby improving overall performance. Enhanced the robustness of the model. The attention mechanism and gated fusion unit can effectively filter noise and irrelevant information, making the model more robust to perturbations of input sentences. Simplified the model training process. Compared with training intent recognition and slot filling models separately, the joint learning method only needs to train one model, reducing training cost and complexity.

[0126] In an optional implementation, the processing server generates a collaborative control strategy for the experimental equipment according to the control instruction content and the equipment status evaluation result, wherein the collaborative control strategy includes control timing, control parameters, and data interaction methods of multiple experimental equipment, including:

[0127] Constructing a dependency graph of experimental equipment, the dependency graph includes a set of equipment nodes and a set of inter-equipment dependency edges, calculating dependency weights based on inter-equipment data dependency, control coupling, and timing constraints, and assigning the dependency weights to each edge in the set of dependency edges;

[0128] Calculate the device control timing based on the dependency graph, for each device node, calculate the earliest start time of the device by the earliest completion time of its predecessor node and the device's own execution time, calculate the latest start time of the device by the latest completion time of its successor node and the device's own execution time, and calculate the device execution time margin based on the earliest start time and the latest start time;

[0129] The execution time margin of the equipment is input as a timing constraint into a multi-objective optimization model, the objective function of the multi-objective optimization model includes equipment operation efficiency, energy consumption and task completion time, the constraint conditions of the multi-objective optimization model include equipment operation parameter range constraints and control conflict constraints between equipment, and the multi-objective optimization model is solved by a genetic algorithm to obtain the optimal control parameters of the equipment;

[0130] A communication message structure between devices is constructed based on the optimal control parameters, and the communication message structure includes device identification, message type, data content, timestamp, and priority information. A multi-device collaborative control strategy is generated based on the device execution time margin, the optimal control parameters, and the communication message structure, and the collaborative control strategy includes control timing, control parameters, and data interaction methods for multiple experimental devices.

[0131] The invention discloses a collaborative control method of experimental equipment, which is used for automatic control of multiple experimental equipment in complex experimental scenarios, so as to improve experimental efficiency, reduce energy consumption, and ensure the completion of experimental tasks on time.

[0132] First, a dependency graph of experimental equipment is constructed. The dependency graph describes the dependencies between different experimental equipment so that the subsequent generation of control strategies can take into account the mutual influence between equipment. The dependency graph contains a set of device nodes and a set of inter-device dependency edges. The set of device nodes represents all the equipment involved in the experiment. The set of inter-device dependency edges represents the dependency between devices. For example, if device A needs the data of device B to start running, there is a dependency edge pointing from device B to device A. Each dependency edge is assigned a dependency weight, which is calculated based on the data dependency, control coupling and timing constraints between devices. For example, if device A strongly depends on the data of device B, the weight of the dependency edge between them will be higher. Suppose there are three devices: device A, device B and device C. Device A depends on the output data of device B, and device C depends on the output data of device A and device B. Then the weight of the dependency edge from device B to device A is set to 0.8, the weight of the dependency edge from device A to device C is set to 0.6, and the weight of the dependency edge from device B to device C is set to 0.5.

[0133] Next, the device control timing is calculated based on the constructed dependency graph. For each device node, its earliest start time and latest start time need to be calculated. The earliest start time of a device is determined by the earliest completion time of all its predecessor nodes, and the largest one is taken as the earliest start time of the device. The latest start time of a device is determined by the latest start time of all its successor nodes, and the smallest one is taken and subtracted from the execution time of the device itself to obtain the latest start time of the device. The execution time margin of the device is obtained by subtracting the earliest start time from the latest start time. Continuing with the previous example, assume that the execution time of device B is 10 minutes, the execution time of device A is 5 minutes, and the execution time of device C is 7 minutes. If the earliest start time of device B is 0 minutes, then the earliest start time of device A is 10 minutes, and the earliest start time of device C is 10+5=15 minutes. Assuming that the latest start time of device C is 25 minutes, then the latest start time of device A is 25-7=18 minutes, and the latest start time of device B is 18-5=13 minutes. Therefore, the execution time margin of device B is 13-0=13 minutes, the execution time margin of device A is 18-10=8 minutes, and the execution time margin of device C is 25-15=10 minutes.

[0134] Then, the calculated device execution time margin is input into the multi-objective optimization model as a timing constraint. The objective function of the model includes device operation efficiency, energy consumption, and task completion time. The constraints include device operation parameter range constraints and inter-device control conflict constraints. The multi-objective optimization model is solved using a genetic algorithm to obtain the optimal control parameters for each device. For example, for device A, the possible optimal control parameters are the temperature set to 25 degrees Celsius and the pressure set to 1 standard atmosphere. For device B, the possible optimal control parameter is the speed set to 1000 rpm. For device C, the possible optimal control parameter is the voltage set to 220 volts.

[0135] Finally, the communication message structure between devices is constructed based on the obtained optimal control parameters. The communication message structure contains device identification, message type, data content, timestamp, and priority information. Based on the device execution time margin, optimal control parameters, and communication message structure, a multi-device collaborative control strategy is generated. The strategy includes the control timing, control parameters, and data interaction methods of multiple experimental devices. For example, the control strategy may instruct device B to start running at 0 minutes and send data to device A at 10 minutes. Device A starts running after receiving the data at 10 minutes, and sends the data to device C at 15 minutes. Device C starts running after receiving the data at 15 minutes.

[0136] The solution of this application can:

[0137] Improve experimental efficiency: By optimizing the equipment control timing and parameters, reduce equipment idle time and resource waste, thereby improving experimental efficiency. Reduce energy consumption: By optimizing equipment operating parameters, reduce equipment energy consumption and save experimental costs. Ensure that tasks are completed on time: By considering the dependencies and timing constraints between devices, ensure that experimental tasks can be completed on time.

[0138] Figure 2 FIG. 1 is a schematic diagram of the structure of the human-computer interaction control system of the intelligent laboratory according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0139] The first unit is used to collect image information of multiple experimental equipment in the laboratory through multiple visual acquisition devices set up in the laboratory, and transmit the image information to a processing server; the processing server analyzes and processes the image information based on a preset equipment status deep learning model, and identifies the operating status parameters of the multiple experimental equipment, wherein the operating status parameters include equipment switch status, equipment operating indicator status, equipment display status, and equipment fault indication status; the processing server compares the identified operating status parameters with preset equipment normal operating parameters to generate an equipment status evaluation result;

[0140] The second unit is used to collect the user's voice command information through a voice collection device set up in the laboratory, and transmit the voice command information to the processing server; the processing server performs semantic analysis on the voice command information, extracts the control command content, and determines the target experimental equipment to be controlled and the corresponding control action according to the control command content, wherein the control action includes equipment power on and off, equipment parameter adjustment, and equipment data collection; after determining the legitimacy of the control command content, the processing server combines the equipment status evaluation result to determine whether the target experimental equipment meets the conditions for executing the control action;

[0141] The third unit is used for the processing server to generate a collaborative control strategy for the experimental equipment according to the control instruction content and the equipment status evaluation results, and the collaborative control strategy includes the control timing, control parameters, and data interaction mode of multiple experimental equipment; the processing server sends control instructions to the multiple experimental equipment through the laboratory field bus network to realize collaborative control of the multiple experimental equipment.

[0142] According to a third aspect of the embodiments of the present invention,

[0143] An electronic device is provided, comprising:

[0144] processor;

[0145] a memory for storing processor-executable instructions;

[0146] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0147] A fourth aspect of the embodiments of the present invention is:

[0148] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0149] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The intelligent laboratory human-computer interaction control method is characterized by: include: Collecting image information of multiple experimental equipment in the laboratory through multiple visual acquisition devices arranged in the laboratory, and transmitting the image information to a processing server; The processing server analyzes and processes the image information based on a preset device status deep learning model, identifies the operating status parameters of the multiple experimental devices, and the operating status parameters include the device switch status, the device operating indicator status, the device display status, and the device fault indication status; the processing server compares the identified operating status parameters with the preset device normal operating parameters to generate an equipment status evaluation result; Collecting the user's voice command information through a voice collection device arranged in the laboratory, and transmitting the voice command information to the processing server; The processing server performs semantic analysis on the voice command information, extracts the control command content, and determines the target experimental equipment to be controlled and the corresponding control action according to the control command content, wherein the control action includes equipment power on / off, equipment parameter adjustment, and equipment data collection; after determining the legitimacy of the control command content, the processing server determines whether the target experimental equipment meets the conditions for executing the control action in combination with the equipment status evaluation result; The processing server generates a collaborative control strategy for the experimental equipment according to the control instruction content and the equipment status evaluation result, wherein the collaborative control strategy includes the control timing, control parameters, and data interaction mode of multiple experimental equipment; the processing server sends the control instruction to the multiple experimental equipment through the laboratory field bus network to realize the collaborative control of the multiple experimental equipment; The processing server analyzes and processes the image information based on a preset device status deep learning model, and identifies the operating status parameters of the multiple experimental devices, including: The image information is input into a YOLOv5 target detection network of a device state deep learning model, wherein a spatial attention mechanism module is provided in the YOLOv5 target detection network, and the device area in the image information is feature enhanced by the spatial attention mechanism module, wherein the spatial attention mechanism module performs average pooling and maximum pooling on the input image information to obtain pooling features, and the pooling features are input into a seven-by-seven convolutional layer for convolution operation, and a spatial attention weight matrix is ​​generated by a sigmoid activation function; The spatial attention weight matrix is ​​fused with the feature map corresponding to the image information to obtain a fused feature, the fused feature is input into a ResNet backbone network provided with a feature pyramid network, and multi-scale feature extraction is performed on the fused feature through the feature pyramid network, wherein the feature pyramid network performs upsampling on the fused feature, and performs feature splicing with the lateral connection features of the same layer, and outputs a multi-scale feature map through a convolution layer; The multi-scale feature map is input into a multi-task learning framework, which includes a device switch state recognition branch, an operating indicator light state recognition branch, and a display state recognition branch, and the losses of the three recognition branches are jointly optimized through a weighted loss function; the optimized state recognition result is input into a bidirectional long short-term memory network to establish a time series state model, and the bidirectional long short-term memory network predicts the device state at the current moment based on the historical state sequence characteristics; The state feature vector output by the bidirectional long short-term memory network is input into a Gaussian mixture model, wherein the Gaussian mixture model includes multiple Gaussian components, each of which has a corresponding mixing coefficient, a mean vector and a covariance matrix. By calculating the probability density distribution of the state feature vector, it is determined whether the equipment is in an abnormal state and the operating state parameters of the multiple experimental equipment are identified.

2. The method according to claim 1, characterized in that The method further comprises training the Gaussian mixture model: The K-means clustering algorithm is used to pre-group the experimental equipment state training data acquired in advance, and the initial mean vector of each Gaussian component is calculated according to the pre-grouping result, and the initial mean vector is obtained by calculating the mean of all samples in each group; Calculate the initial covariance matrix of each Gaussian component, where the initial covariance matrix is ​​obtained by calculating the difference product of the samples in each group and the corresponding initial mean vector; Calculating an initial mixing coefficient of each Gaussian component, wherein the initial mixing coefficient is obtained by calculating the ratio of the number of samples in each group to the total number of samples; The initial mean vector, the initial covariance matrix, and the initial mixing coefficient are input into the expectation maximization algorithm, and the posterior probability of each sample belonging to each Gaussian component is calculated in the expectation step, and the posterior probability is obtained by calculating the product of the mixing coefficient of each Gaussian component and the probability density and normalizing them; in the maximization step, the mean vector, covariance matrix, and mixing coefficient of the Gaussian component are updated based on the posterior probability, and the updating process is iteratively performed until the model parameters converge; The converged Gaussian mixture model is applied to abnormal state detection, by calculating the logarithmic likelihood and Mahalanobis distance of the sample to be detected, wherein the logarithmic likelihood is obtained by calculating the logarithmic value of the weighted probability density of the sample on all Gaussian components, and the Mahalanobis distance is obtained by calculating the minimum value of the standardized distance from the sample to the center of each Gaussian component; the logarithmic likelihood and the Mahalanobis distance are weightedly combined by a weight coefficient to obtain an abnormal detection score; A dynamic threshold is calculated based on a sliding time window, and the dynamic threshold is obtained by calculating the weighted sum of the mean and standard deviation of the historical detection scores; the temporal feature anomaly detection score and the spatial feature anomaly detection score are multimodally fused, and the multimodal fusion is achieved by assigning fusion weights to the temporal anomaly detection score and the spatial anomaly detection score respectively and summing them. When the anomaly detection score after the multimodal fusion exceeds the dynamic threshold, it is determined to be an abnormal state.

3. The method according to claim 1, characterized in that The processing server performs semantic analysis on the voice command information, extracts the control command content, and determines the target experimental equipment to be controlled and the corresponding control action according to the control command content, including: Extracting Mel-frequency cepstral coefficient features from the input voice command information to obtain Mel-frequency cepstral coefficient features; inputting the Mel-frequency cepstral coefficient features into a Transformer encoder-decoder network, wherein the Transformer encoder-decoder network implements feature encoding by calculating attention weights between a query matrix, a key-value matrix, and a value matrix to obtain encoding features; The coding feature is input into a bidirectional intention-slot joint learning model, the bidirectional intention-slot joint learning model calculates the intention recognition probability based on the coding feature, the intention recognition probability and the coding feature are input into a slot recognition module to obtain a slot sequence probability, and the intention recognition probability is multiplied by the slot sequence probability to obtain a semantic understanding probability distribution; Construct a semantic dependency analysis graph, map the semantic understanding probability distribution to the node features of the semantic dependency analysis graph, iteratively update the node features based on the graph neural network, and obtain updated node features by aggregating weighted features of adjacent nodes and performing nonlinear transformation in each iteration. After multiple rounds of iterations, a graph representation containing complete semantic dependency relationships is obtained; Matching the graph representation with a preset control instruction template library, calculating the similarity between the semantic parameters in the graph representation and the parameters of each template to obtain a matching score, and selecting the template with the highest matching score as the target control instruction template; constructing a state transfer matrix based on the historical dialogue state record, performing conflict detection between the target control instruction template and the state transfer matrix, and eliminating instruction conflicts; The control instruction content is extracted from the target control instruction template, and the target experimental equipment to be controlled and the corresponding control action are determined according to the control instruction content.

4. The method according to claim 3, characterized in that The bidirectional intention-slot joint learning model calculates the intention recognition probability based on the coding feature, inputs the intention recognition probability and the coding feature into the slot recognition module to obtain the slot sequence probability, and multiplies the intention recognition probability by the slot sequence probability to obtain the semantic understanding probability distribution, including: Constructing a bidirectional long short-term memory network, the bidirectional long short-term memory network includes a forward long short-term memory network unit and a backward long short-term memory network unit, inputting the encoding features into the forward long short-term memory network unit and the backward long short-term memory network unit respectively, obtaining a forward intention feature vector and a backward intention feature vector, and concatenating the forward intention feature vector with the backward intention feature vector to obtain a bidirectional intention feature vector; Constructing an intention attention mechanism based on the bidirectional intention feature vector, performing a nonlinear transformation on the bidirectional intention feature vector through a trainable weight matrix and a bias vector, calculating an attention weight distribution, performing a weighted summation of the attention weight distribution and the bidirectional intention feature vector to obtain an intention context vector, and calculating the intention recognition probability based on the intention context vector through a softmax function; The intention context vector is concatenated with the hidden feature sequence corresponding to the coding feature, and the concatenated result is input into a gated fusion unit. The gated fusion unit calculates a gated vector through a trainable gated parameter matrix, and adaptively fuses the gated vector with the coding feature to obtain a fused slot feature representation. A conditional random field model is constructed, and the fused slot feature representation is input into the conditional random field model. The transfer scores between adjacent slot labels are calculated based on the state transition matrix, and the slot emission score of each time step is calculated. The transfer score and the emission score are added to obtain the overall score of the sequence, and the conditional probability of the slot labeling sequence is calculated by normalization.

5. The method according to claim 1, characterized in that: The processing server generates a collaborative control strategy for the experimental equipment according to the control instruction content and the equipment status evaluation result. The collaborative control strategy includes the control timing, control parameters, and data interaction mode of multiple experimental equipment. Constructing a dependency graph of experimental equipment, the dependency graph includes a set of equipment nodes and a set of inter-equipment dependency edges, calculating dependency weights based on inter-equipment data dependency, control coupling, and timing constraints, and assigning the dependency weights to each edge in the set of dependency edges; Calculate the device control timing based on the dependency graph, for each device node, calculate the earliest start time of the device by the earliest completion time of its predecessor node and the device's own execution time, calculate the latest start time of the device by the latest completion time of its successor node and the device's own execution time, and calculate the device execution time margin based on the earliest start time and the latest start time; The execution time margin of the equipment is input as a timing constraint into a multi-objective optimization model, the objective function of the multi-objective optimization model includes equipment operation efficiency, energy consumption and task completion time, the constraint conditions of the multi-objective optimization model include equipment operation parameter range constraints and control conflict constraints between equipment, and the multi-objective optimization model is solved by a genetic algorithm to obtain the optimal control parameters of the equipment; A communication message structure between devices is constructed based on the optimal control parameters, and the communication message structure includes device identification, message type, data content, timestamp, and priority information. A multi-device collaborative control strategy is generated based on the device execution time margin, the optimal control parameters, and the communication message structure, and the collaborative control strategy includes control timing, control parameters, and data interaction methods for multiple experimental devices.

6. A smart laboratory human-computer interaction control system, used to implement the method as described in any one of claims 1 to 5, characterized in that: include: The first unit is used to collect image information of multiple experimental equipment in the laboratory through multiple visual collection devices arranged in the laboratory, and transmit the image information to a processing server; The processing server analyzes and processes the image information based on a preset device status deep learning model to identify the operating status parameters of the multiple experimental devices, wherein the operating status parameters include the device switch status, the device operation indicator light status, the device display screen status, and the device fault indication status; The processing server compares the identified operating status parameters with preset normal operating parameters of the equipment to generate an equipment status assessment result; The second unit is used to collect the user's voice command information through a voice collection device arranged in the laboratory, and transmit the voice command information to the processing server; The processing server performs semantic analysis on the voice command information, extracts the control command content, and determines the target experimental equipment to be controlled and the corresponding control action according to the control command content, wherein the control action includes equipment power on / off, equipment parameter adjustment, and equipment data collection; after determining the legitimacy of the control command content, the processing server determines whether the target experimental equipment meets the conditions for executing the control action in combination with the equipment status evaluation result; The third unit is used for the processing server to generate a collaborative control strategy for the experimental equipment according to the control instruction content and the equipment status evaluation results, and the collaborative control strategy includes the control timing, control parameters, and data interaction mode of multiple experimental equipment; the processing server sends control instructions to the multiple experimental equipment through the laboratory field bus network to realize collaborative control of the multiple experimental equipment.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images