Machine vision-based experimental error detection method, system, and device

Abstract

The application discloses a kind of experimental error detection method, system and equipment based on machine vision, which is applied to the experimental error detection of polymer composite material preparation, the method comprises: based on the image monitoring information of current experimental process obtains the current operation action of experimental robot, generates the posture time sequence of current operation action, updates the experimental operation log of experimental robot based on current operation action and posture time sequence, and generates the time constraint condition, space constraint condition and operation constraint condition corresponding to current operation action, based on constraint condition, the current operation action of experimental robot is carried out time dimension error detection, space dimension error detection and operation dimension error detection, dynamic constraint generation is realized by updating operation log, based on the constraint condition of multiple dimensions, multi-dimensional experimental error detection is carried out to experimental robot, more accurate risk monitoring and response are realized, and the safety and reliability of automatic experimental robot are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method, system, and device for detecting experimental errors based on machine vision. Background Technology

[0002] With the development of science and technology, the experimental research and production process of polymer composite materials has gradually replaced manual labor with robots. Experimental robots are suitable for polymer material research and development experiments. Compared with traditional manual experiments, robots can effectively improve experimental efficiency.

[0003] The preparation and experimental processes of polymer composite materials are complex, and existing automated experimental robots rely on preset programs and limited sensor data, making it difficult to detect operational errors in a timely and accurate manner (such as collisions with shelves or equipment, reversed steps, or abnormal reagent addition). This leads to risks such as experimental failure, material waste, and chemical leaks. The core problem lies in the current experimental robots' limitations: incomplete data perception (relying solely on physical parameters and lacking operational logic analysis), weak risk prediction capabilities (unable to correlate time, space, and operational data to predict hidden risks), insufficient dynamic response (requiring human intervention and unable to terminate errors in real time), and poor interpretability. Currently, due to the single dimension of sensory data, experimental robots cannot accurately and promptly detect erroneous behaviors during experiments, resulting in significant safety hazards and failure risks. Summary of the Invention

[0004] The main objective of this invention is to provide a machine vision-based experimental error detection method, system, and device, which aims to solve the technical problem that existing technologies cannot accurately and timely detect the erroneous behavior of experimental robots during experiments due to the single dimension of the perceptual data of experimental robots, resulting in significant safety hazards and failure risks during experiments.

[0005] To achieve the above objectives, this invention provides a machine vision-based experimental error detection method, which is applied to error detection in polymer composite material preparation experiments. The method includes the following steps:

[0006] The current experimental process of the experimental robot is monitored by images to obtain image monitoring information;

[0007] The current operation action of the experimental robot is obtained based on the image monitoring information, and the posture time sequence of the current operation action is generated. The current operation action consists of one or more operation postures, and the posture time sequence consists of operation postures and timestamps corresponding to each operation posture.

[0008] The experimental operation log of the experimental robot is updated based on the current operation action and the posture time series. The experimental operation log is constructed based on each operation action of the experimental robot during the experiment and the posture time series corresponding to each operation action.

[0009] The updated experimental operation log generates the constraints corresponding to the current operation action, including: time constraints, space constraints, and operation constraints.

[0010] Based on the constraints, multidimensional experimental error detection is performed on the current operation of the experimental robot. The multidimensional experimental error detection includes time dimension error detection, spatial dimension error detection, and operation dimension error detection.

[0011] Optionally, the step of performing multi-dimensional experimental error detection on the current operation of the experimental robot based on the constraints includes:

[0012] Based on the current operation action, extract spatial dimension features and operation dimension features;

[0013] Extract time dimension features based on the attitude time series;

[0014] The time channel attention weight, spatial channel attention weight, and operation channel attention weight are assigned according to the experimental steps corresponding to the current operation action of the experimental robot.

[0015] Based on the time channel attention weight, the spatial channel attention weight, and the operation channel attention weight, the time dimension features, spatial dimension features, and operation dimension features are fused by multi-head attention to obtain the multi-dimensional experimental features of the experimental robot.

[0016] Obtain the experimental rule information corresponding to the experimental steps, and assign experimental constraint weights based on the experimental rule information;

[0017] The constraints are quantized to obtain time constraint vector, space constraint vector and operation constraint vector;

[0018] Based on the experimental constraint weights, the time constraint vector, spatial constraint vector, and operational constraint vector are mapped to the same space as the multidimensional experimental features to obtain the multidimensional constraint features.

[0019] Multidimensional experimental error detection is performed on the multidimensional experimental features based on the multidimensional constraint features.

[0020] Optionally, the step of performing feature quantization on the constraints to obtain a time constraint vector, a spatial constraint vector, and an operational constraint vector includes:

[0021] Perform feature analysis on the time constraints to obtain time constraint feature information:

[0022] ;

[0023] in, express and The time deviation distance between them Used to represent time-constrained feature information. Represents the time series of observed attitude. This represents the template attitude time series. Indicates the length of the time series. Represents the time series of observed attitude In the The value at each point in time. Represents the template attitude time series In the The value at each time point;

[0024] Obtain key time point information for the experimental steps, and determine time attention parameters based on the key time point information:

[0025] ;

[0026] in, Represents the temporal attention parameter. Represents the query matrix. A query vector used to represent a time series. Represents the key matrix. Key vectors used to represent time series Indication matrix, Key vectors used to represent time series This represents the dimension of the key vector. Indicates transpose;

[0027] The time constraint conditions are quantized based on the time constraint feature information and the time attention parameters to obtain a time constraint vector;

[0028] Based on the experimental equipment layout information, a device topology is constructed, and the features of each device node in the topology are quantized to obtain the node spatial feature vector of each device node in the topology:

[0029] ;

[0030] in, Represents device node In the The feature vector of the layer, This represents the activation function, which is a non-linear function. Indicates the first The weight matrix of the layer, Represents device node In the The feature vector of the layer, Represents device node The set of adjacent nodes, and Represents device node and The number of adjacent nodes, Represents device node In the The feature vector of the layer;

[0031] Point cloud data is collected from each device node, and the local spatial geometric features of a local area, which consists of one or more device nodes, are analyzed based on the point cloud data.

[0032] ;

[0033] in, Representing local spatial geometric features, This refers to a multilayer perceptron, which is a feedforward neural network. Point cloud data representing a local spatial region, This represents the point cloud coordinates of each device node within a local space. This represents the max pooling operation;

[0034] The spatial constraint conditions are quantified based on the local spatial geometric features and the node spatial feature vectors to obtain the spatial constraint vectors.

[0035] The operational constraints are semantically encoded to obtain the embedding vectors for each operational step:

[0036] ;

[0037] in, Indicates the first Embedding vectors for each operation step express The location of each operation step is embedded. express Word embedding for each operation step Indicates the first Text of each operation step;

[0038] An operation graph network structure is constructed based on the embedding vectors, and the dependencies between operation nodes are obtained based on the operation graph network structure. The dependencies are the edge features between operation nodes in the operation graph network structure.

[0039] ;

[0040] in, Indicates dependency relationship, and The characteristics of the operation nodes are represented by the operation nodes and their features. Indicates the step interval;

[0041] The violation of constraints in the operation steps is evaluated based on the dependencies to obtain the degree of impact of the violation of each operation step;

[0042] Construct an operational constraint reward function based on the degree of impact of the violation:

[0043] ;

[0044] in, Indicates the total reward. This indicates a reward for completing the task. Indicates the penalty coefficient. Representative violated Article The negative reward generated by the operational constraint. Indicates the total number of operational constraints;

[0045] The operational constraints are quantified based on the operational constraint reward function and the graph network structure to obtain the operational constraint vector.

[0046] Optionally, generating the constraint conditions corresponding to the current operation action based on the updated experimental operation log includes:

[0047] Determine the experimental shelf corresponding to the current operation based on the updated experimental operation log;

[0048] Collect the point cloud data of the experimental shelf, and construct the global coordinate system of the experimental shelf and the inter-layer coordinate system of each shelf surface in the experimental shelf based on the point cloud data;

[0049] The experimental shelf is divided into multiple grid units based on the global coordinate system and the inter-layer coordinate system, and each grid unit is assigned a corresponding unit ID based on its physical coordinates.

[0050] The spatial constraints and current placement status of each grid cell are determined based on the image monitoring data of each grid cell.

[0051] The spatial constraints corresponding to the current experimental step are generated based on the unit spatial constraints and the current placement state.

[0052] Optionally, generating the spatial constraints corresponding to the current experimental step based on the unit space constraints and the current placement state includes:

[0053] Based on the current placement state, determine the idle cells in the grid cells;

[0054] Obtain the process and material information for the current experimental task;

[0055] Based on the process information and the material information, determine the material label, material usage order, and material usage operation for each experimental material in the current experimental task;

[0056] Based on the cell space constraints of each grid cell, the material usage order, and the material usage operation, the experimental material is clamped into the idle cell, and a material placement mapping relationship is established between the material label and the cell ID;

[0057] The shelf status database is updated based on the material placement, and the current placement status of each grid cell in the shelf status database is updated. The shelf status database maintains the material placement mapping relationship corresponding to each grid cell and the current placement status of each grid cell.

[0058] Based on the shelf point cloud data, construct the shelf outline boundary, the tabletop outline boundary of each material platform, and the unit outline boundary of each grid unit of the experimental shelf.

[0059] Multidimensional spatial constraints are generated based on the shelf outline boundary, the shelf outline boundary, the unit outline boundary, and the shelf status database. The multidimensional spatial constraints include shelf boundary constraints, inter-layer boundary constraints, unit boundary constraints, and material logic constraints.

[0060] Based on the point cloud data, the three-dimensional model of the experimental shelf is divided into multiple three-dimensional mesh units, and the three-dimensional Euclidean distance between each three-dimensional mesh unit and each grid unit is calculated.

[0061] Risk quantification is performed on each three-dimensional grid cell based on the three-dimensional Euclidean distance and the shelf status database to obtain the risk quantification result.

[0062] Based on the risk quantification results, the multidimensional space constraints are weighted to obtain quantized weight values.

[0063] Construct a multidimensional space weight matrix based on the quantized weight values ​​and the multidimensional space constraints.

[0064] The spatial constraints corresponding to the current experimental step are generated based on the multidimensional spatial weight matrix.

[0065] Optionally, generating the constraint conditions corresponding to the current operation action based on the updated experimental operation log includes:

[0066] Encode each operation action in the updated experimental operation log into an action encoding vector:

[0067] ;

[0068] in, Represents the action encoding vector. Indicates the motion encoder. Indicates an operation action;

[0069] Generate a temporal coding vector based on the attitude time series corresponding to each operation action:

[0070] ;

[0071] in, This is a temporal encoding vector used to represent the temporal dependencies between operations. This represents a pre-built time series analysis model. These represent the various operations in the updated experimental operation log;

[0072] Generate the current operation encoding vector based on the action encoding vector and the timing encoding vector;

[0073] The current operation encoding vector is compared with the operation specification text vector, and the operation constraints corresponding to the current operation action are generated based on the comparison result.

[0074] Optionally, after performing multi-dimensional experimental error detection on the current operation of the experimental robot based on the constraints, the method further includes:

[0075] The experimental process of the experimental robot is monitored by sensor data to acquire multimodal sensor data, which includes gas sensor data, temperature sensor data, sound sensor data, pressure sensor data and vibration sensor data.

[0076] Obtain the experimental material information and experimental equipment information associated with the experimental steps corresponding to the current operation action of the experimental robot;

[0077] Based on the experimental material information and the experimental equipment information, emergency event information for the current operation is generated. The emergency event information includes the target harmful gas type, test tube temperature threshold, loudness threshold, vibration frequency threshold, and test tube reaction pressure threshold.

[0078] Emergency response conditions are generated based on the emergency event information;

[0079] Based on the multimodal sensor data, determine whether the current experimental scenario of the experimental robot meets the emergency event response conditions;

[0080] In response to the current experimental scenario meeting the emergency event response conditions, a preset emergency event response strategy is executed, which includes controlling the experimental robot to stop performing the experiment.

[0081] The emergency response conditions include:

[0082] The target hazardous gas type was detected in the current environment;

[0083] The temperature of the current material reaction test tube in the current experimental scenario exceeds the test tube temperature threshold.

[0084] The loudness of the reaction sound in the current material reaction tube exceeds the loudness threshold.

[0085] The chemical reaction amplitude in the current material reaction tube exceeds the vibration frequency threshold.

[0086] The internal pressure of the current material reaction tube exceeds the reaction pressure threshold of the tube.

[0087] Optionally, the multimodal sensor data further includes image sensor data, and the emergency event response condition further includes detecting a crack on the surface of the current material reaction tube;

[0088] The step of determining whether the current experimental scenario of the experimental robot meets the emergency event response conditions based on the multimodal sensor data includes:

[0089] Acquire normal chemical reaction image sample data of the experimental step corresponding to the current operation action of the experimental robot;

[0090] Based on the normal chemical reaction image sample data, chemical reaction image features are extracted, including the geometric features and gray-scale distribution features of the chemical reaction;

[0091] The current test tube image of the current material reaction test tube and the historical test tube image before the material was injected into the current material reaction test tube are obtained based on the image sensor data.

[0092] A test tube template image is generated based on the historical test tube images, and the test tube illumination characteristics are obtained based on the test tube template image;

[0093] Based on the light characteristics of the test tube, the current test tube image is shaded, and the shaded current test tube image is processed into grayscale to obtain candidate test tube images;

[0094] Based on the chemical reaction image features, extract the chemical reaction product features from the candidate test tube images, and separate the chemical reaction product features from the candidate test tube images to obtain the target test tube image;

[0095] Crack detection is performed on the target test tube image, and the current experimental scenario of the experimental robot is determined based on the crack detection results to determine whether the emergency event response conditions are met.

[0096] Furthermore, to achieve the above objectives, this invention also proposes a machine vision-based experimental error detection system, which is applied to error detection in polymer composite material preparation experiments. The machine vision-based experimental error detection system includes:

[0097] The image monitoring module is used to monitor the current experimental process of the experimental robot and acquire image monitoring information;

[0098] The operation posture analysis module is used to obtain the current operation action of the experimental robot based on the image monitoring information and generate the posture time series of the current operation action. The current operation action consists of one or more operation postures, and the posture time series consists of operation postures and timestamps corresponding to each operation posture.

[0099] The operation log update module is used to update the experimental operation log of the experimental robot based on the current operation action and the posture time series. The experimental operation log is constructed based on each operation action of the experimental robot during the experiment and the posture time series corresponding to each operation action.

[0100] The constraint generation module is used to generate constraints corresponding to the current operation based on the updated experimental operation log. The constraints include: time constraints, space constraints, and operation constraints.

[0101] An experimental error detection module is used to perform multi-dimensional experimental error detection on the current operation of the experimental robot based on the constraints. The multi-dimensional experimental error detection includes time dimension error detection, spatial dimension error detection, and operation dimension error detection.

[0102] Furthermore, to achieve the above objectives, this application also proposes a machine vision-based experimental error detection device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the machine vision-based experimental error detection method described above.

[0103] In addition, to achieve the above objectives, this application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the machine vision-based experimental error detection method described above.

[0104] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the machine vision-based experimental error detection method described above.

[0105] This invention acquires image monitoring information by monitoring the current experimental process of an experimental robot. Based on this information, it obtains the robot's current operation action and generates a posture time series for that action. The current operation action consists of one or more operation postures, and the posture time series comprises the operation postures and their corresponding timestamps. The invention updates the experimental operation log of the robot based on the current operation action and the posture time series. The experimental operation log is constructed based on each operation action and its corresponding posture time series during the experiment. Constraints are generated for the current operation action based on the updated log. These constraints include temporal, spatial, and operational constraints. The current operation action is then assessed based on these constraints. The invention employs multi-dimensional experimental error detection, encompassing temporal, spatial, and operational dimensions. By visually monitoring the experimental robot's current actions and updating the experimental operation log based on the current actions and their posture time series, it achieves spatiotemporal monitoring of the robot. Dynamic constraint generation through log updates effectively avoids the inaccuracy of error detection caused by the lag of static constraints. Multi-dimensional error detection based on temporal, spatial, and operational constraints enables more accurate risk monitoring and response, effectively avoiding unreliable detection results due to a single data source. This allows for rapid response to erroneous behaviors of the experimental robot, significantly improving the safety and reliability of automated experimental robots. Attached Figure Description

[0106] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0107] Figure 1 This is a schematic diagram of the structure of a machine vision-based experimental error detection device in the hardware operating environment involved in the embodiments of the present invention;

[0108] Figure 2 This is a flowchart illustrating the first embodiment of the machine vision-based experimental error detection method of the present invention.

[0109] Figure 3 This is a schematic diagram of an emergency event response process in one embodiment of the machine vision-based experimental error detection method of the present invention;

[0110] Figure 4 This is a flowchart illustrating the second embodiment of the machine vision-based experimental error detection method of the present invention.

[0111] Figure 5 This is a structural block diagram of the first embodiment of the machine vision-based experimental error detection system of the present invention.

[0112] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0113] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0114] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a machine vision-based experimental error detection device for the hardware operating environment involved in the embodiments of the present invention.

[0115] like Figure 1As shown, the machine vision-based experimental error detection device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. Optionally, the memory 1005 may also be a storage system independent of the aforementioned processor 1001.

[0116] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on machine vision-based experimental error detection devices and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0117] like Figure 1 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a machine vision-based experimental error detection program.

[0118] exist Figure 1 In the machine vision-based experimental error detection device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the machine vision-based experimental error detection device of the present invention can be set in the machine vision-based experimental error detection device. The machine vision-based experimental error detection device calls the machine vision-based experimental error detection program stored in the memory 1005 through the processor 1001 and executes the machine vision-based experimental error detection method provided in the embodiment of the present invention.

[0119] This invention provides a machine vision-based experimental error detection method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the machine vision-based experimental error detection method of the present invention.

[0120] In this embodiment, the method is applied to error detection in polymer composite material preparation experiments. The machine vision-based error detection method includes the following steps:

[0121] Step S10: Perform image monitoring on the current experimental process of the experimental robot and obtain image monitoring information.

[0122] It should be noted that this embodiment can be applied to the chemical preparation process of polymer composite materials (such as fiber-reinforced composite materials, phthalonitrile, epoxy resin, low-viscosity epoxy resin, electrolyte and ionogel hydrogel materials, etc.) to detect errors in the experimental behavior, actions and operations of the experimental robot. When the experimental robot is found to have violated regulations or made errors, it can issue an error alarm in a timely manner and control the experimental robot to stop the experiment or control the experimental robot to take risk rescue measures.

[0123] It should be understood that the executing entity of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a terminal electronic device capable of performing the above functions. The following description uses a machine vision-based experimental error detection device (hereinafter referred to as the detection device) as an example to illustrate this embodiment and the subsequent embodiments.

[0124] It should be noted that the above-mentioned experimental robot can be an automated experimental workstation with a robotic arm. The robotic arm can include a robotic arm body and an end effector. The robotic arm body can be a multi-degree-of-freedom body (e.g., six degrees of freedom) with integrated intelligent joints. The end effector can be connected to multiple execution ends. For example, the end effector can be connected to multiple replaceable execution modules (e.g., gripping module, pipetting module, ultrasonic module, etc.) through mechanical connection devices or electromagnetic connection devices.

[0125] In some embodiments, the detection device can perform image monitoring on the experimental robot through an image sensor, acquire image monitoring data, perform image processing (e.g., noise reduction, binarization, grayscale processing, etc.) and target detection on the image monitoring data, and acquire image monitoring information of the experimental robot. The image monitoring information may include the motion posture and motion trajectory of the experimental robot's robotic arm body, as well as the currently used module and currently executed operation of the experimental robot's end effector.

[0126] Step S20: Based on the image monitoring information, obtain the current operation action of the experimental robot and generate the posture time series of the current operation action.

[0127] It should be noted that the current operation consists of one or more operation postures, and the posture time sequence consists of the operation postures and the timestamps corresponding to each operation posture.

[0128] In some embodiments, the detection device may perform noise reduction (e.g., noise reduction by Gaussian filtering), contrast enhancement (e.g., contrast enhancement processing by the CLAHE algorithm), and edge sharpening on the image monitoring data.

[0129] In some embodiments, the detection device can segment the region of interest (ROI) from the preprocessed image monitoring data using background subtraction or Mask R-CNN, identify and analyze the features of the ROI, obtain the current operation action of the experimental robot, and generate the posture time series of the current operation action. The ROI may include the activity area of ​​the robotic arm body of the experimental robot, the activity area of ​​the end effector, and the area of ​​the experimental robot's current operating workbench and material preparation table, etc.

[0130] In some embodiments, the detection device can obtain point cloud data of the region of interest by preprocessing the image monitoring data and segmenting the region of interest, perform key point detection based on the power data, obtain the coordinates of the key points, input the coordinates of the key points into a pre-trained LSTM classifier, match the current operation posture based on the posture feature library (such as "grabbing reagent bottle", "pipetting and injecting", "stirring and mixing"), and identify the current operation action of the experimental robot based on the operation posture.

[0131] In some embodiments, the detection device can divide sub-action stages using a time-series segmentation algorithm (such as DTW-based dynamic time warping), identify the operational posture of the experimental robot in each frame of the image monitoring data, record the corresponding timestamp, associate the timestamp with the operational posture, construct structured data based on the image frame sequence and timestamp, and generate a posture time series.

[0132] In some embodiments, the detection device can also obtain the current operation of the experimental robot by acquiring the operation instruction data of the experimental robot, aligning the operation instructions in the operation instruction data with the operation actions identified in the image monitoring information.

[0133] Step S30: Update the experimental operation log of the experimental robot based on the current operation action and the posture time series.

[0134] It should be noted that the experimental operation log is constructed based on the time series of each operation action and the corresponding posture of the experimental robot during the experiment. The detection device can record the operation action and the corresponding posture time series at each moment or time node, generate the experimental operation log based on the operation action and posture time series, and maintain and update the experimental operation log in real time.

[0135] For example, a log entry in the experimental operation log may include: the experimental robot grabbing reagent bottle number one (action ID is ACTION-011) and the posture time sequence corresponding to grabbing reagent bottle number one.

[0136] Step S40: Generate the constraints corresponding to the current operation based on the updated experimental operation log.

[0137] It should be noted that the constraints include: time constraints, space constraints, and operational constraints.

[0138] It is understood that this embodiment generates the constraint conditions corresponding to the current operation based on the updated experimental operation log, thereby dynamically generating constraint conditions based on the operation actions and attitude time series recorded in the experimental operation log. This avoids the problem that the static constraint conditions generated before the experiment cannot adapt to the complex dynamic environment of the experimental process, and adjusts the constraint conditions in real time to improve the accuracy of error detection.

[0139] In some embodiments, time constraints may be conditions that constrain the time window for the experimental robot to perform operations. For example, the stirring time for the experimental robot to perform material mixing must meet a preset time, the curing time for the material curing reaction must meet a preset curing time, and the reaction time of certain hazardous reactions (such as strongly exothermic reactions or reactions that produce toxic substances) must be controlled within a preset time.

[0140] In some embodiments, spatial constraints can be constraints on the activity space of the robotic arm and end effector of the experimental robot. For example, during the experiment, the robotic arm and end effector may collide with material racks, workbenches, experimental vessels, test tubes, etc. Therefore, it is necessary to constrain the motion trajectory and actions of the experimental robot to avoid spatial collisions. For example, spatial constraints may include joint space activity constraints, end effector activity constraints, etc.

[0141] In some embodiments, operational constraints may be constraints used to determine whether the operation sequence, operation type, and operation logic of the experimental robot are correct. For example, operational constraints may include whether the current operation of the experimental robot is correctly associated with the operation logic of the previous operation, whether the gripper pressure of the experimental robot exceeds the gripping force threshold, and whether the material grasped by the experimental robot meets the requirements of the experimental process.

[0142] Furthermore, in order to accurately constrain the spatial dimensions of the experimental robot, step S40 above may include:

[0143] Step S401: Determine the experimental shelf corresponding to the current operation based on the updated experimental operation log;

[0144] Step S402: Collect the point cloud data of the experimental shelf, and construct the global coordinate system of the experimental shelf and the inter-layer coordinate system of each shelf surface in the experimental shelf based on the point cloud data;

[0145] Step S403: Divide each material platform of the experimental shelf into multiple grid units according to the global coordinate system and the inter-layer coordinate system, and assign a corresponding unit ID to each grid unit based on the physical coordinates of each grid unit;

[0146] Step S404: Determine the spatial constraints and current placement status of each grid cell based on the image monitoring data of each grid cell;

[0147] Step S405: Generate the spatial constraints corresponding to the current experimental step based on the unit spatial constraints and the current placement state.

[0148] In some embodiments, the testing device can determine the consumables (such as materials and utensils) required for the experiment based on the updated experimental operation log, and locate the experimental shelf corresponding to the current operation by querying the shelf where the consumables are stored based on the experimental database.

[0149] In some embodiments, the detection device can collect point cloud data of the experimental shelf and construct a global coordinate system of the experimental shelf and an inter-layer coordinate system of each shelf surface in the experimental shelf based on the point cloud data; divide each material surface of the experimental shelf into multiple grid units according to the global coordinate system and the inter-layer coordinate system, and assign a corresponding unit ID to each grid unit based on the physical coordinates of each grid unit; determine the unit spatial constraints and current placement state of each grid unit based on the image monitoring data of each grid unit; and generate the spatial constraints corresponding to the current experimental step according to the unit spatial constraints and the current placement state.

[0150] Furthermore, to improve the accuracy of spatial constraints, step S405 may include:

[0151] Step S4051: Determine the free cells in the grid cells based on the current placement state;

[0152] Step S4052: Obtain the process information and material information for the current experimental task;

[0153] Step S4053: Based on the process information and the material information, determine the material label, material usage order, and material usage operation of each experimental material in the current experimental task;

[0154] Step S4054: According to the cell space constraints of each grid cell, the material usage order and the material usage operation, the experimental material is clamped to the idle cell, and a material placement mapping relationship is established between the material label and the cell ID;

[0155] Step S4055: Update the shelf status database based on the material placement, and update the current placement status of each grid cell in the shelf status database. The shelf status database maintains the material placement mapping relationship corresponding to each grid cell and the current placement status of each grid cell.

[0156] Step S4056: Construct the shelf outline boundary, the tabletop outline boundary, and the cell outline boundary of each grid unit of the experimental shelf based on the shelf point cloud data.

[0157] Step S4057: Generate multi-dimensional spatial constraints based on the shelf outline boundary, the shelf outline boundary, the unit outline boundary and the shelf status database. The multi-dimensional spatial constraints include shelf boundary constraints, inter-layer boundary constraints, unit boundary constraints and material logic constraints.

[0158] Step S4058: Based on the point cloud data, divide the three-dimensional model of the experimental shelf into multiple three-dimensional mesh units, and calculate the three-dimensional Euclidean distance between each three-dimensional mesh unit and each grid unit;

[0159] Step S4059: Perform risk quantification on each three-dimensional grid cell based on the three-dimensional Euclidean distance and the shelf status database to obtain the risk quantification result;

[0160] Step S40510: Based on the risk quantification result, perform weight quantization on the multidimensional space constraint to obtain quantized weight values;

[0161] Step S40511: Construct a multidimensional space weight matrix based on the quantized weight values ​​and the multidimensional space constraints;

[0162] Step S40512: Generate the spatial constraints corresponding to the current experimental step based on the multidimensional spatial weight matrix.

[0163] In some embodiments, the detection device identifies the target laboratory shelf as a "chemical storage shelf" on the east side of the laboratory. Subsequently, the shelf is scanned using 3D LiDAR to generate high-precision point cloud data. Based on point cloud segmentation technology, a global coordinate system for the shelf (with the laboratory floor center as the origin) and inter-layer coordinate systems for each shelf level are constructed (e.g., the first shelf level is 0.8 meters high, and the second shelf level is 1.2 meters high). The detection device divides each shelf level into 10cm × 10cm grid units, assigning each grid a unique ID (e.g., "L2-03-05" indicates the 3rd row and 5th column of the 2nd shelf). Static spatial constraints are defined in conjunction with shelf load limits (e.g., a maximum of 5kg per grid), physical dimensions, and hazard markings (e.g., a "no-use" label for corrosive reagents). The detection device monitors the occupancy status of each grid in real time using a top camera and infrared sensors (e.g., detecting that a grid on the 2nd shelf is occupied or contains overweight items). Based on current operational requirements (e.g., the need to place 2kg reagent bottles) and dynamic status, select grids that meet the requirements of load-bearing capacity, are not occupied, and are located in the designated reagent storage area (e.g., "L1-02-04"). Generate the allowed access coordinate range and the prohibited operation area, thereby generating spatial constraints.

[0164] Furthermore, in order to accurately constrain the operational behavior of the experimental robot, step S40 above may include:

[0165] Step S4001: Encode each operation action in the updated experimental operation log into an action encoding vector:

[0166] ;

[0167] in, Represents the action encoding vector. Indicates the motion encoder. Indicates an operation action;

[0168] Step S4002: Generate a temporal coding vector based on the attitude time series corresponding to each operation action:

[0169] ;

[0170] in, This is a temporal encoding vector used to represent the temporal dependencies between operations. This represents a pre-built time series analysis model. These represent the various operations in the updated experimental operation log;

[0171] Step S4003: Generate the current operation encoding vector based on the action encoding vector and the timing encoding vector;

[0172] Step S4004: Compare the current operation encoding vector with the operation specification text vector, and generate the operation constraints corresponding to the current operation action based on the comparison result.

[0173] In some embodiments, the detection device can encode each operation action in the updated operation log into a uniform-dimensional action encoding vector using a pre-trained BERT model, generate a temporal encoding vector using an LSTM network to capture the temporal features of the operation, and fuse the action encoding vector and the temporal encoding vector through concatenation and a fully connected layer to generate the current operation encoding vector. The current operation encoding vector is compared with the text operation vector recorded in the preset operation planning text, and the cosine similarity between the two is calculated to determine the matching text action, obtain the operation specification information corresponding to the text action, and generate the operation constraints corresponding to the current operation action based on the calculated cosine similarity between the two.

[0174] Step S50: Perform multi-dimensional experimental error detection on the current operation of the experimental robot based on the constraints.

[0175] It should be noted that the multidimensional experimental error detection includes time dimension error detection, spatial dimension error detection, and operational dimension error detection.

[0176] It is understood that this embodiment performs time dimension error detection on the current operation of the experimental robot based on time constraints, spatial dimension error detection on the current operation of the experimental robot based on spatial constraints, and operation dimension error detection on the current operation of the experimental robot based on operation constraints.

[0177] In some embodiments, the detection device can quantize the features of time constraints, spatial constraints, and operational constraints, and then map the feature vectors of each dimension to the same dimensional space for fusion to obtain a multimodal constraint vector. Based on the multimodal constraint vector, multidimensional experimental error detection is performed on the current operation of the experimental robot.

[0178] In some embodiments, the detection device can quantify the risk features of the operational constraint monitoring information, the temporal constraint monitoring information, the spatial constraint monitoring information, and the module constraint monitoring information to obtain risk quantification parameters in multiple dimensions, including operational risk quantification parameters, temporal risk quantification parameters, spatial risk quantification parameters, and module risk quantification parameters; merge the risk quantification parameters in multiple dimensions into a risk input matrix; generate a query weight matrix, a key weight matrix, and a value weight matrix based on the risk input matrix; establish linear transformation relationships between the risk quantification parameters in each dimension of the risk input matrix and the query weight matrix, the key weight matrix, and the value weight matrix, respectively; calculate the query vector, key vector, and value vector of the risk quantification parameters in each dimension based on the linear transformation relationships; calculate a similarity matrix based on the query vector, key vector, and value vector; generate an attention weight matrix based on the similarity matrix; obtain the attention weights corresponding to the risk quantification parameters in each dimension based on the attention weight matrix; aggregate the risk quantification parameters in multiple dimensions based on the attention weights to obtain comprehensive risk quantification parameters; and perform experimental error detection on the experimental robot based on the comprehensive risk quantification parameters.

[0179] In some embodiments, the detection device can monitor whether the current experimental operation of the experimental robot meets the operational constraints based on the image monitoring information, obtain operational constraint monitoring information, determine the operation time of the current experimental operation of the experimental robot based on the image monitoring information, and monitor whether the operation time meets the time constraints, obtain time constraint monitoring information, construct the dynamic three-dimensional trajectory of the experimental robot based on the image monitoring information, and monitor whether the dynamic three-dimensional trajectory meets the spatial constraints, obtain spatial constraint monitoring information, and perform experimental error detection on the experimental robot based on the operational constraint monitoring information, the time constraint monitoring information, and the spatial constraint monitoring information.

[0180] In some embodiments, the detection device may acquire control signals of experimental operation modules associated with the current experimental step of the experimental robot, and determine the operation target information of the experimental operation modules; acquire module constraints of the experimental operation modules based on the operation target information; monitor whether the control signals meet the module constraints to obtain module constraint monitoring information; and perform experimental error detection on the experimental robot based on the operation constraint monitoring information, the time constraint monitoring information, the spatial constraint monitoring information, and the module constraint monitoring information.

[0181] Furthermore, in order to respond promptly to emergencies during the experiment, prevent the scale of risk impact from escalating, and reduce risk losses, refer to Figure 3 , Figure 3This is a flowchart illustrating an emergency response in one embodiment. In some embodiments, after step S50, the following may be included:

[0182] Step S501: Monitor sensor data during the experimental process of the experimental robot and acquire multimodal sensor data;

[0183] Step S502: Obtain the experimental material information and experimental equipment information associated with the experimental step corresponding to the current operation action of the experimental robot;

[0184] Step S503: Generate emergency event information for the current operation based on the experimental material information and the experimental equipment information;

[0185] Step S504: Generate emergency response conditions based on the emergency event information;

[0186] Step S505: Determine whether the current experimental scenario of the experimental robot meets the emergency event response conditions based on the multimodal sensor data;

[0187] Step S506: In response to the current experimental scenario meeting the emergency event response conditions, execute the preset emergency event response strategy.

[0188] It should be noted that the multimodal sensor data includes gas sensor data, temperature sensor data, sound sensor data, pressure sensor data, and vibration sensor data. The emergency event information includes the target hazardous gas type, test tube temperature threshold, loudness threshold, vibration frequency threshold, and test tube reaction pressure threshold.

[0189] It should be noted that the preset emergency response strategy includes controlling the experimental robot to stop performing the experiment. In some embodiments, the preset emergency response strategy may also include emergency rescue operations, such as controlling the experimental robot to extinguish the fire and issue a fire alarm when the emergency is a chemical material reaction and fire.

[0190] It should be noted that the emergency response conditions include: the detection of the target harmful gas type in the current environment; the tube temperature of the current material reaction tube in the current experimental scenario exceeding the tube temperature threshold; the loudness of the reaction sound in the current material reaction tube exceeding the loudness threshold; the amplitude of the chemical reaction in the current material reaction tube exceeding the vibration frequency threshold; and the internal pressure of the current material reaction tube exceeding the tube reaction pressure threshold.

[0191] Furthermore, in some embodiments, the multimodal sensor data further includes image sensor data, and the emergency event response condition further includes detecting a crack on the surface of the current material reaction test tube; in order to accurately identify cracks on the test tube surface and effectively avoid material leakage and test tube breakage, the above step S505 may include:

[0192] Step S5051: Obtain normal chemical reaction image sample data of the experimental step corresponding to the current operation action of the experimental robot;

[0193] Step S5052: Extract chemical reaction image features based on the normal chemical reaction image sample data, wherein the chemical reaction image features include the geometric features and grayscale distribution features of the chemical reaction;

[0194] Step S5053: Obtain the current test tube image of the current material reaction test tube and the historical test tube image before the material was injected into the current material reaction test tube based on the image sensor data;

[0195] Step S5054: Generate a test tube template image based on the historical test tube images, and obtain the test tube illumination characteristics based on the test tube template image;

[0196] Step S5055: Perform shadow correction on the current test tube image based on the test tube illumination characteristics, and perform grayscale processing on the shadow-corrected current test tube image to obtain candidate test tube images;

[0197] Step S5056: Extract chemical reaction product features from the candidate test tube images based on the chemical reaction image features, and separate the chemical reaction product features from the candidate test tube images to obtain the target test tube image;

[0198] Step S5057: Perform crack detection on the target test tube image, and determine whether the current experimental scenario of the experimental robot meets the emergency event response conditions based on the crack detection results.

[0199] It should be noted that when chemical materials and substances in a test tube react, they produce reaction states (such as foam, flocculent matter, etc.), which cause interference from color, texture, and noise in the test tube images acquired by machine vision. Therefore, this embodiment extracts chemical reaction image features based on normal chemical reaction image sample data, and performs noise reduction processing on the test tube images based on the chemical reaction image features, thereby eliminating the interference of chemical reaction states and accurately identifying the image features on the surface of the test tube.

[0200] In a specific implementation, the detection device acquires normal chemical reaction image sample data corresponding to the experimental steps of the current operation action of the experimental robot. Based on this data, it extracts chemical reaction image features, including geometric features and grayscale distribution features of the chemical reaction. It also acquires the current test tube image of the current material reaction test tube and historical test tube images before material injection, based on the image sensor data. A test tube template image is generated based on the historical test tube images, and the test tube illumination features are acquired from the template image. Shadow correction is then applied to the current test tube image based on these illumination features, and the shadow-corrected image is then processed for grayscale. The process involves obtaining candidate test tube images, extracting chemical reaction product features from the candidate test tube images based on the chemical reaction image features, separating the chemical reaction product features from the candidate test tube images to obtain target test tube images, performing crack detection on the target test tube images, and determining whether the current experimental scenario of the experimental robot meets the emergency event response conditions based on the crack detection results. This effectively eliminates noise interference from the chemical reaction state in the test tubes, corrects shadows on the test tube images, balances the noise from uneven lighting, and extracts high-quality target test tube images. This effectively improves the accuracy of test tube crack identification, promptly identifies cracked test tubes, and effectively avoids the risk of chemical material leakage caused by test tube breakage.

[0201] This embodiment acquires image monitoring information by monitoring the current experimental process of the experimental robot. Based on this information, it obtains the current operation action of the experimental robot and generates a posture time series of the current operation action. The current operation action consists of one or more operation postures, and the posture time series consists of operation postures and timestamps corresponding to each posture. Based on the current operation action and the posture time series, the experimental operation log of the experimental robot is updated. The experimental operation log is constructed based on each operation action of the experimental robot during the experiment and the corresponding posture time series. Based on the updated experimental operation log, constraints corresponding to the current operation action are generated. These constraints include time constraints, spatial constraints, and operational constraints. Based on these constraints, the current operation action of the experimental robot is... The preceding operation involves multi-dimensional experimental error detection, including time-dimensional error detection, spatial-dimensional error detection, and operational-dimensional error detection. This embodiment monitors the experimental robot's current operation via vision, updating the experimental operation log based on the current operation and its posture time series. This enables monitoring of the experimental robot from a spatiotemporal perspective. Dynamic constraint generation through log updates effectively avoids the inaccuracy of experimental error detection caused by the lag of static constraints. Multi-dimensional experimental error detection based on constraints in the time, space, and operational dimensions achieves more accurate risk monitoring and response, effectively avoiding unreliable detection results due to a single data source. This enables rapid response to erroneous behaviors of the experimental robot, significantly improving the safety and reliability of the automated experimental robot.

[0202] refer to Figure 4 , Figure 4 This is a flowchart illustrating the second embodiment of the machine vision-based experimental error detection method of the present invention.

[0203] Based on the first embodiment described above, in this embodiment, step S50 further includes:

[0204] Step S51: Extract spatial dimension features and operation dimension features based on the current operation action.

[0205] In some embodiments, the detection device can extract point cloud data of the robot end effector and the robotic arm body when the experimental robot performs the current operation, and obtain the spatial trajectory, operation type and operation-related target (e.g., picking up test tubes, stirring materials, etc.) during the movement process of the current operation based on the point cloud data.

[0206] Step S52: Extract time dimension features based on the attitude time series.

[0207] It is understood that the detection device can standardize or normalize the operation time of the current operation (e.g., object grasping time, robotic arm movement time, stirring time, etc.) based on the posture time series, use feature engineering to extract statistics (e.g., mean, variance, peak value) or time-domain features, and use feature-based image registration methods to extract key time points of the time series (e.g., operation start / end time).

[0208] Step S53: Assign time channel attention weights, spatial channel attention weights, and operational channel attention weights according to the experimental steps corresponding to the current operation action of the experimental robot.

[0209] It should be noted that different types of experimental steps are assigned different attention weights. For example, in the step of stirring materials, attention needs to be paid to the stirring time and stirring action (such as stirring amplitude and stirring force). Therefore, more weight can be assigned to the time channel attention and operation channel attention in the step of stirring materials. In the step of picking up materials, attention needs to be paid to the experimental robot colliding with equipment, worktable, material platform, consumables, etc. Therefore, more weight can be assigned to the spatial channel attention.

[0210] Step S54: Based on the time channel attention weight, the spatial channel attention weight, and the operation channel attention weight, perform multi-head attention fusion on the time dimension features, spatial dimension features, and operation dimension features to obtain the multi-dimensional experimental features of the experimental robot.

[0211] In some embodiments, the detection device may perform multi-head attention fusion of the time dimension features, spatial dimension features, and operational dimension features based on the following formula:

[0212] ;

[0213] ;

[0214] ;

[0215] ;

[0216] in, , and These represent the attention weights for the time channel, the spatial channel, and the operational channel, respectively. , and These represent time-dimensional features, spatial-dimensional features, and operational-dimensional features, respectively. , and These represent time-weighted features, space-weighted features, and operation-weighted features, respectively. Indicates multidimensional experimental characteristics; It is a feedforward neural network used for nonlinear mapping; Representation layer normalization is used to standardize the input features to stabilize the training process and accelerate convergence; This indicates that multiple attention heads are processed in parallel to capture correlations in different dimensions.

[0217] Step S55: Obtain the experimental rule information corresponding to the experimental step, and assign experimental constraint weights based on the experimental rule information.

[0218] In some embodiments, the detection device can parse the corresponding experimental rule information from the experimental rule text corresponding to the experimental steps, and obtain the corresponding constraints based on the experimental rule information. For example, time constraints may include experimental steps needing to be completed within a specific time period (e.g., "the reaction needs to be completed within 30 minutes"); spatial constraints may include experimental equipment or materials needing to meet spatial layout requirements (e.g., "reagent A and reagent B need to be stored in different areas"); and operational constraints may include operational steps needing to follow a specific order or specification (e.g., "the catalyst must be added only after heating to 100°C").

[0219] In some embodiments, the detection device can assign constraint weights to constraints of different dimensions based on experimental rule information. For example, the spatial constraints of experimental step A are the most important, so more weight can be assigned to the spatial constraints. If the experimental step is to pick up and move a test tube, it is necessary to avoid collisions with obstacles, avoid excessive picking force that could damage the test tube, and avoid picking up the wrong test tube. Therefore, more weight needs to be assigned to the spatial constraints and operational constraints in this step.

[0220] Step S56: Perform feature quantization on the constraints to obtain the time constraint vector, spatial constraint vector, and operational constraint vector.

[0221] In some embodiments, the detection device can transform abstract constraint rules into numerical feature vectors to improve error detection efficiency and analysis accuracy. For example, time constraints can be encoded into numerical vectors of time constraints; spatial constraints can be transformed into numerical representations of coordinates or topological relationships; and operational constraints can be transformed into numerical vectors of operational sequences or logical relationships.

[0222] Furthermore, to improve constraint management efficiency and error detection accuracy, step S56 may include:

[0223] Step S561: Perform feature analysis on the time constraints to obtain time constraint feature information.

[0224] It should be noted that the detection equipment can detect time deviations in operation actions (such as "insufficient heating time" or "excessive stirring interval") using Dynamic Time Warping (DTW) and LSTM-Attention models, as shown in the following formula:

[0225] ;

[0226] in, express and The time deviation distance between them Used to represent time-constrained feature information. Represents the time series of observed attitude. This represents the template attitude time series. Indicates the length of the time series. Represents the time series of observed attitude In the The value at each point in time. Represents the template attitude time series In the The value at each time point;

[0227] Step S562: Obtain key time point information of the experimental steps, and determine time attention parameters based on the key time point information.

[0228] In some embodiments, the detection device can capture key time points using a Transformer model, as shown in the following formula:

[0229] ;

[0230] in, Represents the temporal attention parameter. Represents the query matrix. A query vector used to represent a time series. Represents the key matrix. Key vectors used to represent time series Indication matrix, Key vectors used to represent time series This represents the dimension of the key vector. Indicates transpose;

[0231] Step 563: Quantize the time constraint conditions based on the time constraint feature information and the time attention parameters to obtain the time constraint vector.

[0232] In some embodiments, the time constraint vector may include time deviation distance, periodic deviation distance, time attention parameter, etc.

[0233] Step S564: Construct the equipment topology based on the experimental equipment layout information, and perform feature quantization on each equipment node in the equipment topology to obtain the node space feature vector of each equipment node in the equipment topology.

[0234] In some embodiments, the detection device can model the topological relationships of the experimental device layout using a graph neural network to extract spatial feature vectors, as shown in the following formula:

[0235] ;

[0236] in, Represents device node In the The feature vector of the layer, This represents the activation function, which is a non-linear function. Indicates the first The weight matrix of the layer, Represents device node In the The feature vector of the layer, Represents device node The set of adjacent nodes, and Represents device node and The number of adjacent nodes, Represents device node In the The feature vector of the layer;

[0237] Step S565: Collect point cloud data from each device node, and analyze the local spatial geometric features of the local area based on the point cloud data.

[0238] It should be noted that the local area consists of one or more device nodes, and the detection device can analyze the local spatial geometric features of each local area, referring to the following formula:

[0239] ;

[0240] in, Representing local spatial geometric features, This refers to a multilayer perceptron, which is a feedforward neural network. Point cloud data representing a local spatial region, This represents the point cloud coordinates of each device node within a local space. This represents the max pooling operation;

[0241] Step S566: Quantize the spatial constraints based on the local spatial geometric features and the node spatial feature vectors to obtain the spatial constraint vectors.

[0242] In some embodiments, the spatial constraint vector may include spatial deviation, topological similarity, 3D point cloud features, etc.

[0243] Step S567: Semantically encode the operation constraints to obtain the embedding vectors for each operation step:

[0244] ;

[0245] in, Indicates the first Embedding vectors for each operation step express The location of each operation step is embedded. express Word embedding for each operation step Indicates the first Text of each operation step;

[0246] Step S568: Construct an operation graph network structure based on the embedding vector, and obtain the dependency relationships between each operation node based on the operation graph network structure.

[0247] It should be noted that this embodiment can capture the global dependencies of operation steps by learning the embedding vectors of nodes through a graph neural network. These dependencies are the edge features between operation nodes in the operation graph network structure.

[0248] ;

[0249] in, Indicates dependency relationship, and The characteristics of the operation nodes are represented by the operation nodes and their features. Indicates the step interval;

[0250] Step S569: Evaluate the violation of constraints of the operation steps based on the dependencies to obtain the degree of impact of the violation of each operation step.

[0251] In some embodiments, the detection device may quantify the dependency strength between operation steps based on the embedding vector and edge features, and classify the dependency type according to the edge features (such as logical dependencies, resource conflicts) and the semantics of the embedding vector.

[0252] In some embodiments, the detection device can assess the violation of constraints in operational steps by constructing an impact assessment model, for example, defining an impact assessment formula by combining dependency strength, severity of constraint violation, and recovery cost:

[0253] ;

[0254] in, Indicates the severity of the violation of the constraint. This indicates the cost of recovery.

[0255] Step S570: Construct an operational constraint reward function based on the degree of impact of the violation:

[0256] ;

[0257] in, Indicates the total reward. This indicates a reward for completing the task. Indicates the penalty coefficient. Representative violated Article The negative reward generated by the operational constraint. Indicates the total number of operational constraints;

[0258] Step S571: Quantize the operation constraint conditions according to the operation constraint reward function and the graph network structure to obtain the operation constraint vector.

[0259] In some embodiments, the operation constraint vector may include step sequence deviation, parameter compliance, operation reward parameters, etc.

[0260] Step S57: Based on the experimental constraint weights, map the time constraint vector, spatial constraint vector, and operational constraint vector to the same space as the multidimensional experimental features to obtain multidimensional constraint features.

[0261] Understandably, the detection equipment can map time, space, and operational constraint vectors to a high-dimensional space identical to the experimental features through weighted projection, facilitating subsequent fusion and analysis.

[0262] Step S58: Perform multidimensional experimental error detection on the multidimensional experimental features based on the multidimensional constraint features.

[0263] Understandably, detection equipment can map multidimensional constraint features to a space with the same dimension as multidimensional experimental features to perform multidimensional experimental error detection.

[0264] In some embodiments, the detection device may acquire control signals of experimental operation modules associated with the current experimental step of the experimental robot, and determine the operation target information of the experimental operation modules; acquire module constraints of the experimental operation modules based on the operation target information; monitor whether the control signals meet the module constraints to obtain module constraint monitoring information; and perform experimental error detection on the experimental robot based on the operation constraint monitoring information, the time constraint monitoring information, the spatial constraint monitoring information, and the module constraint monitoring information.

[0265] This embodiment extracts spatial and operational features based on the current operation action, and extracts temporal features based on the posture time series. It assigns temporal, spatial, and operational attention weights according to the experimental steps corresponding to the current operation action of the experimental robot. Based on these weights, it performs multi-head attention fusion to obtain the multi-dimensional experimental features of the experimental robot, acquires the experimental rule information corresponding to the experimental steps, assigns experimental constraint weights based on the experimental rule information, and performs feature quantization on the constraint conditions. The process involves obtaining time constraint vectors, spatial constraint vectors, and operational constraint vectors. Based on the experimental constraint weights, these vectors are mapped to the same space as the multidimensional experimental features to obtain multidimensional constraint features. Multidimensional experimental error detection is then performed on these features, thereby achieving the fusion of constraints across multiple dimensions. The operational actions of the experimental robot are decoupled into feature vectors of multiple dimensions and then coupled into a fused feature vector in the same dimensional space. This allows for error detection from the time, space, and operational dimensions, while simultaneously performing error detection on the multidimensional constraint features and the multidimensional experimental features of the experimental robot in the same space, thus improving error detection efficiency.

[0266] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a machine vision-based experimental error detection program, wherein the machine vision-based experimental error detection program, when executed by a processor, implements the steps of the machine vision-based experimental error detection method described above.

[0267] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0268] The aforementioned computer-readable storage medium may be included in a machine vision-based experimental error detection device; or it may exist independently and not assembled into a machine vision-based experimental error detection device.

[0269] Furthermore, this invention also proposes a computer program product, including a machine vision-based experimental error detection program, which, when executed by a processor, implements the steps of the machine vision-based experimental error detection method described above.

[0270] The specific implementation of the computer program product of the present invention is basically the same as the embodiments of the above-described experimental error detection method based on machine vision, and will not be repeated here.

[0271] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the machine vision-based experimental error detection system of the present invention.

[0272] like Figure 5 As shown in the figure, the machine vision-based experimental error detection system proposed in this embodiment of the invention is applied to error detection in polymer composite material preparation experiments. The machine vision-based experimental error detection system includes:

[0273] The image monitoring module 10 is used to monitor the current experimental process of the experimental robot and acquire image monitoring information.

[0274] The operation posture analysis module 20 is used to obtain the current operation action of the experimental robot based on the image monitoring information and generate the posture time sequence of the current operation action. The current operation action consists of one or more operation postures, and the posture time sequence consists of operation postures and timestamps corresponding to each operation posture.

[0275] The operation log update module 30 is used to update the experimental operation log of the experimental robot based on the current operation action and the posture time series. The experimental operation log is constructed based on each operation action and the posture time series corresponding to each operation action during the experiment.

[0276] The constraint generation module 40 is used to generate the constraint conditions corresponding to the current operation action based on the updated experimental operation log. The constraint conditions include: time constraint conditions, space constraint conditions and operation constraint conditions.

[0277] The experimental error detection module 50 is used to perform multi-dimensional experimental error detection on the current operation of the experimental robot based on the constraints. The multi-dimensional experimental error detection includes time dimension error detection, spatial dimension error detection and operation dimension error detection.

[0278] This embodiment acquires image monitoring information by monitoring the current experimental process of the experimental robot. Based on this information, it obtains the current operation action of the experimental robot and generates a posture time series of the current operation action. The current operation action consists of one or more operation postures, and the posture time series consists of operation postures and timestamps corresponding to each posture. Based on the current operation action and the posture time series, the experimental operation log of the experimental robot is updated. The experimental operation log is constructed based on each operation action of the experimental robot during the experiment and the corresponding posture time series. Based on the updated experimental operation log, constraints corresponding to the current operation action are generated. These constraints include time constraints, spatial constraints, and operational constraints. Based on these constraints, the current operation action of the experimental robot is... The preceding operation involves multi-dimensional experimental error detection, including time-dimensional error detection, spatial-dimensional error detection, and operational-dimensional error detection. This embodiment monitors the experimental robot's current operation via vision, updating the experimental operation log based on the current operation and its posture time series. This enables monitoring of the experimental robot from a spatiotemporal perspective. Dynamic constraint generation through log updates effectively avoids the inaccuracy of experimental error detection caused by the lag of static constraints. Multi-dimensional experimental error detection based on constraints in the time, space, and operational dimensions achieves more accurate risk monitoring and response, effectively avoiding unreliable detection results due to a single data source. This enables rapid response to erroneous behaviors of the experimental robot, significantly improving the safety and reliability of the automated experimental robot.

[0279] The machine vision-based experimental error detection system provided in this application, employing the machine vision-based experimental error detection method described in the above embodiments, can solve the technical problems of machine vision-based experimental error detection. Compared with the prior art, the beneficial effects of the machine vision-based experimental error detection system provided in this application are the same as those of the machine vision-based experimental error detection method provided in the above embodiments, and other technical features of the machine vision-based experimental error detection system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0280] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0281] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0282] In addition, for technical details not described in detail in this embodiment, please refer to the machine vision-based experimental error detection method provided in any embodiment of the present invention, which will not be repeated here.

[0283] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0284] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0285] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0286] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A machine vision-based experimental error detection method, characterized in that, The method is applied to error detection in polymer composite material preparation experiments, and the machine vision-based error detection method includes: The current experimental process of the experimental robot is monitored by images to obtain image monitoring information; The current operation action of the experimental robot is obtained based on the image monitoring information, and the posture time sequence of the current operation action is generated. The current operation action consists of one or more operation postures, and the posture time sequence consists of operation postures and timestamps corresponding to each operation posture. The experimental operation log of the experimental robot is updated based on the current operation action and the posture time series. The experimental operation log is constructed based on each operation action of the experimental robot during the experiment and the posture time series corresponding to each operation action. The updated experimental operation log generates the constraints corresponding to the current operation action, including: time constraints, space constraints, and operation constraints. Based on the constraints, multidimensional experimental error detection is performed on the current operation of the experimental robot. The multidimensional experimental error detection includes time dimension error detection, spatial dimension error detection and operation dimension error detection. The multi-dimensional experimental error detection based on the constraints of the experimental robot's current operation includes: Based on the current operation action, extract spatial dimension features and operation dimension features; Extract time dimension features based on the attitude time series; The time channel attention weight, spatial channel attention weight, and operation channel attention weight are assigned according to the experimental steps corresponding to the current operation action of the experimental robot. Based on the time channel attention weight, the spatial channel attention weight, and the operation channel attention weight, the time dimension features, spatial dimension features, and operation dimension features are fused by multi-head attention to obtain the multi-dimensional experimental features of the experimental robot. Obtain the experimental rule information corresponding to the experimental steps, and assign experimental constraint weights based on the experimental rule information; The constraints are quantized to obtain time constraint vector, space constraint vector and operation constraint vector; Based on the experimental constraint weights, the time constraint vector, spatial constraint vector, and operational constraint vector are mapped to the same space as the multidimensional experimental features to obtain the multidimensional constraint features. Multidimensional experimental error detection is performed on the multidimensional experimental features based on the multidimensional constraint features.

2. The machine vision-based experimental error detection method as described in claim 1, characterized in that, The step of performing feature quantization on the constraints to obtain time constraint vector, space constraint vector, and operational constraint vector includes: Perform feature analysis on the time constraints to obtain time constraint feature information: in, express and The time deviation distance between them Used to represent time-constrained feature information. Represents the time series of observed attitude. This represents the template attitude time series. Indicates the length of the time series. Represents the time series of observed attitude In the The value at each point in time. Represents the template attitude time series In the The value at each time point; Obtain key time point information for the experimental steps, and determine time attention parameters based on the key time point information: in, Represents the temporal attention parameter. Represents the query matrix. A query vector used to represent a time series. Represents the key matrix. Key vectors used to represent time series Indication matrix, Key vectors used to represent time series The dimension of the key vector. Indicates transpose; The time constraint conditions are quantized based on the time constraint feature information and the time attention parameters to obtain a time constraint vector; Based on the experimental equipment layout information, a device topology is constructed, and the features of each device node in the topology are quantized to obtain the node spatial feature vector of each device node in the topology: in, Represents device node In the The feature vector of the layer, This represents the activation function, which is a non-linear function. Indicates the first The weight matrix of the layer, Represents device node In the The feature vector of the layer, Represents device node The set of adjacent nodes, and Represents device node and The number of adjacent nodes, Represents device node In the The feature vector of the layer; Point cloud data is collected from each device node, and local spatial geometric features of a local area are analyzed based on the point cloud data. The local area consists of one or more device nodes. in, Representing local spatial geometric features, This refers to a multilayer perceptron, which is a feedforward neural network. Point cloud data representing a local spatial region, This represents the point cloud coordinates of each device node within a local space. This represents the max pooling operation; The spatial constraint conditions are quantified based on the local spatial geometric features and the node spatial feature vectors to obtain the spatial constraint vectors. The operational constraints are semantically encoded to obtain the embedding vectors for each operational step: in, Indicates the first Embedding vectors for each operation step express The location of each operation step is embedded. express Word embedding for each operation step Indicates the first Text of each operation step; An operation graph network structure is constructed based on the embedding vectors, and the dependencies between operation nodes are obtained based on the operation graph network structure. The dependencies are the edge features between operation nodes in the operation graph network structure. in, Indicates dependency relationship, and This represents the characteristics of the operation nodes and the operation nodes. Indicates the step interval; The violation of constraints in the operation steps is evaluated based on the dependencies to obtain the degree of impact of the violation of each operation step; Construct an operational constraint reward function based on the degree of impact of the violation: in, Indicates the total reward. This indicates a reward for completing the task. Indicates the penalty coefficient. Representative violated Article The negative reward generated by the operational constraint. Indicates the total number of operational constraints; The operational constraints are quantified based on the operational constraint reward function and the graph network structure to obtain the operational constraint vector.

3. The machine vision-based experimental error detection method as described in any one of claims 1 or 2, characterized in that, The step of generating the constraint conditions corresponding to the current operation action based on the updated experimental operation log includes: Determine the experimental shelf corresponding to the current operation based on the updated experimental operation log; Collect the point cloud data of the experimental shelf, and construct the global coordinate system of the experimental shelf and the inter-layer coordinate system of each shelf surface in the experimental shelf based on the point cloud data; The experimental shelf is divided into multiple grid units based on the global coordinate system and the inter-layer coordinate system, and each grid unit is assigned a corresponding unit ID based on its physical coordinates. The spatial constraints and current placement status of each grid cell are determined based on the image monitoring data of each grid cell. The spatial constraints corresponding to the current experimental step are generated based on the unit spatial constraints and the current placement state.

4. The machine vision-based experimental error detection method as described in claim 3, characterized in that, The step of generating spatial constraints corresponding to the current experimental step based on the unit spatial constraints and the current placement state includes: Based on the current placement state, determine the idle cells in the grid cells; Obtain the process and material information for the current experimental task; Based on the process information and the material information, determine the material label, material usage order, and material usage operation for each experimental material in the current experimental task; Based on the cell space constraints of each grid cell, the material usage order, and the material usage operation, the experimental material is clamped into the idle cell, and a material placement mapping relationship is established between the material label and the cell ID; The shelf status database is updated based on the material placement, and the current placement status of each grid cell in the shelf status database is updated. The shelf status database maintains the material placement mapping relationship corresponding to each grid cell and the current placement status of each grid cell. Based on the shelf point cloud data, construct the shelf outline boundary, the tabletop outline boundary of each material platform, and the unit outline boundary of each grid unit of the experimental shelf. Multidimensional spatial constraints are generated based on the shelf outline boundary, the shelf outline boundary, the unit outline boundary, and the shelf status database. The multidimensional spatial constraints include shelf boundary constraints, inter-layer boundary constraints, unit boundary constraints, and material logic constraints. Based on the point cloud data, the three-dimensional model of the experimental shelf is divided into multiple three-dimensional mesh units, and the three-dimensional Euclidean distance between each three-dimensional mesh unit and each grid unit is calculated. Risk quantification is performed on each three-dimensional grid cell based on the three-dimensional Euclidean distance and the shelf status database to obtain the risk quantification result. Based on the risk quantification results, the multidimensional space constraints are weighted to obtain quantized weight values. Construct a multidimensional space weight matrix based on the quantized weight values ​​and the multidimensional space constraints. The spatial constraints corresponding to the current experimental step are generated based on the multidimensional spatial weight matrix.

5. The machine vision-based experimental error detection method as described in claim 4, characterized in that, The step of generating the constraint conditions corresponding to the current operation action based on the updated experimental operation log includes: Encode each operation action in the updated experimental operation log into an action encoding vector: in, Represents the action encoding vector. Indicates the motion encoder. Indicates an operation action; Generate a temporal coding vector based on the attitude time series corresponding to each operation action: in, This is a temporal encoding vector used to represent the temporal dependencies between operations. This represents a pre-built time series analysis model. These represent the various operations in the updated experimental operation log; Generate the current operation encoding vector based on the action encoding vector and the timing encoding vector; The current operation encoding vector is compared with the operation specification text vector, and the operation constraints corresponding to the current operation action are generated based on the comparison result.

6. The machine vision-based experimental error detection method as described in any one of claims 1 or 2, characterized in that, After performing multi-dimensional experimental error detection on the current operation of the experimental robot based on the constraints, the method further includes: The experimental process of the experimental robot is monitored by sensor data to acquire multimodal sensor data, which includes gas sensor data, temperature sensor data, sound sensor data, pressure sensor data and vibration sensor data. Obtain the experimental material information and experimental equipment information associated with the experimental steps corresponding to the current operation action of the experimental robot; Based on the experimental material information and the experimental equipment information, emergency event information for the current operation is generated. The emergency event information includes the target harmful gas type, test tube temperature threshold, loudness threshold, vibration frequency threshold, and test tube reaction pressure threshold. Emergency response conditions are generated based on the emergency event information; Based on the multimodal sensor data, determine whether the current experimental scenario of the experimental robot meets the emergency event response conditions; In response to the current experimental scenario meeting the emergency event response conditions, a preset emergency event response strategy is executed, which includes controlling the experimental robot to stop performing the experiment. The emergency response conditions include: The target hazardous gas type was detected in the current environment; The temperature of the current material reaction test tube in the current experimental scenario exceeds the test tube temperature threshold. The loudness of the reaction sound in the current material reaction tube exceeds the loudness threshold. The chemical reaction amplitude in the current material reaction tube exceeds the vibration frequency threshold. The internal pressure of the current material reaction tube exceeds the reaction pressure threshold of the tube.

7. The machine vision-based experimental error detection method as described in claim 6, characterized in that, The multimodal sensor data also includes image sensor data, and the emergency event response conditions also include detecting a crack on the surface of the current material reaction tube; The step of determining whether the current experimental scenario of the experimental robot meets the emergency event response conditions based on the multimodal sensor data includes: Acquire normal chemical reaction image sample data of the experimental step corresponding to the current operation action of the experimental robot; Based on the normal chemical reaction image sample data, chemical reaction image features are extracted, including the geometric features and gray-scale distribution features of the chemical reaction; The current test tube image of the current material reaction test tube and the historical test tube image before the material was injected into the current material reaction test tube are obtained based on the image sensor data. A test tube template image is generated based on the historical test tube images, and the test tube illumination characteristics are obtained based on the test tube template image; Based on the light characteristics of the test tube, the current test tube image is shaded, and the shaded current test tube image is processed into grayscale to obtain candidate test tube images; Based on the chemical reaction image features, extract the chemical reaction product features from the candidate test tube images, and separate the chemical reaction product features from the candidate test tube images to obtain the target test tube image; Crack detection is performed on the target test tube image, and the current experimental scenario of the experimental robot is determined based on the crack detection results to determine whether the emergency event response conditions are met.

8. A machine vision-based experimental error detection system, characterized in that, The system is used for error detection in polymer composite material preparation experiments. The machine vision-based experimental error detection system includes: The image monitoring module is used to monitor the current experimental process of the experimental robot and acquire image monitoring information; The operation posture analysis module is used to obtain the current operation action of the experimental robot based on the image monitoring information and generate the posture time series of the current operation action. The current operation action consists of one or more operation postures, and the posture time series consists of operation postures and timestamps corresponding to each operation posture. The operation log update module is used to update the experimental operation log of the experimental robot based on the current operation action and the posture time series. The experimental operation log is constructed based on each operation action of the experimental robot during the experiment and the posture time series corresponding to each operation action. The constraint generation module is used to generate the constraint conditions corresponding to the current operation action based on the updated experimental operation log. The constraint conditions include: time constraint conditions, space constraint conditions and operation constraint conditions. An experimental error detection module is used to perform multi-dimensional experimental error detection on the current operation of the experimental robot based on the constraints. The multi-dimensional experimental error detection includes time dimension error detection, spatial dimension error detection and operation dimension error detection. The experimental error detection module is further configured to: extract spatial dimension features and operational dimension features based on the current operation action; extract temporal dimension features based on the posture time series; allocate temporal channel attention weights, spatial channel attention weights, and operational channel attention weights according to the experimental steps corresponding to the current operation action of the experimental robot; perform multi-head attention fusion on the temporal dimension features, spatial dimension features, and operational dimension features based on the temporal channel attention weights, spatial channel attention weights, and operational channel attention weights to obtain multi-dimensional experimental features of the experimental robot; obtain experimental rule information corresponding to the experimental steps, and allocate experimental constraint weights based on the experimental rule information; perform feature quantization on the constraint conditions to obtain temporal constraint vectors, spatial constraint vectors, and operational constraint vectors; map the temporal constraint vectors, spatial constraint vectors, and operational constraint vectors to the same space as the multi-dimensional experimental features based on the experimental constraint weights to obtain multi-dimensional constraint features; and perform multi-dimensional experimental error detection on the multi-dimensional experimental features based on the multi-dimensional constraint features.

9. A machine vision-based experimental error detection device, characterized in that, The machine vision-based experimental error detection device includes: a memory, a processor, and a machine vision-based experimental error detection program stored in the memory and executable on the processor, wherein the machine vision-based experimental error detection program is configured to implement the machine vision-based experimental error detection method as described in any one of claims 1 to 7.

Images