Industrial multi-modal intelligent interaction method and system based on large language model agent

Abstract

The application relates to the technical field of data processing, and discloses an industrial multi-modal intelligent interaction method and system based on a large language model agent. The method comprises the following steps: fusing industrial field multi-modal collection data sets into a physical state vector, jointly encoding the physical state vector and a natural language scheduling instruction to obtain a multi-modal interaction context, modulating a word element probability distribution in real time in a word element generation process by using a step-by-step anchor point confidence, and injecting an uncertainty semantic vector to drive safe degradation instruction generation when the confidence continuously decreases. The application solves the problems in the prior art that, in the word element generation process of a large language model agent, semantic drift between language reasoning paths and industrial field physical states is continuously accumulated, post-checking cannot intervene in the completed reasoning process, and when a physical constraint signal fails, the system cannot complete safe degradation without interrupting the operation of the industrial field.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an industrial multimodal intelligent interaction method and system based on a large language model agent. Background Technology

[0002] With the deep integration of the Industrial Internet and artificial intelligence technologies, intelligent logistics execution systems centered on AMRs and AGVs are widely used in highly complex industrial production scenarios such as automobile manufacturing and automated warehousing. To achieve efficient scheduling of AMR / AGV clusters and real-time monitoring of production lines, existing technologies are gradually introducing large language model agents as the core decision-making unit for industrial interaction. By inputting operators' natural language scheduling commands into the large language model agent, the agent understands the semantics of the commands and generates corresponding scheduling responses, driving the AMR / AGV cluster to perform industrial tasks such as handling, sorting, and warehousing, thus realizing natural language interaction between humans and machines. Regarding multimodal information fusion, existing technologies typically convert industrial field data such as visual images and sensor readings into natural language descriptive text and then concatenate it into the input prompts of the large language model, conveying the physical state information of the industrial field to the large language model in text form.

[0003] However, the method of converting multimodal sensor data from the industrial site into natural language descriptions and then concatenating them into input prompts results in a lack of computable mapping between the physical dimensions of the sensors and the semantics of the language. Physical information and language reasoning exist in two independent information spaces, and the large language model agent cannot impose mathematical constraints on the real-time physical state of the industrial site during the generation of scheduling instructions. More importantly, all existing verification schemes for the consistency of large language model outputs, including retrieval enhancement generation, post-fact checking, and prompt constraints, only verify the physical consistency of the output results after the large language model has completed the complete output sequence. The verification results cannot be applied back to the completed reasoning process, causing the semantic deviation between the language reasoning path and the physical state of the industrial site to accumulate continuously during the word-by-word autoregressive generation process of the large language model agent. This deviation can only be detected after the complete scheduling instruction is output, by which time the erroneous scheduling instruction has already entered the execution queue. In industrial control scenarios with millisecond-level responses in AMR / AGV, this can easily lead to safety accidents such as equipment collisions and path conflicts. Summary of the Invention

[0004] This application provides an industrial multimodal intelligent interaction method and system based on a large language model agent. It solves the problems in the prior art where semantic drift between the language reasoning path and the physical state of the industrial site accumulates continuously during the word-by-word generation process of the large language model agent, and post-verification cannot intervene in the completed reasoning process. It also solves the problem that the system cannot complete the safety degradation without interrupting the operation of the industrial site when the physical constraint signal fails. Thus, it realizes the real-time constraint of the physical state on the language reasoning process and the autonomous safety degradation under systemic anomalies in the industrial multimodal intelligent interaction scenario.

[0005] Firstly, this application provides an industrial multimodal intelligent interaction method based on a large language model agent, the industrial multimodal intelligent interaction method based on a large language model agent comprising: Step S1: The autonomous mobile robot pose data stream, visual image frames, cargo load readings, and production line alarm status codes are timestamped to obtain a multimodal acquisition dataset. Step S2: Project the modal features of each modality in the multimodal acquisition dataset onto the hidden layer dimension space of the large language model and then fuse them by weight to obtain the physical state vector; Step S3: Jointly encode the natural language scheduling instructions input by the operator with the physical state vector to obtain a multimodal interaction context; Step S4: During the word-by-word generation process of the large language model agent with the multimodal interaction context as input, the stepwise anchor confidence is calculated based on the cosine similarity between the physical state vector and the current hidden layer state vector, and the original word probability distribution and the physical constraint preference distribution are weighted and mixed according to the anchor confidence to obtain the modulated word probability distribution. Step S5: When the continuous window mean and the descent slope of the anchor confidence are both lower than a preset threshold, an uncertainty semantic vector is constructed based on the difference between the physical state vector and the current hidden layer state vector. The uncertainty semantic vector is then injected into the large language model agent to generate a security degradation instruction, which is then sent to the autonomous mobile robot and the automated guided vehicle for execution.

[0006] Secondly, this application provides an industrial multimodal intelligent interaction system based on a large language model agent, the industrial multimodal intelligent interaction system based on a large language model agent comprising: The alignment module is used to align the autonomous mobile robot pose data stream, visual image frames, cargo load readings, and production line alarm status codes with timestamps to obtain a multimodal acquisition dataset. The weighting module is used to project the modal features in the multimodal acquisition dataset onto the hidden layer dimension space of the large language model and then fuse them in a weighted manner to obtain the physical state vector. The encoding module is used to jointly encode the natural language scheduling instructions input by the operator and the physical state vector to obtain a multimodal interaction context; The calculation module is used to calculate the stepwise anchor confidence based on the cosine similarity between the physical state vector and the current hidden layer state vector during the word-by-word generation process of the large language model agent with the multimodal interaction context as input, and to perform a weighted mixture of the original word probability distribution and the physical constraint preference distribution according to the anchor confidence to obtain the modulated word probability distribution. The execution module is used to construct an uncertain semantic vector based on the difference between the physical state vector and the current hidden layer state vector when the continuous window mean and the descent slope of the anchor confidence are both lower than a preset threshold, and to inject the uncertain semantic vector into the large language model agent to generate a security degradation instruction, and to send the security degradation instruction to the autonomous mobile robot and the automated guided vehicle for execution.

[0007] Thirdly, an industrial multimodal intelligent interaction device based on a large language model agent is provided, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the industrial multimodal intelligent interaction device based on a large language model agent to execute the aforementioned industrial multimodal intelligent interaction method based on a large language model agent.

[0008] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the aforementioned industrial multimodal intelligent interaction method based on a large language model agent.

[0009] The technical solution provided in this application constructs a multimodal acquisition dataset by aligning the pose data stream of an autonomous mobile robot, visual image frames, cargo load readings, and production line alarm status codes with timestamps. Then, the features of each modality are projected onto the hidden layer dimension space of a large language model and weighted fused to obtain a physical state vector. This allows for a unified expression of four heterogeneous modalities in the industrial field within the same mathematical space, enabling cosine similarity calculation between physical information and the language reasoning process of the large language model within the same dimensional space. This overcomes the deficiency in existing technologies where sensor data is converted into natural language text and spliced ​​into input, resulting in no computable mapping relationship between physical dimensions and linguistic semantics. Furthermore, the natural language scheduling instructions input by the operator and the physical state vector are jointly encoded through a cross-modal attention layer to obtain a multimodal interaction context. This establishes an explicit attention weight association between each semantic unit in the scheduling instructions and the physical state of the industrial field, solving the problems in existing technologies where natural language scheduling instructions and the physical state of the industrial field are independent, and the large language model agent cannot perceive physical constraints at the semantic level.

[0010] The core technical contribution of this application lies in the fact that, during the word-by-word generation process of the large language model agent with multimodal interactive context as input, the confidence of the progressive anchor point is calculated in real time by using the cosine similarity between the physical state vector and the current hidden layer state vector. The original word probability distribution and the physical constraint preference distribution are then weighted and mixed to obtain the modulated word probability distribution. This embeds the physical state of the industrial site into the probabilistic decision-making process at each word generation moment, fundamentally different from the inherent defect of the prior art that post-verification cannot intervene in the completed reasoning process. This allows physical reality to play a constraining role in the language reasoning process in real time, rather than just being verified at the output end. Furthermore, when the continuous window mean and the descent slope of the stepwise anchor confidence are both below a preset threshold, an uncertain semantic vector is constructed based on the difference between the physical state vector and the current hidden state vector. This vector is then injected into the large language model agent to generate a safety degradation instruction, which is then sent to the autonomous mobile robot and automated guided vehicle for execution. This enables the system to match a degradation strategy corresponding to the severity based on the quantized drift amplitude when the semantic drift exceeds the progressive modulation repair capability due to the continuous failure of the physical constraint signal. This allows the system to complete autonomous safety degradation without interrupting the operation of the industrial site, thus solving the problem of the lack of an effective degradation mechanism under systemic anomaly conditions. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of one embodiment of the industrial multimodal intelligent interaction method based on a large language model agent in this application. Figure 2 This is a schematic diagram of one embodiment of the industrial multimodal intelligent interaction system based on a large language model agent in this application. Figure 3 This is a schematic block diagram of the structure of an industrial multimodal intelligent interactive device based on a large language model agent in an embodiment of the present invention. Detailed Implementation

[0013] This application provides an industrial multimodal intelligent interaction method and system based on a large language model agent. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0014] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the industrial multimodal intelligent interaction method based on a large language model agent in this application includes: Step S1: The autonomous mobile robot pose data stream, visual image frames, cargo load readings, and production line alarm status codes are timestamped to obtain a multimodal acquisition dataset. It is understood that the executing entity of this application can be an industrial multimodal intelligent interaction system based on a large language model agent, or it can be a terminal or a server; the specific implementation is not limited here. This application's embodiments use a server as an example for illustration.

[0015] Specifically, the four types of data—autonomous mobile robot pose data stream, visual image frames, warehouse load readings, and production line alarm status codes—are forcibly aligned within the same acquisition cycle with a timestamp deviation not exceeding 20 milliseconds. The 20-millisecond setting is based on the common standard for industrial AMR / AGV control cycles. A cycle shorter than this ensures that each modal data corresponds to the on-site state at the same physical moment. Among them, the autonomous mobile robot pose data stream refers to the sequence of position and attitude parameters output in real time by the AMR's own sensors during operation in the industrial field, including two-dimensional plane coordinates, orientation angle, and velocity vector; the visual image frames refer to the RGB images output by the vision sensors deployed on the production site according to the acquisition cycle; the warehouse load readings refer to the current load status values ​​output by the sensors of each warehouse location in the automated warehouse; and the production line alarm status codes refer to the production line abnormality type codes output by the Andon management platform. The four types of data after timestamp alignment together constitute a multimodal acquisition dataset. The data in this dataset have clear physical correspondences and describe the industrial field situation in four dimensions: robot motion state, scene semantic state, warehouse physical state, and production line abnormal state.

[0016] Step S2: Project the modal features of each modality in the multimodal acquisition dataset onto the hidden layer dimension space of the large language model and then fuse them with weights to obtain the physical state vector; Specifically, the physical state vector refers to the weighted fusion vector obtained by uniformly mapping the four types of modal features to the hidden layer dimension space of the large language model. Its dimension is the same as that of the hidden layer of the large language model, which is 4096 dimensions in this embodiment. This vector and the hidden layer state vector of the large language model are in the same mathematical space, so that physical information and language reasoning can be calculated for cosine similarity in the same space. After each modal feature is mapped by the linear projection layer, it is weighted and summed with a pose weight of 0.40, a visual weight of 0.25, a storage weight of 0.20, and an alarm weight of 0.15. The sum of the four weights is strictly equal to 1. The weight ratio is set according to the degree of dominance of pose information in robot motion decision-making in industrial scheduling scenarios. The pose weight is the highest, reflecting the decisive influence of pose data on the generation of scheduling instructions. The visual weight is the second highest, reflecting the auxiliary role of scene semantics in path planning. The storage weight and alarm weight decrease in turn, reflecting their reference status in scheduling decision-making. The above weight coefficients are determined by gradient descent during the offline training stage with the industrial scheduling accuracy as the optimization objective.

[0017] Step S3: Jointly encode the natural language scheduling instructions input by the operator with the physical state vector to obtain the multimodal interaction context; Specifically, the multimodal interaction context refers to the fusion representation obtained by jointly encoding the natural language scheduling instructions input by the operator and the physical state vector. This fusion representation carries both the operator's intent information and the physical information of the industrial site. The natural language scheduling instructions refer to the scheduling commands for AMR or AGV input by the operator in natural language form, such as "transfer the goods in area A to area B". After word segmentation and word embedding, the instructions are input into the large language model encoder to obtain the instruction semantic vector. The instruction semantic vector and the physical state vector are concatenated and input into the cross-modal attention layer. The cross-modal attention layer performs attention operation with the instruction semantic vector as the query vector and the physical state vector as the key vector, so that each semantic unit in the natural language scheduling instructions establishes an explicit association with the current physical state of the industrial site. The fusion encoding result is the multimodal interaction context, which serves as the input for word-by-word generation by the large language model agent in step S4.

[0018] Step S4: In the process of generating word-by-word pronouns by the large language model agent with multimodal interaction context as input, the stepwise anchor confidence is calculated based on the cosine similarity between the physical state vector and the current hidden layer state vector, and the original word probability distribution and the physical constraint preference distribution are weighted and mixed according to the anchor confidence to obtain the modulated word probability distribution. Specifically, the stepwise anchor confidence refers to the quantitative scalar of the semantic consistency between the current language inference path and the physical state of the industrial site at each step of word generation in the large language model agent. Its value ranges from (0,1), with higher values ​​indicating greater consistency between the current inference direction and physical reality. In the specific calculation, the output hidden vector of the last Transformer layer of the large language model agent for the currently generated word sequence is extracted as the current hidden state vector. This vector is then normalized to the L2 norm with the physical state vector, and a dot product operation is performed to obtain the cosine. Similarity, with cosine similarity ranging from [-1, 1], is adjusted by a scaling factor of 5.0 and a bias term of -2.0 before being input into the Sigmoid function and mapped to the (0, 1) interval to obtain the original confidence score for the current step. The scaling factor of 5.0 and the bias term of -2.0 ensure that when the cosine similarity is higher than 0.4, the original confidence score is greater than 0.5, and when it is lower than 0.2, the original confidence score is lower than 0.27, thus establishing a clear numerical boundary between acceptable and unacceptable physical consistency. The original confidence score for the current step is then weighted by a time-decay factor of 0.7. The stepwise anchor confidence is obtained by weighting the previous anchor confidence with a complement of 0.3 and then summing the results. A time-decay coefficient of 0.7 ensures that the current physical state has a dominant influence on the confidence while retaining the smoothing effect of historical confidence. The physical constraint preference distribution refers to the word probability distribution obtained by mapping the physical state vector to the vocabulary dimension via a linear transformation through the physical preference projection layer, scaling it with a temperature coefficient of 0.5, and then processing it using softmax. The temperature coefficient of 0.5, which is less than 1, is used to sharpen the distribution, making the advantage of high-preference word probabilities more significant. Modulation word probability distribution... The modulation lexical probability distribution is obtained by weighting the original lexical probability distribution and the physical constraint preference distribution using the stepwise anchor confidence as the first mixed weight and the difference between the stepwise anchor confidence and 1 as the second mixed weight. When the stepwise anchor confidence approaches 1, the modulation lexical probability distribution approaches the original lexical probability distribution. When the stepwise anchor confidence approaches 0, the modulation lexical probability distribution approaches the physical constraint preference distribution. Thus, the physical state of the industrial site is embedded into the language reasoning process in real time at each lexical generation moment, which is different from the existing technology of performing post-verification after the large language model has completed the complete output.

[0019] Step S5: When the continuous window mean of the anchor confidence and the descent slope are both lower than the preset threshold, an uncertain semantic vector is constructed based on the difference between the physical state vector and the current hidden layer state vector. The uncertain semantic vector is then injected into the large language model agent to generate a security degradation instruction, which is then sent to the autonomous mobile robot and the automated guided vehicle for execution.

[0020] Specifically, the deterministic semantic vector refers to the vector obtained by subtracting the physical state vector from the current hidden state vector at the time of triggering the circuit breaker and then performing layer normalization. The direction of this vector captures the drift direction between physical reality and the language reasoning space, and its L2 norm reflects the magnitude of the drift. The continuous window length is set to 10 lexical generation steps. The window length of 10 is based on the average lexical sequence length of industrial scheduling instructions. Within this length, a continuous decrease in confidence is sufficient to determine a systematic deviation rather than a transient disturbance. The mean circuit breaker threshold is set to 0.35, and the slope circuit breaker threshold is set to -0.02. Both thresholds must be met simultaneously to trigger the circuit breaker. Only when the mean is below the threshold and the slope does not continuously decrease is it determined to be a transient disturbance and the circuit breaker is not triggered. This dual-condition design avoids the influence of instantaneous... Sensor noise causes false triggering; when the L2 norm of the uncertainty semantic vector is lower than the first amplitude threshold of 0.5, the safety degradation level is level one, generating a deceleration continuation command and switching to manual confirmation mode; when the L2 norm is between 0.5 and the second amplitude threshold of 1.2, the safety degradation level is level two, generating a wait-in command and switching to conservative path replanning mode, with a replanning time window of 30 seconds; when the L2 norm is higher than 1.2, the safety degradation level is level three, generating a return to the safe zone command and sending a system-level alarm signal to the Andon management platform to trigger manual intervention. The above three thresholds are determined by industrial safety assessment experiments. After structured parsing, the safety degradation command is converted into a control message that can be directly executed by the autonomous mobile robot and the automated guided vehicle and sent to the corresponding device terminal.

[0021] In one specific embodiment, step S1 includes: Kalman filtering is applied to the pose data stream of the autonomous mobile robot to obtain the pose feature matrix composed of the two-dimensional coordinates, orientation angle and velocity vector of each autonomous mobile robot. Visual image frames are input to a visual encoder for feature extraction to obtain visual semantic feature vectors; The load readings of the storage locations are normalized to obtain the storage location occupancy vector. The alarm status codes of the production line are processed by unique hot encoding to obtain the alarm status vector; Based on a unified acquisition period timestamp, the pose feature matrix, visual semantic feature vector, cargo space occupancy vector, and alarm status vector are timestamped to obtain a multimodal acquisition dataset.

[0022] Specifically, when performing Kalman filtering on the pose data stream of the autonomous mobile robot, the Kalman filter takes the original position and attitude measurements output by the AMR's own sensors as the observation input, uses the AMR kinematic model as the state transition equation, iteratively predicts and updates the original measurement noise, and outputs the smooth position and attitude estimates of each AMR at the current acquisition time, specifically including two-dimensional plane coordinates, orientation angle, and velocity vector. These four quantities are arranged in rows to obtain a pose feature matrix with 4 rows and the number of columns equal to the number of currently online AMRs. When extracting features from the visual image frames input to the visual encoder, the visual encoder uses a ViT-B / 16 backbone network. It segments the input RGB image into a 16×16 pixel image block sequence and then encodes it using Transformer to output a 768-dimensional visual semantic feature vector. This vector captures the scene semantic information of the production site. The visual encoder uses a ViT-B / 16 backbone network, which takes the RGB image frames acquired from the production site as input and segments the input image into... A fixed-size sequence of image patches of 16×16 pixels is generated. Each image patch is flattened into a one-dimensional vector by a linear projection layer and then a positional encoding is added to obtain an image patch embedding sequence. The image patch embedding sequence is concatenated with an additional classification label vector and then input into a backbone network consisting of 12 stacked Transformer encoders. Each Transformer encoder layer contains a multi-head self-attention module and a feedforward neural network module. The multi-head self-attention module has 12 attention heads, each with a dimension of 64. The hidden layer dimension of the feedforward neural network module is 3072. The 12 Transformer encoder layers sequentially perform feature extraction and semantic abstraction on the image patch embedding sequence. Finally, the 768-dimensional hidden layer vector corresponding to the classification label vector at the output of the last Transformer encoder layer is taken as the global semantic representation of the entire frame image. This 768-dimensional vector is the visual semantic feature vector, which captures scene semantic information such as cargo location, aisle occupancy, and robot distribution contained in the RGB image of the production site.

[0023] When normalizing the load readings of the storage locations, the original load values ​​output by each storage location sensor are divided by the rated load limit of the corresponding storage location to obtain a storage location occupancy vector with a value range of 0 to 1. The vector dimension is the total number of storage locations in the automated warehouse. When performing one-hot encoding on the production line alarm status codes, the total number of alarm categories is used as the vector dimension. The position corresponding to the currently active alarm category is set to 1, and the other positions are set to 0 to obtain an alarm status vector with the dimension of the total number of alarm categories.

[0024] When performing timestamp alignment processing on the above four types of outputs based on a unified acquisition cycle timestamp, the acquisition cycle is set to 20 milliseconds. This value is determined according to the common standard of industrial AMR / AGV control cycle. A control cycle shorter than this ensures that the pose feature matrix, visual semantic feature vector, storage location occupancy vector, and alarm status vector all reflect the industrial site status at the same physical moment. This avoids the introduction of timing misalignment errors caused by inconsistent acquisition times of different modal data, which would result in different physical moments corresponding to different modal information in subsequent physical status vectors. After the four types of data are aligned under the same timestamp, they together constitute a multimodal acquisition dataset. The multimodal acquisition dataset refers to the ordered set of four types of modal data that have been forcibly aligned by timestamp within a single acquisition cycle. Among them, the pose feature matrix corresponds to the robot motion state, the visual semantic feature vector corresponds to the semantic state of the production site scene, the storage location occupancy vector corresponds to the physical state of the warehouse, and the alarm status vector corresponds to the abnormal state of the production line. Together, they completely describe the physical reality of the industrial site within a single acquisition cycle.

[0025] In one specific embodiment, step S2 includes: The pose feature matrix in the multimodal acquisition dataset is subjected to row-direction max pooling to obtain the global pose summary vector. The global pose summary vector is then mapped to the hidden layer dimension space of the large language model through a linear projection layer to obtain the pose modality embedding vector. The visual semantic feature vectors in the multimodal acquisition dataset are mapped to the hidden layer dimension space of the large language model through a linear projection layer to obtain the visual modality embedding vectors. The location occupancy vector in the multimodal data collection is compressed by a sparse coding layer and then mapped to the hidden layer dimension space of the large language model through a linear projection layer to obtain the warehouse modality embedding vector. The alarm state vectors in the multimodal acquisition dataset are mapped to the hidden layer dimension space of the large language model through a linear projection layer to obtain the alarm modality embedding vectors. Based on the preset modal weight coefficients, the pose modal embedding vector, visual modal embedding vector, storage modal embedding vector, and alarm modal embedding vector are weighted and summed to obtain the physical state vector.

[0026] Specifically, when performing row-direction max pooling on the pose feature matrix in the multimodal acquisition dataset, the pose feature matrix has 4 rows and the number of columns is the current number of online AMRs. Row-direction max pooling takes the maximum value of each row along the column direction, compressing the pose feature matrix with 4 rows and the number of columns into a 4-dimensional global pose summary vector. The global pose summary vector extracts the maximum response values ​​of all current online AMRs in four dimensions: two-dimensional coordinates, orientation angle, and velocity vector, reflecting the overall motion boundary state of the AMR cluster in the industrial field at the current moment. The global pose summary vector is mapped to the hidden layer dimension space of the large language model through a linear projection layer. The input dimension of this linear projection layer is 4 and the output dimension is 4096. The mapping result is the pose modality embedding vector. The semantic feature vector has a dimension of 768. After mapping through a linear projection layer with an input dimension of 768 and an output dimension of 4096, the visual modality embedding vector is obtained. The dimension of the storage location occupancy vector is the total number of storage locations in the automated warehouse. This dimension is usually much larger than 4096 and contains a large number of sparse elements close to zero. After being compressed to 256 dimensions by a sparse coding layer, it is then mapped through a linear projection layer with an input dimension of 256 and an output dimension of 4096 to obtain the storage modality embedding vector. The sparse coding layer sets the compression target dimension of 256 to keep the computational load within a reasonable range while retaining the main structural information of the storage location occupancy distribution. The alarm status vector has a dimension of the total number of alarm categories. After mapping through a linear projection layer with an input dimension of the total number of alarm categories and an output dimension of 4096, the alarm modality embedding vector is obtained.

[0027] When weighting and summing the pose modal embedding vector, visual modal embedding vector, storage modal embedding vector, and alarm modal embedding vector according to preset modal weight coefficients, the pose weight coefficient is set to 0.40, the visual weight coefficient to 0.25, the storage weight coefficient to 0.20, and the alarm weight coefficient to 0.15. The sum of the four weight coefficients is strictly equal to 1. The weight coefficients are set based on the degree of influence of each modal information on scheduling decisions in industrial scheduling scenarios. The pose weight has the highest value, reflecting the decisive influence of the AMR cluster's motion state on the generation of scheduling instructions. The visual weight is next, reflecting the auxiliary role of scene semantics in path planning. The storage weight and alarm weight decrease in that order, reflecting their reference status in scheduling decisions. The above weight coefficients are obtained from offline... During the training phase, the industrial scheduling accuracy is optimized using gradient descent. The embedding vectors of the four modalities are all located in the hidden layer dimension space of the large language model with a dimension of 4096. The weighted summation result is the physical state vector. The physical state vector refers to the fusion vector that uniformly expresses the physical information of the four modalities in the industrial site in the hidden layer dimension space of the large language model. Its dimension is the same as that of the hidden layer state vector of the large language model. In the word generation process, the physical state vector and the current hidden layer state vector generated at each step of the large language model are in the same mathematical space. The cosine similarity between the two directly reflects the semantic consistency between the current language reasoning path and the physical state of the industrial site, thus imposing physical constraints on the language reasoning process at each word generation moment.

[0028] In one specific embodiment, step S3 includes: The natural language scheduling instructions input by the operator are segmented and embedded to obtain the instruction word sequence; The instruction word sequence is input into the large language model encoder for semantic encoding processing to obtain the instruction semantic vector; The physical state vector and the instruction semantic vector are concatenated to obtain the physical-instruction joint vector. The physical-instruction joint vector is fused and encoded through a cross-modal attention layer to obtain a multimodal interaction context.

[0029] Specifically, when performing word segmentation and lexical embedding on the natural language scheduling instructions input by the operator, the word segmentation operation divides the natural language scheduling instructions into several lexical units based on the byte-pair encoding vocabulary used by the large language model. The vocabulary size of the byte-pair encoding vocabulary is 32,000. Each lexical unit is mapped into a dense vector with a dimension of 4096 through the lexical embedding layer. The weight matrix of the lexical embedding layer has a dimension of 32,000 rows and 4096 columns. The embedding vectors of all lexical units are arranged in the input order to form the instruction lexical unit sequence. The length of the instruction lexical unit sequence depends on the number of lexical units in the natural language scheduling instructions after word segmentation. The instruction lexical unit sequence is then input into the large language model encoder for processing. During semantic encoding, the large language model encoder performs multi-layer self-attention operations on the embedding vectors of all words in the instruction word sequence. There are 32 attention layers, 32 attention heads, and each head has a dimension of 128. This captures the semantic dependencies between words. The encoder then performs mean pooling on all word hidden vectors output from the last layer along the sequence length, compressing the variable-length word hidden vector sequence into a single vector of fixed dimension, resulting in an instruction semantic vector with a dimension of 4096. The mean pooling operation is designed to unify the variable-length sequence into a fixed-dimensional representation while preserving all word semantic information, so that it can be concatenated with the physical state vector later.

[0030] When concatenating the physical state vector and the instruction semantic vector, the 4096-dimensional physical state vector and the 4096-dimensional instruction semantic vector are directly concatenated end-to-end along the feature dimension. The physical state vector is arranged in the first 4096 dimensions, and the instruction semantic vector is arranged in the last 4096 dimensions, resulting in a physical-instruction joint vector of dimension 8192. The first 4096 dimensions of the physical-instruction joint vector correspond to the physical state information of the industrial site, and the last 4096 dimensions correspond to the semantic information of the operator's natural language scheduling instructions. The two parts of information coexist in this vector through concatenation, but cross-modal semantic association has not yet been established. When the physical-instruction joint vector is fused and encoded through a cross-modal attention layer, the cross-modal attention layer uses the last 4096 dimensions of the physical-instruction joint vector, i.e., the instruction semantic vector, as the basis for encoding. Semantic information is used as the query vector, and the first 4096 dimensions of the physical-instruction joint vector, i.e., physical state information, are used as the key and value vectors. A scaled dot product attention operation is performed with 16 attention heads, each with a dimension of 256. The product of 16 and 256 equals 4096, which is consistent with the input dimension. The scaling factor is the square root of the dimension of 256 for each head, i.e., 16. The operation result is output through a linear projection layer with a dimension of 4096 to obtain the multimodal interaction context. The multimodal interaction context refers to the attention-weighted fusion representation obtained by using each semantic unit in the natural language scheduling instruction as the query and the physical state of the industrial site as the key value. Each semantic unit of the scheduling instruction has been explicitly associated with the current physical state of the industrial site with attention weights. The multimodal interaction context has a dimension of 4096.

[0031] In one specific embodiment, step S4, which calculates the progressive anchor confidence based on the cosine similarity between the physical state vector and the current hidden layer state vector, includes: Before the large language model agent performs the k-th step of word generation, extract the output hidden layer vector of the last Transformer layer of the large language model agent for the currently generated word sequence to obtain the current hidden layer state vector; After normalizing the current hidden state vector and the physical state vector with the L2 norm, a dot product operation is performed to obtain the cosine similarity of the current step. The value range of the cosine similarity of the current step is [-1, 1]. The higher the value, the higher the semantic consistency between the current word generation direction and the physical state of the industrial site. Multiply the current step cosine similarity by a preset scaling factor, add a preset bias term, and input the result into the Sigmoid function for interval mapping to obtain the current step original confidence score with a value range of (0,1). The current step's original confidence level and the previous step's anchor point confidence level are multiplied by a preset time-series decay coefficient and its complement, and then summed to obtain the stepwise anchor point confidence level. The preset time-series decay coefficient controls the weight of the current step's original confidence level and the historical anchor point confidence level in the stepwise anchor point confidence level.

[0032] Specifically, before the large language model agent performs the k-th step of lexical generation, when extracting the output hidden vector of the last Transformer layer of the large language model agent for the currently generated lexical sequence, the large language model agent contains a total of 32 Transformer layers. The output of the last, i.e., the 32nd Transformer layer, is a hidden vector sequence with a dimension of 4096. The hidden vector corresponding to the position of the last lexical in the currently generated lexical sequence is taken as the current hidden state vector. This vector has a dimension of 4096, reflecting the large language model agent's approach to the next step based on the currently generated lexical sequence. The semantic tendency of each word generation direction; when the current hidden state vector and the physical state vector are respectively normalized by L2 norm, the L2 norm normalization operation divides the vector by the square root of the sum of the squares of its elements. After normalization, the L2 norm of both vectors is 1. The dot product operation of the two normalized vectors is performed by multiplying the elements one by one and then summing them. The current step cosine similarity is strictly in the range of [-1,1]. When the current step cosine similarity is 1, it means that the current hidden state vector and the physical state vector are completely consistent in direction. When it is -1, it means that the two vectors are completely opposite in direction. When it is 0, it means that the two vectors are orthogonal and that there is no semantic relationship.

[0033] Multiplying the current cosine similarity by a preset scaling factor of 5.0 and adding a preset bias term of -2.0, when inputting this into the Sigmoid function for interval mapping, the Sigmoid function maps any real number to the (0,1) interval. The scaling factor of 5.0 ensures that the Sigmoid function's output value varies significantly within the range of cosine similarity. The bias term of -2.0 ensures that the Sigmoid function's output value is greater than 0.5 when the cosine similarity is 0.4 and less than 0.27 when the cosine similarity is less than 0.2. This creates a clear numerical boundary between acceptable and unacceptable physical consistency, thus obtaining the value range. The current step's original confidence is (0,1). The current step's original confidence and the previous step's anchor confidence are multiplied by a preset temporal decay coefficient of 0.7 and its complement of 0.3, respectively, and then summed to obtain the stepwise anchor confidence. The preset temporal decay coefficient of 0.7 makes the current step's original confidence dominate the stepwise anchor confidence, and the complement of 0.3 makes the historical anchor confidence retain a smoothing effect on the stepwise anchor confidence. When k equals 1 in the first step of word generation, the initial value of the previous step's anchor confidence is set to 1.0. This initial value is set based on the assumption that language reasoning and physical state are completely consistent at the start of generation. Subsequently, it is gradually updated by the actual calculated current step's original confidence as word generation progresses.

[0034] In one specific embodiment, step S4 involves weighting and mixing the original lexical probability distribution and the physical constraint preference distribution according to the anchor confidence level to obtain the modulated lexical probability distribution, including: The large language model agent performs softmax normalization on the original logits vector output by the entire vocabulary in the k-th step to obtain the original probability distribution of each word selected in the vocabulary. The physical state vector is input into the physical preference projection layer and linearly transformed to map to the physical preference logits vector with the same dimension as the vocabulary. Then, the physical preference logits vector is scaled by a preset temperature coefficient and softmax processing is performed to obtain the physical constraint preference distribution that reflects the degree of preference of each word in the physical state of the industrial site. The preset temperature coefficient is less than 1 to sharpen the physical constraint preference distribution and make the probability advantage of high-preference words more significant. Using the stepwise anchor confidence as the first mixed weight and the difference between the stepwise anchor confidence and 1 as the second mixed weight, the original word probability distribution and the physical constraint preference distribution are weighted and summed to obtain the modulated word probability distribution. When the stepwise anchor confidence approaches 1, the modulated word probability distribution approaches the original word probability distribution, and when the stepwise anchor confidence approaches 0, the modulated word probability distribution approaches the physical constraint preference distribution.

[0035] Specifically, when the large language model agent performs softmax normalization on the original score vector output by the entire vocabulary in step k, the weight matrix of the output layer of the large language model agent has a dimension of 4096 rows and 32000 columns. Multiplying the current hidden layer state vector of dimension 4096 with this weight matrix yields the original score vector of dimension 32000. The real value at each position in the original score vector corresponds to the original tendency intensity of the corresponding word in the vocabulary being selected in the current generation step. Softmax normalization is performed on this original score vector. Softmax takes the exponent of each real value at each position and divides it by the sum of the exponent values ​​of all 32000 positions, obtaining the original word probability distribution composed of the selection probabilities of each of the 32000 words. Each element in the original word probability distribution takes a value in the range (0,1) and the sum of all elements is strictly equal to 1. The physical state vector is input into the physical... When the preference projection layer undergoes linear transformation, the weight matrix of the physical preference projection layer has a dimension of 4096 rows and 32000 columns. Multiplying the physical state vector of dimension 4096 with this weight matrix yields a physical preference original score vector of dimension 32000. The real value at each position in the physical preference original score vector reflects the preference intensity of the industrial site physical state for the corresponding word in the vocabulary. After scaling all elements in the physical preference original score vector by division with a preset temperature coefficient of 0.5, softmax processing is performed. The temperature coefficient of 0.5 is less than 1, which increases the numerical difference between the elements after scaling. After softmax processing, the probability value of the high preference word is more significantly different from the probability value of the low preference word, resulting in a physical constraint preference distribution that reflects the degree of preference of the industrial site physical state for each word. The physical constraint preference distribution has the same dimension of 32000 and the sum of all elements is strictly equal to 1.

[0036] Using the stepwise anchor confidence as the first mixture weight and the difference between the stepwise anchor confidence and 1 as the second mixture weight, when weighted summing is performed on the original lexical probability distribution and the physical constraint preference distribution respectively, the sum of the first mixture weight and the second mixture weight is strictly equal to 1. Multiplying each of the 32,000 elements in the original lexical probability distribution by the first mixture weight and each of the 32,000 elements in the physical constraint preference distribution by the second mixture weight, and then summing them digit by digit, yields a modulated lexical probability distribution with the same dimension of 32,000 and a sum of all elements strictly equal to 1. The modulated lexical probability distribution refers to the probability distribution obtained after applying physical constraints to the original lexical generation probability of the large language model agent at each lexical generation time based on the physical state of the industrial site. The resulting mixed probability distribution; when the stepwise anchor confidence is 1, the first mixed weight is 1 and the second mixed weight is 0, the modulated lexical probability distribution is exactly the same as the original lexical probability distribution, and the physical constraint preference distribution does not interfere with lexical generation; when the stepwise anchor confidence is 0, the first mixed weight is 0 and the second mixed weight is 1, the modulated lexical probability distribution is exactly the same as the physical constraint preference distribution, and the original lexical generation probability proxies of the large language model are completely replaced by the physical constraint preference distribution determined by the physical state vector; when the stepwise anchor confidence takes the middle value, the modulated lexical probability distribution mixes the two proportionally, and the physical state intervenes in the lexical generation process with a weight positively correlated with the degree of semantic drift.

[0037] In one specific embodiment, step S5 includes: After each word generation step is completed, the mean of the continuous window and the descent slope obtained by least squares linear regression of the historical sequence of anchor confidence within the continuous window length are calculated respectively. When the mean of the continuous window is lower than the mean circuit breaker threshold and the descent slope is lower than the slope circuit breaker threshold, the circuit breaker decision is triggered. Based on the circuit breaker decision, the physical state vector is subtracted from the current hidden layer state vector at the moment of circuit breaker triggering and then subjected to layer normalization to obtain the uncertainty semantic vector. The L2 norm of the uncertainty semantic vector reflects the drift magnitude between the physical state vector and the current hidden layer state vector. The security downgrade level is obtained by comparing the L2 norm of the uncertain semantic vector with the preset first amplitude threshold and second amplitude threshold. The security downgrade level is as follows: when the L2 norm is lower than the first amplitude threshold, the security downgrade level is level 1; when the L2 norm is between the first amplitude threshold and the second amplitude threshold, the security downgrade level is level 2; and when the L2 norm is higher than the second amplitude threshold, the security downgrade level is level 3. The uncertain semantic vector is transformed by a linear projection layer and injected as a prefix hidden state into the large language model agent. The large language model agent is driven to generate corresponding security degradation instructions according to the security degradation level, and the security degradation instructions are sent to the autonomous mobile robot and the automated guided vehicle for execution.

[0038] Specifically, after each lexical generation step is completed, the mean and descent slope of the continuous window are calculated based on the historical sequence of the stepwise anchor confidence within the continuous window length. The continuous window length is set to 10 lexical generation steps. This value is based on the average lexical sequence length of industrial scheduling instructions. Within this length, a continuous decrease in the stepwise anchor confidence is sufficient to determine systematic semantic drift rather than a transient disturbance. The continuous window mean is the arithmetic mean of the 10 stepwise anchor confidence values ​​within the window. The descent slope is calculated using least squares linear regression. Least squares linear regression uses the sequence numbers 1 to 10 of the 10 lexical generation steps within the window as independent variables and the corresponding 10 stepwise anchor confidence values ​​as dependent variables. The slope parameter of the linear equation is solved by minimizing the sum of squares of the differences between the actual value and the linearly fitted value. A negative slope parameter indicates that the confidence level of the stepwise anchor points is decreasing within the window, and the larger the absolute value of the slope, the faster the decrease. The mean circuit breaker threshold is set to 0.35, and the slope circuit breaker threshold is set to -0.02. When the mean of the continuous window is lower than 0.35 and the decreasing slope is lower than -0.02, both conditions are met simultaneously to trigger the circuit breaker decision. The mean circuit breaker threshold of 0.35 is set based on the fact that when the confidence level of the stepwise anchor points is lower than this value, the semantic consistency between the language reasoning path and the physical state is unacceptable. The slope circuit breaker threshold of -0.02 is set based on the fact that the confidence level of the stepwise anchor points within the window corresponding to this slope decreases by an average of 0.02 per step and a cumulative decrease of 0.2 over 10 steps, which is a continuous system deviation rather than a short-term disturbance. Both conditions must be met simultaneously to avoid false triggering of the circuit breaker due to instantaneous sensor noise.

[0039] Based on the circuit breaker decision, after subtracting the physical state vector from the current hidden state vector at the moment of circuit breaker triggering and then performing layer normalization, the physical state vector has a dimension of 4096, and the current hidden state vector also has a dimension of 4096. Subtracting them element-wise yields a difference vector of dimension 4096. This difference vector reflects the drift direction and magnitude between physical reality and the language reasoning space. Layer normalization is then performed on this difference vector. Layer normalization subtracts the mean from the difference vector, divides it by its standard deviation, and then multiplies it by the learnable scaling parameter and the learnable bias parameter. The output is an uncertain semantic vector of dimension 4096. Layer normalization stabilizes the numerical range of the uncertain semantic vector, thus facilitating subsequent linear projection. Image processing; when comparing the L2 norm of the uncertain semantic vector with the first amplitude threshold of 0.5 and the second amplitude threshold of 1.2, the L2 norm is the square root of the sum of squares of all 4096 elements of the uncertain semantic vector. When the L2 norm is lower than 0.5, it is judged as mild drift and the safety degradation level is level 1. When the L2 norm is between 0.5 and 1.2, it is judged as moderate drift and the safety degradation level is level 2. When the L2 norm is higher than 1.2, it is judged as severe drift and the safety degradation level is level 3. The first amplitude threshold of 0.5 and the second amplitude threshold of 1.2 are set based on the classification results of the AMR / AGV misoperation risk level under the three types of drift amplitude in the industrial safety assessment experiment.

[0040] When the uncertain semantic vector is transformed by a linear projection layer and injected as the prefix hidden state into the large language model agent, the input dimension of the linear projection layer is 4096 and the output dimension is 4096. Multiplying the uncertain semantic vector by the weight matrix of the linear projection layer and adding a bias vector yields a prefix hidden state of dimension 4096. The prefix hidden state refers to a special vector that replaces the initial hidden state during normal reasoning of the large language model agent. Injecting this vector into the input of the first Transformer layer of the large language model agent allows the agent's downgraded reasoning to proceed from the direction of physical drift bias rather than re-reasoning from a blank state. When the security downgrade level is level one, the large language model agent operates in the prefix hidden state... Under state guidance, a deceleration continuation command is generated, corresponding to the AMR / AGV speed multiplied by a decay coefficient of 0.5 and switching to manual confirmation mode; when the safety degradation level is level two, an in-place waiting command is generated, and all AMR / AGVs in the current scheduling task immediately stop moving and switch to conservative path replanning mode, with a replanning time window of 30 seconds; when the safety degradation level is level three, a return to safe zone command is generated, and all AMR / AGVs return to the nearest safe docking point according to the pre-stored safe path and send a system-level alarm signal to the Andon management platform to trigger manual intervention. After structured parsing, the safety degradation command is converted into a control message that can be directly executed by the autonomous mobile robot and the automated guided vehicle and sent to the corresponding device terminal.

[0041] The above describes the industrial multimodal intelligent interaction method based on a large language model agent in the embodiments of this application. The following describes the industrial multimodal intelligent interaction system based on a large language model agent in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 2 One embodiment of the industrial multimodal intelligent interaction system based on a large language model agent in this application includes: The alignment module is used to align the autonomous mobile robot pose data stream, visual image frames, cargo load readings, and production line alarm status codes with timestamps to obtain a multimodal acquisition dataset. The weighting module is used to project the modal features in the multimodal acquisition dataset onto the hidden layer dimension space of the large language model and then fuse them in a weighted manner to obtain the physical state vector. The encoding module is used to jointly encode the natural language scheduling instructions input by the operator and the physical state vector to obtain a multimodal interaction context; The calculation module is used to calculate the stepwise anchor confidence based on the cosine similarity between the physical state vector and the current hidden layer state vector during the word-by-word generation process of the large language model agent with the multimodal interaction context as input, and to perform a weighted mixture of the original word probability distribution and the physical constraint preference distribution according to the anchor confidence to obtain the modulated word probability distribution. The execution module is used to construct an uncertain semantic vector based on the difference between the physical state vector and the current hidden layer state vector when the continuous window mean and the descent slope of the anchor confidence are both lower than a preset threshold, and to inject the uncertain semantic vector into the large language model agent to generate a security degradation instruction, and to send the security degradation instruction to the autonomous mobile robot and the automated guided vehicle for execution.

[0042] above Figure 2 The industrial multimodal intelligent interaction system based on a large language model agent in this embodiment of the invention is described in detail from the perspective of modular functional entities. The industrial multimodal intelligent interaction device based on a large language model agent in this embodiment of the invention is described in detail from the perspective of hardware processing.

[0043] Reference Figure 3 This invention also provides an industrial multimodal intelligent interaction device based on a large language model agent. This device can be a server, and its internal structure can be as follows: Figure 3As shown, the industrial multimodal intelligent interactive device based on a large language model agent includes a processor, memory, display screen, input device, network interface, and database connected via a system bus. The processor, designed as a computer, provides computing and control capabilities. The memory of the industrial multimodal intelligent interactive device based on a large language model agent includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the industrial multimodal intelligent interactive device based on a large language model agent is used to store the data corresponding to this embodiment. The network interface of the industrial multimodal intelligent interactive device based on a large language model agent is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the above-described method.

[0044] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the industrial multimodal intelligent interactive device based on a large language model agent to which the present invention is applied.

[0045] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the industrial multimodal intelligent interaction method based on a large language model agent.

[0046] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0047] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an industrial multimodal intelligent interactive device (which may be a personal computer, server, or network device, etc.) based on a large language model agent to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0048] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An industrial multimodal intelligent interaction method based on a large language model agent, characterized in that, The method includes: Step S1: The autonomous mobile robot pose data stream, visual image frames, cargo load readings, and production line alarm status codes are timestamped to obtain a multimodal acquisition dataset. Step S2: Projecting the modal features of each modality in the multimodal acquisition dataset to the hidden layer dimension space of the large language model and then weighted and fused to obtain a physical state vector, includes: performing row-direction max pooling on the pose feature matrix in the multimodal acquisition dataset to obtain a global pose summary vector, and mapping the global pose summary vector to the hidden layer dimension space of the large language model through a linear projection layer to obtain a pose modality embedding vector; mapping the visual semantic feature vector in the multimodal acquisition dataset to the hidden layer dimension space of the large language model through a linear projection layer to obtain a visual modality embedding vector. The input vector is obtained by compressing the location occupancy vector in the multimodal acquisition dataset through a sparse coding layer and then mapping it to the hidden layer dimension space of the large language model through a linear projection layer. The alarm state vector in the multimodal acquisition dataset is also obtained by mapping it to the hidden layer dimension space of the large language model through a linear projection layer. The alarm state vector is obtained by weighting and summing the pose modality embedding vector, the visual modality embedding vector, the storage modality embedding vector, and the alarm modality embedding vector according to preset modality weight coefficients. Step S3: Jointly encode the natural language scheduling instructions input by the operator and the physical state vector to obtain a multimodal interaction context, including: performing word segmentation and lexical embedding processing on the natural language scheduling instructions input by the operator to obtain an instruction lexical sequence; inputting the instruction lexical sequence into a large language model encoder for semantic encoding processing to obtain an instruction semantic vector; concatenating the physical state vector and the instruction semantic vector to obtain a physical-instruction joint vector; and performing fusion encoding processing on the physical-instruction joint vector through a cross-modal attention layer to obtain the multimodal interaction context. Step S4: During the word-by-word generation process of the large language model agent with the multimodal interaction context as input, the stepwise anchor confidence is calculated based on the cosine similarity between the physical state vector and the current hidden layer state vector, and the original word probability distribution and the physical constraint preference distribution are weighted and mixed according to the anchor confidence to obtain the modulated word probability distribution. Step S5: When the continuous window mean and the descent slope of the anchor confidence are both lower than a preset threshold, an uncertainty semantic vector is constructed based on the difference between the physical state vector and the current hidden layer state vector. The uncertainty semantic vector is then injected into the large language model agent to generate a security degradation instruction, which is then sent to the autonomous mobile robot and the automated guided vehicle for execution.

2. The industrial multimodal intelligent interaction method based on a large language model agent according to claim 1, characterized in that, Step S1 includes: Kalman filtering is applied to the pose data stream of the autonomous mobile robot to obtain the pose feature matrix composed of the two-dimensional coordinates, orientation angle and velocity vector of each autonomous mobile robot. Visual image frames are input to a visual encoder for feature extraction to obtain visual semantic feature vectors; The load readings of the storage locations are normalized to obtain the storage location occupancy vector. The alarm status codes of the production line are processed by unique hot encoding to obtain the alarm status vector; Based on a unified acquisition period timestamp, the pose feature matrix, the visual semantic feature vector, the cargo space occupancy vector, and the alarm status vector are timestamped to obtain the multimodal acquisition dataset.

3. The industrial multimodal intelligent interaction method based on a large language model agent according to claim 1, characterized in that, Step S4, which calculates the progressive anchor confidence based on the cosine similarity between the physical state vector and the current hidden layer state vector, includes: Before the large language model agent performs the k-th step of word generation, extract the output hidden layer vector of the last Transformer layer of the large language model agent for the currently generated word sequence to obtain the current hidden layer state vector; After normalizing the current hidden state vector and the physical state vector with the L2 norm, a dot product operation is performed to obtain the current step cosine similarity. The value range of the current step cosine similarity is [-1, 1], where a higher value indicates a higher degree of semantic consistency between the current word generation direction and the physical state of the industrial site. The cosine similarity of the current step is multiplied by a preset scaling factor, and then a preset bias term is added. The result is then input into the Sigmoid function for interval mapping to obtain the original confidence of the current step with a value range of (0,1). The current step original confidence and the previous step anchor confidence are multiplied by a preset time decay coefficient and its complement, and then summed to obtain the stepwise anchor confidence. The preset time decay coefficient controls the weight of the current step original confidence and the historical anchor confidence in the stepwise anchor confidence.

4. The industrial multimodal intelligent interaction method based on a large language model agent according to claim 3, characterized in that, In step S4, the original lexical probability distribution and the physical constraint preference distribution are weighted and mixed according to the anchor confidence level to obtain the modulated lexical probability distribution, including: The large language model agent performs softmax normalization on the original logits vector output by the entire vocabulary in the k-th step to obtain the original word probability distribution of each word selected in the vocabulary. The physical state vector is input into the physical preference projection layer for linear transformation and mapped to a physical preference logits vector with the same dimension as the vocabulary. The physical preference logits vector is then scaled by a preset temperature coefficient and softmax processing is performed to obtain the physical constraint preference distribution that reflects the degree of preference of each word in the physical state of the industrial site. The preset temperature coefficient is less than 1 to sharpen the physical constraint preference distribution and make the probability advantage of high-preference words more significant. Using the stepwise anchor confidence as the first mixed weight and the difference between the stepwise anchor confidence and 1 as the second mixed weight, the original lexical probability distribution and the physical constraint preference distribution are weighted and summed to obtain the modulated lexical probability distribution. When the stepwise anchor confidence approaches 1, the modulated lexical probability distribution approaches the original lexical probability distribution. When the stepwise anchor confidence approaches 0, the modulated lexical probability distribution approaches the physical constraint preference distribution.

5. The industrial multimodal intelligent interaction method based on a large language model agent according to claim 1, characterized in that, Step S5 includes: After each word generation step is completed, the mean of the continuous window and the descent slope obtained by least squares linear regression of the historical sequence of the step-by-step anchor confidence within the continuous window length are calculated respectively. When the mean of the continuous window is lower than the mean circuit breaker threshold and the descent slope is lower than the slope circuit breaker threshold, the circuit breaker decision is triggered. Based on the circuit breaker decision, the physical state vector is subtracted from the current hidden layer state vector at the time of circuit breaker triggering, and then subjected to layer normalization to obtain the uncertainty semantic vector. The L2 norm of the uncertainty semantic vector reflects the drift amplitude between the physical state vector and the current hidden layer state vector. The L2 norm of the uncertain semantic vector is compared with a preset first amplitude threshold and a second amplitude threshold to obtain a security downgrade level. The security downgrade level is level 1 when the L2 norm is lower than the first amplitude threshold, level 2 when the L2 norm is between the first amplitude threshold and the second amplitude threshold, and level 3 when the L2 norm is higher than the second amplitude threshold. The uncertain semantic vector is transformed by a linear projection layer and injected as a prefix hidden state into the large language model agent. The large language model agent is driven to generate the corresponding security degradation instruction according to the security degradation level, and the security degradation instruction is sent to the autonomous mobile robot and the automated guided vehicle for execution.

6. An industrial multimodal intelligent interaction system based on a large language model agent, characterized in that, For implementing the industrial multimodal intelligent interaction method based on a large language model agent as described in any one of claims 1-5, the industrial multimodal intelligent interaction system based on a large language model agent comprises: The alignment module is used to align the autonomous mobile robot pose data stream, visual image frames, cargo load readings, and production line alarm status codes with timestamps to obtain a multimodal acquisition dataset. The weighting module is used to project the modal features of each mode in the multimodal acquisition dataset onto the hidden layer dimension space of the large language model and then fuse them in a weighted manner to obtain a physical state vector. This includes: performing row-direction max pooling on the pose feature matrix in the multimodal acquisition dataset to obtain a global pose summary vector; mapping the global pose summary vector to the hidden layer dimension space of the large language model via a linear projection layer to obtain a pose modality embedding vector; and mapping the visual semantic feature vector in the multimodal acquisition dataset to the hidden layer dimension space of the large language model via a linear projection layer to obtain a visual modality embedding vector. Embedding vectors: The location occupancy vector in the multimodal acquisition dataset is compressed through a sparse coding layer and then mapped to the hidden layer dimension space of a large language model through a linear projection layer to obtain the warehouse modality embedding vector; The alarm state vector in the multimodal acquisition dataset is mapped to the hidden layer dimension space of a large language model through a linear projection layer to obtain the alarm modality embedding vector; According to preset modality weight coefficients, the pose modality embedding vector, the visual modality embedding vector, the warehouse modality embedding vector, and the alarm modality embedding vector are weighted and summed to obtain the physical state vector; An encoding module is used to jointly encode the natural language scheduling instructions input by the operator and the physical state vector to obtain a multimodal interaction context. This includes: segmenting and embedding the natural language scheduling instructions input by the operator to obtain an instruction lexical sequence; inputting the instruction lexical sequence into a large language model encoder for semantic encoding to obtain an instruction semantic vector; concatenating the physical state vector and the instruction semantic vector to obtain a physical-instruction joint vector; and performing fusion encoding on the physical-instruction joint vector through a cross-modal attention layer to obtain the multimodal interaction context. The calculation module is used to calculate the stepwise anchor confidence based on the cosine similarity between the physical state vector and the current hidden layer state vector during the word-by-word generation process of the large language model agent with the multimodal interaction context as input, and to perform a weighted mixture of the original word probability distribution and the physical constraint preference distribution according to the anchor confidence to obtain the modulated word probability distribution. The execution module is used to construct an uncertain semantic vector based on the difference between the physical state vector and the current hidden layer state vector when the continuous window mean and the descent slope of the anchor confidence are both lower than a preset threshold, and to inject the uncertain semantic vector into the large language model agent to generate a security degradation instruction, and to send the security degradation instruction to the autonomous mobile robot and the automated guided vehicle for execution.

7. An industrial multimodal intelligent interactive device based on a large language model agent, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, and the processor executing the computer program to implement the industrial multimodal intelligent interaction method based on a large language model agent as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it causes the processor to execute the industrial multimodal intelligent interaction method based on a large language model agent as described in any one of claims 1 to 5.

Images