Industrial cognitive decision-making system and method based on large language model and multi-modal cooperation

Abstract

The invention relates to industrial control, in particular to an industrial cognitive decision-making system and method based on a large language model and multi-modal collaboration, and the method comprises the steps: obtaining multi-modal data of an industrial scene, and constructing a multi-modal feature library; generating a matched feature index set according to space-time attributes of the multi-modal feature library; comparing the working condition of the industrial equipment with the feature index set, and calibrating a multi-modal feature set associated with the industrial equipment so as to generate a behavior baseline of the industrial equipment; acquiring a task process of the industrial equipment, predicting a behavior sequence, and estimating a potential abnormal event in combination with the behavior baseline; according to the potential abnormal events, large language model decision making is carried out on the industrial equipment, a coordination strategy of the industrial equipment is generated, and therefore the working process of the industrial equipment is adjusted. According to the method, space-time distribution of industrial scene multi-modal data is integrated, a basis is provided for behavior baseline setting of different industrial devices, behavior prediction and strategy coordination are carried out through a large language model, and accurate and reliable decision making of device behaviors is achieved.

Description

Technical Field

[0001] This invention relates to industrial control, and in particular to an industrial cognitive decision-making system and method based on large language models and multimodal collaboration. Background Technology

[0002] Industrial scenarios are characterized by multiple devices, tasks, and interactive elements, resulting in rapid data updates and multimodal coexistence. The complex and ever-changing environmental conditions of industrial scenarios necessitate that industrial robots and other equipment handle numerous complex situations during operation. Currently, most industrial equipment employs intelligent operating modes, capable of collecting and analyzing real-time multimodal data from its surroundings to control its operational status. However, this approach places high demands on the data processing capabilities of industrial equipment, increasing its workload and limiting control to a single device, failing to coordinate and control all equipment within the industrial scenario. Furthermore, current equipment control relies on an emergency braking mode, immediately halting its current behavior upon encountering an anomaly, without proactively altering behavior to address abnormal situations, thus reducing the predictability of behavioral decisions. Therefore, integrating multimodal data from industrial scenarios and implementing process coordination decisions tailored to the current operational state of industrial equipment is crucial for improving the accuracy and compatibility of behavioral decisions in industrial environments. Summary of the Invention

[0003] Considering that existing industrial equipment control in industrial scenarios focuses on individual devices and is limited to the surrounding environment, it fails to provide comprehensive control over all industrial equipment within the entire industrial environment. Furthermore, the single control mode for industrial equipment is incompatible with its current behavior, reducing the accuracy and reliability of overall management and control in industrial scenarios. Therefore, this invention provides an industrial cognitive decision-making system based on a large language model and multimodal collaboration, comprising: The feature library construction module is used to acquire multimodal data from industrial scenarios and thereby construct a multimodal feature library for the industrial scenarios. The index set generation module is used to generate a matching feature index set based on the spatiotemporal attributes of the multimodal feature library. A multimodal feature set calibration module is used to calibrate the multimodal feature set associated with the industrial equipment by comparing it with the feature index set based on the actual working conditions of the industrial equipment. A behavior baseline generation module is used to generate a behavior baseline for the industrial equipment based on the multimodal feature set. The behavior sequence prediction module is used to obtain the task progress of the industrial equipment and predict the behavior sequence. An abnormal event determination module is used to estimate potential abnormal events of the industrial equipment based on the behavior sequence and the behavior baseline. The coordination strategy generation module is used to perform large language model decision-making on the industrial equipment based on the potential abnormal events, and generate a coordination strategy for the industrial equipment. The workflow adjustment module is used to adjust the workflow of the industrial equipment according to the coordination strategy.

[0004] Optionally, the feature library construction module is used to acquire multimodal data from industrial scenarios to construct a multimodal feature library for the industrial scenarios, including: Collect multimodal data streams generated by multimodal sensors in industrial scenarios, and perform differentiation preprocessing and filtering preprocessing on the multimodal data streams; Based on the positioning tags and built-in clock tags of the multimodal sensors, the multimodal data stream is processed, calibrated, and converted to construct a multimodal feature library for the industrial scenario; The index set generation module is used to generate a matching feature index set based on the spatiotemporal attributes of the multimodal feature library, including: Obtain the spatial label and temporal label of each modal feature in the multimodal feature library to generate a four-dimensional index for each modal feature; wherein, the four-dimensional index includes a three-dimensional spatial index and a one-dimensional temporal index; based on the four-dimensional index, generate a feature index set for all modal features.

[0005] Optionally, the multimodal feature set calibration module is used to calibrate the multimodal feature set associated with the industrial equipment by comparing it with the feature index set based on the actual working conditions of the industrial equipment, including: Obtain confirmed work task information of industrial equipment, and extract four-dimensional information corresponding to the work period from the confirmed work task information; wherein, the four-dimensional information includes three-dimensional spatial information and one-dimensional time information of the industrial equipment during the work period; Based on the four-dimensional information, a four-dimensional index matching the industrial equipment is found from the feature index set; based on the four-dimensional index, a multimodal feature set associated with the industrial equipment is identified from the multimodal feature library; The behavior baseline generation module is used to generate the behavior baseline of the industrial equipment based on the multimodal feature set, including: Identify all multimodal data within the multimodal feature set to obtain the environmental behavior of the industrial scene; wherein, the environmental behavior includes the spatial migration behavior and self-movement behavior of the environmental objects during the work period; Based on the behavior of the environmental objects, the behavioral constraints of the environmental objects on the industrial equipment are determined; the behavioral constraints are physically mapped to generate the behavioral baseline of the industrial equipment; wherein, the behavioral baseline refers to the types and number of behaviors that the industrial equipment is allowed to perform during operation.

[0006] Optionally, the behavior sequence prediction module is used to obtain the task progress of the industrial equipment and predict the behavior sequence, including: The task process loaded by the industrial equipment in a future time period is analyzed to obtain the expected execution behavior sequence; the expected execution behavior sequence is filtered according to the behavior execution constraints of the industrial equipment to predict the allowed execution behavior sequence of the industrial equipment. The abnormal event determination module is used to estimate potential abnormal events of the industrial equipment based on the behavior sequence and the behavior baseline, including: By comparing the allowed execution behavior sequence with the behavior baseline, the allowed execution behavior segments of the industrial equipment that conflict with the behavior baseline are obtained, thereby estimating the potential abnormal events of the industrial equipment; wherein, the potential abnormal events correspond to the behavior segments that the industrial equipment is actually not allowed to execute.

[0007] Optionally, the coordination strategy generation module is used to perform large language model decision-making on the industrial equipment based on the potential abnormal events, and generate a coordination strategy for the industrial equipment, including: Based on the potential abnormal events corresponding to the actual not allowed behavior segments of the industrial equipment, a large language model knowledge retrieval is performed on the industrial equipment to obtain a set of behavior segments that can replace the behavior segments. Based on the behavioral change constraints of the industrial equipment, a large language model is used to make decisions on the set of behavioral fragments to generate a coordination strategy for the industrial equipment; wherein, the coordination strategy refers to the strategy of the industrial equipment to replace or cancel the behavioral fragments that are actually not allowed to be executed. The workflow adjustment module is used to adjust the workflow of the industrial equipment according to the coordination strategy, including: According to the coordination strategy, locate the task process nodes of the industrial equipment that need to be changed; according to the execution time interval of the task process nodes that need to be changed, adjust the workflow of the industrial equipment.

[0008] As one aspect of the present invention, embodiments of the present invention also provide an industrial cognitive decision-making method based on a large language model and multimodal collaboration, including: Step S1: Acquire multimodal data of industrial scenarios to construct a multimodal feature library for the industrial scenarios; generate a matching feature index set based on the spatiotemporal attributes of the multimodal feature library; Step S2: Based on the actual working conditions of the industrial equipment, compare it with the feature index set to identify the multimodal feature set associated with the industrial equipment; generate the behavioral baseline of the industrial equipment based on the multimodal feature set; Step S3: Obtain the task progress of the industrial equipment and predict the behavior sequence; based on the behavior sequence and the behavior baseline, estimate the potential abnormal events of the industrial equipment; Step S4: Based on the potential abnormal events, perform large language model decision-making on the industrial equipment to generate a coordination strategy for the industrial equipment; adjust the workflow of the industrial equipment according to the coordination strategy.

[0009] Optionally, in step S1, multimodal data of the industrial scene is acquired to construct a multimodal feature library for the industrial scene; based on the spatiotemporal attributes of the multimodal feature library, a matching feature index set is generated, including: Collect multimodal data streams generated by multimodal sensors in industrial scenarios, and perform differentiation preprocessing and filtering preprocessing on the multimodal data streams; Based on the positioning tags and built-in clock tags of the multimodal sensors, the multimodal data stream is processed, calibrated, and converted to construct a multimodal feature library for the industrial scenario; Obtain the spatial label and temporal label of each modal feature in the multimodal feature library to generate a four-dimensional index for each modal feature; wherein, the four-dimensional index includes a three-dimensional spatial index and a one-dimensional temporal index; based on the four-dimensional index, generate a feature index set for all modal features.

[0010] Optionally, in step S2, based on the actual working conditions of the industrial equipment, a multimodal feature set associated with the industrial equipment is identified by comparing it with the feature index set; based on the multimodal feature set, a behavioral baseline of the industrial equipment is generated, including: Obtain confirmed work task information of industrial equipment, and extract four-dimensional information corresponding to the work period from the confirmed work task information; wherein, the four-dimensional information includes three-dimensional spatial information and one-dimensional time information of the industrial equipment during the work period; Based on the four-dimensional information, a four-dimensional index matching the industrial equipment is found from the feature index set; based on the four-dimensional index, a multimodal feature set associated with the industrial equipment is identified from the multimodal feature library; Identify all multimodal data within the multimodal feature set to obtain the environmental behavior of the industrial scene; wherein, the environmental behavior includes the spatial migration behavior and self-movement behavior of the environmental objects during the work period; Based on the behavior of the environmental objects, the behavioral constraints of the environmental objects on the industrial equipment are determined; the behavioral constraints are physically mapped to generate the behavioral baseline of the industrial equipment; wherein, the behavioral baseline refers to the types and number of behaviors that the industrial equipment is allowed to perform during operation.

[0011] Optionally, in step S3, the task progress of the industrial equipment is acquired, and a behavioral sequence is predicted; based on the behavioral sequence and the behavioral baseline, potential abnormal events of the industrial equipment are estimated, including: The task process loaded by the industrial equipment in a future time period is analyzed to obtain the expected execution behavior sequence; the expected execution behavior sequence is filtered according to the behavior execution constraints of the industrial equipment to predict the allowed execution behavior sequence of the industrial equipment. By comparing the allowed execution behavior sequence with the behavior baseline, the allowed execution behavior segments of the industrial equipment that conflict with the behavior baseline are obtained, thereby estimating the potential abnormal events of the industrial equipment; wherein, the potential abnormal events correspond to the behavior segments that the industrial equipment is actually not allowed to execute.

[0012] Optionally, in step S4, based on the potential abnormal events, a large language model decision is performed on the industrial equipment to generate a coordination strategy for the industrial equipment; based on the coordination strategy, the workflow of the industrial equipment is adjusted, including: Based on the potential abnormal events corresponding to the actual not allowed behavior segments of the industrial equipment, a large language model knowledge retrieval is performed on the industrial equipment to obtain a set of behavior segments that can replace the behavior segments. Based on the behavioral change constraints of the industrial equipment, a large language model is used to make decisions on the set of behavioral fragments to generate a coordination strategy for the industrial equipment; wherein, the coordination strategy refers to the strategy of the industrial equipment to replace or cancel the behavioral fragments that are actually not allowed to be executed. According to the coordination strategy, locate the task process nodes of the industrial equipment that need to be changed; according to the execution time interval of the task process nodes that need to be changed, adjust the workflow of the industrial equipment.

[0013] The beneficial effects of the above-mentioned technical solutions provided in the embodiments of the present invention include at least the following: This invention provides an industrial cognitive decision-making system and method based on large language models and multimodal collaboration. The system acquires multimodal data from industrial scenarios and constructs a multimodal feature library. Based on the spatiotemporal attributes of the multimodal feature library, a matching feature index set is generated. The system compares the actual working conditions of industrial equipment with the feature index set to identify the multimodal feature set associated with the industrial equipment, thereby generating a behavioral baseline for the industrial equipment. The system acquires the task progress of the industrial equipment, predicts behavioral sequences, and estimates potential abnormal events based on the behavioral baseline. Based on the potential abnormal events, the system performs large language model-based decision-making on the industrial equipment to generate coordination strategies, thereby adjusting the workflow of the industrial equipment. By integrating the spatiotemporal distribution of multimodal data from industrial scenarios, the system provides a basis for setting behavioral baselines for different industrial equipment. Furthermore, it utilizes large language models for behavior prediction and strategy coordination, achieving accurate and reliable decision-making regarding equipment behavior.

[0014] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the structure of an industrial cognitive decision-making system based on a large language model and multimodal collaboration provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the industrial cognitive decision-making method based on a large language model and multimodal collaboration provided in an embodiment of the present invention. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0018] Please see Figure 1 As shown, an embodiment of this application provides an industrial cognitive decision-making system based on a large language model and multimodal collaboration. This industrial cognitive decision-making system based on a large language model and multimodal collaboration includes: The feature library construction module is used to acquire multimodal data from industrial scenarios and construct a multimodal feature library for industrial scenarios. The index set generation module is used to generate a matching feature index set based on the spatiotemporal attributes of the multimodal feature library. The multimodal feature set calibration module is used to calibrate the multimodal feature set associated with the industrial equipment by comparing it with the feature index set based on the actual working conditions of the industrial equipment. The behavior baseline generation module is used to generate the behavior baseline of industrial equipment based on the multimodal feature set. The behavior sequence prediction module is used to acquire the task progress of industrial equipment and predict the behavior sequence. An anomaly event determination module is used to estimate potential anomalies in industrial equipment based on behavior sequences and behavior baselines. The coordination strategy generation module is used to make large language model decisions for industrial equipment based on potential abnormal events and generate coordination strategies for the industrial equipment. The workflow adjustment module is used to adjust the workflow of industrial equipment according to the coordination strategy.

[0019] This industrial cognitive decision-making system based on large language models and multimodal collaboration integrates the spatiotemporal distribution of multimodal data in industrial scenarios to provide a basis for setting behavioral baselines for different industrial equipment. It also utilizes large language models for behavior prediction and strategy coordination to achieve accurate and reliable decision-making on equipment behavior.

[0020] In another embodiment, the feature library construction module is used to acquire multimodal data from industrial scenarios to construct a multimodal feature library for industrial scenarios, including: Collect multimodal data streams generated by multimodal sensors in industrial scenarios, and perform differentiation preprocessing and filtering preprocessing on the multimodal data streams; Based on the positioning tags and built-in clock tags of multimodal sensors, the multimodal data stream is processed, calibrated, and converted to build a multimodal feature library for industrial scenarios; The index set generation module is used to generate a matching feature index set based on the spatiotemporal attributes of the multimodal feature library, including: Obtain the spatial and temporal labels of each modal feature in the multimodal feature library to generate a four-dimensional index for each modal feature. The four-dimensional index includes a three-dimensional spatial index and a one-dimensional temporal index. Based on the four-dimensional index, generate a feature index set for all modal features.

[0021] In the above technical solution, a distributed array of sensors of different types, such as cameras, sound sensors, and vibration sensors, is deployed within industrial settings like factories and workshops. Each type of sensor collects data on the corresponding modalities within the industrial setting, forming a multimodal monitoring system. It can be understood that all sensors within each type of sensor cluster can be positioned according to a pre-defined spatial plan within the industrial setting to monitor a corresponding spatial range. This allows each type of sensor cluster to achieve full-area coverage monitoring of the industrial setting. Each sensor within each type of cluster generates modal data for a corresponding spatial range within the industrial setting. By deploying these different types of sensor clusters, multimodal data covering the entire industrial setting can be acquired. This multimodal data may include, but is not limited to, image data, sound data, and vibration data of the industrial environment. It is understandable that each sensor is affected by its own internal and external factors during operation, and the sensor data stream generated will have noise interference. In order to accurately distinguish and separate the sensor data streams generated by different types of sensors and to perform noise reduction and filtering on the sensor data streams, a cloud platform is used to collect the multimodal data streams generated by multimodal sensors. The multimodal data streams are distinguished and preprocessed according to their data formats to obtain several single-modal data streams. Then, Kalman filtering is performed on each single-modal data stream to reduce the interference components of the single-modal data stream.

[0022] Each sensor is fixedly installed at a corresponding location within the industrial environment. The sensor data stream it generates has a matching location tag (i.e., a tag indicating the sensor's installation location). Simultaneously, each sensor has a built-in system clock, which generates a clock tag for each data point generated. Therefore, each data point within the data stream can be uniquely represented using its corresponding location tag, clock tag, data value, and dimensions. To accurately and uniformly represent the multimodal data in the industrial environment, location tags and clock tags are added and calibrated to each data point within the multimodal data stream based on the location tag and built-in clock tag of each sensor. Then, the data packet containing the data, location tag, and clock tag is vectorized. This allows each data point within the multimodal data stream to be converted into a feature vector containing three elements: the data value, location, and time. This constructs a multimodal feature library from the multimodal data stream. As described above, each data point can be uniquely represented using its location tag and clock tag; correspondingly, each feature vector in the multimodal feature library can be uniquely represented by its location and time elements. To quickly and accurately locate and calibrate the multimodal feature library, spatial labels (i.e., positional elements) and time labels (i.e., time elements) of each modal feature (i.e., feature vector) within the library are obtained. This generates a four-dimensional index for each modality. The spatial labels contain three-dimensional spatial coordinates, forming a three-dimensional spatial index, while the time labels contain time information, forming a one-dimensional time index. Each modal feature corresponds one-to-one with each four-dimensional index. All four-dimensional indices are combined to form a feature index set. Thus, each four-dimensional index within this feature index set has a one-to-one mapping relationship with each modal feature in the multimodal feature library. By querying this feature index library, the required modal features can be found within the multimodal feature library, providing a basis for subsequently determining multimodal data affecting industrial equipment in industrial scenarios.

[0023] In another embodiment, the multimodal feature set calibration module is used to calibrate the multimodal feature set associated with the industrial equipment by comparing it with the feature index set based on the actual working conditions of the industrial equipment, including: Obtain confirmed work task information of industrial equipment, and extract the corresponding four-dimensional information during the work period from the confirmed work task information; wherein, the four-dimensional information includes the three-dimensional spatial information and one-dimensional time information of the industrial equipment during the work period; Based on the four-dimensional information, find the four-dimensional index that matches the industrial equipment from the feature index set; based on the four-dimensional index, identify the multimodal feature set associated with the industrial equipment from the multimodal feature library; The behavior baseline generation module is used to generate behavior baselines for industrial equipment based on a multimodal feature set, including: The behavior of environmental objects in industrial scenarios is obtained by identifying all multimodal data within the multimodal feature set; the behavior of environmental objects includes their spatial migration behavior and their own action behavior during operation. Based on the behavior of environmental objects, determine the behavioral constraints of environmental objects on industrial equipment; perform physical space mapping on the behavioral constraints to generate the behavioral baseline of industrial equipment; whereby the behavioral baseline refers to the types and number of behaviors that industrial equipment is allowed to perform during operation.

[0024] In the above technical solution, a large number of industrial robots and other industrial equipment exist simultaneously in the industrial setting. Each piece of equipment operates independently and can execute work tasks issued from the cloud. It is understood that industrial equipment possesses free movement and free motion capabilities. Free movement refers to the ability of the equipment to freely migrate within the industrial setting, while free motion refers to the ability of its robotic arms / manipulators to move freely in six degrees of freedom. When an industrial equipment receives and parses the issued work task, it generates corresponding migration motion tasks and robotic arm / manipulator motion tasks. During the execution of these migration motion tasks and robotic arm / manipulator motion tasks, corresponding four-dimensional information is generated. This four-dimensional information is used to characterize the three-dimensional spatial information and one-dimensional temporal information of each migration motion, as well as the three-dimensional spatial information and one-dimensional temporal information of each robotic arm / manipulator motion. Thus, there is a one-to-one correspondence between the above four-dimensional information and the migration motion and robotic arm / manipulator motion of the industrial equipment. Considering that industrial equipment interacts with the industrial environment during migration and robotic arm / manipulation, the multimodal environment of the industrial environment (such as the multimodal environment generated by personnel or other robots in the industrial environment) interferes with the industrial equipment, restricting its original migration and robotic arm / manipulation. Therefore, the multimodal environment of the industrial environment constrains the migration and robotic arm / manipulation of the industrial equipment. The migration and robotic arm / manipulation of the industrial equipment can only be executed normally if they are within the acceptable range of the above constraints; otherwise, they cannot be executed.

[0025] To identify whether the migration movements and robotic arm / manipulation actions of industrial equipment during its work tasks within an industrial environment are compatible with the aforementioned constraints, it is necessary to comprehensively and accurately calibrate the environmental impacts (such as personnel or other robots) of the industrial environment at their corresponding three-dimensional spatial positions and times during the equipment's confirmed work tasks. It is understandable that different three-dimensional spatial positions within an industrial environment impose different constraints on the equipment at different times. This difference in constraints stems from the varying behaviors of environmental objects at different three-dimensional spatial positions at different times. Therefore, determining the behaviors of environmental objects at different three-dimensional spatial positions at different times is fundamental to determining the constraints imposed on the industrial equipment by these different positions. To this end, firstly, based on the confirmed work task information of the industrial equipment, the corresponding four-dimensional information during the equipment's operation is extracted; this four-dimensional information is compared with a feature index set to find four-dimensional indices that are identical to the aforementioned four-dimensional information or meet preset similarity conditions; then, modal features corresponding to the aforementioned four-dimensional indices are calibrated and selected from a multimodal feature library, thereby forming a multimodal feature set associated with the industrial equipment. It is understandable that the aforementioned multimodal feature set associated with industrial equipment refers to the environmental modalities encountered by industrial equipment at a corresponding three-dimensional spatial location within an industrial scene at a given time.

[0026] The aforementioned multimodal feature set associated with industrial equipment directly reflects the behavioral state of environmental objects in the corresponding time and three-dimensional spatial location within the industrial scene. Therefore, all multimodal data within the feature set are identified to obtain the spatial migration behavior of environmental objects in the industrial scene (such as the migration path and speed of environmental objects) and their own action behavior (such as the trajectory of movable parts of the environmental objects in a six-degree-of-freedom space). Based on the aforementioned spatial migration behavior and their own action behavior, a blank space range is determined that allows the industrial equipment to perform actions during the period when the environmental object performs an action. This blank space range serves as a behavioral constraint on the industrial equipment. This blank space range is mapped to the physical space of the industrial scene and, combined with the external contour of the industrial equipment, a behavioral baseline for the industrial equipment is generated. This behavioral baseline accurately characterizes the types of behaviors that the industrial equipment is allowed to perform in the aforementioned physical space during operation and the number of times each type of behavior is implemented, providing guidance for subsequent adjustments to the actual behavior of the industrial equipment.

[0027] In another embodiment, the behavior sequence prediction module is used to acquire the task progress of industrial equipment and predict behavior sequences, including: The task processes loaded by industrial equipment in the future time period are analyzed to obtain the expected execution behavior sequence; the expected execution behavior sequence is filtered according to the behavior execution constraints of the industrial equipment to predict the allowed execution behavior sequence of the industrial equipment; Anomaly detection module is used to estimate potential anomalies in industrial equipment based on behavior sequences and behavior baselines, including: By comparing the allowed execution behavior sequence with the behavior baseline, the allowed execution behavior segments that conflict with the behavior baseline of the industrial equipment are obtained, thereby estimating the potential abnormal events of the industrial equipment; where the potential abnormal events correspond to the behavior segments that the industrial equipment is actually not allowed to execute.

[0028] In the above technical solution, the operation of industrial equipment in industrial scenarios is based on work instructions issued from the cloud. When the cloud issues work instructions to the industrial equipment, the industrial equipment loads the corresponding task process, which controls the industrial equipment to perform corresponding behaviors. It can be understood that there is a one-to-one correspondence between the task process and the behavior performed by the industrial equipment. Therefore, the task processes loaded by the industrial equipment in future time periods are analyzed to obtain the expected sequence of execution behaviors. This expected sequence includes the expected behaviors that the industrial equipment is expected to perform at each point in time within the future period. Considering the inherent limitations on the execution of industrial equipment (such as the slope limitations of obstacles that can be crossed for migration operations, and the limitations on the displacement and angle ranges of robotic arm / manipulator movements), the industrial equipment cannot completely and correctly execute the above sequence of behaviors. Therefore, the expected sequence of execution behaviors is filtered based on the behavioral execution limitations of the industrial equipment to predict the allowed sequence of execution behaviors. Thus, the allowed sequence of execution behaviors is the actual sequence of behaviors that the industrial equipment can execute.

[0029] Furthermore, each behavior in the permitted execution behavior sequence is compared with the aforementioned behavior baseline. If a behavior conflicts with the baseline (e.g., the behavior type does not match the baseline), the behavior is marked as potentially triggering an abnormal event in the industrial equipment; otherwise, it is marked as not triggering such an event. All behaviors in the permitted execution behavior sequence that conflict with the baseline are then integrated into a permitted execution behavior segment where the industrial equipment conflicts with the baseline. Based on this segment, potential abnormal events in the industrial equipment are estimated. This potential abnormal event refers to anomalies such as collisions with environmental objects in the industrial setting during the execution of the permitted execution behavior segment. Furthermore, the behavior segment that the industrial equipment is actually not allowed to execute, corresponding to the potential abnormal event, is the permitted execution behavior segment where the industrial equipment conflicts with the baseline.

[0030] In another embodiment, the coordination strategy generation module is used to perform large language model decision-making on industrial equipment based on potential abnormal events, and generate a coordination strategy for the industrial equipment, including: Based on the behavioral fragments that industrial equipment is actually not allowed to perform corresponding to potential abnormal events, a large language model knowledge retrieval is performed on the industrial equipment to obtain a set of behavioral fragments that can replace the behavioral fragments. Based on the constraints on the behavior changes of industrial equipment, a large language model is used to make decisions on the set of behavior fragments to generate coordination strategies for the industrial equipment. The coordination strategy refers to the strategy of industrial equipment to replace or cancel behavior fragments that are not actually allowed to be executed. The workflow adjustment module is used to adjust the workflow of industrial equipment according to coordination strategies, including: Based on the coordination strategy, identify the task process nodes that need to be changed for industrial equipment; based on the execution time interval of the task process nodes that need to be changed, adjust the workflow of the industrial equipment.

[0031] In the above technical solution, in specific operation, based on the behavioral content attributes of the industrial equipment's actual prohibited behavioral fragments corresponding to potential abnormal events, a large language model knowledge retrieval is performed on the industrial equipment to obtain a set of alternative behavioral fragments. These alternative behavioral fragments refer to those that satisfy preset similarity conditions in behavioral content attributes and meet the aforementioned behavioral benchmarks. Then, based on the industrial equipment's behavioral change restrictions (such as the allowed changes in the direction and amplitude of the industrial equipment's actions), a large language model is used to select and decide on the alternative behavioral fragments, generating a strategy for replacing or canceling the actual prohibited behavioral fragments. For example, at least a portion of the behaviors of the alternative behavioral fragments can be used to replace the actual prohibited behavioral fragments. Furthermore, based on the above strategy, the task process node requiring change is located (i.e., the node where the actual prohibited behavioral fragment corresponds to the task process of the industrial equipment). Then, based on the time interval between the current time point and the time point where the task process node requires change is located, the industrial equipment's workflow is pre-adjusted to correspond to the task process requiring change, ensuring that the industrial equipment adapts to behavioral decision changes in advance.

[0032] Please see Figure 2 As shown, an embodiment of this application provides an industrial cognitive decision-making method based on a large language model and multimodal collaboration. This industrial cognitive decision-making method based on a large language model and multimodal collaboration includes: Step S1: Acquire multimodal data of industrial scenarios to construct a multimodal feature library for industrial scenarios; generate a matching feature index set based on the spatiotemporal attributes of the multimodal feature library; Step S2: Based on the actual working conditions of the industrial equipment, compare it with the feature index set to identify the multimodal feature set associated with the industrial equipment; based on the multimodal feature set, generate the behavioral baseline of the industrial equipment; Step S3: Obtain the task progress of the industrial equipment and predict the behavior sequence; based on the behavior sequence and the behavior baseline, estimate the potential abnormal events of the industrial equipment; Step S4: Based on potential abnormal events, perform large language model decision-making on industrial equipment to generate coordination strategies for industrial equipment; adjust the workflow of industrial equipment according to the coordination strategies.

[0033] This industrial cognitive decision-making method based on large language models and multimodal collaboration provides a basis for setting behavioral baselines for different industrial equipment by integrating the spatiotemporal distribution of multimodal data in industrial scenarios. It also utilizes large language models for behavior prediction and strategy coordination to achieve accurate and reliable decision-making on equipment behavior.

[0034] In another embodiment, in step S1, multimodal data of an industrial scenario is acquired to construct a multimodal feature library for the industrial scenario; based on the spatiotemporal attributes of the multimodal feature library, a matching feature index set is generated, including: Collect multimodal data streams generated by multimodal sensors in industrial scenarios, and perform differentiation preprocessing and filtering preprocessing on the multimodal data streams; Based on the positioning tags and built-in clock tags of multimodal sensors, the multimodal data stream is processed, calibrated, and converted to build a multimodal feature library for industrial scenarios; Obtain the spatial and temporal labels of each modal feature in the multimodal feature library to generate a four-dimensional index for each modal feature. The four-dimensional index includes a three-dimensional spatial index and a one-dimensional temporal index. Based on the four-dimensional index, generate a feature index set for all modal features.

[0035] In another embodiment, in step S2, based on the actual working conditions of the industrial equipment, a multimodal feature set associated with the industrial equipment is identified by comparing it with the feature index set; based on the multimodal feature set, a behavioral baseline of the industrial equipment is generated, including: Obtain confirmed work task information of industrial equipment, and extract the corresponding four-dimensional information during the work period from the confirmed work task information; wherein, the four-dimensional information includes the three-dimensional spatial information and one-dimensional time information of the industrial equipment during the work period; Based on the four-dimensional information, find the four-dimensional index that matches the industrial equipment from the feature index set; based on the four-dimensional index, identify the multimodal feature set associated with the industrial equipment from the multimodal feature library; The behavior of environmental objects in industrial scenarios is obtained by identifying all multimodal data within the multimodal feature set; the behavior of environmental objects includes their spatial migration behavior and their own action behavior during operation. Based on the behavior of environmental objects, determine the behavioral constraints of environmental objects on industrial equipment; perform physical space mapping on the behavioral constraints to generate the behavioral baseline of industrial equipment; whereby the behavioral baseline refers to the types and number of behaviors that industrial equipment is allowed to perform during operation.

[0036] In another embodiment, in step S3, the task progress of the industrial equipment is acquired, and the behavior sequence is predicted; based on the behavior sequence and the behavior baseline, potential abnormal events of the industrial equipment are estimated, including: The task processes loaded by industrial equipment in the future time period are analyzed to obtain the expected execution behavior sequence; the expected execution behavior sequence is filtered according to the behavior execution constraints of the industrial equipment to predict the allowed execution behavior sequence of the industrial equipment; By comparing the allowed execution behavior sequence with the behavior baseline, the allowed execution behavior segments that conflict with the behavior baseline of the industrial equipment are obtained, thereby estimating the potential abnormal events of the industrial equipment; where the potential abnormal events correspond to the behavior segments that the industrial equipment is actually not allowed to execute.

[0037] In another embodiment, in step S4, based on potential abnormal events, a large language model decision is made for the industrial equipment to generate a coordination strategy for the industrial equipment; based on the coordination strategy, the workflow of the industrial equipment is adjusted, including: Based on the behavioral fragments that industrial equipment is actually not allowed to perform corresponding to potential abnormal events, a large language model knowledge retrieval is performed on the industrial equipment to obtain a set of behavioral fragments that can replace the behavioral fragments. Based on the constraints on the behavior changes of industrial equipment, a large language model is used to make decisions on the set of behavior fragments to generate coordination strategies for the industrial equipment. The coordination strategy refers to the strategy of industrial equipment to replace or cancel behavior fragments that are not actually allowed to be executed. Based on the coordination strategy, identify the task process nodes that need to be changed for industrial equipment; based on the execution time interval of the task process nodes that need to be changed, adjust the workflow of the industrial equipment.

[0038] The industrial cognitive decision-making method based on large language models and multimodal collaboration of the present invention has the same operation and effect as the aforementioned industrial cognitive decision-making system based on large language models and multimodal collaboration. Therefore, the industrial cognitive decision-making method based on large language models and multimodal collaboration will not be described again here.

[0039] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. This disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims. Thus, if these modifications and variations of the invention fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. An industrial cognitive decision system based on large language model and multi-modal collaboration, characterized in that, include: The feature library construction module is used to acquire multimodal data from industrial scenarios and thereby construct a multimodal feature library for the industrial scenarios. The index set generation module is used to generate a matching feature index set based on the spatiotemporal attributes of the multimodal feature library. A multimodal feature set calibration module is used to calibrate the multimodal feature set associated with the industrial equipment by comparing it with the feature index set based on the actual working conditions of the industrial equipment. A behavior baseline generation module is used to generate a behavior baseline for the industrial equipment based on the multimodal feature set. The behavior sequence prediction module is used to obtain the task progress of the industrial equipment and predict the behavior sequence. An abnormal event determination module is used to estimate potential abnormal events of the industrial equipment based on the behavior sequence and the behavior baseline. The coordination strategy generation module is used to perform large language model decision-making on the industrial equipment based on the potential abnormal events, and generate a coordination strategy for the industrial equipment. The workflow adjustment module is used to adjust the workflow of the industrial equipment according to the coordination strategy.

2. The industrial cognitive decision-making system based on large language models and multimodal collaboration as described in claim 1, characterized in that: The feature library construction module is used to acquire multimodal data from industrial scenarios and thereby construct a multimodal feature library for the industrial scenarios, including: Collect multimodal data streams generated by multimodal sensors in industrial scenarios, and perform differentiation preprocessing and filtering preprocessing on the multimodal data streams; Based on the positioning tags and built-in clock tags of the multimodal sensors, the multimodal data stream is processed, calibrated, and converted to construct a multimodal feature library for the industrial scenario; The index set generation module is used to generate a matching feature index set based on the spatiotemporal attributes of the multimodal feature library, including: Obtain the spatial and temporal labels of each modal feature in the multimodal feature library to generate a four-dimensional index for each modal feature; wherein the four-dimensional index includes a three-dimensional spatial index and a one-dimensional temporal index; and generate a feature index set for all modal features based on the four-dimensional index.

3. The industrial cognitive decision-making system based on large language models and multimodal collaboration as described in claim 1, characterized in that: The multimodal feature set calibration module is used to calibrate the multimodal feature set associated with the industrial equipment by comparing it with the feature index set based on the actual working conditions of the industrial equipment, including: Obtain confirmed work task information of industrial equipment, and extract four-dimensional information corresponding to the work period from the confirmed work task information; wherein, the four-dimensional information includes three-dimensional spatial information and one-dimensional time information of the industrial equipment during the work period; Based on the four-dimensional information, a four-dimensional index matching the industrial equipment is found from the feature index set; based on the four-dimensional index, a multimodal feature set associated with the industrial equipment is identified from the multimodal feature library; The behavior baseline generation module is used to generate the behavior baseline of the industrial equipment based on the multimodal feature set, including: Identify all multimodal data within the multimodal feature set to obtain the environmental behavior of the industrial scene; wherein, the environmental behavior includes the spatial migration behavior and self-movement behavior of the environmental objects during the work period; Based on the behavior of the environmental objects, the behavioral constraints of the environmental objects on the industrial equipment are determined; the behavioral constraints are physically mapped to generate the behavioral baseline of the industrial equipment; wherein, the behavioral baseline refers to the types and number of behaviors that the industrial equipment is allowed to perform during operation.

4. The industrial cognitive decision-making system based on large language models and multimodal collaboration as described in claim 1, characterized in that: The behavior sequence prediction module is used to obtain the task progress of the industrial equipment and predict the behavior sequence, including: The task process loaded by the industrial equipment in a future time period is analyzed to obtain the expected execution behavior sequence; the expected execution behavior sequence is filtered according to the behavior execution constraints of the industrial equipment to predict the allowed execution behavior sequence of the industrial equipment. The abnormal event determination module is used to estimate potential abnormal events of the industrial equipment based on the behavior sequence and the behavior baseline, including: By comparing the allowed execution behavior sequence with the behavior baseline, the allowed execution behavior segments of the industrial equipment that conflict with the behavior baseline are obtained, thereby estimating the potential abnormal events of the industrial equipment; wherein, the potential abnormal events correspond to the behavior segments that the industrial equipment is actually not allowed to execute.

5. The industrial cognitive decision-making system based on large language models and multimodal collaboration as described in claim 1, characterized in that: The coordination strategy generation module is used to perform large language model decision-making on the industrial equipment based on the potential abnormal events, and generate a coordination strategy for the industrial equipment, including: Based on the potential abnormal events corresponding to the actual not allowed behavior segments of the industrial equipment, a large language model knowledge retrieval is performed on the industrial equipment to obtain a set of behavior segments that can replace the behavior segments. Based on the behavioral change constraints of the industrial equipment, a large language model is used to make decisions on the set of behavioral fragments to generate a coordination strategy for the industrial equipment; wherein, the coordination strategy refers to the strategy of the industrial equipment to replace or cancel the behavioral fragments that are actually not allowed to be executed. The workflow adjustment module is used to adjust the workflow of the industrial equipment according to the coordination strategy, including: According to the coordination strategy, locate the task process nodes of the industrial equipment that need to be changed; according to the execution time interval of the task process nodes that need to be changed, adjust the workflow of the industrial equipment.

6. An industrial cognitive decision-making method based on a large language model and multi-modal cooperation, characterized in that, include: Step S1: Acquire multimodal data of industrial scenarios to construct a multimodal feature library for the industrial scenarios; generate a matching feature index set based on the spatiotemporal attributes of the multimodal feature library; Step S2: Based on the actual working conditions of the industrial equipment, compare it with the feature index set to identify the multimodal feature set associated with the industrial equipment; generate the behavioral baseline of the industrial equipment based on the multimodal feature set; Step S3: Obtain the task progress of the industrial equipment and predict the behavior sequence; based on the behavior sequence and the behavior baseline, estimate the potential abnormal events of the industrial equipment; Step S4: Based on the potential abnormal events, perform large language model decision-making on the industrial equipment to generate a coordination strategy for the industrial equipment; adjust the workflow of the industrial equipment according to the coordination strategy.

7. The industrial cognitive decision-making method based on large language models and multimodal collaboration as described in claim 6, characterized in that: In step S1, multimodal data of the industrial scene is acquired to construct a multimodal feature library for the industrial scene; based on the spatiotemporal attributes of the multimodal feature library, a matching feature index set is generated, including: Collect multimodal data streams generated by multimodal sensors in industrial scenarios, and perform differentiation preprocessing and filtering preprocessing on the multimodal data streams; Based on the positioning tags and built-in clock tags of the multimodal sensors, the multimodal data stream is processed, calibrated, and converted to construct a multimodal feature library for the industrial scenario; Obtain the spatial and temporal labels of each modal feature in the multimodal feature library to generate a four-dimensional index for each modal feature; wherein the four-dimensional index includes a three-dimensional spatial index and a one-dimensional temporal index; and generate a feature index set for all modal features based on the four-dimensional index.

8. The industrial cognitive decision-making method based on large language models and multimodal collaboration as described in claim 6, characterized in that: In step S2, based on the actual working conditions of the industrial equipment, the feature index set is compared with the feature index set to identify the multimodal feature set associated with the industrial equipment; Based on the multimodal feature set, a behavioral baseline for the industrial equipment is generated, including: Obtain confirmed work task information of industrial equipment, and extract four-dimensional information corresponding to the work period from the confirmed work task information; wherein, the four-dimensional information includes three-dimensional spatial information and one-dimensional time information of the industrial equipment during the work period; Based on the four-dimensional information, a four-dimensional index matching the industrial equipment is found from the feature index set; based on the four-dimensional index, a multimodal feature set associated with the industrial equipment is identified from the multimodal feature library; Identify all multimodal data within the multimodal feature set to obtain the environmental behavior of the industrial scene; wherein, the environmental behavior includes the spatial migration behavior and self-movement behavior of the environmental objects during the work period; Based on the behavior of the environmental objects, the behavioral constraints of the environmental objects on the industrial equipment are determined; the behavioral constraints are physically mapped to generate the behavioral baseline of the industrial equipment; wherein, the behavioral baseline refers to the types and number of behaviors that the industrial equipment is allowed to perform during operation.

9. The industrial cognitive decision-making method based on large language models and multimodal collaboration as described in claim 6, characterized in that: In step S3, the task progress of the industrial equipment is obtained, and the behavior sequence is predicted; Based on the behavioral sequence and the behavioral baseline, potential anomalous events of the industrial equipment are estimated, including: The task process loaded by the industrial equipment in a future time period is analyzed to obtain the expected execution behavior sequence; the expected execution behavior sequence is filtered according to the behavior execution constraints of the industrial equipment to predict the allowed execution behavior sequence of the industrial equipment. By comparing the allowed execution behavior sequence with the behavior baseline, the allowed execution behavior segments of the industrial equipment that conflict with the behavior baseline are obtained, thereby estimating the potential abnormal events of the industrial equipment; wherein, the potential abnormal events correspond to the behavior segments that the industrial equipment is actually not allowed to execute.

10. The industrial cognitive decision-making method based on large language models and multimodal collaboration as described in claim 6, characterized in that: In step S4, based on the potential abnormal events, a large language model decision is made on the industrial equipment to generate a coordination strategy for the industrial equipment. According to the coordination strategy, the workflow of the industrial equipment is adjusted, including: Based on the potential abnormal events corresponding to the actual not allowed behavior segments of the industrial equipment, a large language model knowledge retrieval is performed on the industrial equipment to obtain a set of behavior segments that can replace the behavior segments. Based on the behavioral change constraints of the industrial equipment, a large language model is used to make decisions on the set of behavioral fragments to generate a coordination strategy for the industrial equipment; wherein, the coordination strategy refers to the strategy of the industrial equipment to replace or cancel the behavioral fragments that are actually not allowed to be executed. According to the coordination strategy, locate the task process nodes of the industrial equipment that need to be changed; according to the execution time interval of the task process nodes that need to be changed, adjust the workflow of the industrial equipment.

Images