Cigarette station illegal operation early warning method and system based on artificial intelligence

Abstract

The invention provides a cigarette station violation operation early warning method and system based on artificial intelligence. The method comprises the following steps: firstly, obtaining a multi-modal monitoring data set of a target cigarette station, wherein the multi-modal monitoring data set comprises tobacco leaf quality, equipment operation logs and environmental parameters; performing time sequence behavior feature extraction on the set to obtain a tobacco processing flow time sequence behavior track sequence, equipment operation compliance and environment abnormal fluctuation feature set, and then calling a pre-trained multi-mode violation identification model to perform dynamic weight distribution on the features to generate a fusion violation feature vector; the method comprises the following steps: obtaining a fusion rule-violation feature vector, determining rule-violation operation probability distribution based on the fusion rule-violation feature vector, further generating a cigarette station rule-violation early warning strategy, finally feeding back the cigarette station rule-violation early warning strategy to a cigarette station supervision terminal, triggering a rule-violation operation interception instruction, and realizing effective early warning and interception of cigarette station rule-violation operation by using multi-modal data and an artificial intelligence technology.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based method and system for early warning of illegal operations at a tobacco station. Background Art

[0002] In the production and operation of the tobacco industry, tobacco stations are the key link in tobacco leaf processing and management. The standardization and safety of their operations are crucial to ensuring the quality of tobacco products, maintaining production order, and ensuring the safety of personnel. However, traditional monitoring methods for illegal operations in tobacco stations mainly rely on manual inspections and regular sampling. This method is not only inefficient and difficult to achieve real-time and comprehensive monitoring of the entire production process of tobacco stations, but also limited by human factors, prone to missed inspections and false inspections, resulting in some potential illegal operations that cannot be discovered and corrected in a timely manner.

[0003] With the development of information technology, some tobacco stations have begun to introduce automated monitoring systems. However, existing automated monitoring systems often use fixed analysis modes and lack in-depth exploration of the correlation between different data sources, making it difficult to extract valuable violation feature information from massive monitoring data. When faced with the complex and ever-changing production environment of tobacco stations, it is impossible to dynamically adjust the analysis strategy according to real-time conditions, resulting in low accuracy and timeliness in identifying illegal operations. Moreover, the existing early warning mechanism is usually based on simple threshold judgments. An early warning is triggered only when the monitoring data exceeds the preset threshold. This method cannot comprehensively consider the impact of multiple factors on illegal operations, and is prone to false alarms or missed reports, which cannot meet the actual needs of tobacco stations for accurate early warning of illegal operations. Summary of the invention

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides an artificial intelligence-based cigarette station illegal operation warning method, the method comprising: Acquire a multimodal monitoring data set of a target tobacco station, wherein the multimodal monitoring data set includes tobacco leaf quality monitoring data, equipment operation log data, and environmental parameter monitoring data; Performing temporal behavior feature extraction processing on the multimodal monitoring data set to obtain a temporal behavior trajectory sequence in the tobacco processing flow, an equipment operation compliance feature set, and an environmental abnormal fluctuation feature set; Calling a pre-trained multimodal violation recognition model, dynamically assigning weights to the temporal behavior trajectory sequence, the device operation compliance feature set, and the environmental abnormal fluctuation feature set to generate a fused violation feature vector; Determine the probability distribution of illegal operations of the target cigarette station based on the fused illegal feature vector, and generate a cigarette station illegal warning strategy according to the probability distribution of illegal operations; The cigarette station violation warning strategy is fed back to the cigarette station supervision terminal to trigger the illegal operation interception instruction.

[0005] On the other hand, an embodiment of the present invention also provides an artificial intelligence-based tobacco station illegal operation warning system, including a processor and a machine-readable storage medium, the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0006] Based on the above aspects, the embodiment of the present application realizes accurate monitoring and intelligent early warning of the entire production process of the tobacco station, significantly improving the comprehensiveness and accuracy of the identification of illegal operations in the tobacco station. Specifically, by obtaining a set of monitoring data covering multiple dimensions such as tobacco leaf quality, equipment operation and environmental parameters, on this basis, by extracting time-series behavioral features from multimodal monitoring data, not only the dynamic behavior trajectory of the tobacco leaf processing process is accurately portrayed, but also the compliance characteristics of equipment operation and the abnormal fluctuation pattern of environmental parameters are deeply excavated, providing rich and relevant feature information for subsequent violation identification. Furthermore, the pre-trained multimodal violation identification model intelligently fuses feature information from different sources through a dynamic weight allocation mechanism to generate a highly representative fused violation feature vector, which effectively improves the sensitivity and specificity of violation feature identification. Based on the probability distribution of illegal operations determined by the fused violation feature vector, it is possible to accurately quantify the violation risks of each link in the tobacco station, and then generate targeted violation warning strategies, realizing the full chain intelligence from risk identification to warning decision-making. Ultimately, by providing real-time feedback of the early warning strategy to the tobacco station supervision terminal and triggering instructions to intercept illegal operations, a closed-loop violation prevention and control system was formed. This not only greatly improved the timely detection rate and handling efficiency of illegal operations in tobacco stations, but also effectively reduced production accidents and quality risks caused by illegal operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a schematic diagram of the execution flow of the artificial intelligence-based cigarette station illegal operation warning method provided in an embodiment of the present invention.

[0008] Figure 2 It is a schematic diagram of exemplary hardware and software components of the artificial intelligence-based tobacco station illegal operation warning system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 It is a flow chart of an artificial intelligence-based early warning method for illegal operation of a cigarette station provided by an embodiment of the present invention. The artificial intelligence-based early warning method for illegal operation of a cigarette station is introduced in detail below.

[0010] Step S110, obtaining a multimodal monitoring data set of a target tobacco station, wherein the multimodal monitoring data set includes tobacco leaf quality monitoring data, equipment operation log data, and environmental parameter monitoring data.

[0011] In this embodiment, a certain tobacco station is equipped with a monitoring system for comprehensive monitoring of the tobacco processing process. In detail, a plurality of tobacco baking areas are arranged in the tobacco station, and various sensors and equipment recording systems are installed in each tobacco baking area. For example, in the tobacco baking room, humidity sensors and temperature sensors are installed, which can record the temperature and humidity data of the environment in which the tobacco leaves are located every 10 minutes, so as to reflect the quality changes of the tobacco leaves during the baking process. At the same time, tobacco appearance detection equipment is also set up, which can take pictures and analyze the color, shape and other characteristics of the tobacco leaves every hour. The above data together constitute part of the tobacco quality monitoring data.

[0012] In addition, the equipment operation log data is automatically recorded by the control system of each device. For example, the operation console of the baking equipment can record in detail the startup time and shutdown time of each equipment, and various operation instructions entered by the operator, including temperature adjustment instructions, ventilation control instructions, etc. For other auxiliary equipment, such as conveyors, sorting machines, etc., similar operation recording systems are also set up to accurately record the operating status and operation process of the equipment.

[0013] Furthermore, the environmental parameter monitoring data covers multiple environmental indicators in the tobacco station. Specifically, multiple temperature, humidity, and dust concentration sensors are installed in different areas of the tobacco station, which can also collect data once a minute, including real-time temperature and humidity changes in the area, and the cumulative amount of dust concentration in the air. Therefore, through the above-mentioned monitoring equipment, a multimodal monitoring data set can be obtained, which covers tobacco leaf quality monitoring data, equipment operation log data, and environmental parameter monitoring data.

[0014] Step S120, performing temporal behavior feature extraction processing on the multimodal monitoring data set to obtain a temporal behavior trajectory sequence in the tobacco processing flow, an equipment operation compliance feature set, and an environmental abnormal fluctuation feature set.

[0015] In this embodiment, for tobacco leaf quality monitoring data, the baking stage division process is first performed. Taking a typical tobacco leaf baking process as an example, according to the changing trend of temperature and humidity, the entire baking process is divided into three main stages: yellowing period, color fixing period, and dry tendon period. In detail, during the yellowing period, the temperature is usually controlled between 35°C and 42°C, the humidity is maintained at a high level, and the tobacco leaves gradually turn yellow over time. Through a detailed analysis of each stage, a tobacco leaf baking curve is generated, which clearly shows the changes in temperature and humidity over time. At the same time, the duration characteristics of each stage are extracted, for example, the yellowing period lasts for 20 hours. The temperature fluctuation variance feature is obtained by calculating the variance of the temperature data recorded every 10 minutes during the stage. Assuming that the temperature fluctuation variance during the yellowing period is 3.5 after calculation, it means that the temperature fluctuates within a certain range. The humidity-related offset feature refers to the difference between the actual humidity and the preset humidity standard during the stage. For example, the humidity-related offset during the yellowing period is +5%, indicating that the actual humidity is slightly higher than the preset humidity. The above features together constitute a temporal behavior trajectory sequence.

[0016] Furthermore, the operation instruction parsing and processing is performed on the equipment operation log data. Taking the baking equipment as an example, the equipment power on / off time interval feature is calculated by recording the power on time and power off time of the equipment. For example, in a baking process, the equipment is powered on at 9 am and powered off at 5 pm, so the power on / off time interval is 8 hours. The operation instruction compliance label is used to determine whether each operation instruction is compliant based on the preset operation rule library. For example, the preset rule stipulates that the temperature adjustment range during the color setting period cannot exceed ±5°C. If the temperature adjustment instruction input by the operator is within the temperature adjustment range of the color setting period, the compliance label of the temperature adjustment instruction is "compliant", otherwise it is "violation". The operation frequency abnormality index feature is calculated by counting the number of operation instructions per unit time and comparing it with the historical normal operation frequency. Assuming that the number of operation instructions per hour is between 10 and 15 under normal circumstances, and the number of operation instructions in a certain hour reaches 20, the operation frequency abnormality index is calculated to be 1.3, indicating that the operation frequency is relatively high. The above features constitute the equipment operation compliance feature set.

[0017] Furthermore, the environmental parameter monitoring data is processed by sliding window mean. Taking a 10-minute sliding window as an example, the temperature extreme difference feature in the window can be calculated, that is, the difference between the highest temperature and the lowest temperature in the window. For example, in a 10-minute window, the highest temperature is 28°C and the lowest temperature is 25°C, then the temperature extreme difference is 3°C. The humidity change slope feature is obtained by calculating the rate of change of humidity in the window over time. Assuming that in this window, the humidity changes from 60% to 62%, the humidity change slope is calculated to be 0.2% / minute. The dust concentration accumulation feature is to accumulate the dust concentration data collected every minute in the window. For example, the dust concentration accumulation in the window is 50μg / m³. The above features constitute the abnormal environmental fluctuation feature set.

[0018] Step S130 , calling a pre-trained multimodal violation recognition model, performing dynamic weight allocation processing on the temporal behavior trajectory sequence, the device operation compliance feature set, and the environmental abnormal fluctuation feature set, to generate a fused violation feature vector.

[0019] In this embodiment, the pre-trained multimodal violation recognition model has been trained on a large amount of historical data and has strong recognition capabilities. In detail, the temporal behavior trajectory sequence can be input into the temporal coding network in the multimodal violation recognition model. In the temporal coding network, the bidirectional LSTM layer performs forward and backward time step traversal processing on the sequence. Taking the temporal behavior trajectory sequence of the tobacco leaf baking stage mentioned above as an example, the bidirectional LSTM layer will start from the first time step and calculate the forward hidden state vector and backward hidden state vector of each time step in turn. For example, in the first time step, according to the input temperature, humidity and other characteristics of the time step, combined with the hidden state vector of the previous time step, the forward hidden state vector h1_f and the backward hidden state vector h1_b are calculated. Then the forward hidden state vector and the backward hidden state vector are spliced ​​to obtain a set of bidirectional fused hidden state vectors. Next, the set is input into the time attention layer, and the time attention layer assigns weight coefficients to each time step according to the importance of different time steps in the sequence. For example, after calculation, the time step weight coefficient of the color fixing period in the whole baking process is relatively high because this stage has a greater impact on the quality of tobacco leaves. Finally, the bidirectional fusion hidden state vector set is weighted and summed according to these attention weight coefficients to generate a long-term and short-term dependency feature vector, which contains the time dependency information of different stages in the tobacco leaf processing process.

[0020] The device operation compliance feature set is input into the compliance assessment network. The convolutional neural network performs multi-scale convolution kernel sliding processing on the device operation compliance feature set, for example, using 3×3 and 5×5 convolution kernels to generate a set of local spatial feature maps. These feature maps capture the local information of the device operation features at different scales. The local spatial feature map set is subjected to maximum pooling processing to extract key information and generate a spatial invariant feature vector. The vector is input into the rule matching layer and feature matched with the compliance rules in the preset device operation rule library. For example, the rule library specifies the value range of certain parameters of the device during operation. By comparing the parameter values ​​in the device operation records with the standards in the rule library, a rule matching score set is generated. The rule matching score set is subjected to threshold filtering processing to extract the device operation features corresponding to the matching scores that exceed the compliance threshold, and generate a spatial local violation feature set.

[0021] The set of abnormal environmental fluctuation features is input into the environmental association network. First, the environmental graph structure is constructed according to the environmental parameter type. The graph nodes represent different environmental parameter sensors, such as temperature sensors, humidity sensors, dust concentration sensors, etc., and the graph edges represent the spatial adjacency relationship between sensors. For example, there is an edge connection between temperature sensors and humidity sensors at adjacent positions. The environmental graph structure is input into the graph neural network for node feature aggregation processing. Each node generates the environmental association feature vector of each node based on the feature information of other nodes connected to it. These vectors are processed by the abnormal propagation layer, and the abnormal propagation weight coefficient between adjacent nodes is calculated. For example, if a temperature sensor detects an abnormal increase in temperature, and the adjacent humidity sensor detects an abnormal decrease in humidity, the abnormal propagation weight coefficient between them is calculated through a certain algorithm. The environmental association feature vectors are weighted updated according to these coefficients to generate a set of environmental association abnormal feature vectors.

[0022] The dynamic weight allocation layer in the multimodal violation recognition model is called to process the three vectors generated above. First, the long-term and short-term dependency feature vectors, the spatial local violation feature set, and the environment-related abnormal feature vector set are input into the cross-modal attention mechanism to calculate the similarity between modalities. For example, the similarity between the long-term and short-term dependency feature vectors and the spatial local violation feature set is calculated, and the cross-modal attention weight matrix is ​​obtained by inner product and other methods. Each element in the matrix represents the degree of correlation between different modal features. The cross-modal attention weight matrix is ​​input into the weight normalization layer for row-wise normalization to generate a modal weight allocation matrix. According to the matrix, the long-term and short-term dependency feature vectors, the spatial local violation feature set, and the environment-related abnormal feature vector set are linearly weighted to generate a fused violation feature vector, which fuses the violation information of different modalities.

[0023] Step S140, determining the probability distribution of illegal operations of the target cigarette station based on the fused illegal feature vector, and generating a cigarette station violation warning strategy according to the probability distribution of illegal operations.

[0024] In this embodiment, the fused violation feature vector can be input into the violation classification network in the multimodal violation recognition model. The fully connected layer performs nonlinear transformation processing on the fused violation feature vector, maps it to a high-dimensional space, extracts more complex feature information, and generates a high-dimensional violation feature vector. The vector is processed by the probability output layer to generate a violation operation category confidence vector. For example, the vector may contain elements such as equipment violation operation confidence, environmental out-of-control operation confidence, and tobacco leaf processing violation operation confidence, which respectively represent the possibility assessment of different types of violation operations. The violation operation category confidence vector is input into the probability normalization layer, and these confidence values ​​are converted into probability values ​​through an algorithm to generate a probability distribution of violation operations. Assume that after calculation, the probability of equipment violation operation is 0.3, the probability of environmental out-of-control operation is 0.2, and the probability of tobacco leaf processing violation operation is 0.4.

[0025] Then, based on these probability values, a warning strategy for tobacco station violations is generated. When the probability of equipment illegal operation exceeds the first probability threshold, assuming that the first probability threshold is 0.25, a mandatory verification instruction for equipment operation and a device locking strategy are generated. The mandatory verification instruction for equipment operation requires a comprehensive check of the key parameters and operation logic of the equipment to ensure that the equipment operates normally. The device locking strategy is to temporarily lock the equipment to prevent further illegal operations. When the probability of environmental out-of-control operation exceeds the second probability threshold, assuming that the second probability threshold is 0.15, an environmental parameter adjustment instruction and a ventilation equipment startup strategy are generated. The environmental parameter adjustment instruction adjusts parameters such as temperature, humidity, and dust concentration to the normal range according to the abnormal conditions of the current environmental parameters. The ventilation equipment startup strategy is to turn on the ventilation equipment to improve the air quality in the tobacco station. When the probability of illegal operation of tobacco leaf processing exceeds the third probability threshold, assuming that the third probability threshold is 0.35, a baking process suspension instruction and a tobacco leaf quality re-inspection strategy are generated. The baking process suspension instruction immediately stops the current baking operation to prevent further impact on tobacco leaf quality. The tobacco leaf quality re-inspection strategy requires that the already baked tobacco leaves be re-tested to ensure that the tobacco leaf quality meets the standards. These instructions are combined to generate a smoke station violation warning strategy.

[0026] Step S150, feeding back the cigarette station violation warning strategy to the cigarette station supervision terminal to trigger an illegal operation interception instruction.

[0027] In this embodiment, the device locking strategy can be encapsulated as a first control instruction set and sent to the baking device controller of the target tobacco station through the device control interface. For example, the specific parameters and instruction information of the device locking are packaged and sent to the control unit of the baking device. After receiving the instruction, the device controller immediately executes the device locking operation to prohibit any unauthorized operation.

[0028] The ventilation equipment startup strategy is encapsulated as a second control instruction set and sent to the ventilation equipment controller of the target smoke station through the environmental control interface. After receiving the instruction, the ventilation equipment controller starts the ventilation equipment according to the instruction requirements and adjusts the air circulation and environmental parameters in the smoke station.

[0029] The tobacco leaf quality re-inspection strategy is encapsulated as a third control instruction set and sent to the tobacco leaf quality inspection terminal of the target tobacco station through the quality inspection terminal interface. After receiving the instruction, the quality inspection terminal organizes staff to conduct quality re-inspection of the designated tobacco leaves.

[0030] When the first control instruction set, the second control instruction set, and the third control instruction set are all successfully executed, the tobacco station supervision terminal generates a signal that the illegal operation interception is completed. At the same time, the probability distribution of illegal operations is updated according to the new operation conditions and monitoring data. For example, after equipment calibration and adjustment, the probability of equipment illegal operation may be reduced to 0.1; after the ventilation equipment is started, the probability of environmental out-of-control operation is reduced to 0.1; after the tobacco leaf quality is re-inspected, the probability of illegal operation in tobacco leaf processing is adjusted according to the re-inspection results, such as adjusted to 0.3. Through such a cyclic processing, the tobacco station can promptly detect and handle illegal operations to ensure the smooth progress of the tobacco leaf processing process and product quality.

[0031] Based on the above steps, the embodiment of the present application realizes accurate monitoring and intelligent early warning of the entire production process of the tobacco station, significantly improving the comprehensiveness and accuracy of the identification of illegal operations in the tobacco station. Specifically, by obtaining a set of monitoring data covering multiple dimensions such as tobacco leaf quality, equipment operation and environmental parameters, on this basis, by extracting time-series behavioral features from multimodal monitoring data, not only the dynamic behavior trajectory of the tobacco leaf processing process is accurately portrayed, but also the compliance characteristics of equipment operation and the abnormal fluctuation pattern of environmental parameters are deeply excavated, providing rich and relevant feature information for subsequent violation identification. Furthermore, the pre-trained multimodal violation identification model intelligently fuses feature information from different sources through a dynamic weight allocation mechanism to generate a highly representative fused violation feature vector, effectively improving the sensitivity and specificity of violation feature identification. Based on the probability distribution of illegal operations determined by the fused violation feature vector, it is possible to accurately quantify the violation risks of each link in the tobacco station, and then generate targeted violation warning strategies, realizing the full chain intelligence from risk identification to warning decision-making. Ultimately, by providing real-time feedback of the early warning strategy to the tobacco station supervision terminal and triggering instructions to intercept illegal operations, a closed-loop violation prevention and control system was formed. This not only greatly improved the timely detection rate and handling efficiency of illegal operations in tobacco stations, but also effectively reduced production accidents and quality risks caused by illegal operations.

[0032] In a possible implementation, step S120 includes: Step S121, dividing the tobacco leaf quality monitoring data into baking stages to generate a tobacco leaf baking curve and a corresponding temperature control stage sequence.

[0033] In this embodiment, in a specific tobacco leaf baking process, relevant data is continuously collected from the time the tobacco leaves are sent to the baking room. As time goes by, the entire baking process is divided into different stages according to the changes in temperature and humidity. In the initial stage, the temperature rises slowly from room temperature, and the humidity is also maintained at a relatively stable high level. This stage is determined to be the early stage of yellowing. After that, the temperature rises further and stabilizes in a specific range, and the humidity begins to gradually decrease. This stage is the late stage of yellowing. As the baking process advances, the temperature rises significantly, the humidity decreases rapidly, and the color fixing period is entered. Finally, the temperature is maintained at a high level until the tobacco leaves are completely dry, which is the dry tendon period. By accurately recording the start and end time of each stage, and sorting out the corresponding temperature and humidity data, a detailed tobacco leaf baking curve is generated. The curve uses time as the horizontal axis and temperature and humidity as the vertical axis to intuitively show the dynamic changes of temperature and humidity during the entire baking process. At the same time, the time nodes of each stage are clarified, and a corresponding temperature control stage sequence is formed, such as the early yellowing period from 0 hours to 6 hours, the late yellowing period from 6 hours to 14 hours, the color fixing period from 14 hours to 26 hours, and the drying period from 26 hours to 40 hours.

[0034] Step S122, extracting the duration characteristics, temperature fluctuation variance characteristics and humidity associated offset characteristics of each stage in the temperature control stage sequence to obtain the time series behavior trajectory sequence.

[0035] For example, for the duration feature, take the early stage of yellowing as an example, from the start time 0 hours to the end time 6 hours, the duration is 6 hours. For the temperature fluctuation variance feature, in the early stage of yellowing, the temperature data is recorded every 10 minutes. Assume that a total of 36 temperature data are recorded (36 10-minute data in 6 hours). First calculate the average value of these temperature data, add all the temperature data, get the sum and divide it by the number of data 36 to get the average value. Then, calculate the square of the difference between each temperature data and the average value, add the above square values, and divide it by the number of data 36 to finally get the temperature fluctuation variance. For example, after detailed calculation, the temperature fluctuation variance in the early stage of yellowing is 2.5. For the humidity correlation offset feature, the humidity standard value in the early stage of yellowing is preset to 70%, and the actual monitored humidity data is 72% after average calculation. Then the humidity correlation offset is the actual average value minus the standard value, that is, 72%-70%=+2%. In the same way, calculate the other stages to obtain the entire time series behavior trajectory sequence.

[0036] Step S123, performing operation instruction parsing processing on the device operation log data, extracting the device power on / off time interval characteristics, operation instruction compliance labels and operation frequency abnormality index characteristics, and obtaining the device operation compliance feature set.

[0037] Taking the baking equipment as an example, the equipment operation log records the operation of the equipment in detail. The log clearly shows the start-up time and shutdown time of the equipment. For example, during this baking process, the equipment was turned on at 8 am and turned off at 6 pm. The shutdown time is converted to 18 o'clock in the 24-hour system, so the start-up and shutdown time interval is 18-8=10 hours. For the operation instruction compliance label, a series of equipment operation rules are preset, such as the temperature adjustment range cannot exceed ±5℃ during the color setting period. In the log, check the temperature adjustment instructions entered by the operator. If the temperature adjustment instruction entered at a certain moment in this stage makes the temperature change within the allowable range, the compliance label of the instruction is "compliant"; if it exceeds the range, it is "violation". The calculation of the operation frequency abnormality index feature first counts the number of equipment operation instructions in a certain period of time. For example, in the 12 hours of the color setting period, a total of 100 operation instructions were recorded. According to historical data, the normal number of operation instructions in this stage is about 8 per hour, and the normal number for 12 hours should be 8×12=96. The operation frequency abnormality index is obtained by dividing the actual number of operation instructions by the normal number of operation instructions, that is, 100÷96≈1.04. By performing similar processing on the operation logs of each device, the device operation compliance feature set is obtained.

[0038] Step S124, performing sliding window mean processing on the environmental parameter monitoring data, extracting the temperature extreme difference characteristics, humidity change slope characteristics and dust concentration accumulation characteristics within the window, and obtaining the environmental abnormal fluctuation feature set.

[0039] For example, a 15-minute sliding window can be set, and data processing can be performed with 15 minutes as a window from the moment when the environmental parameter monitoring data starts to be recorded. For the temperature extreme difference feature within the window, in a certain 15-minute window, the highest temperature recorded is 30°C and the lowest temperature is 26°C. The temperature extreme difference is the highest temperature minus the lowest temperature, that is, 30-26=4°C. For the humidity change slope feature, the humidity is 65% at the beginning of the window and 63% at the end. The time span is 15 minutes. The humidity change slope is the humidity change divided by the time, that is, (63%-65%) ÷ 15 ≈ -0.13% / minute. For the dust concentration accumulation feature, in the 15-minute window, the dust concentration data is recorded once a minute, and these 15 data are added in sequence. Assuming that the sum after addition is 80μg / m³, this is the dust concentration accumulation in the window. As the window slides in sequence, the same calculation is performed on each window, and finally a set of environmental abnormal fluctuation features is obtained.

[0040] In a possible implementation, step S130 includes: Step S131, inputting the temporal behavior trajectory sequence into the temporal coding network in the multimodal violation recognition model to generate a tobacco leaf processing behavior coding vector.

[0041] In this embodiment, the time series behavior trajectory sequence of the tobacco station contains the characteristic information of each stage of tobacco leaf baking, such as the duration characteristics, temperature fluctuation variance characteristics and humidity associated offset characteristics mentioned above. In the time series coding network, the bidirectional LSTM layer begins to traverse the time series behavior trajectory sequence in forward and backward time steps. Taking the entire process of tobacco leaf baking as an example, starting from the first time step, the bidirectional LSTM layer calculates the forward hidden state vector and the backward hidden state vector based on the specific feature data of the input time step and the hidden state vector of the previous time step. For example, in the first 10-minute time step at the beginning of baking, the input is the temperature, humidity and other related feature data at the beginning of the yellowing period. The bidirectional LSTM layer uses specific calculation rules and combines the information of the previous time step (because it is the starting step, it can be regarded as the initial hidden state vector) to calculate the forward hidden state vector of the time step, which contains the key information of tobacco leaf baking in the forward time dimension from the beginning to the current moment; at the same time, through reverse calculation, the backward hidden state vector is obtained, which contains the relevant information from the end of the entire baking process to the current moment.

[0042] After that, the forward hidden state vector and the backward hidden state vector are concatenated. The two vectors are combined in a specific order and manner to form a set of bidirectional fused hidden state vectors. The set of bidirectional fused hidden state vectors integrates the information of the forward and reverse time steps, and more comprehensively reflects the characteristics of the tobacco processing process at different time points. Next, the set of bidirectional fused hidden state vectors is input into the temporal attention layer. The temporal attention layer assigns a weight coefficient to each time step according to the importance of different time steps in the sequence to the overall tobacco processing behavior. For example, in the entire baking process, the color fixation period plays a key role in determining the final quality of tobacco leaves, so the weight coefficient of the time step in this stage will be relatively high. The temporal attention layer determines the attention weight coefficient of each time step through calculation and analysis. Finally, the set of bidirectional fused hidden state vectors is weighted and summed according to these attention weight coefficients. Specifically, each bidirectional fused hidden state vector is multiplied by the corresponding attention weight coefficient, and then all products are added together to generate a long-term and short-term dependency feature vector, that is, a tobacco processing behavior encoding vector. The tobacco leaf processing behavior encoding vector comprehensively and concisely summarizes the time dependencies and behavioral characteristics of different stages in the tobacco leaf processing process.

[0043] Step S132: input the device operation compliance feature set into the compliance assessment network in the multimodal violation identification model to generate a device operation risk coding vector.

[0044] In this embodiment, the equipment operation compliance feature set of the tobacco station covers the equipment power on / off time interval features, operation instruction compliance labels, and operation frequency abnormality index features. The convolutional neural network in the compliance assessment network performs multi-scale convolution kernel sliding processing on the equipment operation compliance feature set. For example, convolution kernels of sizes 3×3 and 5×5 are used to perform sliding operations on the equipment operation compliance feature data. During the sliding process, the convolution kernel calculates and extracts the feature data of the local area according to its own parameters and calculation rules to generate a set of local spatial feature maps. These feature maps capture the local information of the equipment operation features at different scales, such as the change pattern of operation instructions in different time periods, the local fluctuation of operation frequency, etc.

[0045] Subsequently, the local spatial feature map set is subjected to maximum pooling. The maximum pooling operation selects the maximum value in each local spatial feature map to generate a spatial invariant feature vector. The spatial invariant feature vector retains the key information in the local spatial feature map and ignores some relatively unimportant details. The spatial invariant feature vector is input into the rule matching layer, which performs feature matching with the compliance rules in the preset device operation rule library. For example, the rule library specifies the operating frequency range of the device at a specific stage, the correct format of the instruction, and other rules. The rule matching layer compares the device operation features in the spatial invariant feature vector with these rules one by one, calculates the matching degree of each rule with the current device operation feature, and generates a rule matching score set. After that, the rule matching score set is subjected to threshold filtering. A compliance threshold is set, such as 0.6, and the device operation features corresponding to the matching scores exceeding the compliance threshold are extracted to generate a spatial local violation feature set, that is, a device operation risk coding vector. The device operation risk coding vector highlights the feature information that may have violation risks during the device operation process.

[0046] Step S133: input the environmental abnormal fluctuation feature set into the environmental association network in the multimodal violation recognition model to generate an environmental abnormality coding vector.

[0047] In this embodiment, an environmental graph structure can be constructed according to the environmental parameter types in the abnormal fluctuation feature set of the tobacco station environment. Graph nodes represent different environmental parameter sensors, such as temperature sensors, humidity sensors, dust concentration sensors, etc.; graph edges represent the spatial adjacency relationship between sensors, such as the existence of connecting edges between temperature sensors and humidity sensors at adjacent positions. The constructed environmental graph structure is input into the graph neural network for node feature aggregation processing. The graph neural network will aggregate and calculate the features of each node based on the connection relationship between the nodes in the graph structure. For example, a temperature sensor node will integrate the feature information of the humidity sensor node, dust concentration sensor node, etc. connected to it, and generate the environment-related feature vectors of each node through a specific calculation method.

[0048] Next, these environment-related feature vectors are processed by the anomaly propagation layer. The anomaly propagation layer calculates the anomaly propagation weight coefficients between adjacent nodes. For example, if a temperature sensor detects a sudden increase in temperature, and the adjacent humidity sensor detects an abnormal decrease in humidity, a series of calculations, combined with preset algorithms and rules, are used to calculate the anomaly propagation weight coefficient between them. The anomaly propagation weight coefficient reflects the degree of association and propagation possibility of the anomaly between two adjacent nodes. The environment-related feature vectors are weighted updated according to these anomaly propagation weight coefficients. Specifically, each environment-related feature vector is multiplied by the corresponding anomaly propagation weight coefficient, and then adjusted and updated accordingly to generate a set of environment-related anomaly feature vectors, that is, an environment anomaly coding vector. The environment anomaly coding vector integrates the association information between environmental parameters and possible abnormal fluctuations.

[0049] Step S134, calling the dynamic weight allocation layer in the multimodal violation identification model, and generating a set of modal weight allocation coefficients based on the cross-modal correlation scores between the tobacco processing behavior coding vector, the equipment operation risk coding vector and the environmental anomaly coding vector.

[0050] In this embodiment, the previously generated tobacco leaf processing behavior coding vector, equipment operation risk coding vector and environmental anomaly coding vector can be input into the cross-modal attention mechanism for inter-modal similarity calculation. For example, by calculating the inner product of the tobacco leaf processing behavior coding vector and the equipment operation risk coding vector, the similarity of the two vectors in the feature dimension is measured to obtain the first cross-modal correlation score. Similarly, the inner product of the tobacco leaf processing behavior coding vector and the environmental anomaly coding vector is calculated to obtain the second cross-modal correlation score; the inner product of the equipment operation risk coding vector and the environmental anomaly coding vector is calculated to obtain the third cross-modal correlation score. These three cross-modal correlation scores are input into the softmax function for normalization. The softmax function calculates these three scores so that their sum is 1, and each score is between 0 and 1, thereby generating a set of modal weight distribution coefficients.

[0051] Step S135, performing weighted fusion processing on the tobacco leaf processing behavior coding vector, the equipment operation risk coding vector and the environmental anomaly coding vector according to the modal weight allocation coefficient set to generate the fused violation feature vector.

[0052] Specifically, the tobacco leaf processing behavior coding vector is multiplied by the corresponding modal weight distribution coefficient, the equipment operation risk coding vector is multiplied by its corresponding coefficient, and the environmental anomaly coding vector is multiplied by the corresponding coefficient. Then these three products are added together to finally generate a fused violation feature vector. The fused violation feature vector integrates the violation feature information of tobacco leaf processing behavior, equipment operation, and environmental factors.

[0053] In a possible implementation, step S134 includes: Step S1341, performing inner product processing on the tobacco leaf processing behavior coding vector and the equipment operation risk coding vector to generate a first cross-modal correlation score.

[0054] In this embodiment, the tobacco leaf processing behavior coding vector is a vector that comprehensively summarizes the time dependency and behavior characteristics of different stages in the tobacco leaf processing process after being processed by the temporal coding network; the equipment operation risk coding vector is generated by the compliance assessment network, highlighting the characteristic information of possible violation risks in the equipment operation process. Both vectors have a certain dimension. Assume that the tobacco leaf processing behavior coding vector is a vector containing 10 elements, which respectively represent characteristics such as the duration ratio of different baking stages and the temperature change trend; the equipment operation risk coding vector is also a vector of 10 elements, representing characteristics such as the abnormal degree of the time interval between power on and power off, and the frequency of violation of operation instructions.

[0055] In this embodiment, when performing inner product processing, the elements at the corresponding positions of the two vectors are multiplied, and then all the products are added. For example, the first element of the tobacco leaf processing behavior coding vector is 0.2, which represents the proportion of the yellowing period to the total baking time; the first element of the equipment operation risk coding vector is 0.3, which represents the degree of deviation of the start-up time interval from the normal standard. Multiplying these two elements yields 0.2×0.3=0.06. In the same way, the products of the elements at other corresponding positions are calculated in sequence, such as the second element product is 0.1×0.4=0.04, the third element product is 0.3×0.2=0.06, and so on. Add all 10 groups of products, 0.06+0.04+0.06+... (a total of 10 items added), after detailed calculation, the final sum is the first cross-modal correlation score, assuming that the first cross-modal correlation score is 0.4. The first cross-modal correlation score reflects the degree of correlation between the two modes of tobacco leaf processing behavior and equipment operation risk.

[0056] Step S1342, performing inner product processing on the tobacco leaf processing behavior coding vector and the environmental anomaly coding vector to generate a second cross-modal correlation score.

[0057] In this embodiment, the environmental anomaly coding vector integrates the correlation information between environmental parameters and possible abnormal fluctuations. It is also assumed that the environmental anomaly coding vector is also a vector of 10 elements, representing features such as temperature extremes and humidity accumulation anomalies. In the same way as the previous inner product processing method, the tobacco leaf processing behavior coding vector is multiplied by the elements at the corresponding positions of the environmental anomaly coding vector and the summed. For example, the first element 0.2 of the tobacco leaf processing behavior coding vector is multiplied by the first element 0.1 of the environmental anomaly coding vector to obtain 0.2×0.1=0.02, and the second element product is 0.1×0.3=0.03 and so on. Add all 10 groups of products, 0.02+0.03+... (a total of 10 items added), and after calculation, the second cross-modal correlation score is obtained, assuming it to be 0.3. The second cross-modal correlation score reflects the correlation between tobacco leaf processing behavior and environmental anomalies.

[0058] Step S1343: perform inner product processing on the equipment operation risk coding vector and the environment anomaly coding vector to generate a third cross-modal correlation score.

[0059] In detail, according to the same inner product calculation method, the elements of the corresponding positions of the equipment operation risk coding vector and the environmental anomaly coding vector are multiplied. For example, the first element 0.3 of the equipment operation risk coding vector is multiplied by the first element 0.1 of the environmental anomaly coding vector to obtain 0.3×0.1=0.03, and the second element product is 0.4×0.2=0.08, and so on. Add these 10 groups of products, 0.03+0.08+... (a total of 10 items are added), and after calculation, the third cross-modal correlation score is obtained, which is assumed to be 0.2. This third cross-modal correlation score shows the degree of correlation between equipment operation risk and environmental anomalies.

[0060] Step S1344: input the first cross-modal correlation score, the second cross-modal correlation score, and the third cross-modal correlation score into a softmax function for normalization processing to generate the modality weight allocation coefficient set.

[0061] In this embodiment, the first cross-modal correlation score of 0.4, the second cross-modal correlation score of 0.3, and the third cross-modal correlation score of 0.2 are input into the softmax function for normalization processing to generate a set of modal weight allocation coefficients. The normalization process of the softmax function is as follows: First, the exponential values ​​of the three scores are calculated, that is, the score power of e. e is a natural constant, which is approximately 2.718. For the first cross-modal correlation score of 0.4, the 0.4 power of e is calculated, that is, the 0.4 power of 2.718, which is approximately 1.5. For the second cross-modal correlation score of 0.3, the 0.3 power of e is calculated, and the 0.3 power of 2.718 is approximately 1.35. For the third cross-modal correlation score of 0.2, the 0.2 power of e is calculated, and the 0.2 power of 2.718 is approximately 1.22.

[0062] Then add these three index values, 1.5+1.35+1.22=4.07. Next, calculate the proportion of each index value in the total, which is the modal weight allocation coefficient. The modal weight allocation coefficient corresponding to the first cross-modal correlation score is 1.5÷4.07≈0.37, the modal weight allocation coefficient corresponding to the second cross-modal correlation score is 1.35÷4.07≈0.33, and the modal weight allocation coefficient corresponding to the third cross-modal correlation score is 1.22÷4.07≈0.3. The above three coefficients together constitute the modal weight allocation coefficient set, which respectively represent the weights of the three modes of tobacco processing behavior, equipment operation risks and environmental anomalies in the subsequent fusion process.

[0063] In a possible implementation, step S140 includes: Step S141, calling the violation classification network in the multimodal violation recognition model, performing full connection mapping processing on the fused violation feature vector, and generating a violation operation category confidence vector.

[0064] In this embodiment, the fused violation feature vector is obtained by weighted fusion of the tobacco leaf processing behavior coding vector, the equipment operation risk coding vector and the environmental anomaly coding vector, which integrates various aspects of violation feature information. The fully connected mapping process in the violation classification network is to multiply each element of the fused violation feature vector with a series of preset weight values, then add all the products and add a bias value to achieve feature conversion and mapping.

[0065] Assume that the fused violation feature vector is a vector containing 15 elements, which represent violation information of different dimensions, such as the degree of abnormal temperature fluctuation during tobacco leaf baking, the frequency of violation of equipment operation instructions, and the amplitude of environmental parameters exceeding the normal range. For each output node in the violation classification network (corresponding to different violation operation categories), there is a set of 15 weight values ​​and a bias value. Taking the output node corresponding to the equipment violation operation category as an example, the first weight value is assumed to be 0.2, which is multiplied by the first element of the fused violation feature vector (assuming that it represents the degree of abnormality of the equipment startup time interval) to obtain a product. The second weight value is 0.3, which is multiplied by the second element of the fused violation feature vector (assuming that it represents the frequency of equipment operation instruction format errors) to obtain another product. Similarly, after multiplying the 15 weight values ​​with the corresponding elements of the fused violation feature vector, these 15 products are added. Assuming that the sum after addition is 3.5, plus the preset bias value of 0.5, the result 4 is a preliminary value of the confidence of the equipment violation operation category.

[0066] In the same way, the above calculations are performed for the output nodes corresponding to the environmental out-of-control operation category and the tobacco processing illegal operation category. Assume that the preliminary value of the environmental out-of-control operation category is 2.8, and the preliminary value of the tobacco processing illegal operation category is 3.2. These preliminary values ​​are further transformed nonlinearly (such as using the ReLU function, etc.), the negative values ​​are converted to 0, the numerical range is adjusted, and finally the confidence vector of the illegal operation category is generated. The confidence vector of the illegal operation category contains the confidence of the equipment illegal operation, the confidence of the environmental out-of-control operation, and the confidence of the tobacco processing illegal operation, which are the results of the above calculations and nonlinear transformations, respectively, and are assumed to be [0.6, 0.4, 0.5].

[0067] Step S142, inputting the illegal operation category confidence vector into the probability normalization layer to generate the illegal operation probability distribution, wherein the illegal operation probability distribution includes the probability of equipment illegal operation, the probability of environmental out-of-control operation and the probability of tobacco leaf processing illegal operation.

[0068] In this embodiment, the function of the probability normalization layer is to convert each value in the confidence vector of the illegal operation category into a probability value, and the sum of these probability values ​​is 1. The specific calculation process is as follows: First, calculate the sum of all elements in the confidence vector of the illegal operation category, that is, 0.6+0.4+0.5=1.5. Then, for the equipment illegal operation confidence of 0.6, calculate its proportion in the total, that is, 0.6÷1.5=0.4, and this 0.4 is the probability of equipment illegal operation. For the environmental out-of-control operation confidence of 0.4, its probability is calculated to be 0.4÷1.5≈0.27. For the tobacco leaf processing illegal operation confidence of 0.5, its probability is calculated to be 0.5÷1.5≈0.33. In this way, a probability distribution of illegal operation is generated, in which the probability of equipment illegal operation is 0.4, the probability of environmental out-of-control operation is 0.27, and the probability of tobacco leaf processing illegal operation is 0.33.

[0069] Step S143: When the probability of the device illegal operation exceeds a first probability threshold, a device operation mandatory verification instruction and a device locking strategy are generated.

[0070] Step S144: when the probability of the environmental out-of-control operation exceeds a second probability threshold, an environmental parameter adjustment instruction and a ventilation equipment startup strategy are generated.

[0071] Step S145, when the probability of illegal operation in tobacco leaf processing exceeds a third probability threshold, a baking process pause instruction and a tobacco leaf quality re-inspection strategy are generated.

[0072] In this embodiment, a tobacco station violation warning strategy can be generated based on the probability distribution of illegal operations. Set the first probability threshold to 0.3, the second probability threshold to 0.2, and the third probability threshold to 0.3. For the probability of equipment illegal operation is 0.4, since it exceeds the first probability threshold of 0.3, a mandatory verification instruction for equipment operation and an equipment locking strategy are generated at this time. The mandatory verification instruction for equipment operation requires a comprehensive inspection of each key component and parameter of the equipment, such as checking whether the heating system of the baking equipment is normal, whether the temperature sensor is accurate, and whether the ventilation duct is unobstructed. The equipment locking strategy is to immediately stop the operation of the equipment and prohibit any unauthorized operation to prevent possible further illegal operations from causing damage to the equipment or tobacco leaf processing.

[0073] For the probability of environmental out-of-control operation of 0.27, it exceeds the second probability threshold of 0.2, so an environmental parameter adjustment instruction and a ventilation equipment startup strategy are generated. The environmental parameter adjustment instruction formulates a specific adjustment plan based on the actual situation of the current environmental parameters. For example, if the temperature in the current smoke station is too high, the instruction will require the power of the heating equipment to be reduced; if the humidity exceeds the standard, the instruction will instruct the dehumidification equipment to be started. The ventilation equipment startup strategy is to turn on the ventilation equipment, strengthen the circulation of air in the smoke station, improve air quality, reduce dust concentration, etc., to ensure that the environmental parameters are within the normal working range.

[0074] For the probability of illegal operation in tobacco leaf processing of 0.33, which exceeds the third probability threshold of 0.3, a baking process pause instruction and a tobacco leaf quality re-inspection strategy are generated. The baking process pause instruction will immediately interrupt the current baking operation to avoid further deterioration of tobacco leaf quality due to possible illegal operations. The tobacco leaf quality re-inspection strategy requires re-inspection of tobacco leaves that have been baked to the current stage. The inspection items include moisture content, color, aroma and other aspects of the tobacco leaves to determine whether the quality of the tobacco leaves meets the standards.

[0075] Step S146, combining the equipment operation mandatory verification instruction, the environmental parameter adjustment instruction and the baking process pause instruction to generate the smoke station violation warning strategy.

[0076] In this embodiment, the tobacco station violation warning strategy comprehensively covers the response measures for different probability situations of illegal operations, aiming to timely discover and deal with potential violations during the operation of the tobacco station, and ensure the quality of tobacco leaf processing and the normal operation of the tobacco station.

[0077] In a possible implementation, step S150 includes: Step S151, encapsulating the device locking strategy into a first control instruction set, and sending it to the baking device controller of the target tobacco station through the device control interface.

[0078] In this embodiment, the device locking strategy is to prevent the equipment from continuing to operate when there is a risk of illegal operation, causing more serious problems. In this tobacco station, the device locking strategy contains key information such as detailed equipment identification, locking time, prohibited operation type, etc. For example, what needs to be locked this time is the type A baking equipment located in baking area No. 3. The tobacco station supervision system encapsulates the equipment locking strategy, and integrates and encodes the equipment number "Type A baking equipment in baking area No. 3", the locking start time "current time + 5 minutes", the locking duration "60 minutes", and prohibited operation instructions such as "temperature adjustment instructions, ventilation control instructions". Then, through the device control interface, these encapsulated first control instruction sets are sent to the baking equipment controller of the target tobacco station. The device control interface is a system specifically used for equipment communication and control to ensure that the instructions can be accurately transmitted to the corresponding equipment controller.

[0079] Step S152: encapsulate the ventilation equipment startup strategy into a second control instruction set, and send it to the ventilation equipment controller of the target smoke station through the environmental control interface.

[0080] In this embodiment, the ventilation equipment startup strategy aims to improve the environmental parameters in the smoke station so that they reach the normal operating range. Assume that the ventilation equipment startup strategy requires starting the Type B ventilation equipment located in Area 2, setting the ventilation intensity to "medium speed" and the ventilation time to "last for 30 minutes". The smoke station supervision system will also encapsulate this information, and integrate the information such as the ventilation equipment number "Type B ventilation equipment in Area 2", the ventilation intensity setting "medium speed", the ventilation start time "current time + 10 minutes" and the ventilation time "30 minutes" into a second control instruction set. Afterwards, the instruction set is accurately sent to the ventilation equipment controller of the target smoke station through the environmental control interface, which is responsible for ensuring the reliable transmission of instructions between environmental control related equipment.

[0081] Step S153: encapsulate the tobacco leaf quality re-inspection strategy into a third control instruction set, and send it to the tobacco leaf quality inspection terminal of the target tobacco station through the quality inspection terminal interface.

[0082] In this embodiment, the tobacco leaf quality re-inspection strategy stipulates the batches of tobacco leaves that need to be re-inspected, the specific items and standards for the re-inspection, etc. For example, this time it is necessary to re-inspect the 5th batch of tobacco leaves that have been baked to the color-fixing period in baking room No. 1. The re-inspection items include "moisture content, color uniformity, aroma concentration", etc., and each item has a clear standard value. The tobacco station supervision system organizes this information, and encodes the detailed information such as the tobacco leaf batch "the 5th batch of tobacco leaves in the color-fixing period in baking room No. 1", the re-inspection items "moisture content, color uniformity, aroma concentration" and the corresponding standard values ​​into a third control instruction set, and then sends the instruction set to the tobacco leaf quality inspection terminal of the target tobacco station through the quality inspection terminal interface. The quality inspection terminal interface ensures that the instructions can reach the quality inspection equipment accurately.

[0083] Step S154, when the first control instruction set, the second control instruction set and the third control instruction set are all successfully executed, a violation operation interception completion signal is generated and the violation operation probability distribution is updated.

[0084] In this process, after receiving the first control instruction set, the type A baking equipment in the baking area 3 locked the equipment at the specified time and prohibited the corresponding operation according to the instruction requirements; after receiving the second control instruction set, the type B ventilation equipment in the baking area 2 started on time and operated according to the set ventilation intensity and time; after receiving the third control instruction set, the tobacco leaf quality inspection terminal in the baking room 1 immediately re-inspected the fifth batch of color-fixing tobacco leaves according to the specified items and standards. When all these operations are completed successfully, the tobacco station supervision terminal confirms that the three control instruction sets have been successfully executed, and then generates a signal for the completion of the illegal operation interception.

[0085] At the same time, the probability distribution of illegal operations is updated according to the new operating conditions and monitoring data. For example, after the equipment is locked and inspected, it is found that the risk factors of equipment violations have been eliminated, and the probability of equipment illegal operations has been reduced from 0.4 to 0.1; after the ventilation equipment is started, the environmental parameters return to normal, and the probability of environmental out-of-control operations drops from 0.27 to 0.1; after the tobacco leaf quality is re-inspected, it is found that the tobacco leaf quality basically meets the standards, but there are some minor problems. After adjustment, the probability of illegal operations in tobacco leaf processing is adjusted from 0.33 to 0.2. Through such timely feedback and updates, the tobacco station can more accurately grasp the operating status, effectively prevent and deal with illegal operations, and ensure the stability of the tobacco leaf processing process and the reliability of product quality.

[0086] In a possible implementation, the training method of the multimodal violation recognition model includes: Step S210, obtaining a historical cigarette station illegal operation data set, wherein the historical cigarette station illegal operation data set includes a historical multimodal monitoring data set and corresponding illegal operation labels.

[0087] During the long-term operation of the tobacco station, a wealth of historical data has been accumulated. The historical multimodal monitoring data set contains information from multiple aspects. The tobacco leaf quality monitoring data records the detailed parameters of different batches of tobacco leaves during the baking process. For example, the temperature and humidity data of each batch of tobacco leaves are recorded every 5 minutes from the time they enter the baking room. These data reflect the quality changes of the tobacco leaves at various times. The equipment operation log data records the operation of the equipment in detail, including the equipment's startup time, shutdown time, each operation instruction entered by the operator, and the time when the instruction was executed. Environmental parameter monitoring data is collected in real time through sensors distributed at different locations in the tobacco station, recording the changes in environmental parameters such as temperature, humidity, and dust concentration over time.

[0088] At the same time, corresponding illegal operation labels are provided for these historical data. For example, when the temperature control of a batch of tobacco leaves exceeds the preset reasonable range during the baking process, resulting in the quality of the tobacco leaves being affected, the relevant data of the batch of tobacco leaves will be marked as "tobacco leaf processing violation"; if there is an error in the equipment operation instruction, such as performing a specific operation within an unallowed time period, the equipment operation record will be marked as "equipment illegal operation"; when the environmental parameters exceed the normal range for a long time and affect the tobacco leaf processing environment, the corresponding environmental parameter monitoring data will be marked as "environmental out-of-control operation". The above illegal operation labels provide clear learning goals for subsequent model training.

[0089] Step S220 , performing time series behavior feature extraction processing on the historical multimodal monitoring data set to obtain a historical time series behavior trajectory sequence, a historical equipment operation compliance feature set, and a historical environment abnormal fluctuation feature set.

[0090] In a possible implementation, step S220 includes: Step S221, performing abnormal baking stage labeling processing on the tobacco leaf quality monitoring data in the historical multimodal monitoring data set to generate a labeled tobacco leaf baking stage sequence.

[0091] Taking a batch of tobacco leaves as an example, when analyzing its baking process, different baking stages are determined based on the changing trends of temperature and humidity and the comparison with the preset standards. For example, in the early stage of baking, the temperature gradually rises from room temperature to 38°C, and the humidity is maintained at around 75%. This stage is marked as "normal yellowing early stage"; then the temperature rises rapidly to 48°C, and the humidity drops to 60%, but the temperature rise rate exceeds the standard range, and this stage is marked as "abnormal yellowing late stage"; as baking continues, the temperature stabilizes at 55°C for color fixation, and this stage is marked as "normal color fixation period"; finally, it enters the dry rib stage, the temperature is maintained at 68°C, but the humidity fluctuates greatly, and is marked as "abnormal dry rib stage". In this way, a sequence of tobacco leaf baking stages with annotations is generated.

[0092] Step S222, extracting the temperature deviation characteristics, humidity stability index characteristics and baking time abnormality coefficient characteristics of each stage in the labeled tobacco leaf baking stage sequence, and obtaining the historical time series behavior trajectory sequence.

[0093] Taking "late stage of abnormal yellowing" as an example, when calculating the temperature deviation feature, first determine the normal temperature range preset for this stage. Assume that the upper limit of the normal temperature is 45°C and the lower limit is 42°C, while the actual average temperature for this stage is 48°C. Then the temperature deviation is the difference between the actual average temperature and the midpoint of the normal temperature range. The midpoint of the normal temperature range is (45+42)÷2=43.5°C, and the temperature deviation is 48-43.5=4.5°C. The calculation of the humidity stability index is to count the fluctuations of the humidity data in this stage. Assuming that a total of 30 humidity data are recorded in this stage, first calculate the average value of these data, add all humidity data, and then divide by the number of data 30 to get the average value. Then calculate the square of the difference between each humidity data and the average value, add these square values, and divide by the number of data 30 to get the humidity variance. Assuming the humidity variance is 5, the humidity stability index is 1 divided by the square root of the humidity variance, that is, 1÷√5≈0.45. The calculation of the baking time anomaly coefficient is to compare the actual baking time of the stage with the preset normal baking time. Assuming that the preset normal baking time of the "late yellowing" stage is 4 hours, and the actual baking time is 5 hours, the baking time anomaly coefficient is (5-4) ÷ 4 = 0.25. Using the same method, calculate each stage and finally obtain the historical time series behavior trajectory sequence.

[0094] Step S223, performing operation instruction compliance labeling processing on the device operation log data in the historical multimodal monitoring data set, generating labeled device operation frequency features, operation interval abnormality features and instruction sequence conflict features, and obtaining the historical device operation compliance feature set.

[0095] Take a baking device as an example. During a period of time, the operation instructions of the device were recorded. When counting the operation frequency of the device, it is assumed that the device received a total of 60 operation instructions within 8 hours of working time. Under normal circumstances, the number of operation instructions of the device within 8 hours should be between 40 and 50. Then the operation frequency characteristic of the device is the ratio of the actual number of operation instructions to the midpoint of the normal number of operation instructions. The midpoint of the normal number of operation instructions is (40+50) ÷ 2 = 45, and the operation frequency characteristic of the device is 60 ÷ 45 ≈ 1.33. The calculation of the abnormal operation interval characteristic is to analyze the time interval between adjacent operation instructions. Assume that the normal time interval between two adjacent operation instructions should be 10 minutes, but in the actual record, the time interval between two adjacent instructions is 5 minutes. Then the abnormal operation interval characteristic is the difference between the normal time interval and the actual time interval divided by the normal time interval, that is, (10-5) ÷ 10 = 0.5. The instruction sequence conflict characteristic is to check whether the execution order of the operation instructions conforms to the preset rules. For example, the preset rules stipulate that ventilation operation cannot be performed at the same time during the equipment heating stage, but in the actual record, ventilation instructions are sent at the same time during the heating stage. In this case, the instruction sequence conflict feature is marked as "conflict exists" and the content and time of the conflicting instructions are recorded. Through a comprehensive analysis of the equipment operation log data, related features with labels are generated to obtain a set of historical equipment operation compliance features.

[0096] Step S224, performing time window anomaly labeling processing on the environmental parameter monitoring data in the historical multimodal monitoring data set, generating environmental temperature mutation characteristics, humidity accumulation anomaly characteristics and dust concentration exceeding standard characteristics with timestamps, and obtaining the historical environmental abnormal fluctuation feature set.

[0097] Taking a 1-hour time window as an example, the environmental parameters are monitored within a certain period of time. The calculation of the environmental temperature mutation feature is to compare the difference between the highest temperature and the lowest temperature within the time window. Assuming the highest temperature is 32°C and the lowest temperature is 25°C, the temperature mutation feature is 32-25=7°C. The calculation of the humidity accumulation anomaly feature is to count the cumulative change in humidity within the time window. Assuming that the humidity is 60% at the beginning of the time window and 70% at the end, the humidity accumulation anomaly feature is 70%-60%=10%. The dust concentration exceeding standard feature is to compare the average dust concentration within the time window with the preset standard concentration. Assuming that the preset standard dust concentration is 50μg / m³, and the average dust concentration within the time window is calculated to be 65μg / m³, then the dust concentration exceeding standard feature is (65-50) ÷ 50=0.3, and the timestamp of the time window is recorded. By analyzing the environmental parameters of different time windows, the relevant features with timestamps are generated, and the historical environmental abnormal fluctuation feature set is obtained.

[0098] Step S230, constructing an initial multimodal violation recognition model, inputting the historical time series behavior trajectory sequence, the historical equipment operation compliance feature set and the historical environment abnormal fluctuation feature set into the initial multimodal violation recognition model, and generating a historical fusion violation feature vector.

[0099] For example, based on the cross entropy loss between the historical fused violation feature vector and the violation operation label, the initial multimodal violation recognition model is trained until convergence. The initial multimodal violation recognition model consists of multiple parts, such as the temporal coding network, compliance assessment network, environmental association network, dynamic weight allocation layer, and violation classification network mentioned above. The historical temporal behavior trajectory sequence is input into the temporal coding network, which extracts the long-term and short-term dependency features in the sequence through the processing of the bidirectional LSTM layer and the temporal attention layer, and generates the corresponding encoding vector. The historical equipment operation compliance feature set is input into the compliance assessment network, and the spatial local violation features are extracted through the operation of the convolutional neural network and the rule matching layer to obtain the corresponding encoding vector. The historical environmental abnormal fluctuation feature set is input into the environmental association network, and the environmental association abnormal features are extracted with the help of the analysis of the graph neural network and the abnormal propagation layer to generate the corresponding encoding vector.

[0100] Step S240 , training the initial multimodal violation recognition model until convergence based on the cross entropy loss between the historical fusion violation feature vector and the violation operation label.

[0101] For example, the inner product of the historical temporal behavior trajectory sequence encoding vector and the encoding vector of the historical equipment operation compliance feature set is calculated to obtain a cross-modal correlation score; similarly, the inner product of the historical temporal behavior trajectory sequence encoding vector and the encoding vector of the historical environmental abnormal fluctuation feature set is calculated, as well as the inner product of the encoding vector of the historical equipment operation compliance feature set and the encoding vector of the historical environmental abnormal fluctuation feature set is calculated to obtain two other cross-modal correlation scores. These three scores are input into the softmax function for normalization to generate a set of modal weight allocation coefficients. The three encoding vectors are weighted fused according to the coefficient set to generate a historical fused violation feature vector.

[0102] The calculation process of cross entropy loss is as follows: for each sample, the historical fusion violation feature vector output by the model is processed by the violation classification network to obtain a confidence vector for different violation operation categories. Assuming that there are three types of violation operation categories: "equipment violation operation", "environmental out-of-control operation" and "tobacco leaf processing violation operation", the confidence vector output by the model is [0.3, 0.4, 0.3], and the actual violation operation label of the sample is "equipment violation operation", and the corresponding label vector is [1, 0, 0]. When calculating the cross entropy loss, for the "equipment violation operation" category, use -(actual label probability × log (model output confidence)), that is, -(1×log(0.3)); for the "environmental out-of-control operation" category, it is -(0×log(0.4))=0; for the "tobacco leaf processing violation operation" category, it is -(0×log(0.3))=0. Add these three results to get the cross entropy loss of the sample. For the entire training data set, add the cross entropy losses of all samples and average them to get the average cross entropy loss. By continuously adjusting the parameters of the initial multimodal violation identification model, the average cross entropy loss is gradually reduced until the model converges. At this time, the model can more accurately identify the illegal operations of the smoke station based on the input multimodal monitoring data.

[0103] For example, in a possible implementation, step S230 includes: Step S231, constructing a temporal coding network, wherein the temporal coding network includes a bidirectional LSTM layer and a temporal attention layer, which is used to extract long-term and short-term dependency features in the historical temporal behavior trajectory sequence.

[0104] For example, in a possible implementation, step S231 includes: Step S2311, input the historical temporal behavior trajectory sequence into the bidirectional LSTM layer for forward and backward time step traversal processing to generate a forward hidden state vector and a backward hidden state vector for each time step.

[0105] Step S2312, concatenating the forward hidden state vector and the backward hidden state vector to generate a bidirectional fused hidden state vector set.

[0106] Taking the historical data of the tobacco station as an example, the historical time series behavior trajectory sequence contains rich feature information of each stage in the tobacco baking process, such as temperature deviation characteristics, humidity stability index characteristics and baking time abnormality coefficient characteristics at different stages. During the construction process, the sequence is input into the bidirectional LSTM layer for forward and backward time step traversal processing.

[0107] For a specific tobacco leaf baking process, assume that the entire baking process is divided into 20 time steps to record data. At the first time step, the bidirectional LSTM layer receives the relevant characteristic data of the tobacco leaf baking stage corresponding to the time step, such as the temperature deviation of 1.5 (indicating the difference between the current temperature and the standard temperature), the humidity stability index of 0.8 (an indicator of humidity stability obtained through a series of calculations), and the baking time abnormality coefficient of 0 (indicating that the current baking time is within the normal range). The bidirectional LSTM layer calculates the forward hidden state vector of the first time step based on its internal calculation mechanism and the hidden state vector of the previous time step (at the first time step, the previous time step can be regarded as the initial state). The forward hidden state vector contains the key information of tobacco leaf baking in the forward time dimension from the beginning to the current time step. For example, it may integrate the potential impact of the current and previous temperature and humidity change trends on tobacco leaf quality.

[0108] At the same time, the bidirectional LSTM layer performs reverse calculations from the last time step. Similarly, at the first time step, the backward hidden state vector of the first time step is calculated based on the data of the current time step and the hidden state vector of the subsequent time step (during back propagation). The backward hidden state vector contains relevant information from the end of the entire baking process to the current moment, for example, it may reflect the dependency of the subsequent baking stage on the state of the current stage.

[0109] Similarly, this calculation is performed for each time step to generate the forward hidden state vector and backward hidden state vector for each time step. Then, the forward hidden state vector and the backward hidden state vector are concatenated. For example, the elements of the forward hidden state vector of the first time step are arranged and combined with the corresponding elements of the backward hidden state vector in a specific order to generate a bidirectional fused hidden state vector. This concatenation process is performed for all 20 time steps to form a set of bidirectional fused hidden state vectors.

[0110] Step S2313: input the bidirectional fusion hidden state vector set into the temporal attention layer for time step weight allocation processing to generate the attention weight coefficient of each time step.

[0111] The temporal attention layer assigns weight coefficients to each time step according to its importance in the entire tobacco processing process. For example, during the baking process, the color-fixing period has a greater impact on the final quality of the tobacco leaves, so the weight coefficient of the time step corresponding to the color-fixing period will be relatively high. The temporal attention layer analyzes the characteristics of each time step in the set of bidirectional fusion hidden state vectors and calculates the attention weight coefficient of each time step using a specific algorithm. For example, for the first time step, after calculation and analysis, its attention weight coefficient is 0.05; the second time step is 0.06, and so on. Each time step has a corresponding weight coefficient.

[0112] Step S2314, performing weighted summation processing on the set of bidirectional fusion hidden state vectors according to the attention weight coefficient to generate a long-term and short-term dependency feature vector.

[0113] Specifically, the first bidirectional fusion hidden state vector is multiplied by its corresponding attention weight coefficient 0.05, the second bidirectional fusion hidden state vector is multiplied by 0.06, and so on. All products are added together to finally generate a long-term and short-term dependency feature vector. This long-term and short-term dependency feature vector comprehensively and concisely summarizes the time dependency and behavioral characteristics of different stages in the tobacco processing process.

[0114] Step S232: construct a compliance assessment network, wherein the compliance assessment network includes a convolutional neural network and a rule matching layer, and is used to extract spatial local violation features in the historical device operation compliance feature set.

[0115] For example, in a possible implementation, step S232 includes: Step S2321: input the historical equipment operation compliance feature set into a convolutional neural network for multi-scale convolution kernel sliding processing to generate a local spatial feature map set.

[0116] The historical equipment operation compliance feature set of the tobacco station contains information such as equipment operation frequency features, operation interval abnormal features, and instruction sequence conflict features. The convolutional neural network uses convolution kernels of different sizes, such as 3×3 and 5×5 convolution kernels, to perform sliding operations on the equipment operation compliance feature data. Taking the equipment operation frequency feature as an example, assume that the feature data is represented in matrix form, each row represents a different equipment operation record, and each column represents a different operation frequency-related indicator. The 3×3 convolution kernel starts from the upper left corner of the matrix and moves one unit each time to calculate the elements in the 3×3 area. For example, for an element in a 3×3 area, the convolution kernel performs weighted summation and other operations on the operation frequency data in the area according to its internal preset parameters to generate a new value, which reflects a certain feature of the equipment operation frequency in the local area.

[0117] Step S2322, performing maximum pooling processing on the set of local spatial feature maps to generate a spatial invariant feature vector.

[0118] The set of local spatial feature maps captures the local information of device operation characteristics at different scales, such as the changing pattern of operation frequency in different time periods, the manifestation of abnormal operation intervals in local areas, etc.

[0119] For example, in a 2×2 local spatial feature map, there are four values ​​0.8, 0.6, 0.9, and 0.7. After the maximum pooling process, the maximum value 0.9 is selected. After all local spatial feature maps are processed in this way, they are combined to generate a spatial invariant feature vector. This vector retains the key information in the local spatial feature map and ignores some relatively unimportant details.

[0120] Step S2323: input the spatial invariance feature vector into the rule matching layer, perform feature matching processing with the compliance rules in the preset device operation rule library, and generate a rule matching degree score set.

[0121] The preset device operation rule library contains standards and specifications for various device operations, such as the upper and lower limits of the device operation frequency in a specific time period, the minimum time of the operation interval, etc. The rule matching layer compares the device operation features in the spatial invariance feature vector with these rules one by one. For example, for the operation frequency feature, the preset device operation rule library stipulates that the device operation frequency should be between 10 and 15 times in a certain time period, while the operation frequency reflected in the spatial invariance feature vector is 18 times in this time period. The rule matching layer generates a rule matching score by calculating the degree of difference between the two. Such matching calculations are performed for all rules and device operation features to generate a rule matching score set.

[0122] Step S2324: performing threshold filtering processing on the rule matching score set, extracting device operation features corresponding to matching scores exceeding the compliance threshold, and generating a spatial local violation feature set.

[0123] In this embodiment, a compliance threshold is set, for example, 0.7. The device operation features corresponding to the scores greater than 0.7 in the rule matching score set are extracted to generate a spatial local violation feature set. These features highlight the local information that may have violation risks during the device operation process.

[0124] Step S233, constructing an environment association network, wherein the environment association network includes a graph neural network and an anomaly propagation layer, and is used to extract environment association anomaly features from the historical environment anomaly fluctuation feature set.

[0125] For example, in a possible implementation, step S233 includes: Step S2331 , constructing an environmental graph structure according to the environmental parameter types in the historical environmental abnormal fluctuation feature set, wherein graph nodes represent different environmental parameter sensors, and graph edges represent spatial adjacency relationships between sensors.

[0126] For example, environmental parameters include temperature, humidity, dust concentration, etc. The graph nodes represent sensors that monitor these parameters. For example, node A represents the temperature sensor located in the northwest corner of the smoke station, and node B represents the humidity sensor at an adjacent location. The graph edges represent the spatial adjacency between sensors. If two sensors are close to each other, there is a connecting edge between them.

[0127] Step S2332: input the environment graph structure into the graph neural network for node feature aggregation processing to generate an environment-related feature vector for each node.

[0128] The graph neural network aggregates and calculates the features of each node based on the connection relationship between the nodes in the graph structure. Taking node A as an example, it can collect feature information of the nodes connected to it (such as node B). Assume that the temperature anomaly feature value of node A is 0.6 (indicating the degree to which the temperature deviates from the normal range), and the humidity anomaly feature value of node B is 0.4. The graph neural network uses a specific algorithm to comprehensively calculate the features of node A itself and the features of node B. For example, it may be a weighted average of the feature values ​​of the two (assuming that the weight of node A is 0.6 and the weight of node B is 0.4), that is, (0.6×0.6+0.4×0.4)=0.52, to generate the environment-related feature vector of node A. This process is performed on all nodes to generate the environment-related feature vector of each node.

[0129] Step S2333, performing anomaly propagation layer processing on the environment-related feature vector, and calculating anomaly propagation weight coefficients between adjacent nodes.

[0130] For example, the temperature anomaly of node A may affect the humidity of the adjacent node B. The anomaly propagation layer calculates the anomaly propagation weight coefficient between nodes A and B by analyzing the historical data of nodes A and B and the spatial relationship between them. Assume that after calculation, the anomaly propagation weight coefficient from node A to node B is 0.3, which means that the temperature anomaly of node A has a 0.3 probability of propagating to node B and affecting its humidity.

[0131] Step S2334: performing weighted update processing on the environment-related feature vector according to the anomaly propagation weight coefficient to generate a set of environment-related anomaly feature vectors.

[0132] For example, for the environment-related feature vector of node B, the environment-related feature vector of node B is adjusted according to the abnormal propagation weight coefficient 0.3 from node A to node B. Assuming that the original environment-related feature vector value of node B is 0.5, the updated vector value is 0.5+0.3×(environment-related feature vector value of node A-environment-related feature vector value of node B), that is, 0.5+0.3×(0.52-0.5)=0.506. Such weighted update processing is performed on the environment-related feature vectors of all nodes to generate a set of environment-related abnormal feature vectors.

[0133] Step S234, constructing a dynamic weight allocation layer, wherein the dynamic weight allocation layer includes a cross-modal attention mechanism and a weight normalization layer, which is used to fuse the long-term and short-term dependency features, the spatial local violation features, and the environmental association abnormality features.

[0134] For example, in a possible implementation, step S234 includes: Step S2341, input the long-term and short-term dependency feature vectors, the spatial local violation feature set and the environment-related abnormal feature vector set into the cross-modal attention mechanism to perform inter-modal similarity calculation processing to generate a cross-modal attention weight matrix.

[0135] The long-term and short-term dependency feature vectors reflect the temporal dependency of the tobacco leaf processing process, and the spatial local violation feature set reflects the local violations of the equipment operation. The cross-modal attention mechanism measures the similarity by calculating the inner product between them. Assuming that the long-term and short-term dependency feature vector is vector a, the spatial local violation feature set can be regarded as a set of multiple feature vectors, and one of the representative vectors b is taken. Calculate the inner product of vector a and vector b, multiply each element of vector a by the corresponding element of vector b, and then add all the products. Assuming that the elements of vector a are [0.2, 0.3, 0.4] and the elements of vector b are [0.1, 0.5, 0.2], the inner product is (0.2×0.1+0.3×0.5+0.4×0.2)=0.25. Through such calculations, a cross-modal attention weight matrix is ​​generated, and each element in the matrix represents the degree of correlation between different modal features.

[0136] Step S2342: input the cross-modal attention weight matrix into the weight normalization layer for row-wise normalization to generate a modality weight allocation matrix.

[0137] For example, a row of the cross-modal attention weight matrix has elements [0.2, 0.3, 0.5], and the sum of the elements in this row is 0.2 + 0.3 + 0.5 = 1. The elements in this row are normalized, and the first element becomes 0.2 ÷ 1 = 0.2, the second element becomes 0.3 ÷ 1 = 0.3, and the third element becomes 0.5 ÷ 1 = 0.5, generating a modality weight distribution matrix.

[0138] Step S2343, performing linear weighting processing on the long-term and short-term dependency feature vectors, the spatial local violation feature set and the environment-related abnormal feature vector set according to the modal weight allocation matrix to generate a cross-modal fusion feature vector.

[0139] For example, suppose that the weights of the long-term and short-term dependency feature vectors, the spatial local violation feature set, and the environment-related abnormal feature vector set in the modal weight distribution matrix are 0.4, 0.3, and 0.3, respectively. The long-term and short-term dependency feature vector is vector x, the comprehensive vector of the spatial local violation feature set is vector y, and the comprehensive vector of the environment-related abnormal feature vector set is vector z. Then the cross-modal fusion feature vector is 0.4×vector x+0.3×vector y+0.3×vector z. Through such weighted calculation, the cross-modal fusion feature vector is generated.

[0140] Step S235, constructing a violation classification network, wherein the violation classification network includes a fully connected layer and a probability output layer, and is used to generate the probability distribution of the violation operation.

[0141] For example, in a possible implementation, step S235 includes: Step S2351: input the cross-modal fusion feature vector into a fully connected layer for nonlinear transformation processing to generate a high-dimensional violation feature vector.

[0142] For example, suppose the cross-modal fusion feature vector is a vector containing 10 elements, and the fully connected layer has a 10×20 weight matrix and a bias vector of length 20. Each element of the cross-modal fusion feature vector is multiplied by the corresponding column element of the weight matrix, and then all products are added together, and the corresponding element of the bias vector is added to obtain a new 20-dimensional vector. The new vector is processed by a nonlinear function (such as the ReLU function) to convert negative values ​​to 0, and the vector is further transformed to generate a high-dimensional violation feature vector.

[0143] Step S2352: Perform probability output layer processing on the high-dimensional violation feature vector to generate a violation operation category confidence vector.

[0144] For example, there are three types of illegal operations: equipment illegal operations, environmental out-of-control operations, and tobacco leaf processing illegal operations. Based on the characteristics of the high-dimensional illegal feature vector, the probability output layer calculates that the confidence of equipment illegal operations is 0.3, the confidence of environmental out-of-control operations is 0.4, and the confidence of tobacco leaf processing illegal operations is 0.3, and generates the confidence vector of the illegal operation category.

[0145] Step S2353: input the illegal operation category confidence vector into the probability normalization layer for probability distribution conversion processing to generate the illegal operation probability distribution.

[0146] For example, the confidence vector of the illegal operation category is [0.3, 0.4, 0.3], and the sum of the vector elements is 0.3+0.4+0.3=1. The probability of equipment illegal operation is 0.3÷1=0.3, the probability of environmental out-of-control operation is 0.4÷1=0.4, and the probability of tobacco leaf processing illegal operation is 0.3÷1=0.3, generating a probability distribution of illegal operation. Through such a construction process, the initial multimodal violation recognition model is completed.

[0147] Figure 2 A schematic diagram of exemplary hardware and software components of an artificial intelligence-based cigarette station illegal operation warning system 100 that can implement the concept of the present application is shown in some embodiments of the present application. For example, the processor 120 can be used in the artificial intelligence-based cigarette station illegal operation warning system 100 and is used to perform the functions in the present application.

[0148] The artificial intelligence-based cigarette station illegal operation warning system 100 can be a general server or a special-purpose server, both of which can be used to implement the artificial intelligence-based cigarette station illegal operation warning method of the present application. Although only one server is shown in the present application, for convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0149] For example, the smoke station illegal operation early warning system 100 based on artificial intelligence may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the smoke station illegal operation early warning system 100 based on artificial intelligence may also include program instructions stored in ROM, RAM, or other types of non-temporary storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The smoke station illegal operation early warning system 100 based on artificial intelligence also includes an input / output (I / O) interface 150 between a computer and other input / output devices.

[0150] For ease of explanation, only one processor is described in the artificial intelligence-based early warning system 100 for illegal operation of a cigarette station. However, it should be noted that the artificial intelligence-based early warning system 100 for illegal operation of a cigarette station in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the artificial intelligence-based early warning system 100 for illegal operation of a cigarette station executes steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0151] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer executable instructions are preset. When the processor executes the computer executable instructions, the above-mentioned artificial intelligence-based cigarette station illegal operation warning method is implemented.

[0152] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, various features are sometimes combined into one embodiment, drawing or description thereof.

Claims

1. An artificial intelligence-based early warning method for illegal operation of a cigarette station, characterized in that: The method comprises: Acquire a multimodal monitoring data set of a target tobacco station, wherein the multimodal monitoring data set includes tobacco leaf quality monitoring data, equipment operation log data, and environmental parameter monitoring data; Performing temporal behavior feature extraction processing on the multimodal monitoring data set to obtain a temporal behavior trajectory sequence in the tobacco processing flow, an equipment operation compliance feature set, and an environmental abnormal fluctuation feature set; Calling a pre-trained multimodal violation recognition model, dynamically assigning weights to the temporal behavior trajectory sequence, the device operation compliance feature set, and the environmental abnormal fluctuation feature set to generate a fused violation feature vector; Determine the probability distribution of illegal operations of the target cigarette station based on the fused illegal feature vector, and generate a cigarette station illegal warning strategy according to the probability distribution of illegal operations; The cigarette station violation warning strategy is fed back to the cigarette station supervision terminal to trigger the illegal operation interception instruction.

2. The artificial intelligence-based early warning method for illegal operation of a cigarette station according to claim 1 is characterized in that: The multimodal monitoring data set is subjected to time series behavior feature extraction processing to obtain a time series behavior trajectory sequence in the tobacco processing flow, an equipment operation compliance feature set, and an environmental abnormal fluctuation feature set, including: The tobacco leaf quality monitoring data is processed by dividing the tobacco leaf quality monitoring data into different baking stages to generate a tobacco leaf baking curve and a corresponding temperature control stage sequence. Extracting the duration characteristics, temperature fluctuation variance characteristics and humidity associated offset characteristics of each stage in the temperature control stage sequence to obtain the time series behavior trajectory sequence; Performing operation instruction parsing processing on the device operation log data, extracting the device power on / off time interval characteristics, operation instruction compliance labels and operation frequency abnormality index characteristics, and obtaining the device operation compliance feature set; The environmental parameter monitoring data is processed by sliding window mean, and the temperature extreme difference characteristics, humidity change slope characteristics and dust concentration accumulation characteristics in the window are extracted to obtain the environmental abnormal fluctuation feature set.

3. The artificial intelligence-based early warning method for illegal operation of a cigarette station according to claim 1 is characterized in that: The calling of the pre-trained multimodal violation recognition model, performing dynamic weight allocation processing on the temporal behavior trajectory sequence, the device operation compliance feature set, and the environmental abnormal fluctuation feature set, and generating a fused violation feature vector, includes: Inputting the temporal behavior trajectory sequence into the temporal encoding network in the multimodal violation recognition model to generate a tobacco leaf processing behavior encoding vector; Inputting the equipment operation compliance feature set into the compliance assessment network in the multimodal violation identification model to generate an equipment operation risk coding vector; Inputting the environmental abnormal fluctuation feature set into the environmental association network in the multimodal violation recognition model to generate an environmental abnormality encoding vector; Calling the dynamic weight allocation layer in the multimodal violation identification model to generate a set of modal weight allocation coefficients based on the cross-modal correlation scores between the tobacco leaf processing behavior coding vector, the equipment operation risk coding vector, and the environmental anomaly coding vector; The tobacco leaf processing behavior coding vector, the equipment operation risk coding vector and the environmental anomaly coding vector are weightedly fused according to the modal weight allocation coefficient set to generate the fused violation feature vector.

4. The artificial intelligence-based early warning method for illegal operation of a cigarette station according to claim 3 is characterized in that: The calling of the dynamic weight allocation layer in the multimodal violation identification model generates a set of modal weight allocation coefficients based on the cross-modal correlation scores between the tobacco leaf processing behavior coding vector, the equipment operation risk coding vector and the environmental anomaly coding vector, including: Performing inner product processing on the tobacco leaf processing behavior coding vector and the equipment operation risk coding vector to generate a first cross-modal correlation score; Performing inner product processing on the tobacco leaf processing behavior encoding vector and the environmental anomaly encoding vector to generate a second cross-modal correlation score; Performing inner product processing on the equipment operation risk coding vector and the environment anomaly coding vector to generate a third cross-modal correlation score; The first cross-modal correlation score, the second cross-modal correlation score, and the third cross-modal correlation score are input into a softmax function for normalization processing to generate the modality weight allocation coefficient set.

5. The artificial intelligence-based early warning method for illegal operation of a cigarette station according to claim 3 is characterized in that: The determining the probability distribution of illegal operations of the target cigarette station based on the fused illegal feature vector includes: Calling the violation classification network in the multimodal violation recognition model, performing full connection mapping processing on the fused violation feature vector, and generating a violation operation category confidence vector; Inputting the illegal operation category confidence vector into a probability normalization layer to generate the illegal operation probability distribution; Among them, the probability distribution of illegal operations includes the probability of equipment illegal operations, the probability of environmental out-of-control operations and the probability of illegal operations in tobacco leaf processing.

6. The artificial intelligence-based early warning method for illegal operation of a cigarette station according to claim 5 is characterized in that: The generating of a smoke station violation warning strategy according to the probability distribution of the violation operation includes: When the probability of illegal operation of the device exceeds a first probability threshold, generating a device operation mandatory verification instruction and a device locking strategy; When the probability of the environmental out-of-control operation exceeds a second probability threshold, generating an environmental parameter adjustment instruction and a ventilation equipment startup strategy; When the probability of illegal operation in tobacco leaf processing exceeds a third probability threshold, a baking process suspension instruction and a tobacco leaf quality re-inspection strategy are generated; The equipment operation mandatory verification instruction, the environmental parameter adjustment instruction and the baking process pause instruction are combined to generate the smoke station violation warning strategy.

7. The artificial intelligence-based early warning method for illegal operation of a cigarette station according to claim 6 is characterized in that: Feedback of the cigarette station violation warning strategy to the cigarette station supervision terminal to trigger the illegal operation interception instruction includes: Encapsulating the device locking strategy into a first control instruction set, and sending it to the baking device controller of the target tobacco station through the device control interface; Encapsulating the ventilation equipment startup strategy into a second control instruction set, and sending it to the ventilation equipment controller of the target smoke station through the environmental control interface; Encapsulating the tobacco leaf quality re-inspection strategy into a third control instruction set, and sending it to the tobacco leaf quality inspection terminal of the target tobacco station through the quality inspection terminal interface; When the first control instruction set, the second control instruction set and the third control instruction set are all successfully executed, a violation operation interception completion signal is generated and the violation operation probability distribution is updated.

8. The artificial intelligence-based early warning method for illegal operation of a cigarette station according to claim 1 is characterized in that: The training method of the multimodal violation recognition model includes: Obtain a historical cigarette station illegal operation data set, wherein the historical cigarette station illegal operation data set includes a historical multimodal monitoring data set and corresponding illegal operation labels; Performing time series behavior feature extraction processing on the historical multimodal monitoring data set to obtain a historical time series behavior trajectory sequence, a historical equipment operation compliance feature set, and a historical environmental abnormal fluctuation feature set; Constructing an initial multimodal violation recognition model, inputting the historical time-series behavior trajectory sequence, the historical equipment operation compliance feature set, and the historical environment abnormal fluctuation feature set into the initial multimodal violation recognition model to generate a historical fusion violation feature vector; Based on the cross entropy loss between the historical fusion violation feature vector and the violation operation label, the initial multimodal violation recognition model is trained until convergence.

9. The artificial intelligence-based early warning method for illegal operation of a cigarette station according to claim 8 is characterized in that: The time series behavior feature extraction process is performed on the historical multimodal monitoring data set to obtain a historical time series behavior trajectory sequence, a historical equipment operation compliance feature set and a historical environment abnormal fluctuation feature set, including: Performing abnormal baking stage labeling processing on the tobacco leaf quality monitoring data in the historical multimodal monitoring data set to generate a labeled tobacco leaf baking stage sequence; Extracting the temperature deviation characteristics, humidity stability index characteristics and baking time abnormality coefficient characteristics of each stage in the labeled tobacco leaf baking stage sequence to obtain the historical time series behavior trajectory sequence; Performing operation instruction compliance labeling processing on the equipment operation log data in the historical multimodal monitoring data set, generating labeled equipment operation frequency features, operation interval abnormality features and instruction sequence conflict features, and obtaining the historical equipment operation compliance feature set; The environmental parameter monitoring data in the historical multimodal monitoring data set is subjected to time window anomaly labeling processing to generate environmental temperature mutation characteristics, humidity accumulation anomaly characteristics and dust concentration exceeding standard characteristics with timestamps, and obtain the historical environmental abnormal fluctuation feature set.

10. An artificial intelligence-based warning system for illegal operation of a cigarette station, characterized in that: The artificial intelligence-based smoke station illegal operation warning system includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the artificial intelligence-based smoke station illegal operation warning method described in any one of claims 1 to 9 above.

Images