Target detection method driven by target time sequence causal relationship in multiple industrial scenes

Abstract

The invention provides a target detection method driven by a target time sequence causal relationship under multiple industrial scenes, and aims to solve the problem of multi-scene target detection of real flexible circuit board production. According to the method, in order to solve the problems that a target detection model is high in multi-scene target detection time overhead and weak in multi-scene understanding ability, target time sequence causal cascade Mama is provided, and the target time sequence causal cascade Mama processes features of a Backbone layer and a Neck layer of a yolov9 model to solve parameter values in a multi-scene target time sequence causal quantitative representation graph. The method further provides a relation attention module, and the accuracy of relation feature conversion is enhanced. The target detection method driven by the target time sequence causal relationship under the multiple scenes has the beneficial effects that the production time sequence of the target in the same scene and the causal of the target in different scenes can be modeled, the logic relevance between industrial production scenes is understood, and the real-time performance and the accuracy of the model are further improved.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and deep learning, and specifically relates to a target detection method driven by target temporal causal relationships in multiple industrial scenarios. Background Art

[0002] With the continuous development of artificial intelligence (AI), object detection technology has been widely applied in various fields, such as industrial production, traffic management, medical diagnosis, and national defense security. In industrial production, object detection technology is primarily used in industrial safety monitoring, production line inspection, and quality control scenarios. In safety monitoring scenarios, object detection technology is used to detect objects such as people and their clothing (safety helmets, masks, and work clothes), determine whether there are any anomalies, and promptly issue alerts and take measures to prevent accidents and damage. In production line inspection scenarios, object detection technology is used to detect objects such as equipment, products, and materials on the production line, enabling perception and control of the production line. In quality control scenarios, object detection technology is used to detect defective products and product defects for quality control and management. Therefore, the application of object detection technology can detect objects in the production process, enabling autonomous perception of the production process, ensuring production safety, improving efficiency and quality, and reducing production costs. Currently, industrial object detection involves combining object samples from a single scene into a dataset, then training a model for each dataset to solve the single-scene object detection task. Single-scene object detection focuses on analyzing and detecting object features within a single scene, and therefore may achieve better detection results for objects within that scene. However, single-scene object detection faces the following challenges when dealing with multi-scene object detection tasks: (1) Multi-scenario target detection has a large time overhead. In order to ensure that the target detection task meets the continuity requirements of production, it is necessary to frequently call the target detection models of different task scenarios. This increases the switching frequency between single-scenario target detection models, brings additional time overhead, and affects the real-time performance of production.

[0003] (2) The ability to understand multiple scenarios is weak. Since each single-scenario target detection model is relatively independent, the model ignores the relationship between scene targets. This not only breaks the logical correlation between production scenarios, but also reduces the in-depth cognition and understanding of industrial scenarios.

[0004] In summary, to address the above difficulties, a target detection method driven by target temporal causality in multiple industrial scenarios is proposed. Summary of the Invention

[0005] In light of the above issues, the present invention aims to provide a target detection method driven by temporal causality in industrial multi-scenarios. This method improves the Yolov9 target detection model to extract (capture, model, and apply) the production temporality of targets within the same scenario and the causality of targets across different scenarios, thereby enhancing target detection reliability and understanding of the scenarios.

[0006] A target detection method driven by target temporal causal relationships in multiple industrial scenarios includes the following steps: S1. Constructing an industrial multi-scenario target detection dataset: We obtain images of three production scenarios for flexible printed circuit (FPC) production, including personnel clothing, production line equipment, and product defects. We manually annotate each RGB image to obtain annotation information, which includes target location and target category. Targets include personnel clothing (correct clothing, incorrect clothing), production line equipment (cutting machine, drilling machine, copper sinking wire, copper plating wire, laminating machine, laminating machine, industrial oven, punching machine, exposure machine, printing machine, optical automatic inspection machine), and product defects (common bruises, common contamination, CVL foreign matter, common scratches, common wrinkles, ink shedding, sub-film oxidation, plating leakage, gold surface roughness, ink obligation, and ink contamination). The annotated images are combined to form an industrial multi-scenario target detection dataset, which also implicitly captures target temporality and target causality. S2. Target temporal causal relationship representation: including causal relationship representation, temporal relationship representation and temporal causal relationship representation; Causal relationship representation, that is, to find all safety monitoring scenario targets and production line inspection scenario targets related to quality inspection scenario targets in a flexible manufacturing environment and screen them. Based on industrial mechanism knowledge, find out the possible causal relationship types between all safety monitoring scenario targets, production line inspection scenario targets and quality inspection scenario targets, and express the causal influence of the cause variable on the result variable after removing the confounding factors; temporal relationship representation, first find out the safety monitoring scenarios and production line inspection scenarios with process timing, and construct a temporal relationship representation that satisfies symmetry and transitivity between the targets in the same scenario; temporal causal relationship representation, jointly represent the constructed causal relationship representation and the temporal relationship representation, and obtain the target temporal causal structure diagram based on the joint representation; S3. Quantitative Representation of Temporal Causality: Based on the target temporal causal structure diagram and after determining the confounding factors, we can obtain the target feature probability distribution of the three scenarios at any moment in the temporal causal diagram. This allows us to quantify the temporal causal relationships between the targets and thus establish a quantitative representation diagram of the temporal causal relationships. S4. Quantitative learning of target temporal causal relationships: Construct a target temporal causal cascade Mamba to solve the parameters in the temporal causal graph, and use the relational attention module to optimize the relational feature conversion; S5. Perform industrial multi-scene target detection: Integrate the target temporal causal cascade Mamba into the yolov9 target detection model to obtain the final detection model, input the multi-scene image into the final model, and generate the final industrial multi-scene target detection prediction results.

[0007] The industrial multi-scenario target detection dataset constructed in S1 above comes from three production scenarios of real flexible circuit boards. The target temporality comes from the process temporality during product production (corresponding to the temporality of personnel clothing requirements and the temporality of equipment required for production), and the target causality comes from the fact that the clothing conditions of personnel and the current production equipment under specific processes will produce corresponding product defects.

[0008] In S2 above, it is known that the safety monitoring scenario target and the production line inspection scenario target as causal variables will have a causal impact on the quality inspection scenario target. At this time, there are five types of causal relationships. Using the nonlinear Granger causality prediction model to qualitatively identify the causal relationship, we get: Deterministic causal relationship security monitoring target class ( ) to the quality detection target class ( ), production line detection target class ( ) to the quality detection target class ( ) The predicted value of the quality detection target class at time t is expressed as: , , Where, 、 and They represent the target class of safety monitoring scene, the target class of production line inspection scene and the target class of quality inspection scene respectively. represents a nonlinear function, , Respectively In a relationship and The lag order of , Respectively In a relationship and The lag order of and Represent the error terms of two nonlinear functions respectively.

[0009] Uncertain causality or , and their predicted values ​​are expressed as:

[0010] or , The definition of parameters is similar to the prediction method of deterministic causal relationship.

[0011] The above prediction values ​​were tested using non-parametric test methods to determine whether the cause variable made a significant contribution to the prediction of the result variable.

[0012] At the same time, transfer entropy is used to qualitatively identify the causal relationship type and calculate transfer entropy TE to quantify the information contribution of the cause variable to the result variable: , Where, represents the cause variable, represents the outcome variable, represents the joint probability density function, and represents the conditional probability density function, and Represent the history length respectively, that is, in the calculation How far back in history should we look to determine the future state? represents the significance level parameter. If , it indicates that there is Nonlinear causal effects.

[0013] By comprehensively discriminating the causal relationship type based on the nonlinear Granger causality test and information entropy judgment results, the causal influence of the cause variable on the outcome variable after removing the confounding factors is expressed as: , , , , Where, After removing the influence of confounding factors , After removing the influence of confounding factors , After removing the influence of confounding factors , After removing the influence of confounding factors , Represent safety goals, production line goals and quality goal instances respectively, and Indicates possible confounding factors, do(A=a) means forcing the external variable A to be assigned a. and They represent transfer entropy and The weight of the causal influence after conversion to the do operator.

[0014] The temporal relationship between the security monitoring scenario and the production line inspection scenario is expressed as follows: .

[0015] The constructed temporal relationship satisfies the symmetry and transitivity constraints and is expressed as: , Where, , Represents the distribution characteristics of the target temporal relationship set (such as mean, variance), Represents the target temporal relationship transfer characteristics.

[0016] Then, the target causal relationship and the target temporal relationship are jointly represented, and the constructed temporal causal relationship model is:

[0017] Where, , Represent the temporal and causal search spaces of the target, respectively, and depend on the optimal temporal relationship and optimal causality , and Represents the target temporal relationship pair and the target causal relationship pair, and They represent the search space of causal and temporal relationship pairs and the search space of causal and causal relationship pairs respectively, represents the target pair space, and They are the scoring function for the relationship between targets and the scoring function for the target search space.

[0018] , is defined as follows: , Where, and They represent the temporal relationship and causal relationship between targets respectively. The above formula shows that the search space of the target depends on the corresponding relationship.

[0019] and is defined as follows: , , Where, express The eigenvector of the pair.

[0020] By solving the above formula, we can get the optimal timing relationship and optimal causality , and Target search space and obtain the temporal causal structure diagram.

[0021] The characteristic vectors of the targets in the three scenarios observed at time t in S3 above are , the target characteristic state value observed at time t=1 is recorded as the initial state distribution , then in the same batch of data streams In the t+1th moment relative to the initial moment, the target state distributions of the safety monitoring scene target and the production line inspection scene target are , ,in, Represents the feature vector of the security monitoring scene target at time t, Represents the temporal relationship between the targets at time t and time t+1 in the security monitoring scenario, Represents the feature vector of the production line detection scene target at time t, Represents the temporal relationship between the target at time t and the target at time t+1 in the production line inspection scene.

[0022] Once the confounding factors are determined, the target feature probability distribution representation of the three target data sets at time t in the time series causal structure diagram can be obtained based on the causal relationship expression and the do operator: , Where, represents the fusion operation of the target feature, t and t-1 represent the t moment and t-1 moment of the scene respectively, Represents the causal relationship between scenes, Represents the target class features of the corresponding scene, Represents the probability of the target class feature of the corresponding scene.

[0023] According to the temporal causal structure diagram and the above formula, the temporal causal relationship between targets can be quantitatively represented, thereby establishing a temporal causal relationship quantitative representation diagram.

[0024] In the above S4, the target temporal causal cascade Mamba is constructed to solve the above formula 、 、 、 、 、 The values ​​of the 6 parameters.

[0025] The target temporal causal cascade Mamba receives the features of the image after passing through the Backbone layer and Neck layer of the yolov9 model. The specific operation steps are high-confidence target screening operation, temporal scene relationship feature conversion operation, and causal scene relationship feature conversion operation: High confidence target screening operation: Perform Top-K screening on the features after passing through the Backbone layer and Neck layer of the yolov9 model, and output the top k high confidence target features: , in, is the input feature For each position (h,w) feature vector: , It means taking the first k values ​​of the maximum value.

[0026] Temporal scene relationship feature conversion operation: When predicting the safety monitoring scene target at time t+1, the safety monitoring scene target at time t and the production line inspection scene target at time t+1 are spliced ​​after the high confidence target screening operation. Pass it into the projection layer and output it , , Three matrices, simultaneously Time-invariant state matrix and time-varying matrix Discretize to get and , and then Further with input Multiply, and original state Multiply and add the first two terms to get the current state , and finally the current state With the matrix Multiply as input to get the final temporal relationship feature conversion result The same is true for predicting the production line inspection scene target at time t+1. The formula is as follows:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036] in, , They represent the features of the security monitoring scene and the production line inspection scene after the high-confidence target screening operation at time t, Represents the splicing operation of features, represents the linear projection operation, and Indicates use Using zero-order hold technique and Discretize, represents matrix multiplication, represents matrix addition, and They represent the temporal relationship feature conversion results of the security monitoring scene target and the production line inspection scene target respectively.

[0037] Causal scene relationship feature conversion operation: When predicting the quality inspection scene target at time t, the features of the safety monitoring scene target and the production line inspection scene target at time t are respectively passed into the two projection layers after the high confidence target screening operation, and the output is obtained , , , , , Six matrices, simultaneously and For the time-invariant state matrix A, the time-varying matrix and Discretize to get 、 and , and then and Further separation and input and Multiply, and original state Multiply and add the first three items to get the current state , and finally the current state Respectively with the matrix and Multiply as input to get the final causal feature conversion result and , the formula is as follows:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] in, , They represent the features of the security monitoring scene and the production line inspection scene after the high-confidence target screening operation at time t, represents the linear projection operation, and Respectively indicate the use and Using zero-order hold technique Discretize, and Respectively indicate the use Using zero-order hold technique Discretize and use Using zero-order hold technique Discretize, represents matrix multiplication, represents matrix addition, and They respectively represent the causal feature conversion results of the safety monitoring scenario target and the production line inspection scenario target to the quality inspection scenario target.

[0044] Relational attention module: The obtained security monitoring scene target temporal relationship feature conversion results and production line inspection scene target temporal relationship feature conversion results at time t 、 , the causal relationship feature conversion results of safety monitoring scene target and production line inspection scene target to quality inspection scene target 、 The input is fed into the relational attention module, and the attention weight is output, so that the model can pay more attention to the correct relational expression. The relational conversion result is then multiplied by the attention weight and added to the feature to obtain the relation-enhanced feature.

[0045] The above computational relation attention module first uses the Gaussian kernel to calculate the high confidence target features at time t With the previous The similarity of the high-confidence target feature mean at each moment is then multiplied by the activation function and the result of the relational feature transformation to obtain the result after attention. The formula is as follows:

[0046]

[0047]

[0048] in, represents average pooling, represents the bandwidth parameter of the Gaussian kernel, represents the silu activation function, The weight index representing the relation attention.

[0049] Then, the obtained weight index is multiplied by the corresponding relationship conversion result and added to the feature to obtain the final prediction feature. The operation in the time series scenario is as follows:

[0050]

[0051] in, and They represent the features of the security monitoring scene image and the production line inspection scene image at time t+1 after the high confidence target screening operation, represents matrix addition, represents matrix dot product, and They represent the relational attention weights of the safety monitoring scene relational features and the production line inspection scene relational features at time t respectively.

[0052] The operation of the causal scenario is as follows:

[0053]

[0054] in, Represents the features of the quality detection scene image at time t after the high confidence target screening operation, represents a multilayer perceptron, represents matrix addition, represents matrix dot product, and They represent the attention weights of the relationship from the safety monitoring scene target to the quality inspection scene target and from the production line inspection scene target to the quality inspection scene target at time t respectively.

[0055] In the training process of the target temporal causal cascade Mamba, the mean square error loss is adopted, and the formula is as follows:

[0056] Among them, m represents the total training time, 、 、 Represent the prediction features at time t, 、 、 They represent the high confidence target features obtained from the true labels.

[0057] Finally, the obtained prediction features are predicted to obtain the final prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is the type of causal relationship that exists between the three scenario goals.

[0059] Figure 2 It is a temporal causal structure diagram.

[0060] Figure 3 It is a quantitative representation diagram of temporal causal relationships.

[0061] Figure 4 This is the target timing causal cascade Mamba framework diagram.

[0062] Figure 5 This is a schematic diagram of the yolov9 model.

[0063] Figure 6 It is a schematic diagram of the temporal scene relationship feature conversion operation.

[0064] Figure 7 It is a schematic diagram of the causal scene relationship feature conversion operation.

[0065] Figure 8 This is the effect diagram of the target detection method driven by the target temporal causal relationship in multiple industrial scenarios. DETAILED DESCRIPTION

[0066] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0067] A target detection method driven by target temporal causal relationships in multiple industrial scenarios includes the following steps: S1. Constructing an Industrial Multi-Scenario Object Detection Dataset: We acquired images from three production scenarios related to flexible printed circuit (FPC) production, including images of personnel attire, production line equipment, and product defects. We manually annotated each RGB image to obtain annotation information, including target location and target category. Targets included personnel attire (correct attire, incorrect attire), production line equipment (cutting machine, drilling machine, copper deposition wire, copper plating wire, laminating machine, laminating machine, industrial oven, punching machine, exposure machine, printing machine, and optical automatic inspection machine), and product defects (common bruises, common contamination, CVL foreign matter, common scratches, common wrinkles, ink detachment, sub-film oxidation, plating defects, gold surface roughness, ink impurities, and ink contamination). The annotated images were combined to form the Industrial Multi-Scenario Object Detection Dataset, with an image resolution of 640×640 pixels. The FPCIPMSD dataset contains 5979 images. Labelimg annotation software was used to annotate the FPC defect images. S2. Target temporal causal relationship representation: as shown in the attached Figure 1 As shown in the figure, under the premise that it is known that the safety monitoring scenario target and the production line inspection scenario target as causal variables will have a causal impact on the quality inspection scenario target, there are five types of causal relationships at this time. The nonlinear Granger causality prediction model is used to qualitatively identify the causal relationship, and the following results are obtained: Deterministic causal relationship security monitoring target class ( ) to the quality detection target class ( ), production line detection target class ( ) to the quality detection target class ( ) The predicted value of the quality detection target class at time t is expressed as: , , Where, 、 and They represent the target class of safety monitoring scene, the target class of production line inspection scene and the target class of quality inspection scene respectively. represents a nonlinear function, , Respectively In a relationship and The lag order of , Respectively In a relationship and The lag order of and Represent the error terms of two nonlinear functions respectively.

[0068] Uncertain causality or , and their predicted values ​​are expressed as:

[0069] or , The definition of parameters is similar to the prediction method of deterministic causal relationship.

[0070] The above prediction values ​​were tested using non-parametric test methods to determine whether the cause variable made a significant contribution to the prediction of the result variable.

[0071] At the same time, transfer entropy is used to qualitatively identify the causal relationship type and calculate transfer entropy TE to quantify the information contribution of the cause variable to the result variable:

[0072] Where, represents the cause variable, represents the outcome variable, represents the joint probability density function, and represents the conditional probability density function, and Represent the history length respectively, that is, in the calculation How far back in history should we look to determine the future state? represents the significance level parameter. If , it indicates that there is Nonlinear causal effects.

[0073] By comprehensively discriminating the causal relationship type based on the nonlinear Granger causality test and information entropy judgment results, the causal influence of the cause variable on the outcome variable after removing the confounding factors is expressed as: , , , , Where, After removing the influence of confounding factors , After removing the influence of confounding factors , After removing the influence of confounding factors , After removing the influence of confounding factors , Represent safety goals, production line goals and quality goal instances respectively, and Indicates possible confounding factors, do(A=a) means forcing the external variable A to be assigned a value. and They represent transfer entropy and The weight of causal influence after conversion to the do operator, and They represent transfer entropy and The weight of the causal influence after conversion to the do operator.

[0074] The temporal relationship between the security monitoring scenario and the production line inspection scenario is expressed as follows: .

[0075] The constructed temporal relationship satisfies the symmetry and transitivity constraints and is expressed as: , Where, , Represents the distribution characteristics of the target temporal relationship set (such as mean, variance), Represents the target temporal relationship transfer characteristics.

[0076] Then, the target causal relationship and the target temporal relationship are jointly represented, and the constructed temporal causal relationship model is: , Where, , Represent the temporal and causal search spaces of the target, respectively, and depend on the optimal temporal relationship and optimal causality , and Represents the target temporal relationship pair and the target causal relationship pair, and They represent the search space of causal and temporal relationship pairs and the search space of causal and causal relationship pairs respectively, represents the target pair space, and They are the scoring function for the relationship between targets and the scoring function for the target search space.

[0077] , is defined as follows:

[0078] Where, and They represent the temporal relationship and causal relationship between targets respectively. The above formula shows that the search space of the target depends on the corresponding relationship.

[0079] and is defined as follows: , , Where, express The eigenvector of the pair.

[0080] By solving the above formula, we can get the optimal timing relationship and optimal causality , and Target search space, and obtain the time sequence causal structure diagram, as shown in the attached Figure 2 shown.

[0081] S3. Quantitative Representation of Temporal Causality: Based on the target temporal causal structure diagram and after determining the confounding factors, we can obtain the target feature probability distribution of the three scenarios at any moment in the temporal causal diagram. This allows us to quantify the temporal causal relationships between the targets and thus establish a quantitative representation diagram of the temporal causal relationships. The characteristic vectors of the targets in the three scenarios observed at time t in S3 above are , the target characteristic state value observed at time t=1 is recorded as the initial state distribution , then in the same batch of data streams In the t+1th moment relative to the initial moment, the target state distributions of the safety monitoring scene target and the production line inspection scene target are , ,in, Represents the feature vector of the security monitoring scene target at time t, Represents the temporal relationship between the targets at time t and time t+1 in the security monitoring scenario, Represents the feature vector of the production line detection scene target at time t, Represents the temporal relationship between the target at time t and the target at time t+1 in the production line inspection scene.

[0082] Once the confounding factors are determined, the target feature probability distribution representation of the three target data sets at time t in the time series causal structure diagram can be obtained based on the causal relationship expression and the do operator:

[0083] Where, represents the fusion operation of the target feature, t and t-1 represent the t moment and t-1 moment of the scene respectively, Represents the causal relationship between scenes, Represents the target class features of the corresponding scene, Represents the probability of the target class feature of the corresponding scene.

[0084] According to the temporal causal structure diagram and the above formula, the temporal causal relationship between targets can be quantitatively represented, thereby establishing a temporal causal relationship quantitative representation diagram, as shown in the attached figure. Figure 3 shown.

[0085] S4. Quantitative learning of target temporal causal relationships: as shown in the attached Figure 4 As shown, the target temporal causal cascade Mamba is constructed to solve the parameters in the temporal causal graph, and the relational attention module is used to optimize the relational feature conversion; Target temporal causal cascade Mamba receives the features of the image after passing through the Backbone layer and Neck layer of the yolov9 model. The network structure of the yolov9 target detection model is as shown in the attached figure. Figure 5 As shown, the specific operation steps are high-confidence target screening operation, temporal scene relationship feature conversion operation, and causal scene relationship feature conversion operation: High confidence target screening operation: Perform Top-K screening on the features after passing through the Backbone layer and Neck layer of the yolov9 model, and output the top k high confidence target features:

[0086] in, is the input feature For each position (h,w) feature vector: , It means taking the first k values ​​of the maximum value.

[0087] Temporal scene relationship feature conversion operation: as shown in the attached Figure 6 As shown in the figure, when predicting the safety monitoring scene target at time t+1, the safety monitoring scene target at time t and the production line inspection scene target features at time t+1 are spliced ​​after the high confidence target screening operation. Pass it into the projection layer and output it , , Three matrices, simultaneously Time-invariant state matrix and time-varying matrix Discretize to get and , and then Further with input Multiply, and original state Multiply and add the first two terms to get the current state , and finally the current state With the matrix Multiply as input to get the final temporal relationship feature conversion result The same is true for predicting the production line inspection scene target at time t+1. The formula is as follows:

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097] in, , They represent the features of the security monitoring scene and the production line inspection scene after the high-confidence target screening operation at time t, Represents the splicing operation of features, represents the linear projection operation, and Indicates use Using zero-order hold technique and Discretize, represents matrix multiplication, Represents matrix addition.

[0098] Causal scene relationship feature conversion operation: as shown in the attached Figure 7 As shown in the figure, when predicting the quality inspection scene target at time t, the features of the safety monitoring scene target and the production line inspection scene target at time t are respectively passed into the two projection layers after the high confidence target screening operation, and the output is , , , , , Six matrices, simultaneously and For the time-invariant state matrix A, the time-varying matrix and Discretize to get 、 and , and then and Further separation and input and Multiply, and original state Multiply and add the first three items to get the current state , and finally the current state Respectively with the matrix and Multiply as input to get the final causal feature conversion result and , the formula is as follows:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] in, , They represent the features of the security monitoring scene and the production line inspection scene after the high-confidence target screening operation at time t, represents the linear projection operation, and Respectively indicate the use and Using zero-order hold technique Discretize, and Respectively indicate the use Using zero-order hold technique Discretize and use Using zero-order hold technique Discretize, represents matrix multiplication, represents matrix addition, and They respectively represent the causal feature conversion results of the safety monitoring scenario target and the production line inspection scenario target to the quality inspection scenario target.

[0105] Relational attention module: The obtained security monitoring scene target temporal relationship feature conversion results and production line inspection scene target temporal relationship feature conversion results at time t 、 , the causal relationship feature conversion results of safety monitoring scene target and production line inspection scene target to quality inspection scene target 、 The input is fed into the relational attention module, and the attention weight is output, so that the model can pay more attention to the correct relational expression. The relational conversion result is then multiplied by the attention weight and added to the feature to obtain the relation-enhanced feature.

[0106] The above calculation relation attention operation first uses the Gaussian kernel to calculate the high confidence target feature at time t With the previous The similarity of the high-confidence target feature mean at each moment is then multiplied by the activation function and the result of the relational feature transformation to obtain the result after attention. The formula is as follows:

[0107]

[0108]

[0109] in, represents average pooling, represents the bandwidth parameter of the Gaussian kernel, represents the silu activation function, The weight index representing the relation attention.

[0110] Then, the obtained weight index is multiplied by the corresponding relationship conversion result and added to the feature to obtain the final prediction feature. The operation in the time series scenario is as follows:

[0111]

[0112] in, and They represent the features of the security monitoring scene image and the production line inspection scene image at time t+1 after the high confidence target screening operation, represents matrix addition, represents matrix dot product, and They represent the relational attention weights of the safety monitoring scene relational features and the production line inspection scene relational features at time t respectively.

[0113] The operation of the causal scenario is as follows:

[0114]

[0115] in, Represents the features of the quality detection scene image at time t after the high confidence target screening operation, represents a multilayer perceptron, represents matrix addition, represents matrix dot product, and They represent the attention weights of the relationship from the safety monitoring scene target to the quality inspection scene target and from the production line inspection scene target to the quality inspection scene target at time t respectively.

[0116] In the training process of the target temporal causal cascade Mamba, the mean square error loss is adopted, and the formula is as follows:

[0117] Among them, m represents the total training time, 、 、 Represent the prediction features at time t, 、 、 They represent the high confidence target features obtained from the true labels.

[0118] S5. Perform industrial multi-scene target detection: Integrate the target temporal causal cascade Mamba into the yolov9 target detection model to generate the final industrial multi-scene target detection prediction results. The average precision mean (MAP) is used as the evaluation indicator for defect detection. The MAP of various targets of the trained model on the industrial multi-scene target detection dataset is shown in Table 1. The prediction effect of the industrial multi-scene target detection dataset is visualized, as shown in the attached figure. Figure 8 shown.

[0119] Table 1: Test results of the trained target detection model on the industrial multi-scene target detection dataset

Claims

1. A target detection method driven by target temporal causal relationship in industrial multi-scenario, characterized in that: The following steps are involved: S1. Constructing an industrial multi-scenario target detection dataset: The industrial multi-scenario target detection dataset includes images of personnel wear, production line equipment, and product defects in three production scenarios of flexible circuit board production; S2. Target temporal causal relationship representation: The target temporal causal relationship representation process includes causal relationship representation, temporal relationship representation and temporal causal relationship representation, and obtains a target temporal causal structure diagram; S3. Quantitative representation of temporal causal relationship: Based on the target temporal causal structure diagram, after determining the confounding factor, the target feature probability distribution of the three scenarios at any time in the temporal causal diagram can be obtained, so that the temporal causal relationship between the targets can be quantified, and thus the temporal causal relationship quantitative representation diagram can be established; S4. Perform quantitative learning of target temporal causal relationships: construct a target temporal causal cascade Mamba module to solve the parameters in the temporal causal graph, and use the relational attention module to optimize the relational feature conversion; S5. Perform industrial multi-scenario target detection: Integrate the target temporal causal cascade Mamba into the yolov9 target detection model to obtain the final detection model, input the multi-scene image into the final model, and generate the final industrial multi-scenario target detection prediction results.

2. According to the target detection method driven by target temporal causal relationship in industrial multi-scenario according to claim 1, it is characterized in that: The causal relationship representation is to find all the safety monitoring scene targets and production line detection scene targets related to the quality inspection scene targets in the flexible manufacturing environment and screen them; find out the possible causal relationship types between all safety monitoring scene targets, production line detection scene targets and quality inspection scene targets based on industrial mechanism knowledge, and express the causal influence of the cause variable on the result variable after removing the confounding factors; the temporal relationship representation is to first find the safety monitoring scene and production line detection scene with process timing, and construct a temporal relationship representation that satisfies symmetry and transitivity between the targets in the same scene; The temporal causal relationship representation combines the constructed causal relationship representation and the temporal relationship representation, and obtains the target temporal causal structure diagram according to the combined representation.

3. According to the target detection method driven by target temporal causal relationship in industrial multi-scenario according to claim 1, it is characterized in that: The temporal causal relationship is quantitatively expressed. When the confounding factor is determined, the target feature probability distribution representation of the three target data sets at time t in the temporal causal structure diagram can be obtained according to the causal relationship expression and the do operator: ， In the formula, represents the fusion operation of the target features, t and t-1 represent the time t and time t-1 of the scene respectively, Indicates the causal relationship between scenes, , and They represent the security monitoring scene target class, the production line inspection scene target class and the quality inspection target class respectively. Represents the target class features of the corresponding scene, Represents the probability of the target class feature of the corresponding scene; According to the temporal causal structure diagram and the above formula, the temporal causal relationship between targets can be quantitatively represented, thereby establishing a temporal causal relationship quantitative representation diagram.

4. According to the target detection method driven by target temporal causal relationship in industrial multi-scenario according to claim 1, it is characterized in that: The specific operation steps of the target time series causal cascade Mamba module are high confidence target screening operation, time series scene relationship feature conversion operation, and causal scene relationship feature conversion operation: High confidence target screening operation: Perform Top-K screening on the features after passing through the Backbone layer and Neck layer of the yolov9 model, and output the top k high confidence target features: in, For input features For each feature vector at position (h,w): , It means taking the first k values ​​of the maximum value; Time series scene relationship feature conversion operation: When predicting the safety monitoring scene target at time t+1, the features of the safety monitoring scene target at time t and the production line inspection scene target at time t+1 are concatenated after the high confidence target screening operation. Pass it into the projection layer and output it , , Three matrices, at the same time Time-invariant state matrix and the time-varying matrix Discretize to get and , and then Further with input Multiply, and original state Multiply and add the first two items to get the current state , and finally the current state With the matrix Multiply as input to get the final temporal relationship feature conversion result , the same is true when predicting the production line inspection scene target at time t+1. The formula is as follows: in, and They represent the characteristics of the security monitoring scene and the production line inspection scene after the high-confidence target screening operation at the tth moment, represents the concatenation operation of features, represents a linear projection operation, and Indicates the use Using zero-order hold technique and To discretize, represents matrix multiplication, represents matrix addition, and They represent the temporal relationship feature conversion results of the safety monitoring scene target and the production line inspection scene target respectively; Causal scene relationship feature conversion operation: When predicting the quality inspection scene target at time t, the features of the safety monitoring scene target and the production line inspection scene target at time t are passed to the two projection layers after the high confidence target screening operation, and the output is obtained , , , , , Six matrices, simultaneously and For the time-invariant state matrix A, the time-varying matrix and Discretize to get , and , and then and Further separation and input and Multiply, and original state Multiply and add the first three items to get the current state , and finally the current state Respectively with the matrix and Multiply as input to get the final causal feature conversion result and , the formula is as follows: in, and They represent the characteristics of the security monitoring scene and the production line inspection scene after the high-confidence target screening operation at the tth moment, represents a linear projection operation, and Respectively indicate the use and Using zero-order hold technique To discretize, and Respectively indicate the use Using zero-order hold technique Discretize and use Using zero-order hold technique To discretize, represents matrix multiplication, represents matrix addition, and They respectively represent the causal feature conversion results of the safety monitoring scenario target and the production line inspection scenario target to the quality inspection scenario target.

5. According to the target detection method driven by target temporal causal relationship in industrial multi-scenario according to claim 1, it is characterized in that: The relational attention module converts the temporal relational feature conversion results of the safety monitoring scene target at time t and the temporal relational feature conversion results of the production line inspection scene target at time t. , , the causal relationship feature conversion results of the safety monitoring scene target and the production line inspection scene target to the quality inspection scene target , Input it into the relational attention module, output the attention weight, so that the model can pay more attention to the correct relational expression, and then multiply the relational conversion result by the attention weight and add it to the feature to obtain the relation-enhanced feature: The computational relation attention module first uses a Gaussian kernel to calculate the high-confidence target features at time t. With the previous The similarity of the high confidence target feature mean at each moment is then multiplied by the activation function and the result of the relational feature transformation to obtain the result after attention. The formula is as follows: in, represents average pooling, represents the bandwidth parameter of the Gaussian kernel, represents the silu activation function, The weight index representing the relational attention; Then, the obtained weight index is multiplied by the corresponding relationship conversion result and added to the feature to obtain the final prediction feature. The operation of the time series scenario is as follows: in, and They represent the features of the security monitoring scene image and the production line detection scene image at time t+1 after the high confidence target screening operation, represents matrix addition, represents matrix dot product, and They represent the relational attention weights of the safety monitoring scene relational features and the production line inspection scene relational features at time t respectively; The operation of the cause and effect scenario is as follows: in, Represents the features of the quality inspection scene image at time t after the high confidence target screening operation, represents a multi-layer perceptron, represents matrix addition, represents matrix dot product, and They respectively represent the relationship attention weights from the safety monitoring scene target to the quality inspection scene target and from the production line inspection scene target to the quality inspection scene target at time t.

6. The target detection method driven by target temporal causal relationship in industrial multi-scenario according to claim 1 is characterized in that: The target temporal causal cascade Mamba is trained using mean square error loss: The mean square error loss is calculated: Among them, m represents the total training time, , , They represent the prediction features at time t, , , They represent the high confidence target features obtained from the true labels respectively; Finally, the target temporal causal cascade Mamba is integrated into the yolov9 target detection model to obtain the final detection model.

Images