Assembly sequence anomaly detection method based on multi-level comparison video representation learning

Abstract

The invention relates to an assembly sequence anomaly detection method based on multi-level comparison video representation learning, which comprises three modules, namely a data processing module, a video representation learning module and an assembly sequence anomaly detection module, and is characterized in that the video representation learning module is designed to be a multi-level video comparison module and comprises a global comparison module and a frame-by-frame comparison module; the global comparison module compares the videos on the video level to learn the overall similarity relation between the videos, and the frame-by-frame comparison module aligns the videos on the frame level to learn step sequence information in the videos. And the assembly sequence abnormity detection module detects whether the actual assembly sequence accords with the standard assembly sequence or not by calculating the distance between the multi-level video representations generated by the model, and stops the abnormity in time when the abnormity is detected.

Description

Technical Field

[0001] The present invention relates to the field of industrial Internet, and is mainly an assembly sequence anomaly detection method based on multi-level contrast video representation learning. Background Art

[0002] Assembly is a crucial link in industrial production, which directly affects product quality and production efficiency. In product design, different components are interdependent and need to be assembled in a specific order. If the assembly order is wrong, the production process will not proceed smoothly, which will affect production efficiency. In order to ensure product quality and improve production efficiency, it is necessary to detect whether the actual assembly order conforms to the standard assembly order, and stop it in time when an abnormality is detected to reduce adverse effects. However, due to the complexity and diversity of the assembly process, manual detection methods are often difficult to fully meet the needs in production practice. Therefore, the product manufacturing industry urgently needs more advanced automated detection technology.

[0003] With the rapid development of deep learning, especially the progress of convolutional neural networks and visual transformers, computer vision technology is increasingly used in industrial production, which not only significantly reduces the labor costs of factories, but also realizes highly automated operations, greatly improving product quality and production efficiency. Computer vision technology provides strong support for the development of intelligent manufacturing and Industry 4.0, and promotes the development of industrial production towards a more intelligent, efficient and lean direction.

[0004] As an emerging computer vision technology, the sequential video representation learning method focuses on the temporal order and causal relationship of actions in the video, and can effectively capture the step sequence information in the assembly video, providing a new solution for detecting abnormal assembly sequences. However, existing sequential video representation learning methods have some problems, such as heavy reliance on step timestamp annotations or lack of comprehensive consideration of the overall situation and detailed information of the video. Summary of the invention

[0005] The present invention is an assembly sequence anomaly detection method based on multi-level contrast video representation learning, which includes three modules: a data processing module, a video representation learning module and an assembly sequence anomaly detection module. The video representation learning module designs a multi-level video contrast module, including a global contrast module and a frame-by-frame contrast module. The global contrast module compares videos at the video level to learn the overall similarity relationship between videos, and the frame-by-frame contrast module aligns videos at the frame level to learn the step sequence information in the video. The assembly sequence anomaly detection module detects whether the actual assembly sequence conforms to the standard assembly sequence by calculating the distance between the multi-level video representations generated by the model, and stops it in time when an anomaly is detected.

[0006] The present invention achieves the above-mentioned purpose by adopting the following technical solutions:

[0007] A method for detecting anomalies in assembly sequence based on multi-level contrastive video representation learning comprises a data processing module, a video representation learning module and an anomaly detection module for assembly sequence.

[0008] The data processing module processes the collected video data to provide high-quality training data for model building of the video representation learning module;

[0009] The video representation learning module trains the video data provided by the data processing module and stores parameter information in a database;

[0010] The assembly sequence anomaly detection module uses a high-definition pixel camera to collect real-time monitoring video during the actual assembly process, transmits it to the server for processing, and then returns the processing result to the assembly sequence anomaly detection module for detection.

[0011] The data processing module first collects video data from the assembly process through a camera, and then screens, annotates and formats the data; specifically, since the collected video data may contain irrelevant content, the data must first be screened to determine whether the data meets the requirements, and the data that meets the requirements is retained; then, the retained video data is sequentially annotated for subsequent model training; then, the video is processed into frames, and the height and width of each frame are adjusted to unify the data format; finally, high-quality data suitable for video representation learning model training is generated.

[0012] The video representation learning module includes two parts: a Transformer video encoder and a multi-level video comparison module. Specifically, first, a Transformer encoder is used to generate frame representations of the video; then, the MLP module is applied to aggregate these frame representations into a global representation of the video; next, a multi-level video comparison module is designed, which includes a global comparison module and a frame-by-frame comparison module; the global comparison module compares videos at the video level to learn the overall similarity relationship between videos; the frame-by-frame video comparison module aligns videos at the frame level to learn the step sequence information in the video; finally, the model is optimized and trained through the designed combined loss function to achieve a comprehensive and accurate representation of the assembly video.

[0013] The working logic of the Transformer video encoder is:

[0014] First, the input video is divided into N segments and a frame is randomly selected from each segment. Then, these sampled frames are input into the ResNet-50 network to extract visual features and obtain the feature sequence F. In order to retain the position information of the frame in the sequence, the position embedding E is added to the feature sequence F.pos .

[0015] Next, according to the following formula, the Transformer encoder is used to learn the frame representation of the video. The obtained frame representation contains not only the position information of the frame in the sequence, but also the correlation information between frames. Specifically, the feature sequence F with position embedding is passed to the Transformer encoder. In the encoder, the self-attention mechanism captures the correlation between different frames in the sequence, while the position encoding helps the model understand the relative position relationship between frames.

[0016] FR=TE(F+E pos )

[0017] Among them, TE represents the Transformer encoder, E pos represents the position embedding of the feature sequence F, and FR represents the frame representation of the input video.

[0018] Finally, an MLP module is used to aggregate all frame representations output by the Transformer encoder to obtain a global representation of the input video.

[0019] The working logic of the multi-level video comparison module is as follows:

[0020] 1) Global comparison module

[0021] First, the MLP module is used to obtain the global representation of each video in the video pair, denoted as GR 1 and GR 2 Then, the cosine similarity between video representations is calculated according to the following formula:

[0022]

[0023] Finally, the expected cosine similarity value Cos(GR 1 , G.R. 2 ) is close to 1, the global contrast loss L global The definition is as follows:

[0024] L coarse =1-Cos(GR 1 , G.R. 2 )

[0025] By minimizing the global contrast loss function, different videos with the same assembly order can be clustered more closely together in the feature space of the model;

[0026] 2) Frame-by-frame comparison module

[0027] First, the Transformer encoder is used to obtain all frame representations of each video in the video pair, which are denoted as and

[0028] Then, the matching cost of all frame pairs formed by the two sequences is calculated according to the following formula to obtain the matching cost matrix C:

[0029]

[0030] C i，j represents the matching cost of the i-th row and j-th column in the matching cost matrix C, that is, FR 1 The i-th frame and FR 2 The matching cost of the j-th frame;

[0031] Next, a feasible alignment between sequences is defined as a pairwise corresponding binary matrix M∈{0,1} N×N :

[0032]

[0033] M i，j Indicates whether the elements in the i-th row and j-th column of the matrix M match, that is, FR 1 The i-th frame and FR 2 The jth frame of is matched. If and If it matches, then M i，j =1, otherwise 0;

[0034] Finally, the sequence FR is found by the dynamic time warping algorithm 1 and FR 2 The optimal alignment M* between them is the feasible alignment M with the minimum total matching cost, and the optimal matching cost is used as the frame-by-frame comparison loss L frame :

[0035]

[0036] By minimizing the frame-by-frame contrast loss function, more accurate alignment between sequences can be achieved;

[0037] 3) Training loss function

[0038] In the training phase, video pairs with the same assembly order are used for training, denoted as P i =(V i1 , V i2 ), and classify each video to better learn the step order information in the video. Video classification loss L cls The definition is as follows:

[0039] L cls =CE(f(V i1 ), Y i1 )+CE(f(Vi2 ), Y i2 )

[0040] Among them, CE represents the cross entropy function, f represents the video representation learning model, V i1 and V i2 Represent the two videos in the video pair, Y i1 and Y i2 Respectively represent the video V i1 and V i2 The corresponding true sequence label;

[0041] In summary, as shown in the following formula, the model is trained using a combination of video classification loss, global contrast loss, and frame-by-frame contrast loss:

[0042] L=λ 1 L cls +λ 2 L global +λ 3 L frame

[0043] Among them, λ 1 represents the weight of video classification loss, λ 2 represents the weight of the global contrast loss, λ 3 Represents the weight of the frame-by-frame contrastive loss.

[0044] The assembly sequence anomaly detection module detects whether the actual assembly sequence conforms to the standard assembly sequence by calculating the distance between the multi-level video representations generated by the model, and stops it in time when an anomaly is detected.

[0045] First, a fixed-size video sliding window is set according to the actual situation. The sliding window is applied to the standard assembly video, and the video segment in the window is input into the video representation learning model to generate a multi-level representation of the segment and store it in the embedding vector list of the database for subsequent detection calls. The sliding window moves on the standard assembly video, and this process is repeated until the sliding ends.

[0046] Then, the same sliding window is applied to the assembly video to be tested, and the video segment in the window is input into the video representation learning model to generate a multi-level representation of the segment. At the same time, the multi-level representation of the corresponding standard assembly video segment in the database is retrieved.

[0047] After that, the distance D between the multi-level representations of the video segment to be detected and the standard video segment is calculated:

[0048] D = dist(FR 1 , F R 2 )+dist(GR 1 , G.R. 2 )

[0049] Among them, dist represents the Euclidean distance function, FR 1 and FR 2 Represents the frame representation of the video segment to be detected and the standard video segment, GR 1 and GR 2 Respectively represent the global representation of the two.

[0050] Next, define the detection result R according to the distance D:

[0051]

[0052] Wherein, τ is a threshold value set according to the actual situation. If D>τ, it means that the assembly sequence to be detected is inconsistent with the standard assembly sequence, that is, it is an abnormal assembly sequence.

[0053] Finally, the sliding window moves on the assembly video to be inspected and the above process is repeated. When an abnormality is detected, the sliding stops.

[0054] The present invention adopts the above technical solution to achieve the following beneficial effects:

[0055] The present invention can quickly and accurately detect abnormal assembly sequences in industrial assembly scenarios, thereby improving production efficiency and reducing product failure rates.

[0056] First, a global comparison module is introduced, which compares videos at a global level and learns the overall similarity relationship between videos, helping the model to cluster different videos of the same assembly order more closely together in the feature space, avoiding significant visual differences between different videos due to differences in background and personal preferences, and avoiding erroneous detection results caused by such differences.

[0057] Secondly, a frame-by-frame comparison module is introduced, which aligns videos at the frame level and learns the step sequence information in the video, helping the model to better understand and analyze the temporal order of steps in the video.

[0058] Finally, the distance between the multi-level video representations generated by the model is calculated to detect whether the actual assembly sequence conforms to the standard assembly sequence, and timely stop when an anomaly is detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is the overall architecture diagram of the present invention.

[0060] Figure 2 Schematic diagram of the video representation learning model. DETAILED DESCRIPTION

[0061] The present invention is further described below in conjunction with the accompanying drawings and examples. It should be clear that the following examples are only one or more manifestations of the present invention and are not a complete limitation of the invention described in this application. Any improvements known to those skilled in the art made on the basis of the invention or technical solutions described in this application should fall within the scope of protection claimed in this application.

[0062] A method for detecting anomalies in assembly sequence based on multi-level contrastive video representation learning comprises a data processing module, a video representation learning module and an anomaly detection module for assembly sequence.

[0063] The data processing module first collects video data from the assembly process through a camera, and then screens, annotates and formats the data; specifically, since the collected video data may contain irrelevant content, the data must first be screened to determine whether the data meets the requirements, and the data that meets the requirements is retained; then, the retained video data is sequentially annotated for subsequent model training; then, the video is processed into frames, and the height and width of each frame are adjusted to unify the data format; finally, high-quality data suitable for video representation learning model training is generated.

[0064] The video representation learning module includes two parts: a Transformer video encoder and a multi-level video comparison module. Specifically, first, a Transformer encoder is used to generate frame representations of the video; then, the MLP module is applied to aggregate these frame representations into a global representation of the video; next, a multi-level video comparison module is designed, which includes a global comparison module and a frame-by-frame comparison module; the global comparison module compares videos at the video level to learn the overall similarity relationship between videos; the frame-by-frame video comparison module aligns videos at the frame level to learn the step sequence information in the video; finally, the model is optimized and trained through the designed combined loss function to achieve a comprehensive and accurate representation of the assembly video.

[0065] The working logic of the Transformer video encoder is:

[0066] First, the input video is divided into N segments and a frame is randomly selected from each segment. Then, these sampled frames are input into the ResNet-50 network to extract visual features and obtain the feature sequence F. In order to retain the position information of the frame in the sequence, the position embedding E is added to the feature sequence F. pos .

[0067] Next, according to the following formula, the Transformer encoder is used to learn the frame representation of the video. The obtained frame representation contains not only the position information of the frame in the sequence, but also the correlation information between frames. Specifically, the feature sequence F with position embedding is passed to the Transformer encoder. In the encoder, the self-attention mechanism captures the correlation between different frames in the sequence, while the position encoding helps the model understand the relative position relationship between frames.

[0068] FR=TE(F+E pos )

[0069] Among them, TE represents the Transformer encoder, E pos represents the position embedding of the feature sequence F, and FR represents the frame representation of the input video.

[0070] Finally, an MLP module is used to aggregate all frame representations output by the Transformer encoder to obtain a global representation of the input video.

[0071] The working logic of the multi-level video comparison module is as follows:

[0072] 1) Global comparison module

[0073] First, the MLP module is used to obtain the global representation of each video in the video pair, denoted as GR 1 and GR 2 Then, the cosine similarity between video representations is calculated according to the following formula:

[0074]

[0075] Finally, the expected cosine similarity value Cos(GR 1 , G.R. 2 ) is close to 1, the global contrast loss L global The definition is as follows:

[0076] L coarse =1-Cos(GR 1 , G.R. 2 )

[0077] By minimizing the global contrast loss function, different videos with the same assembly order can be clustered more closely together in the feature space of the model;

[0078] 2) Frame-by-frame comparison module

[0079] First, the Transformer encoder is used to obtain all frame representations of each video in the video pair, which are denoted as and

[0080] Then, the matching cost of all frame pairs formed by the two sequences is calculated according to the following formula to obtain the matching cost matrix C:

[0081]

[0082] C i，j represents the matching cost of the i-th row and j-th column in the matching cost matrix C, that is, FR 1 The i-th frame and FR 2 The matching cost of the j-th frame;

[0083] Next, a feasible alignment between sequences is defined as a pairwise corresponding binary matrix M∈{0,1} N×N :

[0084]

[0085] M i，j Indicates whether the elements in the i-th row and j-th column of the matrix M match, that is, FR 1 The i-th frame and FR 2 The jth frame of is matched. If and If it matches, Mi, j =1, otherwise 0;

[0086] Finally, the sequence FR is found by the dynamic time warping algorithm 1 and FR 2 The optimal alignment M* between them is the feasible alignment M with the minimum total matching cost, and the optimal matching cost is used as the frame-by-frame comparison loss L frame :

[0087]

[0088] By minimizing the frame-by-frame contrast loss function, more accurate alignment between sequences can be achieved;

[0089] 3) Training loss function

[0090] In the training phase, video pairs with the same assembly order are used for training, denoted as P i =(V i1 , V i2 ), and classify each video to better learn the step order information in the video. Video classification loss L cls The definition is as follows:

[0091] L cls =CE(f(V i1 ), Y i1 )+CE(f(V i2 ), Y i2 )

[0092] Among them, CE represents the cross entropy function, f represents the video representation learning model, V i1 and V i2 Represent the two videos in the video pair, Y i1 and Y i2 Respectively represent the video V i1 and V i2 The corresponding true sequence label;

[0093] In summary, the model is trained using a combination of video classification loss, global contrast loss, and frame-by-frame contrast loss as shown in the following formula:

[0094] L=λ 1 L cls +λ 2 L global +λ 3 L frame

[0095] Among them, λ 1 represents the weight of video classification loss, λ 2 represents the weight of the global contrast loss, λ 3 Represents the weight of the frame-by-frame contrastive loss.

[0096] The assembly sequence anomaly detection module detects whether the actual assembly sequence conforms to the standard assembly sequence by calculating the distance between the multi-level video representations generated by the model, and stops it in time when an anomaly is detected.

[0097] First, a fixed-size video sliding window is set according to the actual situation. The sliding window is applied to the standard assembly video, and the video segment in the window is input into the video representation learning model to generate a multi-level representation of the segment and store it in the embedding vector list of the database for subsequent detection calls. The sliding window moves on the standard assembly video, and this process is repeated until the sliding ends.

[0098] Then, the same sliding window is applied to the assembly video to be tested, and the video segment in the window is input into the video representation learning model to generate a multi-level representation of the segment. At the same time, the multi-level representation of the corresponding standard assembly video segment in the database is retrieved.

[0099] After that, the distance D between the multi-level representations of the video segment to be detected and the standard video segment is calculated:

[0100] D = dist(FR 1 , F R 2 )+dist(GR 1 , G.R. 2 )

[0101] Where dist represents the Euclidean distance function, FR1 and FR 2 Represents the frame representation of the video segment to be detected and the standard video segment, GR 1 and GR 2 Respectively represent the global representation of the two.

[0102] Next, define the detection result R according to the distance D:

[0103]

[0104] Wherein, τ is a threshold value set according to the actual situation. If D>τ, it means that the assembly sequence to be detected is inconsistent with the standard assembly sequence, that is, it is an abnormal assembly sequence.

[0105] Finally, the sliding window moves on the assembly video to be inspected and the above process is repeated. When an abnormality is detected, the sliding stops.

[0106] As an embodiment of the present invention, in the process of actually implementing the present invention, the following steps can be followed:

[0107] 1. Data Collection and Processing

[0108] The camera collects video data from the assembly process, and the data processing module processes the collected video data to provide high-quality training data for the video representation learning model. First, the video data that meets the requirements is screened out. Then, the video data is sequentially annotated. Next, the video is processed into frames and the height and width of each frame are adjusted. Finally, high-quality data suitable for video representation learning model training is generated.

[0109] 2. Training of Video Representation Learning Model

[0110] The video representation learning model is responsible for a comprehensive and accurate representation of the assembly video. First, a Transformer encoder is used to generate frame representations of the video. Then, the MLP module is applied to aggregate these frame representations into a global representation of the video. Next, a multi-level video comparison module is designed, which includes a global comparison module and a frame-by-frame comparison module. The global comparison module compares videos at the video level to learn the overall similarity relationship between videos; the frame-by-frame comparison module aligns videos at the frame level to learn the step order information in the video. Finally, the processed video data is input into the video representation learning model for training. During the training process, the designed combined loss function is used, combined with the Adam optimizer for back propagation to optimize the model parameters.

[0111] 3. Design of assembly sequence anomaly detection module

[0112] The assembly sequence anomaly detection module is responsible for detecting whether the actual assembly sequence conforms to the standard assembly sequence and stopping it in time when an anomaly is detected. First, a fixed-size video sliding window is set according to the actual situation. The sliding window is applied to the standard assembly video, and the video clips in the window are input into the video representation learning model to generate its multi-level representation and store it in the embedding vector list of the database. The sliding window moves on the standard assembly video, and this process is repeated until the sliding ends. Then, the same sliding window is applied to the assembly video to be detected, and the video clips in the window are input into the video representation learning model to generate its multi-level representation, and the multi-level representation of the corresponding standard assembly video clip in the database is retrieved at the same time. Next, the distance D between the multi-level representation of the video clip to be detected and the standard video clip is calculated, and a suitable threshold τ is set. If D>τ, it means that the assembly sequence to be detected is inconsistent with the standard assembly sequence, that is, it is an abnormal assembly sequence. Finally, the sliding window moves on the assembly video to be detected and the above process is repeated. When an anomaly is detected, the sliding stops.

[0113] 4. Real-time monitoring and detection

[0114] In the actual assembly process, a high-definition pixel camera is used to collect real-time monitoring video streams and transmit them to the server. The server uses the data processing module to process the video, and then passes the processing results to the assembly sequence anomaly detection module for detection. When it is detected that the assembly sequence is inconsistent with the standard assembly sequence, it is stopped in time to ensure product quality and reduce the product failure rate.

[0115] 5. System integration and deployment:

[0116] Integrate the data processing module, video representation learning module, and assembly sequence anomaly detection module into an overall system, and deploy the system to the factory server to ensure that the system can run stably in actual operations. Monitor the performance of the system in different scenarios and working conditions, and further optimize it based on feedback information.

[0117] 6. Effect evaluation and adjustment

[0118] Evaluate the system's performance, including accuracy, real-time, and intelligence levels. Verify the system's stability and reliability through operations in actual scenarios, and adjust system parameters based on the evaluation results to meet the needs of assembly sequence anomaly detection in different scenarios and working conditions.

Claims

1. A method for assembly sequence anomaly detection based on multi-level contrastive video representation learning, characterized by: It includes a data processing module, a video representation learning module, and an assembly sequence anomaly detection module; The data processing module processes the collected video data to provide high-quality training data for model building of the video representation learning module; The video representation learning module trains the video data provided by the data processing module and stores parameter information in a database; The assembly sequence anomaly detection module uses a high-definition pixel camera to collect real-time monitoring video during the actual assembly process, transmits it to the server for processing, and then returns the processing result to the assembly sequence anomaly detection module for detection.

2. The assembly sequence anomaly detection method based on multi-level contrast video representation learning according to claim 1 is characterized in that: The data processing module first collects video data from the assembly process through a camera, and then screens, annotates and formats the data; specifically, since the collected video data may contain irrelevant content, the data must first be screened to determine whether the data meets the requirements, and the data that meets the requirements is retained; then, the retained video data is sequentially annotated for subsequent model training; then, the video is processed into frames, and the height and width of each frame are adjusted to unify the data format; finally, high-quality data suitable for video representation learning model training is generated.

3. The assembly sequence anomaly detection method based on multi-level contrast video representation learning according to claim 1 is characterized in that: The video representation learning module includes two parts: a Transformer video encoder and a multi-level video comparison module. Specifically, first, a Transformer encoder is used to generate frame representations of a video; then, an MLP module is applied to aggregate these frame representations into a global representation of the video; next, a multi-level video comparison module is designed, which includes a global comparison module and a frame-by-frame comparison module; the global comparison module compares videos at the video level to learn the overall similarity relationship between videos; the frame-by-frame video comparison module aligns videos at the frame level to learn the step order information in the video; Finally, the model is optimized and trained through the designed combined loss function to achieve a comprehensive and accurate representation of the assembly video.

4. The method for detecting anomalies in assembly sequence based on multi-level contrastive video representation learning according to claim 3, characterized in that: The working logic of the Transformer video encoder is: First, the input video is divided into Then, these sampled frames are input into the ResNet-50 network to extract visual features and obtain the feature sequence ; In order to retain the position information of the frame in the sequence, it is a feature sequence Add location embed ; Next, according to the following formula, the Transformer encoder is used to learn the frame representation of the video. The obtained frame representation contains not only the position information of the frame in the sequence, but also the correlation information between frames; specifically, the feature sequence with position embedding Passed to the Transformer encoder; in the encoder, the self-attention mechanism captures the correlation between different frames in the sequence, while the position encoding helps the model understand the relative position relationship between frames; in, represents the Transformer encoder, Represents a feature sequence The position of the embedding, represents the frame representation of the input video; Finally, an MLP module is used to aggregate all frame representations output by the Transformer encoder to obtain a global representation of the input video.

5. The method for detecting anomalies in assembly sequence based on multi-level contrastive video representation learning according to claim 3, characterized in that: The working logic of the multi-level video comparison module is as follows: 1) Global comparison module First, the MLP module is used to obtain the global representation of each video in the video pair, which are denoted as and ; Then, the cosine similarity between the video representations is calculated according to the following formula; Finally, the expected cosine similarity value Close to 1, the global contrast loss is defined as follows: By minimizing the global contrast loss function, different videos with the same assembly order can be clustered more closely together in the feature space of the model; 2) Frame-by-frame comparison module First, the Transformer encoder is used to obtain all frame representations of each video in the video pair, which are denoted as and ; Then, the matching cost of all frame pairs formed by the two sequences is calculated according to the following formula to obtain the matching cost matrix : Represents the matching cost matrix Middle Row and The matching cost of the column is No. Frame and No. Matching cost of frames; Next, a feasible alignment between sequences is defined as a binary matrix of pairwise correspondences ; Representation Matrix Middle Row and Whether the elements of the column match, that is No. Frame and No. Whether the frames match; if and Matching , otherwise 0; Finally, the sequence is found by the dynamic time warping algorithm and The optimal alignment between , that is, the feasible alignment with the minimum total matching cost , and use the optimal matching cost as the frame-by-frame comparison loss ; By minimizing the frame-by-frame contrast loss function, more accurate alignment between sequences can be achieved; 3) Training loss function In the training phase, video pairs with the same assembly order are used for training, denoted as , and classify each video to better learn the step sequence information in the video; Video Classification Loss The definition is as follows: in, represents the cross entropy function, represents the video representation learning model, and Respectively represent the two videos in the video pair, and Respectively represent and video and The corresponding true sequence label; In summary, the model is trained using a combination of video classification loss, global contrast loss, and frame-by-frame contrast loss as shown in the following formula: Among them, represents the weight of video classification loss, represents the weight of the global contrast loss, Represents the weight of the frame-by-frame contrastive loss.

6. The assembly sequence anomaly detection method based on multi-level contrast video representation learning according to claim 1 is characterized in that: The assembly sequence anomaly detection module detects whether the actual assembly sequence conforms to the standard assembly sequence by calculating the distance between the multi-level video representations generated by the model, and stops it in time when an anomaly is detected; First, set a fixed-size video sliding window according to the actual situation; Applying the sliding window to the standard assembly video, the video segment in the window is input into the video representation learning model to generate a multi-level representation of the segment and store it in the embedding vector list of the database for subsequent detection calls; The sliding window moves on the standard assembly video, and this process is repeated until the sliding ends; Then, the same sliding window is applied to the assembly video to be inspected, and the video segment in the window is input into the video representation learning model to generate a multi-level representation of the segment. At the same time, the multi-level representation of the corresponding standard assembly video segment in the database is retrieved. After that, the distance between the multi-level representation of the video segment to be detected and the standard video segment is calculated : in, represents the Euclidean distance function, and Respectively represent the frame representation of the video segment to be detected and the standard video segment, and Respectively represent the global representation of the two; Next, according to the distance Defining test results : Among them, τ is a threshold set according to actual conditions; if , it means that the assembly sequence to be tested is inconsistent with the standard assembly sequence, that is, it is an abnormal assembly sequence; Finally, the sliding window moves on the assembly video to be inspected and the above process is repeated. When an abnormality is detected, the sliding stops.

Images