Weakly supervised slag removal endpoint judgment method and system based on cross-modal alignment

Abstract

The present invention discloses a weakly supervised slag removal endpoint judgment method and system based on cross-modal alignment, comprising: accessing the current slag removal site camera, inputting real-time video stream data into a trained video data processor; the video data processor cuts and packages a total of several frames of data including the current frame based on each current frame, outputs the real-time video clip of the current frame as the video modality input of the trained cross-modal feature extractor, and uses the current steel grade text number and "non-end point" text as the text modality input of the cross-modal feature extractor; the cross-modal feature extractor extracts video clip features and text features, inputs the extracted video clip features and text features into a feature comparison decision module for feature comparison, and finally outputs a judgment result of whether the current frame is the slag removal endpoint corresponding to the current steel grade. The present invention can accurately judge the slag removal endpoint.

Description

Technical Field

[0001] The present invention relates to the technical field of slag removal endpoint judgment, and in particular to a weakly supervised slag removal endpoint judgment method and system based on cross-modal alignment. Background Art

[0002] In the steel refining process, the generation of slag as a reaction by-product is inevitable. The density of slag is lower than that of molten steel, floating on the surface of molten steel, mostly in solid form. If the amount of slag in the molten steel is too high when entering the next process, it will shorten the service life of the ladle, increase the consumption of lime, fluorite, and smelting electricity. If the slag content is too low, the nitrogen content in the molten steel will increase, affecting the final quality of the plate. Therefore, it is particularly necessary to accurately control the content of slag in the ladle before entering the next process. Accurate control of the slag content is also an accurate judgment of the end point of the slag removal process.

[0003] Taking the AOD (argon oxygen decarburization refining) post-furnace process as an example, the slag content control here mainly uses a slag remover to remove the surface slag. Due to the different requirements of different steel grades for the remaining amount of slag after the furnace, and the slag as a by-product of high-temperature chemical reactions, it will continue to generate and solidify during the whole process, so that the remaining slag content does not strictly decrease gradually with the slag removal process, making it difficult to perform mathematical modeling. In addition, due to the high temperature and brightness of the molten steel on site, and the continuous change of the exposed molten steel area during the whole slag removal process, the brightness on site will change sharply, making it impossible to accurately detect the slag content, which makes it difficult to judge the end point of slag removal. The unsupervised image processing method has poor effect due to the lack of robustness. The method based on strong supervision brings strong robustness with the labeled data, but due to the need for a large amount of labeled data, the labeling cost is too high and it cannot be implemented on site. At present, this problem mainly relies on the manual judgment of on-site operators. This method has the problems of strong subjectivity and lack of standardization, and cannot guarantee the objective judgment of the slag removal end point of different steel grades during the slag removal process. Summary of the invention

[0004] The present invention provides a weakly supervised slagging endpoint judgment method and system based on cross-modal alignment to solve the problems existing in the above-mentioned prior art. The technical solution is as follows:

[0005] On the one hand, a weakly supervised slag removal endpoint judgment method based on cross-modal alignment is provided, including:

[0006] S1, connect to the current slag removal site camera, and input the real-time video stream data into the trained video data processor;

[0007] S2, the video data processor cuts and packages a total of several frames of data including the current frame based on each current frame, outputs a real-time video clip of the current frame as the video modality input of the trained cross-modal feature extractor, and uses the current steel grade text number and the "non-end point" text as the text modality input of the cross-modal feature extractor;

[0008] S3, the cross-modal feature extractor extracts video clip features and text features, inputs the extracted video clip features and text features into a feature comparison decision module for feature comparison, and finally outputs a judgment result of whether the current frame is the slag removal endpoint corresponding to the current steel grade.

[0009] Optionally, the training of the video data processor includes:

[0010] Collect the video data and text data of the steel grade of historical slag removal of each steel grade, the single video data includes all frame data starting from the ladle tipping to the ladle angle returning to the normal position, record the steel grade text number of the single slag removal process, and manually mark the slag removal end frame, the three are packaged and saved as a single slag removal data packet, collect multiple single slag removal data packets, and input the collected multiple single slag removal data packets into the video data processor;

[0011] The video data processor extends a number of frames forward and backward with the end frame of a single video as the center according to the set number of video clip frames to form a positive video clip of a single slag removal process, and further extends the positive video clip forward and backward based on the positive video clip, and slices the entire single video with the set number of video clip frames as the length to obtain multiple negative video clips that do not include the end frame;

[0012] The video data processor packages and outputs a single positive video clip and a plurality of negative video clips corresponding to a plurality of single slag removal videos, and inputs the packets into the cross-modal feature extractor.

[0013] Optionally, the cross-modal feature extractor includes a video feature extractor and a text feature extractor;

[0014] The video feature extractor first crops the input video clip at the same position of the frame according to the set cropping size, crops the video clip into several non-overlapping video clip blocks according to the pixel position, each video clip block includes the spatial information of its position, inputs the video clip block as a patch-level visual vector into the visual encoder V1 for visual encoding, outputs spatial visual features with spatial representation ability, flattens the spatial visual features according to the frame dimension to obtain several fream-level visual vectors, adds a frame number position code to each fream-level visual vector to represent the temporal relationship between frames, and simultaneously sends several fream-level visual vectors with the frame number position code to the visual encoder V2, outputs spatiotemporal visual features with both spatial information and temporal information, and outputs the final video features through the average pooling layer;

[0015] The steel grade text number and the "non-end point" text corresponding to the video clip are input into the cross-modal feature extractor. The text feature extractor, after word segmentation of the steel grade text number and the "non-end point" text through the Tokenization layer, inputs the obtained word segmentation vector into the text encoder T1 to obtain text features with the same dimension as the video features output by the video feature extractor. The text features output by the text encoder T1 and the spatial visual features output by the visual encoder V1 are used as the input of the text semantic enhancement module M1 to enhance the visual expression ability of the rich text modality from a spatial perspective. The semantically enhanced text features output by the text semantic enhancement module M1 and the spatiotemporal visual features output by the visual encoder V2 are used as the input of the text semantic enhancement module M2 to further enhance the visual expression ability of the rich text modality from both spatial and temporal perspectives. The final text features are output by the text semantic enhancement module M2.

[0016] Optionally, the structures of the visual encoders V1 and V2 are the same, and the inputs are both visual vectors. After the visual vectors are mapped by the convolution layer conv and the projection layer proj, they are sent to 6 attention modules consisting of self-attention layers and MLP layers connected end to end for visual feature extraction, and the visual features with the same dimension as the input visual vectors are output;

[0017] The input of the text encoder T1 is the word segmentation vector, which is mapped to the text vector by the word embedding model and then sent to 12 attention modules connected end to end for text feature extraction, and the text features are output;

[0018] The text semantic enhancement modules M1 and M2 have the same structure, and simultaneously input visual features and text features, and output text features that fuse visual information, wherein the input visual features include spatial visual information or spatiotemporal visual information, which are matrix multiplied with the input text features and then passed through a softmax layer, and then directly point-multiplied with the input text features through the result of a reshape operation, and the text features are re-weighted, and finally, rich semantic information of the space and temporal information included in the visual features in the process of fusion scraping is obtained, and the feature representation capability of the semantic information is enriched from a more fine-grained perspective.

[0019] Optionally, the training of the cross-modal feature extractor further includes: inputting the video features and text features output by the cross-modal feature extractor into a similarity calculation module, wherein the video features include positive example features and negative example features, and the text features include endpoint text features and non-endpoint text features;

[0020] The similarity calculation module first aligns the video features and the text features in a high-dimensional mapping dimension by convolution with a convolution kernel size of 1, normalizes the aligned features by a layer standardization layer to increase the model stability, outputs the high-dimensional video features and the text features, and respectively calculates the cosine similarities of four groups of feature pairs, namely, "positive example-end point text", "negative example-end point text", "positive example-non-end point text", and "negative example-non-end point text", and obtains a similarity set composed of the cosine similarities of the four groups of feature pairs as output, wherein the cosine similarity calculation process of each group of feature pairs is as follows:

[0021] s=cos(LayerNorm(proj(v)),LayerNorm(proj(c)))

[0022] Where v represents the input video feature, c represents the input text feature, and s represents the similarity calculation result.

[0023] Optionally, the training of the cross-modal feature extractor further includes: alignment optimization of the cross-modal feature extractor, specifically including:

[0024] The similarity set output by the similarity calculation module and the multimodal features extracted by the original cross-modal feature extractor are input into the cross-modal alignment optimization module, and the cross-modal alignment optimization module defines the similarity result of the "positive example-end point text" feature pair and the similarity result of the "negative example-non-end point text" feature pair as the positive sample pair similarity result s p , the similarity results of the “positive example-non-end point text” feature pair and the “negative example-end point text” feature pair are defined as the similarity results of the negative sample pair s n ,s p With s nThe loss is calculated for cross-modal feature alignment, and the loss function is infoNCE loss. The loss value L is calculated as follows:

[0025]

[0026] Where k represents the number of samples of all video clips including positive and negative examples, s k Represents the scoring results of all video clips-text pairs, s k =s p +s n , τ represents the temperature coefficient, which is used to control the distribution of variables in the loss function, and is taken as 1 here;

[0027] Perform gradient back propagation based on the loss value calculation results. During the gradient back propagation process, increase s p In order to shorten the distance between positive sample pairs in the feature space and reduce the loss value, reduce s n In order to increase the distance between negative sample pairs in the feature space and reduce the loss value, the parameters of the cross-modal feature extractor are optimized, the optimized cross-modal feature extractor is fully trained, and then the video features and text features extracted by the cross-modal feature extractor are optimized. The trained cross-modal feature extractor can map video features and text features to the same feature space and ensure that the similarity values ​​between positive sample pairs are large, and the similarity values ​​between negative sample pairs are small.

[0028] Optionally, the feature comparison decision module first calculates the similarity between the video segment feature and the endpoint text feature as a first similarity, and the similarity between the video segment feature and the non-endpoint text feature as a second similarity;

[0029] Compare the calculation results: if the first similarity is higher than the second similarity, record the current frame number, and determine whether the previous two frames have the same result based on the frame number. If the determination is successful, determine that the current frame is the slag removal endpoint frame for the corresponding steel type. If the determination is not successful, determine that the current frame is not the slag removal endpoint frame, and output the determination result in real time.

[0030] On the other hand, a weakly supervised slag removal endpoint judgment system based on cross-modal alignment is provided, the system comprising: an input module, a video data processor, a cross-modal feature extractor and a feature comparison decision module;

[0031] The input module is used to access the current slag removal site camera and input the real-time video stream data into the trained video data processor;

[0032] The video data processor is used to cut and pack a total of several frames of data including the current frame based on each current frame, output a real-time video clip of the current frame as a video modality input of the trained cross-modality feature extractor, and use the current steel grade text number and the "non-end point" text as text modality input of the cross-modality feature extractor;

[0033] The cross-modal feature extractor is used to extract video clip features and text features, wherein the text features include current steel grade text number features and non-end point text features, and the extracted video clip features and text features are input into the feature comparison decision module;

[0034] The feature comparison decision module is used to perform feature comparison and finally output a judgment result of whether the current frame is the slag removal end point corresponding to the current steel grade.

[0035] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned weakly supervised slag removal endpoint judgment method based on cross-modal alignment.

[0036] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned weakly supervised slag removal endpoint judgment method based on cross-modal alignment.

[0037] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0038] The weakly supervised slag removal endpoint prediction method and system proposed in the present invention based on cross-modal alignment can realize automatic detection and discrimination of the slag removal endpoint, enhance the standardization and normalization of the process, and combine the proposed video and text dual-modal modeling scheme. By designing a cross-modal feature extraction module, multi-modal features with both spatial and temporal information can be effectively extracted, the robustness of the extracted features can be improved, and the influence of on-site noise can be reduced. At the same time, a unique video data processor is designed to realize weakly supervised labeling of the slag removal process. Each slag removal process only needs to manually label the slag removal endpoint frame to generate a large number of video samples for model training, which greatly reduces the labeling cost. In addition, through the semantic richness of the introduced text modality and the cross-modal alignment training process, the characteristics between different steel grades can be effectively distinguished, and the same model can be used to simultaneously model the slag removal endpoints of different steel grades. Specifically, the advantages of the present invention are as follows:

[0039] 1) Compared with the manual method of determining the end point of slag removal, the present invention realizes effective modeling of the task of determining the end point of slag removal based on the cross-modal feature space alignment method, thereby improving the objectivity and accuracy of the overall determination process.

[0040] 2) Using text as the second modality, the text has semantic information. After combining with video information through cross-modal alignment, the features of different steel grades can be effectively mapped in the high-dimensional feature space, and the distances between different modalities of the same steel grade are guaranteed to be close. At the same time, due to the natural differences between the text features of different steel grades, the distances between the cross-modal aligned features of different steel grades are far due to the objective differences between the texts after they are trained separately. This effectively gets rid of the limitation of only using the video modality to train one steel grade at a time, and realizes the unified modeling of the same model for multiple steel grades.

[0041] 3) Design a cross-modal feature extraction module to comprehensively utilize the on-site spatial information and temporal information in the video modality and text modality to effectively extract more robust multimodal features and reduce the impact of strong noise in the data collected during the slag removal process.

[0042] 4) A weakly supervised data labeling method only for the slag removal endpoint is designed, and a large number of video samples are automatically generated with the designed video data processor, which greatly reduces the labeling time and reduces the labeling cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 It is a flow chart of a method for determining a slag removal endpoint based on weak supervision and cross-modal alignment provided by an embodiment of the present invention;

[0045] Figure 2 is a general flow chart of the real-time prediction stage provided by an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of a video data processor provided by an embodiment of the present invention;

[0047] Figure 4 is a schematic diagram of a cross-modal feature extractor provided by an embodiment of the present invention;

[0048] Figure 5 is a schematic diagram of a visual encoder and a text encoder provided by an embodiment of the present invention;

[0049] Figure 6 is a schematic diagram of a text semantic enhancement module provided by an embodiment of the present invention;

[0050] Figure 7 is a schematic diagram of a similarity calculation module provided by an embodiment of the present invention;

[0051] Figure 8 is a schematic diagram of a cross-modal alignment optimization module provided by an embodiment of the present invention;

[0052] Fig. 9 It is a general training process flow chart of a weakly supervised slag removal endpoint judgment method based on cross-modal alignment provided by an embodiment of the present invention;

[0053] Fig.10 is a schematic diagram of a feature comparison decision module provided in an embodiment of the present invention;

[0054] Fig.11 It is a block diagram of a weakly supervised slag removal endpoint judgment system based on cross-modal alignment provided by an embodiment of the present invention;

[0055] Fig.12 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0057] The embodiment of the present invention provides a weakly supervised method for determining the end point of slag removal based on cross-modal alignment, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of a weakly supervised slag removal endpoint judgment method based on cross-modal alignment is shown. The processing flow of the method may include the following steps:

[0058] S1, connect to the current slag removal site camera, and input the real-time video stream data into the trained video data processor;

[0059] S2, the video data processor cuts and packages a total of several frames of data including the current frame based on each current frame, outputs a real-time video clip of the current frame as the video modality input of the trained cross-modal feature extractor, and uses the current steel grade text number and the "non-end point" text as the text modality input of the cross-modal feature extractor;

[0060] S3, the cross-modal feature extractor extracts video clip features and text features, inputs the extracted video clip features and text features into a feature comparison decision module for feature comparison, and finally outputs a judgment result of whether the current frame is the slag removal endpoint corresponding to the current steel grade.

[0061] The embodiment of the present invention proposes a weakly supervised slag removal endpoint judgment method based on cross-modal alignment. By designing a video and text cross-modal alignment training method, the model uses the pairing relationship between the cropped positive and negative example video clips and the steel grade text to perform high-dimensional mapping and alignment optimization of the cross-modal feature extractor in a large amount of historical slag removal video data. Through the optimized cross-modal feature extractor, more robust multi-modal features are extracted for high-accuracy automatic slag removal endpoint judgment. Due to the addition of text mode, the cross-modal feature alignment method is used to effectively improve the model's ability to distinguish the slag removal endpoints of different steel grades, and enhance the model's ability to simultaneously model multiple steel grades. At the same time, a weakly supervised annotation method for manually annotating the endpoint frames is designed to adapt to the above-mentioned training framework, thereby reducing the annotation cost and reducing the cost due to video resolution. The labeling error caused by the limitation is improved, and the judgment accuracy and objectivity are improved. For the technical scheme of the embodiment of the present invention: in the training stage, first, collect video data of multiple slag removal of each steel grade, collect and annotate the slag removal end frame in each slag removal process, set the number of video clip frames, take the end frame as the center, and cut to obtain the video clip including the slag removal end frame as the positive video clip of the slag removal end point. At the same time, the remaining clips in this slag removal process are used as negative video clips, together with the text number of the steel grade for this slag removal and the "non-end point" text, which are input into the cross-modal feature extractor for video and text bimodal data feature extraction, and then the extracted positive and negative video features are mapped and the similarity is calculated between the text features, and the loss is calculated according to the similarity calculation result for cross-modal alignment optimization, and infoNCE loss is used as the loss function, and the parameters of the cross-modal feature extractor are repeatedly trained and optimized to obtain the final optimized cross-modal feature extractor. In the real-time prediction stage, the real-time video data of the slag removal site is sliced ​​and processed. The sliced ​​video segment features, the text number information of the steel grade produced this time, and the "non-end point" text are input into the cross-modal feature extractor optimized in the training stage for feature extraction. The extracted video and text features are input into the feature comparison decision module, and the similarity of the video segment features and the steel grade endpoint features and non-end point features is calculated respectively to obtain the final endpoint judgment result, which is used to realize the fully automatic judgment of the slag removal endpoint of the corresponding steel grade. Figure 2-Figure 10 , a weakly supervised slag removal endpoint prediction method based on cross-modal alignment provided by an embodiment of the present invention is described in detail, such as Figure 2 As shown in the figure, it is the overall flow chart of the real-time prediction stage, including:

[0062] S1, connect to the current slag removal site camera, and input the real-time video stream data into the trained video data processor;

[0063] Optionally, the training of the video data processor includes:

[0064] Collect the video data and text data of the steel grade of historical slag removal of each steel grade, the single video data includes all frame data starting from the ladle tipping to the ladle angle returning to the normal position, record the steel grade text number (such as 1A1100) of the single slag removal process, and manually mark the slag removal end frame, the three are packaged and saved as a single slag removal data packet, collect multiple single slag removal data packets, and input the collected multiple single slag removal data packets into the video data processor;

[0065] like Figure 3 As shown, the video data processor extends several frames (for example, 7 frames) forward and backward with the end frame of a single video as the center according to the set number of video clip frames (for example, 15 frames) to form a positive video clip of a single slag removal process, and based on the positive video clip, extends forward and backward again, slices the entire single video with the set number of video clip frames as the length to obtain multiple negative video clips excluding the end frame (for example, extend forward 15 frames from the first frame of the positive video clip to obtain a negative video clip, and then extend forward 15 frames to obtain a negative video clip, and extend backward 15 frames from the last frame of the positive video clip to obtain a negative video clip (the end frame is not the last frame of the single slag removal video) to obtain a negative video clip, and then extend backward 15 frames to obtain a negative video clip, and then continue to extend forward and backward to obtain multiple negative video clips);

[0066] The video data processor packages and outputs a single positive video clip and a plurality of negative video clips corresponding to a plurality of single slag removal videos, and inputs the packets into the cross-modal feature extractor.

[0067] S2, the video data processor cuts and packages the data of several frames (for example, 15 frames) including the current frame based on each current frame, outputs the real-time video clip of the current frame as the video modality input of the trained cross-modal feature extractor, and uses the current steel grade text number and the "non-end point" text as the text modality input of the cross-modal feature extractor;

[0068] Alternatively, if Figure 4 As shown, the cross-modal feature extractor includes a video feature extractor and a text feature extractor;

[0069] The video feature extractor first crops the input video clip at the same position of the frame according to the set cropping size (for example, the size is 32), and crops the video clip into several non-overlapping video clip blocks according to the pixel position (for example, several non-overlapping video clip blocks (patch) with a size of 32×32×15), each video clip block includes the spatial information of its position, and inputs the video clip block as a patch-level visual vector into the visual encoder V1 for visual encoding, outputs spatial visual features with spatial representation ability, flattens the spatial visual features according to the frame dimension to obtain several fream-level visual vectors (for example, 15 fream-level visual vectors with a size of 32×32 are obtained), adds a frame number position code to each fream-level visual vector to represent the temporal relationship between frames, and simultaneously sends several fream-level visual vectors with the frame number position code to the visual encoder V2, outputs spatiotemporal visual features with both spatial information and temporal information, and outputs the final video features (for example, the dimension is 1024) through the average pooling layer;

[0070] The steel grade text number and the "non-end point" text corresponding to the video clip are input into the cross-modal feature extractor. The text feature extractor, after word segmentation of the steel grade text number and the "non-end point" text through the Tokenization layer, inputs the obtained word segmentation vector into the text encoder T1 to obtain text features with the same dimension as the video features output by the video feature extractor (for example, the dimension is 1024). The text features output by the text encoder T1 and the spatial visual features output by the visual encoder V1 are used as the input of the text semantic enhancement module M1 to enhance the visual expression ability of the rich text modality from a spatial perspective. The semantically enhanced text features output by the text semantic enhancement module M1 and the spatiotemporal visual features output by the visual encoder V2 are used as the input of the text semantic enhancement module M2 to further enhance the visual expression ability of the rich text modality from both spatial and temporal perspectives. The text semantic enhancement module M2 outputs the final text features (for example, the dimension is 1024).

[0071] Alternatively, if Figure 5 As shown in the figure, the structures of the visual encoders V1 and V2 are the same, and the inputs are both visual vectors. After the visual vectors are mapped by the convolution layer conv and the projection layer proj, they are sent to the six attention modules consisting of the self-attention layer and the MLP layer connected end to end for visual feature extraction, and the visual features with the same dimension as the input visual vectors are output;

[0072] like Figure 5 As shown, the input of the text encoder T1 is the word segmentation vector, which is mapped to the text vector by the word embedding model and then sent to 12 attention modules connected end to end for text feature extraction, and the text features are output;

[0073] like Figure 6 As shown, the structures of the text semantic enhancement modules M1 and M2 are the same, and visual features and text features are input simultaneously, and text features that fuse visual information are output, wherein the input visual features include spatial visual information or spatiotemporal visual information, which are matrix multiplied with the input text features and then passed through a softmax layer, and then directly multiplied with the input text features through the result of a reshape operation, and the text features are reweighted, and finally the rich semantic information of the space and temporal information included in the visual features in the fusion scraping process is obtained, and the feature representation capability of the semantic information is enriched from a more fine-grained perspective.

[0074] Optionally, the training of the cross-modal feature extractor further includes: inputting the video features and text features output by the cross-modal feature extractor into a similarity calculation module, wherein the video features include positive example features and negative example features, and the text features include endpoint text features and non-endpoint text features;

[0075] like Figure 7 As shown, the similarity calculation module first aligns the video features and text features in high-dimensional mapping dimensions through a convolution with a convolution kernel size of 1, normalizes the aligned features through a layer standardization layer to increase the stability of the model, outputs high-dimensional video features and text features (for example, 2048 dimensions), and calculates the cosine similarities of four groups of feature pairs, namely "positive example-end point text", "negative example-end point text", "positive example-non-end point text", and "negative example-non-end point text", to obtain a similarity set composed of the cosine similarities of the four groups of feature pairs as output, wherein the cosine similarity calculation process of each group of feature pairs is as follows:

[0076] s=cos(LayerNorm(proj(v)),LayerNorm(proj(c)))

[0077] Where v represents the input video feature, c represents the input text feature, and s represents the similarity calculation result.

[0078] Optionally, the training of the cross-modal feature extractor further includes: alignment optimization of the cross-modal feature extractor, specifically including:

[0079] The similarity set output by the similarity calculation module and the multimodal features (video features and text features) extracted by the original cross-modal feature extractor are input into the cross-modal alignment optimization module, such as Figure 8 As shown, the cross-modal alignment optimization module defines the similarity results of the “positive example-end point text” feature pair and the similarity results of the “negative example-non-end point text” feature pair as the positive sample pair similarity results s p, the similarity results of the “positive example-non-end point text” feature pair and the “negative example-end point text” feature pair are defined as the similarity results of the negative sample pair s n ,s p With s n The loss is calculated for cross-modal feature alignment, and the loss function is infoNCE loss. The loss value L is calculated as follows:

[0080]

[0081] Where k represents the number of samples of all video clips including positive and negative examples, s k Represents the scoring results of all video clips-text pairs, s k =s p +s n , τ represents the temperature coefficient, which is used to control the distribution of variables in the loss function, and is taken as 1 here;

[0082] Perform gradient back propagation based on the loss value calculation results. During the gradient back propagation process, increase s p In order to shorten the distance between positive sample pairs in the feature space and reduce the loss value, reduce s n In order to increase the distance between negative sample pairs in the feature space and reduce the loss value, the parameters of the cross-modal feature extractor are optimized, the optimized cross-modal feature extractor is fully trained, and then the video features and text features extracted by the cross-modal feature extractor are optimized. The trained cross-modal feature extractor can map video features and text features to the same feature space and ensure that the similarity values ​​between positive sample pairs are large, and the similarity values ​​between negative sample pairs are small.

[0083] like Fig. 9 As shown, the overall training process of the embodiment of the present invention is: the video data processor, the cross-modal feature extractor, and the similarity calculation module are connected in sequence, and the historical video data, the steel grade text number data and the "non-end point" text data are used as input for training, and the multimodal features (video features and text features) extracted by the optimized cross-modal feature extractor are used as output. During the training process, multiple groups of slag removal process videos are input, and the video data processor generates the video clip data required for training, which are combined with the steel grade text number data and the "non-end point" text data and input into the cross-modal feature extractor together. The feature extraction result is input into the similarity calculation module to obtain a similarity set, and the cross-modal feature extractor is trained for parameter optimization according to the similarity set using the cross-modal alignment optimization module to align the video and text features.

[0084] S3, the cross-modal feature extractor extracts video clip features and text features, inputs the extracted video clip features and text features into a feature comparison decision module for feature comparison, and finally outputs a judgment result of whether the current frame is the slag removal endpoint corresponding to the current steel grade.

[0085] Alternatively, if Fig.10 As shown, the feature comparison decision module first calculates the similarity between the video segment feature and the endpoint text feature as a first similarity, and the similarity between the video segment feature and the non-endpoint text feature as a second similarity;

[0086] Compare the calculation results: if the first similarity is higher than the second similarity, record the current frame number, and determine whether the previous two frames have the same result based on the frame number. If the determination is successful, determine that the current frame is the slag removal endpoint frame for the corresponding steel type. If the determination is not successful, determine that the current frame is not the slag removal endpoint frame, and output the determination result in real time.

[0087] A weakly supervised slag removal endpoint prediction method based on cross-modal alignment is proposed in an embodiment of the present invention. Aiming at the current situation that the quantitative standards of manual judgment methods in the slag removal process are not unified, the existing image processing methods cannot take into account both the strong anti-interference performance of the model and the low annotation cost, and the single-modal model is difficult to simultaneously process the endpoint prediction of multiple steel grades, the video processor is designed by using historical slag removal video data and manually annotated slag removal endpoint frames, and the positive and negative slag removal video clips are automatically generated. The cross-modal feature extractor is jointly trained with the steel grade text number and the "non-end point" text to extract the visual features and text features combining spatial information and temporal information. The modal distance between the positive and negative example videos and the steel grade text in the feature space is aligned by contrast learning through the cross-modal alignment optimization module, and the cross-modal feature extractor is optimized. While reducing the annotation training cost, the objectivity and accuracy of the slag removal endpoint judgment are effectively improved. The text data of each steel grade and the video clip data of the slag removal process are aligned in the feature space, and the slag removal endpoint of the slag removal process of multiple categories of steel grades can be automatically identified and judged by inputting the text number of the steel grade and the real-time slag removal video.

[0088] like Fig.11 As shown, the embodiment of the present invention also provides a weakly supervised slag removal endpoint judgment system based on cross-modal alignment, the system comprising: an input module 1110, a video data processor 1120, a cross-modal feature extractor 1130 and a feature comparison decision module 1140;

[0089] The input module 1110 is used to access the current slag removal site camera and input the real-time video stream data into the trained video data processor;

[0090] The video data processor 1120 is used to cut and pack a total of several frames of data including the current frame based on each current frame, output a real-time video clip of the current frame as a video modality input of the trained cross-modality feature extractor, and use the current steel grade text number and the "non-end point" text as text modality input of the cross-modality feature extractor;

[0091] The cross-modal feature extractor 1130 is used to extract video clip features and text features, wherein the text features include current steel grade text number features and non-end point text features, and the extracted video clip features and text features are input into the feature comparison decision module;

[0092] The feature comparison decision module 1140 is used to perform feature comparison and finally output a judgment result of whether the current frame is the slag removal end point corresponding to the current steel grade.

[0093] An embodiment of the present invention provides a weakly supervised slag removal endpoint judgment system based on cross-modal alignment, and its functional structure corresponds to a weakly supervised slag removal endpoint judgment method based on cross-modal alignment provided by an embodiment of the present invention, which will not be repeated here.

[0094] Fig.12 It is a structural diagram of an electronic device 1200 provided in an embodiment of the present invention. The electronic device 1200 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1201 and one or more memories 1202, wherein at least one instruction is stored in the memory 1202, and the at least one instruction is loaded and executed by the processor 1201 to implement the steps of the above-mentioned weakly supervised slag removal endpoint judgment method based on cross-modal alignment.

[0095] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the weakly supervised slag removal endpoint judgment method based on cross-modal alignment. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0096] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A weakly supervised slag removal endpoint judgment method based on cross-modal alignment, characterized in that: The method comprises: S1, connect to the current slag removal site camera, and input the real-time video stream data into the trained video data processor; S2, the video data processor cuts and packages a total of several frames of data including the current frame based on each current frame, outputs a real-time video clip of the current frame as the video modality input of the trained cross-modal feature extractor, and uses the current steel grade text number and the "non-end point" text as the text modality input of the cross-modal feature extractor; S3, the cross-modal feature extractor extracts video clip features and text features, inputs the extracted video clip features and text features into a feature comparison decision module for feature comparison, and finally outputs a judgment result of whether the current frame is the slag removal endpoint corresponding to the current steel grade; The cross-modal feature extractor includes a video feature extractor and a text feature extractor; The video feature extractor first crops the input video clip at the same position of the frame according to the set cropping size, crops the video clip into several non-overlapping video clip blocks according to the pixel position, each video clip block includes the spatial information of its position, inputs the video clip block as a patch-level visual vector into the visual encoder V1 for visual encoding, outputs spatial visual features with spatial representation ability, flattens the spatial visual features according to the frame dimension to obtain several fream-level visual vectors, adds a frame number position code to each fream-level visual vector to represent the temporal relationship between frames, and simultaneously sends several fream-level visual vectors with the frame number position code to the visual encoder V2, outputs spatiotemporal visual features with both spatial information and temporal information, and outputs the final video features through the average pooling layer; The steel grade text number and the "non-end point" text corresponding to the video clip are input into the cross-modal feature extractor. The text feature extractor, after word segmentation of the steel grade text number and the "non-end point" text via the Tokenization layer, inputs the obtained word segmentation vector into the text encoder T1 to obtain text features with the same dimension as the video features output by the video feature extractor. The text features output by the text encoder T1 and the spatial visual features output by the visual encoder V1 are used as the input of the text semantic enhancement module M1 to enhance the visual expression ability of the rich text modality from a spatial perspective. The semantically enhanced text features output by the text semantic enhancement module M1 and the spatiotemporal visual features output by the visual encoder V2 are used as the input of the text semantic enhancement module M2 to further enhance the visual expression ability of the rich text modality from both spatial and temporal perspectives. The final text features are output by the text semantic enhancement module M2. The structures of the visual encoders V1 and V2 are the same. The inputs are both visual vectors. After the visual vectors are mapped by the convolution layer conv and the projection layer proj, they are sent to the six attention modules consisting of the self-attention layer and the MLP layer connected end to end for visual feature extraction, and the visual features with the same dimension as the input visual vectors are output. The input of the text encoder T1 is the word segmentation vector, which is mapped to the text vector by the word embedding model and then sent to 12 attention modules connected end to end for text feature extraction, and the text features are output; The text semantic enhancement modules M1 and M2 have the same structure, and input visual features and text features at the same time, and output text features that fuse visual information, wherein the input visual features include spatial visual information or spatiotemporal visual information, which are matrix multiplied with the input text features and then passed through a softmax layer, and then directly multiplied with the input text features through the result of a reshape operation, and the text features are reweighted, and finally, rich semantic information of the space and temporal information included in the visual features in the fusion process of the scraping process is obtained, and the feature representation capability of the semantic information is enriched from a more fine-grained perspective; The training of the cross-modal feature extractor further includes: inputting the video features and text features output by the cross-modal feature extractor into a similarity calculation module, wherein the video features include positive example features and negative example features, and the text features include endpoint text features and non-endpoint text features; The similarity calculation module first performs high-dimensional mapping dimension alignment on the video features and the text features through a convolution with a convolution kernel size of 1, normalizes the aligned features through a layer standardization layer to increase the model stability, outputs high-dimensional video features and text features, and respectively calculates the cosine similarities of four groups of feature pairs, namely "positive example-end point text", "negative example-end point text", "positive example-non-end point text", and "negative example-non-end point text", to obtain a similarity set composed of the cosine similarities of the four groups of feature pairs as output, wherein the cosine similarity calculation process of each group of feature pairs is as follows: s=cos(LayerNorm(proj(v)),LayerNorm(proj(c))) Where v represents the input video feature, c represents the input text feature, and s represents the similarity calculation result; The training of the cross-modal feature extractor further includes: optimizing the alignment of the cross-modal feature extractor, specifically including: The similarity set output by the similarity calculation module and the multimodal features extracted by the original cross-modal feature extractor are input into the cross-modal alignment optimization module, and the cross-modal alignment optimization module defines the similarity result of the "positive example-end point text" feature pair and the similarity result of the "negative example-non-end point text" feature pair as the positive sample pair similarity result s p , the similarity results of the "positive example-non-end point text" feature pair and the similarity results of the "negative example-end point text" feature pair are defined as the similarity results of the negative sample pair s n ,s p With s n The loss is calculated for cross-modal feature alignment, and the loss function is infoNCE loss. The loss value L is calculated as follows: Where k represents the number of samples of all video clips including positive and negative examples, s k Represents the scoring results of all video clips-text pairs, s k =s p +s n , τ represents the temperature coefficient, which is used to control the distribution of variables in the loss function, and is taken as 1 here; Perform gradient back propagation based on the loss value calculation results. During the gradient back propagation process, increase s p In order to shorten the distance between positive sample pairs in the feature space and reduce the loss value, reduce s n In order to increase the distance between negative sample pairs in the feature space and reduce the loss value, the parameters of the cross-modal feature extractor are optimized, the optimized cross-modal feature extractor is fully trained, and then the video features and text features extracted by the cross-modal feature extractor are optimized. The trained cross-modal feature extractor can map the video features and text features to the same feature space and ensure that the similarity values ​​between positive sample pairs are large and the similarity values ​​between negative sample pairs are small; The feature comparison decision module first calculates the similarity between the video segment feature and the endpoint text feature as a first similarity, and the similarity between the video segment feature and the non-endpoint text feature as a second similarity; Compare the calculation results: if the first similarity is higher than the second similarity, record the current frame number, and determine whether the previous two frames have the same result based on the frame number. If the determination is successful, determine that the current frame is the slag removal endpoint frame for the corresponding steel type. If the determination is not successful, determine that the current frame is not the slag removal endpoint frame, and output the determination result in real time.

2. The method according to claim 1, characterized in that The training of the video data processor comprises: Collect the video data and text data of the steel grade of historical slag removal of each steel grade, the single video data includes all frame data starting from the ladle tipping to the ladle angle returning to the normal position, record the steel grade text number of the single slag removal process, and manually mark the slag removal end frame, the three are packaged and saved as a single slag removal data packet, collect multiple single slag removal data packets, and input the collected multiple single slag removal data packets into the video data processor; The video data processor extends a number of frames forward and backward with the end frame of a single video as the center according to the set number of video clip frames to form a positive video clip of a single slag removal process, and further extends the positive video clip forward and backward based on the positive video clip, and slices the entire single video with the set number of video clip frames as the length to obtain multiple negative video clips that do not include the end frame; The video data processor packages and outputs a single positive video clip and a plurality of negative video clips corresponding to a plurality of single slag removal videos, and inputs the clips into the cross-modal feature extractor.

3. A weakly supervised slag removal endpoint judgment system based on cross-modal alignment, characterized in that: The system comprises: an input module, a video data processor, a cross-modal feature extractor and a feature comparison and decision module; The input module is used to access the current slag removal site camera and input the real-time video stream data into the trained video data processor; The video data processor is used to cut and pack a total of several frames of data including the current frame based on each current frame, output a real-time video clip of the current frame as a video modality input of the trained cross-modality feature extractor, and use the current steel grade text number and "non-end point" text as text modality input of the cross-modality feature extractor; The cross-modal feature extractor is used to extract video clip features and text features, wherein the text features include current steel grade text number features and non-end point text features, and the extracted video clip features and text features are input into the feature comparison decision module; The feature comparison decision module is used to perform feature comparison and finally output a judgment result of whether the current frame is the slag removal end point corresponding to the current steel grade; The cross-modal feature extractor includes a video feature extractor and a text feature extractor; The video feature extractor first crops the input video clip at the same position of the frame according to the set cropping size, crops the video clip into several non-overlapping video clip blocks according to the pixel position, each video clip block includes the spatial information of its position, inputs the video clip block as a patch-level visual vector into the visual encoder V1 for visual encoding, outputs spatial visual features with spatial representation ability, flattens the spatial visual features according to the frame dimension to obtain several fream-level visual vectors, adds a frame number position code to each fream-level visual vector to represent the temporal relationship between frames, and simultaneously sends several fream-level visual vectors with the frame number position code to the visual encoder V2, outputs spatiotemporal visual features with both spatial information and temporal information, and outputs the final video features through the average pooling layer; The steel grade text number and the "non-end point" text corresponding to the video clip are input into the cross-modal feature extractor. The text feature extractor, after word segmentation of the steel grade text number and the "non-end point" text via the Tokenization layer, inputs the obtained word segmentation vector into the text encoder T1 to obtain text features with the same dimension as the video features output by the video feature extractor. The text features output by the text encoder T1 and the spatial visual features output by the visual encoder V1 are used as the input of the text semantic enhancement module M1 to enhance the visual expression ability of the rich text modality from a spatial perspective. The semantically enhanced text features output by the text semantic enhancement module M1 and the spatiotemporal visual features output by the visual encoder V2 are used as the input of the text semantic enhancement module M2 to further enhance the visual expression ability of the rich text modality from both spatial and temporal perspectives. The final text features are output by the text semantic enhancement module M2. The structures of the visual encoders V1 and V2 are the same. The inputs are both visual vectors. After the visual vectors are mapped by the convolution layer conv and the projection layer proj, they are sent to the six attention modules consisting of the self-attention layer and the MLP layer connected end to end for visual feature extraction, and the visual features with the same dimension as the input visual vectors are output. The input of the text encoder T1 is the word segmentation vector, which is mapped to the text vector by the word embedding model and then sent to 12 attention modules connected end to end for text feature extraction, and the text features are output; The text semantic enhancement modules M1 and M2 have the same structure, and input visual features and text features at the same time, and output text features that fuse visual information, wherein the input visual features include spatial visual information or spatiotemporal visual information, which are matrix multiplied with the input text features and then passed through a softmax layer, and then directly multiplied with the input text features through the result of a reshape operation, and the text features are reweighted, and finally, rich semantic information of the space and temporal information included in the visual features in the fusion process of the scraping process is obtained, and the feature representation capability of the semantic information is enriched from a more fine-grained perspective; The training of the cross-modal feature extractor further includes: inputting the video features and text features output by the cross-modal feature extractor into a similarity calculation module, wherein the video features include positive example features and negative example features, and the text features include endpoint text features and non-endpoint text features; The similarity calculation module first performs high-dimensional mapping dimension alignment on the video features and the text features through a convolution with a convolution kernel size of 1, normalizes the aligned features through a layer standardization layer to increase the model stability, outputs high-dimensional video features and text features, and respectively calculates the cosine similarities of four groups of feature pairs, namely "positive example-end point text", "negative example-end point text", "positive example-non-end point text", and "negative example-non-end point text", to obtain a similarity set composed of the cosine similarities of the four groups of feature pairs as output, wherein the cosine similarity calculation process of each group of feature pairs is as follows: s=cos(LayerNorm(proj(v)),LayerNorm(proj(c))) Where v represents the input video feature, c represents the input text feature, and s represents the similarity calculation result; The training of the cross-modal feature extractor further includes: optimizing the alignment of the cross-modal feature extractor, specifically including: The similarity set output by the similarity calculation module and the multimodal features extracted by the original cross-modal feature extractor are input into the cross-modal alignment optimization module, and the cross-modal alignment optimization module defines the similarity result of the "positive example-end point text" feature pair and the similarity result of the "negative example-non-end point text" feature pair as the positive sample pair similarity result s p , the similarity results of the "positive example-non-end point text" feature pair and the similarity results of the "negative example-end point text" feature pair are defined as the similarity results of the negative sample pair s n ,s p With s n The loss is calculated for cross-modal feature alignment, and the loss function is infoNCE loss. The loss value L is calculated as follows: Where k represents the number of samples of all video clips including positive and negative examples, s k Represents the scoring results of all video clips-text pairs, s k =s p +s n , τ represents the temperature coefficient, which is used to control the distribution of variables in the loss function, and is taken as 1 here; Perform gradient back propagation based on the loss value calculation results. During the gradient back propagation process, increase s p In order to shorten the distance between positive sample pairs in the feature space and reduce the loss value, reduce s n In order to increase the distance between negative sample pairs in the feature space and reduce the loss value, the parameters of the cross-modal feature extractor are optimized, the optimized cross-modal feature extractor is fully trained, and then the video features and text features extracted by the cross-modal feature extractor are optimized. The trained cross-modal feature extractor can map the video features and text features to the same feature space and ensure that the similarity values ​​between positive sample pairs are large and the similarity values ​​between negative sample pairs are small; The feature comparison decision module first calculates the similarity between the video segment feature and the endpoint text feature as a first similarity, and the similarity between the video segment feature and the non-endpoint text feature as a second similarity; Compare the calculation results: if the first similarity is higher than the second similarity, record the current frame number, and determine whether the previous two frames have the same result based on the frame number. If the determination is successful, determine that the current frame is the slag removal endpoint frame for the corresponding steel type. If the determination is not successful, determine that the current frame is not the slag removal endpoint frame, and output the determination result in real time.

4. An electronic device, comprising a processor and a memory, wherein at least one instruction is stored in the memory, wherein: The at least one instruction is loaded and executed by the processor to implement the weakly supervised slag removal endpoint judgment method based on cross-modal alignment as described in claim 1 or 2.

5. A computer-readable storage medium, wherein at least one instruction is stored in the storage medium, characterized in that: The at least one instruction is loaded and executed by the processor to implement the weakly supervised slag removal endpoint judgment method based on cross-modal alignment as described in claim 1 or 2.

Images