Image-text modal scrap steel metal element detection system based on cross attention

Abstract

The invention relates to the technical field of metal element detection, and discloses an image-text modal scrap steel metal element detection system based on cross attention. Comprising an image acquisition module, a text acquisition module, a data preprocessing module, a data set division module, an image feature extraction module, a text feature extraction module, a cross attention mechanism module, a feature fusion module, a classifier module, a model evaluation and optimization module, a metal element database, a result output module and a system deployment and application module. By introducing a cross attention mechanism, visual and text features can be effectively fused, so that when the model processes metals with similar appearances, the model can depend on image features and can utilize the text features to distinguish, the recognition accuracy is remarkably improved, and the system can fully utilize information of image modals and text modals to improve the recognition accuracy. Therefore, the model can analyze the characteristics of the scrap steel more comprehensively and deeply, and the classification effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of metal element detection, and in particular to a scrap steel metal element detection system based on cross-attention graphic modality. Background Art

[0002] With the further deepening of the country's "dual carbon" strategy and increasingly stringent environmental protection requirements, the steel smelting industry faces challenges, including pressure in process, cost and management. Scrap steel, as a green and renewable resource, can replace iron ore for steelmaking, which helps reduce energy consumption, reduce environmental pollution, and achieve green and low-carbon emissions, which is of great significance to the implementation of the dual carbon strategy and national environmental protection policies.

[0003] Due to the large amount of scrap steel used and its complex types, improving steel production and smelting efficiency requires classified testing of scrap steel. An important part of scrap steel testing is scrap steel metal element testing, because the presence of impurity elements in different scrap steels will affect the performance of steel. Traditional scrap steel metal element testing relies on manual identification, which has the disadvantages of strong subjectivity and low efficiency.

[0004] With the rapid development of artificial intelligence technology, scrap steel detection methods based on machine vision have emerged and shown significant advantages. By installing high-definition shooting equipment at the unloading point of scrap metal, obtaining high-resolution images, and using deep learning models for analysis, the detection efficiency and accuracy can be effectively improved. However, the existing scrap steel detection methods based on machine vision still have limitations, as follows:

[0005] Existing scrap steel detection methods based on machine vision mainly rely on image features and have limitations when dealing with metals with similar appearances. For example, titanium alloy and aluminum alloy have similar colors and are difficult to distinguish through image features. The insufficiency of their single modal information reduces the detection accuracy and reliability of the model. Most of the existing scrap steel detection methods can identify the type of metal, but cannot detect the specific metal element composition.

[0006] Therefore, those skilled in the art provide a scrap steel metal element detection system based on cross-attention graphic modality to solve the problems raised in the above background technology. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a scrap steel metal element detection system based on cross-attention graphic modality to solve the problems in the background technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical scheme: a scrap steel metal element detection system based on cross-attention graphic modality, including an image acquisition module, a text acquisition module, a data preprocessing module, a data set division module, an image feature extraction module, a text feature extraction module, a cross-attention mechanism module, a feature fusion module, a classifier module, a model evaluation and optimization module, a metal element database, a result output module, and a system deployment and application module.

[0009] Preferably, the image acquisition module is equipped with a high-definition camera for collecting images of the scrap steel unloading site; the text acquisition module is responsible for writing detailed description texts for the images; the data preprocessing module includes image preprocessing and text preprocessing; the data set division module divides the data set into a training set, a validation set and a test set, and performs K-fold cross-validation; the image feature extraction module uses a pre-trained convolutional neural network to extract the visual features of the image; the text feature extraction module uses a pre-trained language model to encode the text and generate a text feature representation; the cross-attention mechanism module is responsible for calculating the attention weights between image and text features; the feature fusion module applies the weights calculated by the cross-attention mechanism to the image feature. The image and text features are used to generate a unified cross-modal high-dimensional feature vector; the classifier module processes the cross-modal high-dimensional feature vector to generate a probability distribution of scrap steel metal types; the model evaluation and optimization module uses a validation set to evaluate the model performance and perform tuning, and uses an independent test set to evaluate the model performance; the metal element database contains information on different metal types and their elemental compositions, and supports querying elemental composition based on metal types; the result output module determines the scrap steel type and queries the corresponding metal element composition based on the model output results, and outputs the metal type and element composition information; the system deployment and application module transmits the image and text description to the trained scrap steel detection model, and outputs the scrap steel type and metal element determination results.

[0010] Preferably, the data preprocessing module includes an image preprocessing unit and a text preprocessing unit; the image preprocessing unit is used to scale and normalize images; the text preprocessing unit is used to segment words, remove useless information, and standardize text formats;

[0011] The data set partitioning module includes a training set partitioning unit, a test set partitioning unit, and a K-fold cross-validation unit; the training set partitioning unit partitions 80% of the data set into a training set for model training; the test set partitioning unit partitions 20% of the data set into a test set for model evaluation; the K-fold cross-validation unit is used to perform K-fold cross-validation on the training set.

[0012] Preferably, the model evaluation and optimization module includes a performance evaluation unit, a model tuning unit, and a final test unit; the performance evaluation unit uses a validation set to evaluate model performance; the model tuning unit performs model tuning according to the evaluation results; and the final test unit uses an independent test set to evaluate final model performance;

[0013] The metal element database includes a data collection unit, a data entry unit, and a query unit; the data collection unit collects information on different metals and their elemental compositions from sources such as scientific literature, metal material manuals, and chemical databases; the data entry unit enters the collected metal types and their elemental composition information into the database; the query unit queries the database based on the predicted metal types to obtain the corresponding elemental composition information.

[0014] Preferably, the result output module includes a result generation unit, an element composition query unit, and an output unit; the result generation unit determines the type of scrap steel according to the model output result; the element composition query unit obtains the corresponding metal element composition information from the metal element database; the output unit outputs the metal type and element composition information;

[0015] The system deployment and application module includes an image and text input unit and a result return unit; the image and text input unit inputs the image and text description into the trained scrap steel detection model; the result return unit system returns the determination result of the scrap steel type and metal element according to the model output result.

[0016] Preferably, the image feature extraction module uses a pre-trained convolutional neural network to extract image features. Specifically, the formulas for the convolution operation, pooling operation and fully connected layer are as follows:

[0017] in, is the value of the (i, j) position of the kth feature map in the feature map, x is the input image, w is the convolution kernel, b is the bias, and σ is the activation function;

[0018] Pooling operation: Where x is the input image and the pooling window size is m*n.

[0019] Preferably, the cross attention mechanism module is used to calculate the attention weights between image and text features to achieve feature alignment and fusion between modalities. The attention mechanism formula is:

[0020] Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key vector.

[0021] Preferably, the feature fusion module applies the weights calculated by the cross-attention mechanism to the image and text features to generate a unified cross-modal high-dimensional feature vector, and its weighted fusion formula is:

[0022] F fused =αF image +βF text , where α and β are attention weights, F image and F text are image and text features.

[0023] Preferably, the classifier module processes the cross-modal high-dimensional feature vector to generate a probability distribution of the types of scrap metals, and the classifier formula is:

[0024] y = softmax(W2·ReLU(W1·F fused +b1)+b2), where W1 and W2 are weight matrices, b1 and b2 are bias vectors, and F fused is the fused feature vector.

[0025] The present invention provides a scrap steel metal element detection system based on cross-attention graphic modality, which has the following beneficial effects:

[0026] 1. The present invention can effectively integrate visual and text features by introducing a cross-attention mechanism, so that when the model processes metals of similar appearance, it can rely on image features while using text features to distinguish them, significantly improving recognition accuracy. In addition, the system can make full use of information from image and text modalities, so that the model can analyze the characteristics of scrap steel more comprehensively and deeply, thereby improving classification effects.

[0027] 2. The present invention overcomes the problem of insufficient single modal information by introducing text features and cross-attention mechanisms, enhances the robustness and generalization ability of the model, and can detect the specific metal element composition while identifying the type of scrap steel. By establishing a metal element database, the system can query the corresponding element composition information according to the predicted metal type, providing an important reference for effectively removing harmful elements in the smelting process and improving steel production and quality.

[0028] 3. By accurately detecting the types and specific elemental composition of scrap steel, the present invention's system can help the smelting process better control the raw material composition, effectively remove elements harmful to steel, improve smelting efficiency and steel quality, and comply with the requirements of the national environmental protection and dual-carbon strategy. It also uses artificial intelligence technology to complete the detection and classification of scrap steel through machine automation, reduce reliance on manual identification, improve detection efficiency and consistency, and reduce the risk of misjudgment caused by subjective factors.

[0029] 4. Through the consistency and complementarity of multimodal data, the present invention can provide more accurate classification and element detection results when faced with scrap steel of different types, states and uses. It has strong adaptability and can operate stably in complex and changeable practical application scenarios. In addition, the system continuously collects and accumulates scrap steel images and text data, uses advanced deep learning and cross-attention mechanisms, and can continuously optimize model performance and gradually improve the accuracy and efficiency of scrap steel detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of the present invention;

[0031] Figure 2 It is a structural diagram of the present invention;

[0032] Figure 3 It is a schematic diagram of a data preprocessing module of the present invention;

[0033] Figure 4 It is a schematic diagram of the model evaluation and optimization module of the present invention;

[0034] Figure 5 It is a schematic diagram of the metal element database of the present invention;

[0035] Figure 6 It is a schematic diagram of a result output module of the present invention;

[0036] Figure 7 It is a schematic diagram of system deployment and application of the present invention;

[0037] Figure 8 It is a schematic diagram of the model structure of the present invention;

[0038] Fig. 9 It is a schematic diagram of the model structure unit of the present invention;

[0039] Fig.10 It is a schematic diagram of the connection of the model structure units of the present invention. DETAILED DESCRIPTION

[0040] In order to make the technical personnel in the technical field understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in combination with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a partial embodiment of the present invention, not a complete embodiment. Based on the embodiment of the present invention, other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0041] The present invention is described in detail below in conjunction with the accompanying drawings:

[0042] Example:

[0043] Please see attached Figure 1- Attachment Figure 6 The embodiment of the present invention provides a scrap metal element detection system based on cross-attention graphic and text modality, including an image acquisition module, a text acquisition module, a data preprocessing module, a data set division module, an image feature extraction module, a text feature extraction module, a cross-attention mechanism module, a feature fusion module, a classifier module, a model evaluation and optimization module, a metal element database, a result output module, and a system deployment and application module; the image acquisition module is equipped with a high-definition camera device for collecting images of the scrap steel unloading site; the text acquisition module is responsible for writing a detailed description text for the image; the data preprocessing module includes image preprocessing and text preprocessing; the data set division module divides the data set into a training set, a validation set, and a test set, and performs K-fold cross validation; the image feature extraction module uses a pre-trained convolutional neural network to extract the visual features of the image; the text feature extraction module uses a pre-trained language model to extract the text The cross-attention mechanism module is responsible for calculating the attention weights between image and text features; the feature fusion module applies the weights calculated by the cross-attention mechanism to image and text features to generate a unified cross-modal high-dimensional feature vector; the classifier module processes the cross-modal high-dimensional feature vector to generate the probability distribution of scrap metal types; the model evaluation and optimization module uses the validation set to evaluate the model performance and perform optimization, and uses an independent test set to evaluate the model performance; the metal element database contains information on different metal types and their elemental composition, and supports querying elemental composition based on metal types; the result output module determines the type of scrap steel and queries the corresponding metal element composition based on the model output results, and outputs the metal type and element composition information; the system deployment and application module transfers the image and text descriptions to the trained scrap steel detection model, and outputs the scrap steel type and metal element determination results.

[0044] The benefits of the image acquisition module are as follows: the high-definition camera equipment ensures the image quality and provides a reliable data basis for subsequent feature extraction and recognition; the benefits of the text acquisition module supplement the lack of image information, provide rich text features for multimodal fusion, and help improve the accuracy of classification; the benefits of the data preprocessing module standardize the data format and improve the efficiency and effect of model training and feature extraction; the benefits of the data set division module reasonably divide the data to ensure the generalization ability and reliability of model training, avoid overfitting, and improve model performance; the benefits of the image feature extraction module effectively extract the local and global features of the image and improve the understanding and recognition of scrap steel images; the benefits of the text feature extraction module capture the semantic information of the text, generate high-quality text feature vectors, and provide support for multimodal fusion; the benefits of the cross-attention mechanism module bridge the gap between visual models The semantic gap between state and language modality is improved, the fusion effect of multimodal data is improved, and the recognition accuracy of the model is enhanced; the benefit of the feature fusion module is to fuse image and text information, generate a more comprehensive feature representation, and improve the accuracy of scrap steel classification; the benefit of the classifier module is to realize the accurate classification of scrap steel types through multi-layer perceptron; the benefit of the model evaluation and optimization module is to ensure the performance of the model in practical applications, and continuously improve the accuracy and robustness of the model through tuning; the benefit of the metal element database is to provide a comprehensive metal element information library to provide a basis for the detection and query of specific element composition; the benefit of the result output module is to realize the intuitive display of scrap steel detection results, which is convenient for users to understand and apply the detection results; the benefit of the system deployment and application module is to ensure the actual deployment and application of the system, so that the detection model can run effectively in the actual environment.

[0045] The data preprocessing module includes an image preprocessing unit and a text preprocessing unit; the image preprocessing unit is used to scale and normalize images; the text preprocessing unit is used to segment words, remove useless information, and standardize text formats; the data set partitioning module includes a training set partitioning unit, a test set partitioning unit, and a K-fold cross-validation unit; the training set partitioning unit divides 80% of the data set into training sets for model training; the test set partitioning unit divides 20% of the data set into test sets for model evaluation; the K-fold cross-validation unit is used to perform K-fold cross-validation on the training set; the model evaluation and optimization module includes a performance evaluation unit, a model tuning unit, and a final test unit; the performance evaluation unit uses a validation set to evaluate model performance; the model tuning unit performs model tuning based on the evaluation results; the final test unit uses an independent test set to evaluate the final model performance; the metal element database includes a data collection unit, a data entry unit, and a query unit The data collection unit collects information on different metals and their elemental compositions through scientific literature, metal material manuals, chemical databases and other sources; the data entry unit enters the collected metal types and their elemental composition information into the database; the query unit queries the database according to the predicted metal types to obtain the corresponding elemental composition information; the result output module includes a result generation unit, an elemental composition query unit, and an output unit; the result generation unit determines the type of scrap steel according to the model output results; the elemental composition query unit obtains the corresponding metal element composition information from the metal element database; the output unit outputs the metal type and elemental composition information; the system deployment and application module includes an image and text input unit and a result return unit; the image and text input unit inputs the image and text description into the trained scrap steel detection model; the result return unit system returns the determination result of the scrap steel type and metal element according to the model output results.

[0046] The benefits of the image preprocessing unit are to standardize the image size and pixel value, improve the consistency and efficiency of the image input model, and enhance the learning effect of the model; the benefits of the text preprocessing unit are to improve the quality and consistency of text data, and ensure that the model can correctly understand and process text features; the benefits of the training set division unit are to provide sufficient training data to improve the learning ability and accuracy of the model; the benefits of the test set division unit are to provide an independent data set for evaluating the performance of the model in practical applications and ensure the generalization ability of the model; the benefits of K-fold cross-validation are to reduce overfitting and improve the robustness and generalization ability of the model through multiple training and verification; the benefits of the performance evaluation unit are to monitor the performance of the model during the training process, discover and solve problems in a timely manner, and ensure the stability and performance of the model; the benefits of the model tuning unit are to optimize model parameters and improve the accuracy and performance of the model; the benefits of the final test unit are to comprehensively evaluate the final performance of the model , ensuring the reliability and effectiveness of the model in a real environment; the data collection unit benefits provide a comprehensive and authoritative data source to ensure the accuracy and comprehensiveness of the database information; the data entry unit benefits establish and maintain a metal element database to provide data support for subsequent queries and tests; the query unit benefits quickly and accurately provide the required element composition information to support scrap steel detection and classification; the result generation unit benefits provide intuitive classification results for easy user understanding and application; the element composition query unit benefits supplement the scrap steel type information and provide detailed element composition information to support subsequent smelting and processing; the output unit benefits provide a complete test report for user reference and decision-making; the image and text input unit benefits ensure that data is smoothly input into the model and start the scrap steel detection process; the result return unit benefits provide test results in real time, support rapid decision-making and processing, and improve the efficiency of scrap steel classification and processing.

[0047] The image feature extraction module uses a pre-trained convolutional neural network to extract image features. Specifically, the formulas for convolution operation, pooling operation, and fully connected layer are as follows:

[0048] in, is the value of the (i, j) position of the kth feature map in the feature map, x is the input image, w is the convolution kernel, b is the bias, and σ is the activation function;

[0049] Pooling operation: Where x is the input image and the pooling window size is m*n.

[0050] The cross-attention mechanism module is used to calculate the attention weights between image and text features to achieve feature alignment and fusion between modalities. The attention mechanism formula is:

[0051] Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key vector.

[0052] The feature fusion module applies the weights calculated by the cross-attention mechanism to the image and text features to generate a unified cross-modal high-dimensional feature vector. Its weighted fusion formula is:

[0053] F fused =αF image +βF text , where α and β are attention weights, F image and F text are image and text features.

[0054] The classifier module processes the cross-modal high-dimensional feature vector to generate the probability distribution of scrap metal types, and the classifier formula is:

[0055] y=softmax ( W2·ReLU(W1·F fused +b1)+b2), where W1 and W2 are weight matrices, b1 and b2 are bias vectors, and F fused is the fused feature vector.

[0056] Please see attached Figure 7 and attached Figure 8 ,The specific model architecture of the visual language task model based on cross attention is as follows: the model includes image input embedding layer, text input embedding layer, visual encoder, language encoder, and cross-modal encoder;

[0057] The model has images and texts: pictures and words. The input embedding layer converts images and texts into word-level sentence embeddings and target-level graph embeddings.

[0058] The text input embedding layer consists of a word segmenter, a position encoder, a word encoder, and a normalization layer;

[0059] After the sentence is input into the text input embedding layer, it is first divided into n words by the tokenizer: w1, w2, …, w n} , the word encoder converts the classified words into vectors of fixed length, the formula is:

[0060] The position encoder obtains the absolute position of each word in the sentence and converts it into a vector of fixed length. Then add the vectors obtained by position encoding and word encoding and normalize them

[0061] The image input embedding layer consists of an object detector and a feature extractor;

[0062] Object detectors are usually pre-trained object detection models that can identify objects of most categories. Object detectors detect targets on images: {O1,…,O j,…,O m}, each target is represented as a position feature p j and ROL feature f j , the target represents the following output after the fully connected layer and the normalization layer:

[0063]

[0064]

[0065] The visual encoder and language encoder of the model are both composed of N stacked unimodal encoders. Each unimodal encoder is shown in the figure and includes the following: self-attention layer, residual connection, normalization layer, and feedforward layer.

[0066] For the visual encoder, the input is v j , for the language encoder, the input is h i In the self-attention layer, the attention weights between the features are calculated to capture the relationship between the features. In the feedforward layer, the output of the attention layer is nonlinearly transformed to extract high-level features. In the normalization layer, the output of the sub-layer and the input that has not passed through the sub-layer are residually connected, that is:

[0067] LayerNorm(x+sublayer(x));

[0068] The cross-modal encoder is composed of an N-bit stacked structure, that is, the output of the kth layer is used as the input of the k+1th layer. The individual cross-modal encoder contains a bidirectional cross-attention layer, a self-attention layer, and a feed-forward layer, as shown in the attached figure. Fig. 9 As shown;

[0069] The core part of the cross-modal encoder is the cross-attention layer, which is from language to vision and from vision to language. The attention mechanism is to find information related to the query vector in the context vector. The k-1th layer outputs the query vector and the context vector (for example, the context vector Query Vector ), input to the cross attention layer of the kth layer:

[0070] The cross-attention layer is used to exchange information and align the modalities to learn cross-modal representations. To further establish the internal relationship between the modalities, the output of the cross-attention layer is input into the self-attention layer. The self-attention layer obtains:

[0071]

[0072] Finally, the output of the cross-modal encoder of the kth layer is generated by the feed-forward layer, and each sub-layer is followed by a residual connection and a normalization layer, similar to the single-modal encoder, as shown in the attached figure. Fig.10 shown.

[0073] The cross-modal encoder has outputs, namely language, vision and multimodal outputs. The language and visual outputs are generated by the cross-modal encoder, and a special tag is added before the word segmenter operation. The corresponding feature vector of the special tag in the language feature sequence is regarded as the cross-modal output. Finally, the text output of the cross-modal encoder will obtain the metal type.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A scrap steel metal element detection system based on cross-attention graphic modality, characterized in that: It includes image acquisition module, text acquisition module, data preprocessing module, data set partitioning module, image feature extraction module, text feature extraction module, cross-attention mechanism module, feature fusion module, classifier module, model evaluation and optimization module, metal element database, result output module, and system deployment and application module.

2. According to claim 1, a scrap steel metal element detection system based on cross-attention graphic modality is characterized in that: The image acquisition module is equipped with a high-definition camera for collecting images of the scrap steel unloading site; the text acquisition module is responsible for writing detailed description texts for the images; the data preprocessing module includes image preprocessing and text preprocessing; the data set division module divides the data set into a training set, a validation set and a test set, and performs K-fold cross-validation; the image feature extraction module uses a pre-trained convolutional neural network to extract the visual features of the image; the text feature extraction module uses a pre-trained language model to encode the text and generate a text feature representation; the cross-attention mechanism module is responsible for calculating the attention weights between image and text features; the feature fusion module applies the weights calculated by the cross-attention mechanism to the image and text features. The text features are used to generate a unified cross-modal high-dimensional feature vector; the classifier module processes the cross-modal high-dimensional feature vector to generate a probability distribution of scrap steel metal types; the model evaluation and optimization module uses a validation set to evaluate the model performance and perform tuning, and uses an independent test set to evaluate the model performance; the metal element database contains information on different metal types and their elemental compositions, and supports querying elemental composition based on metal types; the result output module determines the scrap steel type and queries the corresponding metal element composition based on the model output results, and outputs the metal type and element composition information; the system deployment and application module transmits the image and text description to the trained scrap steel detection model, and outputs the scrap steel type and metal element determination results.

3. The scrap steel metal element detection system based on cross-attention graphic modality according to claim 1 is characterized in that: The data preprocessing module includes an image preprocessing unit and a text preprocessing unit; the image preprocessing unit is used to scale and normalize images; the text preprocessing unit is used to segment words, remove useless information, and standardize text formats; The data set partitioning module includes a training set partitioning unit, a test set partitioning unit, and a K-fold cross-validation unit; the training set partitioning unit partitions 80% of the data set into a training set for model training; the test set partitioning unit partitions 20% of the data set into a test set for model evaluation; the K-fold cross-validation unit is used to perform K-fold cross-validation on the training set.

4. The scrap steel metal element detection system based on cross-attention graphic modality according to claim 1 is characterized in that: The model evaluation and optimization module includes a performance evaluation unit, a model tuning unit, and a final test unit; the performance evaluation unit uses a validation set to evaluate model performance; the model tuning unit performs model tuning according to the evaluation results; and the final test unit uses an independent test set to evaluate final model performance. The metal element database includes a data collection unit, a data entry unit, and a query unit; the data collection unit collects information on different metals and their elemental compositions from sources such as scientific literature, metal material manuals, and chemical databases; the data entry unit enters the collected metal types and their elemental composition information into the database; the query unit queries the database based on the predicted metal types to obtain the corresponding elemental composition information.

5. The scrap steel metal element detection system based on cross-attention graphic modality according to claim 1 is characterized in that: The result output module includes a result generation unit, an element composition query unit, and an output unit; the result generation unit determines the type of scrap steel according to the model output result; The element composition query unit obtains corresponding metal element composition information from the metal element database; the output unit outputs the metal type and element composition information; The system deployment and application module includes an image and text input unit and a result return unit; the image and text input unit inputs the image and text description into the trained scrap steel detection model; the result return unit system returns the determination result of the scrap steel type and metal element according to the model output result.

6. The scrap steel metal element detection system based on cross-attention graphic modality according to claim 1 is characterized in that: The image feature extraction module uses a pre-trained convolutional neural network to extract image features. Specifically, the formulas for the convolution operation, pooling operation, and fully connected layer are as follows: in, is the value of the (i, j) position of the kth feature map in the feature map, x is the input image, w is the convolution kernel, b is the bias, and σ is the activation function; Pooling operation: Where x is the input image and the pooling window size is m*n.

7. The scrap steel metal element detection system based on cross-attention graphic modality according to claim 1 is characterized in that: The cross-attention mechanism module is used to calculate the attention weights between image and text features to achieve feature alignment and fusion between modalities. The attention mechanism formula is: Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key vector.

8. The scrap steel metal element detection system based on cross-attention graphic modality according to claim 1 is characterized in that: The feature fusion module applies the weights calculated by the cross-attention mechanism to the image and text features to generate a unified cross-modal high-dimensional feature vector. The weighted fusion formula is: F fused =αF image +βF text , where α and β are attention weights, F image and F text are image and text features.

9. The scrap steel metal element detection system based on cross-attention graphic modality according to claim 1 is characterized in that: The classifier module processes the cross-modal high-dimensional feature vector to generate the probability distribution of scrap metal types, and the classifier formula is: y = softmax(W2·ReLU(W1·F fused +b1)+b2), where W1 and W2 are weight matrices, b1 and b2 are bias vectors, and F fused is the fused feature vector.

Images