Concrete mixture detection method and system based on time sequence and attention fusion

Abstract

The invention provides a concrete mixture detection method and system based on time sequence and attention fusion, and belongs to the field of concrete detection.The method comprises the steps that a to-be-recognized concrete mixture image sequence is obtained; extracting spatial features of the to-be-identified concrete mixture image sequence frame by frame, generating time sequence features based on the spatial features, and performing feature weighting calculation on the time sequence features to obtain key frame information; and carrying out classified prediction on the key frame information to obtain a concrete mixture detection result. According to the method, the time sequence characteristics can be acquired from the to-be-identified concrete mixture image data so as to reflect the dynamic change of the concrete, and important information in the key frame can be captured for the time sequence characteristics, so that the classified prediction of the state category of the concrete mixture is realized, the detection precision and reliability of the concrete mixture are improved, and the detection accuracy and reliability of the concrete mixture are improved. And a powerful guarantee is provided for intelligent production of concrete.

Description

Technical Field

[0001] This invention belongs to the field of concrete testing, specifically relating to a method and system for testing concrete mixtures based on the fusion of temporal and attentional methods. Background Technology

[0002] The state of the concrete mixture, such as its uniformity and flowability, directly affects the quality of the concrete. Traditional testing methods involve operators visually observing the concrete's shape and flowability, and using a slump cone to test the slump. This method relies on manual labor, has large errors, and is time-consuming and labor-intensive, failing to meet the real-time and accuracy requirements of modern industrial production.

[0003] To address the aforementioned issues, existing technologies employ machine vision and deep learning-based methods for concrete mixture identification and detection, thereby achieving intelligent recognition and automated detection. However, these methods focus on the static features of images, neglecting the dynamic changes during the concrete mixing process. This makes it difficult to comprehensively capture the flow characteristics and uniformity of the concrete mixture, resulting in low detection accuracy. Summary of the Invention

[0004] To address the problem that existing concrete condition detection methods ignore dynamic changes and have low detection accuracy, this invention provides a concrete mixture detection method based on the fusion of temporal and attentional factors.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for detecting concrete mixtures based on temporal and attention fusion, comprising: Obtain the image sequence of the concrete mixture to be identified; Spatial features are extracted frame by frame from the image sequence of the concrete mixture to be identified. Temporal features are generated based on the spatial features. The temporal features are then weighted to obtain key frame information. The keyframe information is classified and predicted to obtain the detection results of the concrete mixture.

[0006] Optionally, the concrete mixture detection method based on temporal and attention fusion provided by the present invention further includes: The image sequence of the concrete mixture to be identified is input into the backbone network of the convolutional neural network, and the spatial feature sequence is extracted frame by frame. Spatial feature sequences are input into a bidirectional long short-term memory network module for bidirectional dependency extraction to generate temporal features.

[0007] Optionally, the keyframe information includes a context vector. The concrete mixture detection method based on temporal and attention fusion provided by this invention further includes: The temporal attention module performs frame-by-frame weight allocation on the temporal features to obtain the context vector.

[0008] Optionally, the backbone network is the efficient network EfficientNet. The concrete mixture detection method based on temporal and attention fusion provided by this invention further includes: The spatial feature sequence is obtained by hierarchically encoding the image sequence of concrete mixture to be identified using the efficient network EfficientNet.

[0009] Optionally, the backbone network is a ResNet50 layer. The concrete mixture detection method based on temporal and attention fusion provided by this invention further includes: The spatial feature sequence is obtained by multi-stage hierarchical spatial feature encoding of the image sequence of concrete mixture to be identified using ResNet50 layers.

[0010] Optionally, the concrete mixture detection method based on temporal and attention fusion provided by the present invention further includes: The state category of concrete mixture is obtained by mapping keyframe information through a fully connected layer. The probability distribution of the state categories of concrete mixtures is calculated by using the Softmax activation function to obtain the detection results of the concrete mixtures.

[0011] Optionally, the concrete mixture detection method based on temporal and attention fusion provided by the present invention further includes: Real-time acquisition of mixer current curves and mixer blade resistance; When both the mixer current curve and the mixer blade resistance meet the corresponding preset thresholds, the video of the concrete mixture to be identified is obtained. The video of the concrete mixture to be identified is subjected to continuous image frame extraction to obtain the image sequence of the concrete mixture to be identified.

[0012] Optionally, the concrete mixture detection method based on temporal and attention fusion provided by the present invention further includes: The region of interest is cropped from the image sequence of the concrete mixture to be identified, and the image cropping result is obtained; Data augmentation is performed on the image cropping results to obtain the data augmentation results; The data augmentation results are denoised using a bilateral filter to obtain the denoising result. The noise reduction results are normalized to obtain the preprocessed image sequence of the concrete mixture to be identified.

[0013] Optionally, the concrete mixture detection method based on temporal and attention fusion provided by the present invention further includes: Acquire training concrete mixture image data, wherein the training concrete mixture image data is pre-labeled with corresponding concrete state labels; The training concrete mixture image data is input into an untrained convolutional neural network for spatial feature extraction. Based on the spatial extraction results, the data is input into an untrained bidirectional long short-term memory network for temporal feature generation. Key frame information is highlighted through an untrained temporal attention module, and the highlighted key frame information is classified and predicted through an untrained fully connected layer to obtain the prediction results. Based on the prediction results and concrete state labels, the cross-entropy loss function is used to calculate the weight backpropagation update and model parameter optimization based on the loss value, resulting in a convolutional neural network, a bidirectional long short-term memory network, a temporal attention module, and a fully connected layer.

[0014] This invention also provides a concrete mixture detection system based on temporal and attention fusion, comprising: The image acquisition module is used to acquire image sequences of the concrete mixture to be identified; The feature extraction module is used to extract spatial features frame by frame from the image sequence of the concrete mixture to be identified, generate temporal features based on the spatial features, and perform feature weighting calculation on the temporal features to obtain key frame information. The result prediction module is used to classify and predict keyframe information to obtain the detection results of concrete mixture.

[0015] The concrete mixture detection method based on time-series and attention fusion provided by this invention has the following beneficial effects: The concrete mixture detection method based on temporal and attention fusion provided by this invention can collect temporal features from the image data of the concrete mixture to be identified, thereby reflecting the dynamic changes of the concrete. Furthermore, it can capture important information in key frames of the temporal features, realize the classification and prediction of the state category of the concrete mixture, improve the detection accuracy and reliability of concrete mixtures, and provide strong support for the intelligent production of concrete. Attached Figure Description

[0016] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is one of the schematic diagrams of a concrete mixture detection method based on temporal and attention fusion provided in an embodiment of the present invention; Figure 2This is the second schematic diagram of the concrete mixture detection method based on temporal and attention fusion provided in the embodiments of the present invention; Figure 3 This is the third schematic diagram of the concrete mixture detection method based on temporal and attention fusion provided in the embodiments of the present invention; Figure 4 The fourth schematic diagram of the concrete mixture detection method based on temporal and attention fusion provided in the embodiments of the present invention; Figure 5 The fifth schematic diagram of the concrete mixture detection method based on temporal and attention fusion provided in the embodiments of the present invention; Figure 6 This is the sixth schematic diagram of the concrete mixture detection method based on temporal and attention fusion provided in the embodiments of the present invention; Figure 7 This is the seventh schematic diagram of the concrete mixture detection method based on temporal and attention fusion provided in the embodiments of the present invention; Figure 8 Eighth schematic diagram of the concrete mixture detection method based on temporal and attention fusion provided in the embodiments of the present invention; Figure 9 Schematic diagram nine of the concrete mixture detection method based on temporal and attention fusion provided in this embodiment of the invention; Figure 10 This is a schematic diagram of the concrete mixture state detection method provided in an embodiment of the present invention; Figure 11 A schematic diagram of a concrete mixture detection system based on temporal and attention fusion provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of a concrete mixture state detection system provided in an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0019] To address the need for state detection of concrete mixtures, existing methods based on machine vision and deep learning primarily focus on the static features of images, judging the state of concrete based on only a single frame image. This ignores the dynamic changes during the concrete mixing process, making it difficult to comprehensively capture the flow characteristics and uniformity of the concrete mixture, thus leading to inaccurate detection results. Furthermore, recognition based on single frames mainly focuses on predicting slump values, making it difficult to predict other important state categories in the concrete mixture, such as bleeding and segregation, which are crucial to concrete quality. This results in low real-time performance and low prediction accuracy, failing to meet the demand for rapid and accurate feedback in the concrete production process.

[0020] Related technologies have taken into account that Long Short-Term Memory (LSTM) networks can capture temporal features in image sequences, thereby reflecting the dynamic changes of concrete, and have applied them to Convolutional Neural Networks (CNNs) for concrete state prediction. However, LSTM has limited processing capabilities for long-term temporal information, making it difficult to effectively capture important information in keyframes, resulting in limited detection accuracy.

[0021] To address the aforementioned shortcomings, this invention combines an attention mechanism to provide a concrete mixture detection method based on temporal and attention fusion. It utilizes a CNN-LSTM-Attention fusion model to extract static and dynamic features of the concrete mixture, fully leveraging temporal information in image sequences and highlighting keyframe information in long-term temporal sequences. This method can comprehensively identify various state categories in concrete mixtures, such as segregation, bleeding, low slump, good slump, and high slump. It boasts advantages of real-time performance, high precision, and automation, providing strong support for intelligent concrete production.

[0022] Example 1 This invention provides a method for detecting concrete mixtures based on the fusion of temporal and attentional methods, specifically as follows: Figure 1 As shown, it includes the following steps: Step 11: Obtain the image sequence of the concrete mixture to be identified.

[0023] Step 12: Extract spatial features frame by frame from the image sequence of the concrete mixture to be identified, generate temporal features based on the spatial features, perform feature weighting calculation on the temporal features, and obtain key frame information.

[0024] Step 13: Classify and predict the keyframe information to obtain the detection results of the concrete mixture.

[0025] Specifically, the concrete mixture detection method based on temporal and attention fusion provided by this invention first acquires image data of the concrete mixture to be identified, for example, by acquiring video information through a camera installed inside a concrete mixer, and decomposes it into image data such as image sequences. Then, the data is input into a pre-trained CNN, where the CNN extracts spatial features frame by frame, and an LSTM module generates temporal features. An Attention module performs weighted feature calculations to determine keyframe information. Subsequently, based on the Attention module, weights are assigned to the temporal features output by the LSTM module, and weighted calculations are performed to determine the keyframe information. Finally, a fully connected layer maps the keyframe information to specific concrete state categories, outputting the probability distribution of the concrete mixture to be identified, and generating the concrete mixture detection result.

[0026] Furthermore, in the concrete mixture detection method based on temporal and attention fusion provided by this invention, after the concrete mixture detection results are generated, the output results can be fed back to the concrete production system in real time, so that the concrete production system can adjust the concrete mixture formula or process in a timely manner based on the concrete mixture detection results to ensure the quality of the concrete mixture.

[0027] Because the convolutional neural network model used in this invention integrates a long short-term memory network, it can collect temporal features from the image data of the concrete mixture to be identified, thereby reflecting the dynamic changes of the concrete. Furthermore, the convolutional neural network model also integrates an attention module to capture important information in key frames, realizing the classification and prediction of the state category of the concrete mixture, improving the detection accuracy and reliability of the concrete mixture, and providing a strong guarantee for the intelligent production of concrete.

[0028] Based on the above implementation methods, the long short-term memory network module includes a bidirectional long short-term memory network module, such as... Figure 2 As shown, the concrete mixture detection method based on temporal and attention fusion provided by the present invention includes step 12 as follows: Step 121: Input the image sequence of the concrete mixture to be identified into the backbone network of the convolutional neural network, and extract the spatial feature sequence frame by frame.

[0029] Step 122: Input the spatial feature sequence into the bidirectional long short-term memory network module to extract bidirectional dependencies and generate temporal features.

[0030] Specifically, in the concrete mixture detection method based on temporal and attention fusion provided by this invention, the LSTM module can employ a Bidirectional Long Short-Term Memory (BiLSTM) network. BiLSTM has two LSTM units, forward and backward, capable of capturing the bidirectional dependencies of spatial feature sequences, thereby determining richer contextual information in the concrete mixture image sequence. After the backbone network in the CNN extracts spatial features such as edges, textures, colors, local shapes and structures, global structures, and semantic information of the concrete mixture from the image data, BiLSTM captures the bidirectional dependencies of these spatial features to obtain contextual information as temporal features.

[0031] Because the concrete mixture detection method based on temporal and attention fusion provided by this invention uses BiLSTM for LSTM, it fully utilizes the advantage of BiLSTM in capturing the bidirectional dependency of spatial feature sequences, extracts the contextual information of the image sequence, and then determines the temporal features, thereby further improving the accuracy of temporal feature extraction and ensuring the detection accuracy of concrete mixtures.

[0032] Based on the above implementation, the attention module includes a temporal attention module, and the keyframe information includes a context vector, such as... Figure 3 As shown, the concrete mixture detection method based on temporal and attention fusion provided by the present invention includes step 13 as follows: Step 131: The temporal features are weighted frame by frame using the temporal attention module to obtain the context vector.

[0033] Specifically, in the concrete mixture detection method based on temporal and attention fusion provided by this invention, the Attention module performs weighted calculations based on a temporal attention mechanism. The Attention module assigns weights to the temporal features output by LSTM, such as the features at each time step output by BiLSTM, and generates a context vector as keyframe information.

[0034] Because the concrete mixture detection method based on temporal and attention fusion provided by this invention can perform weighted calculation of temporal features and key frame information confirmation by the Attention module, it makes full use of the key information highlighting ability of the Attention mechanism to capture the dynamic features of concrete mixtures more accurately and comprehensively, thereby further improving the detection accuracy of concrete mixtures.

[0035] Based on the above implementation method, the backbone network is the EfficientNet network, such as... Figure 4 As shown, the concrete mixture detection method based on temporal and attention fusion provided by the present invention includes step 121 as follows: Step 1211: Use the EfficientNet network to perform hierarchical multi-dimensional spatial feature encoding on the image sequence of the concrete mixture to be identified, and obtain the spatial feature sequence.

[0036] Specifically, in the concrete mixture detection method based on temporal and attention fusion provided by this invention, the backbone network in the CNN can be EfficientNet. EfficientNet uses a composite scaling strategy and a Mobile Inverted Bottleneck Convolution (MBConv) module, which can perform hierarchical multi-dimensional spatial feature encoding on the image sequence of the concrete mixture to be identified, thereby obtaining spatial features such as the edge, texture, color, local shape and structure, global structure, and semantic information of the concrete mixture.

[0037] Since the concrete mixture detection method based on temporal and attention fusion provided by this invention uses EfficientNet as the backbone network, it utilizes the characteristic of EfficientNet to progressively extract and compress image data to form hierarchical feature representations to obtain spatial feature sequences, thereby improving the accuracy of spatial feature sequence extraction and thus improving the detection accuracy of concrete mixtures.

[0038] Based on the above implementation method, the backbone network is a ResNet 50-layer network, such as... Figure 5 As shown, the concrete mixture detection method based on temporal and attention fusion provided by the present invention includes step 121 as follows: Step 1212: Perform multi-stage hierarchical spatial feature encoding on the image sequence of the concrete mixture to be identified using ResNet50 layers to obtain the spatial feature sequence.

[0039] Specifically, in the concrete mixture detection method based on temporal and attention fusion provided by this invention, the backbone network in the CNN can also be ResNet50. The multiple residual block stages of ResNet50 can progressively downsample the image sequence of the concrete mixture to be identified, extract features at different levels, and thus obtain spatial features such as the edge, texture, color, local shape and structure, global structure, and semantic information of the concrete mixture.

[0040] Since the concrete mixture detection method based on temporal and attention fusion provided by this invention uses ResNet50 as the backbone network and utilizes the characteristic of ResNet50 to progressively downsample and extract features at different levels to determine the spatial feature sequence, the accuracy of spatial feature sequence extraction is improved, thereby improving the detection accuracy of concrete mixtures.

[0041] Based on the above implementation methods, such as Figure 6 As shown, the concrete mixture detection method based on temporal and attention fusion provided by the present invention includes step 13 as follows: Step 131: Map the keyframe information through a fully connected layer to obtain the state category of the concrete mixture; Step 132: Calculate the probability distribution of the state category of the concrete mixture using the Softmax activation function to obtain the detection results of the concrete mixture.

[0042] Specifically, in the concrete mixture detection method based on temporal and attention fusion provided by the present invention, after the Attention module assigns weights and performs weighted calculations to determine key frame information, such as context vectors, the fully connected layer maps the extracted context vectors to specific concrete state categories and outputs the probability distribution of the concrete state of the corresponding image sequence through the Softmax activation function.

[0043] Because the concrete mixture detection method based on temporal and attention fusion provided by this invention can also perform state mapping through a fully connected layer and calculate the probability distribution through an activation function, the recognition effect of concrete mixture is guaranteed.

[0044] Based on the above implementation methods, such as Figure 7 As shown, the concrete mixture detection method based on temporal and attention fusion provided by the present invention includes step 11 as follows: Step 111: Real-time acquisition of mixer current curve and mixer blade resistance; Step 112: When both the mixer current curve and the mixer blade resistance meet the corresponding preset thresholds, acquire the video of the concrete mixture to be identified. Step 113: Extract the image frames continuously from the video of the concrete mixture to be identified to obtain the image sequence of the concrete mixture to be identified.

[0045] Specifically, the present invention also provides a method for obtaining an image sequence of a concrete mixture to be identified: During the operation of the concrete mixer, the current curve data and blade resistance data of the concrete mixer are collected in real time and stability is judged. When the current curve data and blade resistance data of the concrete mixer both meet the corresponding preset thresholds, such as the current curve fluctuation being within the preset range and the blade resistance of the concrete mixer being within the preset range, it is determined that the concrete is mixed evenly. Video segments of the concrete mixture to be identified are continuously collected and continuous image frames are extracted to obtain an image sequence for model prediction.

[0046] In addition, during the model training phase, the operating status of the concrete mixer is determined in the same way. When the concrete mixer is running stably, video clips of the concrete mixture under different states are collected when it is evenly mixed, and the concrete state is labeled, such as segregation, bleeding, low slump, good slump, and high slump, and these are used in the model training.

[0047] Because the concrete mixture detection method based on temporal and attention fusion provided by this invention can also judge the operating status of the concrete mixer, video segments are collected only after the mixer is in a stable working state and the concrete is mixed evenly. This avoids discrepancies between the actual situation and the image acquisition and status prediction when the concrete mixture is not mixed evenly, thus ensuring the accuracy of the concrete mixture status detection.

[0048] Based on the above implementation methods, such as Figure 8 As shown, in the concrete mixture detection method based on temporal and attention fusion provided by the present invention, after step 11 and before step 12, the method further includes: Step 15: Extract the region of interest from the image sequence of the concrete mixture to be identified to obtain the image extraction result; Step 16: Perform data augmentation on the image cropping result to obtain the data augmentation result; Step 17: Denoise the data augmentation result using a bilateral filter to obtain the denoising result; Step 18: Normalize the noise reduction results to obtain the preprocessed image sequence of the concrete mixture to be identified.

[0049] Specifically, in the concrete mixture detection method based on temporal and attention fusion provided by this invention, after acquiring the image sequence converted from video clips, the image sequence can also be denoised and normalized to improve prediction accuracy. For example, after assembling the image sequence, the Region of Interest (ROI) is extracted to remove background interference. Data augmentation is then performed on the extracted image results through random cropping, rotation, and flipping to increase data diversity. A bilateral filter is then used to denoise the data augmentation results, thereby smoothing random noise. Finally, the pixel values ​​in the denoised result are normalized using Z-score or Min-Max normalization to obtain the preprocessed image sequence for input into the CNN. Furthermore, when the image sequence carries corresponding state labels, such as the state labels corresponding to the image data used for model training, the state labels are also converted into one-hot encoding to improve model prediction efficiency.

[0050] The concrete mixture detection method based on temporal and attention fusion provided by this invention also reduces noise interference in image data and improves the detection accuracy of concrete mixtures through preprocessing such as ROI extraction, data augmentation, noise reduction, and normalization.

[0051] Based on the above implementation methods, such as Figure 9 As shown, in the concrete mixture detection method based on temporal and attention fusion provided by the present invention, before step 12, the method further includes: Step 19: Obtain image data of the concrete mixture for training, wherein the image data of the concrete mixture for training is pre-labeled with corresponding concrete state labels. Step 20: Input the training concrete mixture image data into an untrained convolutional neural network for spatial feature extraction. Based on the spatial extraction results, input the data into an untrained bidirectional long short-term memory network for temporal feature generation. Highlight key frame information through an untrained temporal attention module, and classify and predict the highlighted key frame information through an untrained fully connected layer to obtain the prediction results. Step 21: Based on the prediction results and concrete state labels, calculate the cross-entropy loss function, update the weights through backpropagation and optimize the model parameters based on the loss value, and obtain the convolutional neural network, bidirectional long short-term memory network, temporal attention module and fully connected layer.

[0052] Specifically, before making predictions, the CNN model can be trained in the following ways: Pre-labeled images of concrete mixtures, each carrying a corresponding state label, are input into the CNN, LSTM, and Attention modules for training. The network weights are updated using backpropagation, and the model parameters are optimized based on the difference between the prediction results and the state labels calculated using the cross-entropy loss function, for example, through gradient descent or the Adam optimizer. The trained model weights better meet the requirements for concrete mixture detection, providing a foundation for predicting the concrete mixtures to be identified.

[0053] Since the present invention can also use image data corresponding to different states to train CNN models, the trained model predicts the state of concrete mixtures more accurately, thereby further improving the detection accuracy of concrete mixtures.

[0054] Example 2 This invention also provides a specific example of concrete mixture detection based on the fusion of temporal and attention methods, such as... Figure 10 As shown: S1. Acquire video footage of concrete mixtures in different states during homogenization inside the concrete mixer, captured by a camera, and decompose it into an image sequence: First, the uniformity of the concrete mixture is determined based on the mixer current curve and the stability of the mixer blade resistance. When the mixer current fluctuation is small and the mixing blade resistance tends to be stable, it indicates that the mixture is uniformly mixed, and video acquisition is suitable at this time. During model training, video clips of the concrete under different states of uniform mixing can be selected, such as video clips showing segregation, bleeding, low slump, good slump, and high slump. Each video clip is labeled with a corresponding state tag, and the tags are associated with the video data to obtain the concrete mixture image data for training. In the practical application stage, when the mixer current fluctuation is small and the mixing blade resistance tends to be stable, video clips are directly acquired as the concrete mixture image data to be identified. It is important to emphasize that, regardless of whether it is the concrete mixture image data for training or the concrete mixture image data to be identified, the input to the CNN is image frames. For example, continuous image frames are extracted from the acquired video clips at a frequency of 1-3 frames per second to ensure the continuity of the sequence and computational efficiency.

[0055] Video acquisition can be achieved using a high-resolution, high-frame-rate industrial-grade camera, such as a global shutter camera, to reduce motion blur and ensure that the acquired image clarity and frame rate meet the requirements of subsequent analysis. To ensure complete capture of the uniformly mixed state of the mixture, the camera is fixedly mounted on an adjustable-angle bracket, placed above or to the side of the mixer's inner wall. The specific location can be adjusted by those skilled in the art based on actual conditions, including adjusting the lens angle and shooting distance. Furthermore, to ensure uniform image brightness and reduce the impact of shadows and reflections on image data, a uniform LED light source can be used to illuminate the interior of the mixer. The specific light source intensity can be adjusted by those skilled in the art based on actual conditions; this invention does not impose any limitations.

[0056] S2. Preprocessing of the image sequence, including region of interest (ROI) cropping, data augmentation, noise reduction, normalization, and label encoding: After obtaining the image sequence, the region of interest (ROI) is first cropped to remove background interference, retaining only the concrete mixture area inside the mixer. The specific ROI can be manually or automatically defined by someone skilled in the art; it is sufficient that the ROI contains the concrete mixture to meet subsequent prediction requirements. Subsequently, the image sequence within the ROI is augmented through random cropping, rotation, and flipping to increase data diversity and improve model robustness. Next, a bilateral filter is used to denoise the image sequence, smoothing random noise and improving image quality. Pixel values ​​are then normalized to a uniform scale, such as mapping them to the [-1, 1] interval, thereby accelerating model convergence. The specific normalization method is not limited in this invention; Z-score normalization or Min-Max normalization can achieve the desired function. When the image sequence carries corresponding labels, the state labels of the concrete mixture can be converted into one-hot encoded form for easier model training.

[0057] S3: Input the preprocessed image sequence into the CNN unit to extract spatial features frame by frame; input the spatial features into the LSTM unit to generate temporal features; input the temporal features into the Attention unit to calculate weighted features and highlight key frame information; finally, output the state of the concrete mixture through a fully connected layer. The CNN unit is mainly used to extract spatial features from image sequences using a CNN model. ResNet50 or EfficientNet can be used as the backbone network, and the network depth can be adjusted by those skilled in the art based on actual needs. ResNet50 has the advantage that residual connections can effectively alleviate the gradient vanishing problem in deep network training. EfficientNet has the advantage of scaling the network depth, width, and resolution through composite coefficients, achieving higher performance and efficiency. The specific selection of the backbone network is not limited in this invention and can be determined by those skilled in the art based on actual needs. The backbone network transforms each preprocessed frame of the image into a corresponding feature vector, including spatial features such as edges, textures, colors, local shapes, global structures, and semantic information.

[0058] After the backbone network extracts spatial features, an LSTM model, such as BiLSTM, captures the forward and backward dependencies in the image sequence to meet the requirements of temporal information processing. The LSTM unit inputs the spatial feature vector of each frame output by the backbone network into the BiLSTM model in chronological order to obtain the corresponding temporal feature vector. The LSTM unit consists of multiple LSTM layers, where the output of the previous frame is fed into the corresponding LSTM layer along with the input of the current frame, repeating this process to effectively transfer temporal information.

[0059] The Attention unit is used to weight the temporal feature vector output by the BiLSTM using the Attention module, highlighting information from key frames. The Attention unit inputs the temporal feature vector output by the BiLSTM into the Attention module, which assigns a weight to each frame, making the model pay more attention to key frames that are useful for classification and weakening the influence of irrelevant frames, thereby improving the model's prediction accuracy.

[0060] Fully connected layers can map feature vectors of different dimensions to specified state categories, enabling classification and prediction of concrete mixtures based on feature extraction results such as keyframe information. The fully connected layer maps weighted context vectors to the state category space of the concrete mixture, uses the softmax function to output the probability distribution of the concrete states in the corresponding image sequence, and takes the category with the highest probability value for each state category as the prediction result.

[0061] S4: The output results are fed back to the concrete production system in real time, and adjustments are made promptly based on the output results. Once the predicted results are generated and the final concrete mix test results are determined, the test results can be output and displayed, for example, transmitted to the control interface of the concrete production system in digital or graphical form, for data storage by the concrete production system or analysis by operators. For instance, while displaying video images of the concrete mix in real time, the corresponding status category information of the test results, such as segregation, bleeding, low slump, good slump, and high slump, can be marked on the video image, allowing operators to intuitively understand the state of the concrete. For example, if the operator observes that the output status is "good slump," they can send an open command to transport the mixed concrete out.

[0062] Furthermore, on the control interface of the concrete production system, operators can manually or automatically adjust the concrete mix parameters, such as the water-cement ratio, cement dosage, or sand-aggregate ratio, to mix the concrete. These adjustments and new concrete data are also saved and used for continuous training and updating of the fusion model, further improving the prediction accuracy of the fusion model.

[0063] In summary, the concrete mixture state detection method based on the CNN-LSTM-Attention model provided in this embodiment extracts the static and dynamic features of the concrete mixture during the mixing process through image sequence analysis, and utilizes an attention mechanism to highlight key frame information, ultimately achieving accurate prediction of the concrete mixture state. This method integrates the spatial feature extraction capability of CNN, the temporal feature processing capability of LSTM, and the key information highlighting capability of the attention mechanism, thereby enabling more accurate and comprehensive capture of the dynamic features of the concrete mixture and ultimately achieving classification and prediction of multiple state categories of the concrete mixture. This classification and prediction not only better reflects the quality of the concrete mixture but also allows technicians to quickly and intuitively identify problems such as bleeding and segregation in the produced concrete, and can also predict whether the concrete slump is good. It overcomes the shortcomings of traditional methods and existing machine vision methods, thus providing a more reliable and effective solution for concrete mixture state detection.

[0064] Example 3 This invention also provides a concrete mixture detection system based on the fusion of temporal and attentional methods, such as... Figure 11 As shown, it includes: Image acquisition module 201 is used to acquire image sequences of concrete mixtures to be identified; The feature extraction module 202 is used to extract spatial features frame by frame from the image sequence of the concrete mixture to be identified, generate temporal features based on the spatial features, perform feature weighting calculation on the temporal features, and obtain key frame information. The result prediction module 203 is used to classify and predict key frame information to obtain the detection results of concrete mixture.

[0065] Specifically, such as Figure 12 As shown, this embodiment also provides a concrete mixture state detection system based on a CNN-LSTM-Attention model. This system can be integrated into the control system of a concrete production line or operate as a standalone device. The system includes:

[0066] The image acquisition module is responsible for acquiring video data of the concrete mixture inside the mixer. It includes components such as a high-definition camera, LED light source, and dustproof shell. It can be set to a timed acquisition mode or controlled manually.

[0067] The image preprocessing module is responsible for preprocessing the raw image sequences acquired by the image acquisition module.

[0068] The model building and prediction module is responsible for building, training, and deploying the CNN-LSTM-Attention model, and using the model to analyze the preprocessed image sequence to obtain the state category of the concrete mixture.

[0069] The results output and feedback module is responsible for feeding back the status category information output by the model to the concrete production system and displaying the model output results on the control interface in real time. This module can be configured with different feedback methods according to actual needs, such as real-time display and historical query. It can also automatically adjust based on the model's prediction results and can be set with an alarm function to promptly notify operators when the concrete condition is abnormal.

[0070] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0072] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for detecting concrete mixtures based on temporal and attention fusion, characterized in that, include: Obtain the image sequence of the concrete mixture to be identified; Spatial features are extracted frame by frame from the image sequence of the concrete mixture to be identified. Temporal features are generated based on the spatial features. Feature weighting calculation is performed on the temporal features to obtain key frame information. The keyframe information is classified and predicted to obtain the detection results of the concrete mixture.

2. The method for detecting concrete mixtures based on temporal and attention fusion according to claim 1, characterized in that, The step of extracting spatial features frame by frame from the image sequence of the concrete mixture to be identified, and generating temporal features based on the spatial features, includes: The image sequence of the concrete mixture to be identified is input into the backbone network of the convolutional neural network, and the spatial feature sequence is extracted frame by frame. The spatial feature sequence is input into a bidirectional long short-term memory network module for bidirectional dependency extraction to generate the temporal features.

3. The method for detecting concrete mixtures based on temporal and attention fusion according to claim 2, characterized in that, The keyframe information includes a context vector, and the keyframe information is obtained by performing feature weighting calculation on the temporal features, including: The temporal features are weighted frame by frame using a temporal attention module to obtain the context vector.

4. The method for detecting concrete mixtures based on temporal and attention fusion according to claim 2, characterized in that, The backbone network is EfficientNet, which performs hierarchical multi-dimensional spatial feature encoding on the image sequence of the concrete mixture to be identified, thereby obtaining the spatial feature sequence.

5. The method for detecting concrete mixtures based on temporal and attention fusion according to claim 2, characterized in that, The backbone network is a ResNet50 layer. The ResNet50 layer is used to perform multi-stage hierarchical spatial feature encoding on the image sequence of the concrete mixture to be identified, so as to obtain the spatial feature sequence.

6. The method for detecting concrete mixtures based on temporal and attention fusion according to claim 1, characterized in that, The classification and prediction of the keyframe information yields the following concrete mixture detection results: The keyframe information is mapped through a fully connected layer to obtain the state category of the concrete mixture. The probability distribution of the state category of the concrete mixture is calculated by using the Softmax activation function to obtain the detection result of the concrete mixture.

7. The method for detecting concrete mixtures based on temporal and attention fusion according to claim 1, characterized in that, The acquisition of the image sequence of the concrete mixture to be identified includes: Real-time acquisition of mixer current curves and mixer blade resistance; When both the mixer current curve and the mixer blade resistance meet the corresponding preset thresholds, a video of the concrete mixture to be identified is acquired. The video of the concrete mixture to be identified is subjected to continuous image frame extraction to obtain the image sequence of the concrete mixture to be identified.

8. The method for detecting concrete mixtures based on temporal and attention fusion according to claim 7, characterized in that, Before extracting spatial features frame by frame from the image sequence of the concrete mixture to be identified, the process also includes: The region of interest is cropped from the image sequence of the concrete mixture to be identified to obtain the image cropping result; The image cropping result is subjected to data augmentation to obtain the data augmentation result; The data augmentation result is denoised using a bilateral filter to obtain the denoising result. The noise reduction result is normalized to obtain the preprocessed image sequence of the concrete mixture to be identified.

9. The method for detecting concrete mixtures based on temporal and attention fusion according to claim 3, characterized in that, Before extracting spatial features frame by frame from the image sequence of the concrete mixture to be identified, the method further includes: Acquire training concrete mixture image data, wherein the training concrete mixture image data is pre-labeled with corresponding concrete state labels; The training concrete mixture image data is input into an untrained convolutional neural network for spatial feature extraction. Based on the spatial extraction results, it is input into an untrained bidirectional long short-term memory network for temporal feature generation. Key frame information is highlighted through an untrained temporal attention module, and the highlighted key frame information is classified and predicted through an untrained fully connected layer to obtain the prediction result. Based on the prediction results and the concrete state labels, the cross-entropy loss function is used to calculate the weight backpropagation update and model parameter optimization based on the loss value, resulting in the trained convolutional neural network, bidirectional long short-term memory network, temporal attention module and fully connected layer.

10. A concrete mixture detection system based on temporal and attention fusion, characterized in that, include: The image acquisition module is used to acquire image sequences of the concrete mixture to be identified; The feature extraction module is used to extract spatial features frame by frame from the image sequence of the concrete mixture to be identified, generate temporal features based on the spatial features, and perform feature weighting calculation on the temporal features to obtain key frame information. The result prediction module is used to classify and predict the keyframe information to obtain the detection results of the concrete mixture.

Images