Interactive question and answer method and system for b-scan images

Abstract

The application discloses an interactive question and answer method for B-scan images, comprising the following steps: obtaining ground penetrating radar B-scan images containing at least two categories of disease targets; analyzing the B-scan images obtained in step S1 to obtain disease information in the images, and generating question-answer pairs according to the disease information; constructing an initial B-scan image interpretation question and answer model, training the same, and obtaining a B-scan image interpretation question and answer model; inputting the actually collected ground penetrating radar B-scan images and questions into the B-scan image interpretation question and answer model to obtain a question-aiding image and a corresponding natural language answer. The application analyzes the B-scan images, generates question and answer pairs as a training set, constructs an initial B-scan image interpretation question and answer model and optimizes the same, and finally realizes end-to-end B-scan image inversion and natural language interactive question and answer through the model.

Description

Technical Field

[0001] This invention relates to the field of ground-penetrating radar detection, and specifically to an interactive question-and-answer method and system for B-scan images. Background Technology

[0002] Ground penetrating radar (GPR) is a non-destructive testing technique that uses high-frequency electromagnetic waves to detect underground targets. It has advantages such as non-destructiveness, high resolution, and high efficiency. However, GPR data must be interpreted and processed to convert it into intuitive geological structure information before it can be used.

[0003] Traditional interpretation methods rely primarily on human experience, identifying targets by observing hyperbolic scattering characteristics, reflection intensity, and phase changes in B-scan images. This approach is inefficient, highly subjective, and struggles to handle multi-target interference and signal attenuation issues in complex geological environments.

[0004] In recent years, deep learning technologies, represented by YOLOvX and the U-Net series architecture with its encoder-decoder structure, have improved the efficiency of GPR data interpretation, enabling the automatic output of the location and classification results of all diseases in one go. However, the output information is difficult to keep in line with actual engineering needs, and the model cannot conduct multiple rounds of follow-up questions like human experts, making it impossible to interactively refine the interpretation results.

[0005] Visual question answering (VQA) technology, as a cutting-edge direction of multimodal artificial intelligence, enables models to reason directly from images and provide accurate answers using natural language based on questions posed by users and related to the input image. However, existing VQA technologies still struggle to adapt to and process GPR B-scan images with high noise, low texture, and hyperbolic shape characteristics, and also have limitations when facing specialized geological questions involving quantitative attributes such as "quantity," "depth," and "size." Summary of the Invention

[0006] One of the objectives of this invention is to construct an interactive question-answering method for B-scan images, enabling the model to automatically generate and output corresponding natural language answers based on the user-input B-scan image and the input question.

[0007] The interactive question-and-answer method for B-scan images provided by this invention includes the following steps:

[0008] S1. Acquire ground-penetrating radar B-scan images containing at least two categories of diseased targets;

[0009] S2. Analyze the B-scan image obtained in step S1 to obtain the disease information in the B-scan image, and generate question-answer pairs including existence features, quantity features, attribute features and relationship features based on the disease information to construct a training set;

[0010] S3. Construct the initial B-scan image interpretation question answering model, including the front-end processing module, the B-scan image interpretation question answering core sub-network, and the back-end output module;

[0011] A front-end processing module is constructed based on a multi-channel image enhancement strategy to process ground-penetrating radar B-scan images and question-answer pairs, so as to construct the input features for training the core sub-network of B-scan image interpretation and question answering.

[0012] A B-scan image interpretation question answering core sub-network is constructed based on a deep feature modeling and fusion mechanism to extract and fuse input features, thereby obtaining parsed image data and answer text data as intermediate representations.

[0013] Based on the multi-channel feature element-wise addition and dictionary decoding mechanism, a back-end output module is constructed to reconstruct the parsed image data and answer text data respectively, and generate the predictive question auxiliary image and the predictive natural language answer, which serve as the final output of the initial B-scan image interpretation question answering model;

[0014] S4. Train the initial B-scan image interpretation question answering model using the training set to obtain the B-scan image interpretation question answering model;

[0015] S5. The user inputs the actual collected ground-penetrating radar B-scan image and the question into the B-scan image interpretation question-and-answer model to obtain the question-assisting image and the corresponding natural language answer, thus completing the interactive question-and-answer for the B-scan image.

[0016] Step S2 specifically includes the following steps:

[0017] A1. Analyze the B-scan image to obtain information about the disease targets in the image, including: the relative permittivity, burial depth, location, size, shape and number of the disease targets;

[0018] A2. For any one of the various disease targets, generate a first-level question-and-answer pair to ask and answer whether the disease target of the current category exists; if so, generate the corresponding answer and proceed to step A3.

[0019] If not, for the disease target of the current category that does not exist, set the answers to all its questions to the default answers, and check whether all categories have completed the traversal of steps A2 to A5. If yes, proceed to step A6; otherwise, continue to loop through steps A2 to A5.

[0020] A3. For existing disease targets of the current category, generate a second-level question-and-answer pair to inquire about their physical attributes, perform corresponding checks based on the questions asked, and then generate corresponding answers based on the check results; the physical attributes include the location, shape, and size of the disease target;

[0021] Determine whether all samples in the current category have completed the traversal of step A3. If yes, proceed to step A4; otherwise, continue to loop through step A3.

[0022] A4. If all samples in the current category have completed the traversal in step A3, count the number of samples in the current category; then generate a third-level question-and-answer pair based on the statistical results, which is used to ask and answer the question about the number of samples in the current category.

[0023] A5. If the number of samples in the current category is greater than one, then for the current category, check the geometric relationship between each disease target in the same category of samples, and then generate a fourth-level question-and-answer pair based on the check results, which is used to ask and answer questions about the geometric relationship between each disease target in the same category of samples; the geometric relationship includes spatial relationship, scale difference and morphological association;

[0024] If the number of samples in the current category is less than or equal to one, then for the current category, set the default answer to the question of the geometric relationship between each disease target in the same category of samples;

[0025] Check if all categories have completed the traversal of steps A2 to A5. If yes, proceed to step A6; otherwise, continue to loop through steps A2 to A5.

[0026] A6. If all categories have completed the traversal of steps A2 to A5, count the number of categories; if the number of categories is greater than one, generate a fifth-level question-and-answer pair to inquire about the geometric relationship between different categories of disease targets, perform corresponding checks according to the questions, and generate corresponding answers based on the check results.

[0027] If the number of disease target categories is less than or equal to one, then set the default answer to the geometric relationship question between different disease target categories.

[0028] The front-end processing module described in step S3 performs the following processing steps on the B-scan image in sequence:

[0029] The B-scan image is subtracted from the background data and then resized to obtain the adjusted B-scan image. The adjusted B-scan image is then normalized and histogram equalized to obtain the normalized B-scan image. The adjusted B-scan image is then stitched together with the normalized B-scan image data to generate a two-channel B-scan image.

[0030] The B-scan image is mapped to a dielectric constant map, and background clutter is suppressed. The dielectric constant map after background clutter suppression is separated according to the type of disease target to obtain a separated dielectric constant map containing only void disease targets and another separated dielectric constant map containing only crack disease targets. The two separated dielectric constant maps are resized and then stitched together to form a two-channel dielectric constant image.

[0031] The front-end processing module performs the following processing steps on the question-and-answer set in sequence:

[0032] Construct a dictionary containing commonly used terms and phrases in the field of ground penetrating radar; encode the questions in the question-and-answer set according to the dictionary to obtain question codes; encode the answers in the question-and-answer set according to the dictionary to obtain answer tag codes;

[0033] The two-channel B-scan image, the two-channel dielectric constant image, the question code, and the answer label code are used as input features for training the B-scan image interpretation question-answering core sub-network.

[0034] The B-scan image interpretation question answering core sub-network described in step S3 includes: an image encoding layer, an image stitching regularization layer, a text encoding layer, a text regularization layer, a cross-modal fusion layer, and an answer decoding layer;

[0035] The output of the image encoding layer is used as the input of the image stitching regularization layer, the output of the text encoding layer is used as the input of the text regularization layer, the output of the image stitching regularization layer and the output of the text regularization layer are used together as the input of the cross-modal fusion layer, and the output of the cross-modal fusion layer is used as the input of the answer decoding layer.

[0036] The processing procedure of the B-scan image interpretation question answering core subnetwork is as follows:

[0037] The two-channel B-scan image is input into the image coding layer and processed to obtain intermediate image features, final image features, and the two-channel inverted image; the intermediate image features and the final image features are input into the image stitching regularization layer and processed to obtain the second image stitching features;

[0038] The question encoding is input into the text encoding layer and processed to obtain the statement vector; the statement vector is input into the text regularization layer and processed to obtain the regularized statement vector.

[0039] The second image stitching features and the regularized sentence vector are input into the cross-modal fusion layer and processed to obtain the fused feature vector; the fused feature vector is then input into the answer decoding layer and processed to obtain the answer encoding.

[0040] The B-scan image interpretation question-answering core subnetwork outputs a two-channel inverted image and answer encoding.

[0041] The image encoding layer includes an image encoder, an image decoder, and an image projection module;

[0042] The image encoder includes: a first image encoding module to a fifteenth image encoding module.

[0043] The first and second image coding modules have the same structure, each including: a 3x3 convolutional layer, a batch normalization layer and a ReLU activation function layer in sequence.

[0044] The third, sixth, ninth, and twelfth image coding modules have the same structure, and each module includes a residual module and a hole space pyramid pooling layer connected in series.

[0045] The fourth, seventh, tenth, and thirteenth image coding modules have the same structure, and each module includes a max pooling layer.

[0046] The fifteenth image coding module includes a hollow space pyramid pooling layer;

[0047] The fifth and eighth image coding modules have the same structure. Each module includes a two-branch pyramid convolutional module, a batch normalization layer, a ReLU activation function layer, a multi-scale depth convolution with spatially selective feature calibration, a batch normalization layer, and a ReLU activation function layer connected in sequence.

[0048] The eleventh and fourteenth image coding modules have the same structure. Each module includes a four-branch pyramid convolutional module, a batch normalization layer, a ReLU activation function layer, a multi-scale depth convolution with spatially selective feature calibration, a batch normalization layer, and a ReLU activation function layer connected in sequence.

[0049] The image decoder includes: a first image decoding module to a seventh image decoding module.

[0050] Among them, the first, third, fifth and seventh image decoding modules have the same structure, and each module includes a two-dimensional transposed convolutional layer;

[0051] The second, fourth, and sixth image decoding modules have the same structure. Each module includes a four-branch pyramid convolutional module, a batch normalization layer, a ReLU activation function layer, a multi-scale depth convolution with spatially selective feature calibration, a batch normalization layer, and a ReLU activation function layer connected in sequence.

[0052] The first to third image encoding modules are connected in series. The output of the previous image encoding module is used as the input of the next image encoding module. The output of the third image encoding module is connected to the output of the seventh image decoding module and then used as the input of the image projection module.

[0053] Meanwhile, the output of the third image encoding module also serves as the input of the fourth image encoding module; the fourth to sixth image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the sixth image encoding module is connected to the output of the fifth image decoding module and then serves as the input of the sixth image decoding module; the output of the sixth image decoding module serves as the input of the seventh image decoding module.

[0054] Meanwhile, the output of the sixth image encoding module also serves as the input of the seventh image encoding module; the seventh to ninth image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the ninth image encoding module is connected to the output of the third image decoding module and then serves as the input of the fourth image decoding module; the output of the fourth image decoding module serves as the input of the fifth image decoding module.

[0055] Meanwhile, the output of the ninth image encoding module also serves as the input of the tenth image encoding module; the tenth to twelfth image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the twelfth image encoding module is connected to the output of the first image decoding module and then serves as the input of the second image decoding module; the output of the second image decoding module serves as the input of the third image decoding module.

[0056] Meanwhile, the output of the twelfth image encoding module also serves as the input of the thirteenth image encoding module; the thirteenth to fifteenth image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the fifteenth image encoding module serves as the input of the first image decoding module;

[0057] The image projection module includes: a first projection submodule and a second projection submodule connected in series. The structure of the first projection submodule is the same as that of the fifth image encoding module. The structure of the second projection submodule includes a 1x1 convolutional layer and a Sigmoid activation function layer connected in series.

[0058] The processing procedure of the image coding layer is as follows:

[0059] Two-channel B-scan images are acquired, features are extracted and downsampled step by step through the image encoding module, and intermediate image features are output from the fifteenth image encoding module.

[0060] The intermediate image features are input into the first image decoding module. The decoding features are upsampled step by step by the first to seventh image decoding modules. The feature maps corresponding to each stage of the encoder are fused with the decoding features through skip connections. Finally, the final image features are output from the seventh image decoding module.

[0061] The final image features are input into the image projection module and processed to obtain a two-channel inversion image corresponding to the size of the two-channel B-scan image.

[0062] The image coding layer outputs a two-channel inverted image;

[0063] The image stitching regularization layer includes: a size transformation layer, a stitching layer, a convolutional layer, and a random deactivation layer;

[0064] The final image features are input into the size transformation layer for processing to obtain the size-adjusted image features; the size-adjusted image features are then combined with the intermediate image features through a stitching layer to obtain the first image stitching features; the first image stitching features are then extracted through a convolutional layer and regularized through a random deactivation layer to obtain the second image stitching features.

[0065] The text encoding layer is constructed based on the recurrent neural network architecture Skip-Thoughts; the text regularization layer is constructed based on a random deactivation layer.

[0066] The cross-modal fusion layer includes a first linear layer and a second linear layer, a first normalization layer and a second normalization layer, a multi-head attention module, a splicing layer, a dynamic gating module, a gated weighting layer, and a feedforward network; wherein, the dynamic gating module includes, in the order of processing, a linear layer, an activation function GELU, a linear layer, and an activation function Sigmoid; the feedforward network includes, in the order of processing, a linear layer, an activation function GELU, a linear layer, and a normalization layer;

[0067] The processing procedure for the cross-modal fusion layer is as follows:

[0068] The process involves: acquiring second image stitching features, processing them sequentially through a first linear layer and a first normalization layer to obtain normalized image mapping features; acquiring regularized sentence vectors, processing them sequentially through a second linear layer and a second normalization layer to obtain normalized text mapping features with the same dimension as the normalized image mapping features; inputting the normalized image mapping features and normalized text mapping features into a multi-head attention module to obtain image-text interaction features; stitching the image-text interaction features and normalized text mapping features together to obtain image-text stitching features; processing the image-text stitching features through a dynamic gating module to obtain gating weights; inputting the image-text interaction features, normalized text mapping features, and gating weights together into a gating weighting layer to obtain initial fusion features; subjecting the initial fusion features and normalized text mapping features to a first residual normalization process to obtain intermediate fusion features; processing the intermediate fusion features through a feedforward network to obtain feedforward enhancement features; subjecting the feedforward enhancement features and intermediate fusion features to a second residual normalization process to obtain a fusion feature vector; and outputting the fusion feature vector from the cross-modal fusion layer.

[0069] The answer decoding layer is constructed from a multilayer perceptron network.

[0070] The training described in step S4 includes the following steps:

[0071] During training, the following loss function is used to train only the B-scan image interpretation question answering core subnetwork:

[0072]

[0073] In the formula, For the total loss function, As weight, The loss function for the answer. As weight, The inversion loss function is... As weight, The Dice loss function;

[0074]

[0075] In the formula, For batch size, This refers to the index number of the question-answer pairs in the sample. Correct category score, The number of candidate answers, For the first The first question and answer, the first The raw scores of the candidate answers, This is the index number of the candidate answer;

[0076]

[0077] In the formula For the sample size, Let be the predicted probability vector for the nth sample. Let n be the true label vector of the nth sample. It is a smoothing factor;

[0078]

[0079] In the formula, For the sample size, The prediction result for the nth sample. Let be the true label of the nth sample.

[0080] Step S5 includes the following steps:

[0081] The actual B-scan images and questions are acquired. The front-end processing module processes the actual B-scan images and questions to obtain the corresponding two-channel B-scan images and question codes, respectively.

[0082] The two-channel B-scan image and the question code are input into the B-scan image interpretation question-answering core subnetwork for processing to obtain the two-channel inversion image and the answer code. The question can be any of the five levels of questions.

[0083] The two-channel inversion image and the answer encoding are input into the back-end output module for processing to obtain the B-scan inversion image and the answer. The B-scan inversion image is a question-assisting image, and the answer is a natural language answer.

[0084] A second objective of this invention is to provide an interactive question-and-answer system for B-scan images. The system comprises: a B-scan image acquisition module, a disease information question-and-answer pair generation module, an initial B-scan image interpretation question-and-answer model construction module, an initial B-scan image interpretation question-and-answer model training module, and a B-scan image interpretation question-and-answer model application module; these modules are connected in series.

[0085] The B-scan image acquisition module is used to acquire ground-penetrating radar B-scan images containing at least two categories of disease targets and upload the data to the disease information question-and-answer pair generation module.

[0086] The disease information question-answer pair generation module is used to parse the B-scan image obtained in step S1 based on the received data, obtain the disease information in the B-scan image, and generate question-answer pairs including existence features, quantity features, attribute features and relation features based on the disease information to build a training set, and upload the data to the initial B-scan image interpretation question-answering model building module;

[0087] The initial B-scan image interpretation question answering model building module is used to build an initial B-scan image interpretation question answering model based on the received data. It includes a front-end processing module, a B-scan image interpretation question answering core sub-network, and a back-end output module.

[0088] A front-end processing module is constructed based on a multi-channel image enhancement strategy to process ground-penetrating radar B-scan images and question-answer pairs, so as to construct the input features for training the core sub-network of B-scan image interpretation and question answering.

[0089] A B-scan image interpretation question answering core sub-network is constructed based on a deep feature modeling and fusion mechanism to extract and fuse input features, thereby obtaining parsed image data and answer text data as intermediate representations.

[0090] Based on the multi-channel feature element-wise addition and dictionary decoding mechanism, a back-end output module is constructed to reconstruct the parsed image data and answer text data respectively, and generate the predictive question auxiliary image and the predictive natural language answer, which serve as the final output of the initial B-scan image interpretation question answering model;

[0091] The data is then uploaded to the initial B-scan image interpretation question-answering model training module.

[0092] The initial B-scan image interpretation question answering model training module is used to train the initial B-scan image interpretation question answering model using the training set based on the received data, to obtain the B-scan image interpretation question answering model, and upload the data to the B-scan image interpretation question answering model application module;

[0093] The B-scan image interpretation question-and-answer model application module is used to process data received from the user. The user inputs the actual collected ground-penetrating radar B-scan image and the question into the B-scan image interpretation question-and-answer model to obtain the question-assisting image and the corresponding natural language answer, thus completing the interactive question-and-answer process for the B-scan image.

[0094] This invention performs structured analysis on B-scan images and automatically generates question-answer pairs corresponding to the features of ground-penetrating radar data based on the analysis results. Simultaneously, an encoder-decoder network composed of attention mechanisms and multi-scale convolutional modules is introduced, combined with recurrent neural networks, multi-head attention mechanisms, and dynamic gating structures, to construct an initial B-scan image interpretation question-answering model capable of deeply coupling B-scan images with natural language. Then, using the automatically generated question-answer pairs as the training set, multi-task joint training is employed to optimize this initial model. The trained model can simultaneously perform B-scan image inversion and natural language question answering within an end-to-end framework. Finally, users only need to input the acquired B-scan image and the question they wish to ask to obtain the corresponding B-scan inversion image and answer from the model. Furthermore, users can continue to ask new questions based on the inverted image; the model supports multi-turn, continuous, and interactive question answering and exhibits good generalization ability. Attached Figure Description

[0095] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0096] Figure 2 This is a network structure diagram of the B-scan image interpretation question-answering model of the method of the present invention.

[0097] Figure 3 This is a schematic diagram of the network structure of the core subnetwork for B-scan image interpretation question answering in the method of the present invention.

[0098] Figure 4 This is a schematic diagram of the network structure of the image coding layer in the method of the present invention.

[0099] Figure 5 This is a schematic diagram of the network structure of the cross-modal fusion module of the method of the present invention.

[0100] Figure 6 This is a schematic diagram of the interactive question-and-answer results for B-scan images using the method of the present invention. Detailed Implementation

[0101] like Figure 1 The diagram shown is a flowchart of the method of the present invention: The visual question-answering method disclosed in this invention includes the following steps:

[0102] S1. Acquire ground-penetrating radar B-scan images containing at least two categories of disease targets; further, the disease target categories include at least cavities and cracks, and the ground-penetrating radar B-scan images are acquired from a pre-constructed scene using the finite-difference time-domain method, the pre-constructed scene including the target area and the background area;

[0103] S2. Analyze the B-scan image obtained in step S1 to obtain the disease information in the B-scan image, and generate question-answer pairs including existence features, quantity features, attribute features, and relationship features based on the disease information to construct a training set; specifically including the following steps:

[0104] A1. Analyze the B-scan image to obtain information about the disease targets in the image, including: the relative permittivity, burial depth, location, size, shape and number of the disease targets;

[0105] A2. For any one of the various disease targets, generate a first-level question-and-answer pair to ask and answer whether the disease target of the current category exists; if so, generate the corresponding answer and proceed to step A3.

[0106] If not, for the disease target of the current category that does not exist, set the answers to all its questions to the default answers, and check whether all categories have completed the traversal of steps A2 to A5. If yes, proceed to step A6; otherwise, continue to loop through steps A2 to A5.

[0107] A3. For existing disease targets of the current category, generate a second-level question-and-answer pair to inquire about their physical attributes, perform corresponding checks based on the questions asked, and then generate corresponding answers based on the check results; the physical attributes include the location, shape, and size of the disease target;

[0108] Determine whether all samples in the current category have completed the traversal of step A3. If yes, proceed to step A4; otherwise, continue to loop through step A3.

[0109] A4. If all samples in the current category have completed the traversal in step A3, count the number of samples in the current category; then generate a third-level question-and-answer pair based on the statistical results, which is used to ask and answer the question about the number of samples in the current category.

[0110] A5. If the number of samples in the current category is greater than one, then for the current category, check the geometric relationship between each disease target in the same category of samples, and then generate a fourth-level question-and-answer pair based on the check results, which is used to ask and answer questions about the geometric relationship between each disease target in the same category of samples; the geometric relationship includes spatial relationship, scale difference and morphological association;

[0111] If the number of samples in the current category is less than or equal to one, then for the current category, set the default answer to the question of the geometric relationship between each disease target in the same category of samples;

[0112] Check if all categories have completed the traversal of steps A2 to A5. If yes, proceed to step A6; otherwise, continue to loop through steps A2 to A5.

[0113] A6. If all categories have completed the traversal of steps A2 to A5, count the number of categories; if the number of categories is greater than one, generate a fifth-level question-and-answer pair to inquire about the geometric relationship between different categories of disease targets, perform corresponding checks according to the questions, and generate corresponding answers based on the check results.

[0114] If the number of disease target categories is less than or equal to one, then set the default answer to the geometric relationship question between different disease target categories.

[0115] S3. Construct the initial B-scan image interpretation question answering model, including the front-end processing module, the B-scan image interpretation question answering core sub-network, and the back-end output module;

[0116] like Figure 2 The diagram shows the network structure of the B-scan image interpretation question-answering model.

[0117] A front-end processing module is constructed based on a multi-channel image enhancement strategy to process ground-penetrating radar B-scan images and question-answer pairs, so as to construct the input features for training the core sub-network of B-scan image interpretation and question answering.

[0118] A B-scan image interpretation question answering core sub-network is constructed based on a deep feature modeling and fusion mechanism to extract and fuse input features, thereby obtaining parsed image data and answer text data as intermediate representations.

[0119] Based on the multi-channel feature element-wise addition and dictionary decoding mechanism, a back-end output module is constructed to reconstruct the parsed image data and answer text data respectively, and generate the predictive question auxiliary image and the predictive natural language answer, which serve as the final output of the initial B-scan image interpretation question answering model;

[0120] The front-end processing module performs the following processing steps on the B-scan image in sequence:

[0121] The B-scan image is subtracted from the background data and then resized to obtain the adjusted B-scan image. The adjusted B-scan image is then normalized and histogram equalized to obtain the normalized B-scan image. The adjusted B-scan image is then stitched together with the normalized B-scan image data to generate a two-channel B-scan image.

[0122] The B-scan image is mapped to a dielectric constant map, and background clutter is suppressed. The dielectric constant map after background clutter suppression is separated according to the type of disease target, resulting in one separated dielectric constant map containing only void disease targets and another containing only crack disease targets. Further, void disease targets and crack disease targets are determined based on the difference in connected regions; regions with larger connected areas and clustered shapes are identified as void disease targets, while regions with smaller connected areas and linear shapes are identified as crack disease targets. The two separated dielectric constant maps are resized and then stitched together to form a two-channel dielectric constant image. Further, the two separated dielectric constant maps are first resized to 256*256, and then stitched together on the channels to form a 256*256*2 image, thus forming the two-channel dielectric constant image.

[0123] The front-end processing module performs the following processing steps on the question-and-answer set in sequence:

[0124] A dictionary containing commonly used words and phrases in the field of ground penetrating radar is constructed. Furthermore, the dictionary is constructed using the following method: the frequency of questions and answers is statistically analyzed from a professional corpus in the field of ground penetrating radar, and specific disease terms and descriptive phrases are treated as single lexical units, thereby constructing a dictionary suitable for the field of ground penetrating radar.

[0125] The questions in the question-and-answer set are encoded according to the dictionary to obtain question codes; further, the question codes are decodeable integer codes; the answers in the question-and-answer set are encoded according to the dictionary to obtain answer tag codes; further, the answer tag codes are decodeable integer codes.

[0126] The two-channel B-scan image, the two-channel dielectric constant image, the question code, and the answer label code are used as input features for training the B-scan image interpretation question-answering core sub-network.

[0127] like Figure 3 The diagram shown is a schematic of the core sub-network of B-scan image interpretation question answering in the method of the present invention. The core sub-network of B-scan image interpretation question answering includes: an image encoding layer, an image stitching regularization layer, a text encoding layer, a text regularization layer, a cross-modal fusion layer, and an answer decoding layer.

[0128] The output of the image encoding layer is used as the input of the image stitching regularization layer, the output of the text encoding layer is used as the input of the text regularization layer, the output of the image stitching regularization layer and the output of the text regularization layer are used together as the input of the cross-modal fusion layer, and the output of the cross-modal fusion layer is used as the input of the answer decoding layer.

[0129] The processing procedure of the B-scan image interpretation question answering core subnetwork is as follows:

[0130] The two-channel B-scan image is input into the image coding layer and processed to obtain intermediate image features, final image features, and the two-channel inverted image; the intermediate image features and the final image features are input into the image stitching regularization layer and processed to obtain the second image stitching features;

[0131] The question encoding is input into the text encoding layer and processed to obtain the statement vector; the statement vector is input into the text regularization layer and processed to obtain the regularized statement vector.

[0132] The second image stitching features and the regularized sentence vector are input into the cross-modal fusion layer and processed to obtain the fused feature vector; the fused feature vector is then input into the answer decoding layer and processed to obtain the answer encoding.

[0133] The B-scan image interpretation question-answering core subnetwork outputs a two-channel inverted image and answer encoding.

[0134] like Figure 4 The diagram shown is a network structure diagram of the image coding layer of the method of the present invention. The image coding layer includes an image encoder, an image decoder, and an image projection module.

[0135] The image encoder includes: a first image encoding module (E1) to a fifteenth image encoding module (E15).

[0136] The first (E1) and the second (E2) image coding modules have the same structure, and each module includes, in sequence, a 3x3 convolutional layer, a batch normalization layer and a ReLU activation function layer;

[0137] The third (E3), sixth (E6), ninth (E9), and twelfth image coding modules (E12) have the same structure. Each module includes a residual module RSU4F (Residual U-Net with 4 layers) and an atrous spatial pyramid pooling (ASPP) layer connected in series.

[0138] The fourth (E4), seventh (E7), tenth (E10), and thirteenth image coding modules (E13) have the same structure, and each module includes a max pooling layer;

[0139] The fifteenth image coding module (E15) includes a hollow space pyramid pooling layer;

[0140] The fifth (E5) and the eighth (E8) image coding modules have the same structure. Each module includes a two-branch pyramid convolution module (PyConv2), a batch normalization layer, a ReLU activation function layer, a multi-scale depthwise convolution (MSDConv) with spatially selective feature calibration (SSFC), a batch normalization layer, and a ReLU activation function layer, which are connected in sequence.

[0141] The eleventh (E11) and fourteenth (E14) image coding modules have the same structure. Each module includes a four-branch pyramid convolutional module, a batch normalization layer, a ReLU activation function layer, a multi-scale depth convolution with spatially selective feature calibration, a batch normalization layer, and a ReLU activation function layer connected in sequence.

[0142] The image decoder includes: a first image decoding module (D1) to a seventh image decoding module (D7).

[0143] Among them, the first (D1), third (D3), fifth (D5) and seventh image decoding modules (D7) have the same structure, and each module includes a two-dimensional transposed convolutional layer;

[0144] The second (D2), fourth (D4) and sixth image decoding modules (D6) have the same structure. Each module includes a four-branch pyramid convolution module, a batch normalization layer, a ReLU activation function layer, a multi-scale depth convolution with spatially selective feature calibration, a batch normalization layer, and a ReLU activation function layer connected in sequence.

[0145] The first (D1) to the third (E3) image encoding modules are connected in series. The output of the previous image encoding module is used as the input of the next image encoding module. The output of the third image encoding module (E3) is connected to the output of the seventh image decoding module (D7) and then used as the input of the image projection module.

[0146] Meanwhile, the output of the third image encoding module (E3) also serves as the input of the fourth image encoding module (E4); the fourth (E4) to the sixth image encoding modules (E6) are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the sixth image encoding module (E6) is connected to the output of the fifth image decoding module (D5) and then serves as the input of the sixth image decoding module (D6); the output of the sixth image decoding module (D6) serves as the input of the seventh image decoding module (D7);

[0147] Meanwhile, the output of the sixth image encoding module (E6) also serves as the input of the seventh image encoding module (E7); the seventh (E7) to the ninth image encoding modules (E9) are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the ninth image encoding module (E9) is connected to the output of the third image decoding module (D3) and then serves as the input of the fourth image decoding module (D4); the output of the fourth image decoding module (D4) serves as the input of the fifth image decoding module (D5);

[0148] Meanwhile, the output of the ninth image encoding module (E9) also serves as the input of the tenth image encoding module (E10); the tenth (E10) to the twelfth (E12) image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the twelfth image encoding module (E12) is connected to the output of the first image decoding module (D1) and then serves as the input of the second image decoding module (D2); the output of the second image decoding module (D2) serves as the input of the third image decoding module (D3);

[0149] Meanwhile, the output of the twelfth image encoding module (E12) also serves as the input of the thirteenth image encoding module (E13); the thirteenth (E13) to the fifteenth (E15) image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the fifteenth image encoding module (E15) serves as the input of the first image decoding module (D1);

[0150] The image projection module includes a first projection submodule (P1) and a second projection submodule (P2) connected in series. The structure of the first projection submodule (P1) is the same as that of the fifth image coding module (E5). The structure of the second projection submodule (P2) includes a 1x1 convolutional layer and a Sigmoid activation function layer connected in series.

[0151] The processing procedure of the image coding layer is as follows:

[0152] Two-channel B-scan images are acquired, features are extracted and downsampled step by step through the image encoding module, and intermediate image features are output from the fifteenth image encoding module (E15).

[0153] The intermediate image features are input into the first image decoding module (D1). The decoded features are upsampled step by step through the first (D1) to the seventh image decoding module (D1). The feature maps corresponding to each stage of the encoder are fused with the decoded features through skip connections. The final image features are then output from the seventh image decoding module (D7). Thus, feature reconstruction is performed through the decoder path to gradually restore spatial details and accurately locate the target area.

[0154] The final image features are input into the image projection module and processed to obtain a two-channel inversion image corresponding to the size of the two-channel B-scan image.

[0155] The image coding layer outputs a two-channel inverted image;

[0156] like Figure 4 As shown, the input B-scan image has 2 channels. The output features of the first (E1), second (E2), and third image encoding modules (E3) have 32 channels; the output features of the fifth (E5) and sixth (E6) image encoding modules (E6) have 64 channels; the output features of the eighth (E8) and ninth (E9) image encoding modules (E9) have 128 channels; the output features of the eleventh (E11) and twelfth (E12) image encoding modules (E11) have 256 channels; and the output features of the fourteenth (E14) and fifteenth (E15) image encoding modules (E15) have... The number of channels for the output features, i.e., the intermediate image features, is 512. The number of channels for the output features of the first image decoding module (D1) is 512. The number of channels for the output features of the third image decoding module (D3) is 256. The number of channels for the output features of the fifth image decoding module (D5) is 128. The number of channels for the output features of the seventh image decoding module (D7), i.e., the final image features, is 64. The number of channels for the output features of the first projection submodule (P1) is 32. The number of channels for the output features of the second projection submodule (P2), i.e., the inverted image, is 2.

[0157] The image stitching regularization layer includes: a size transformation layer, a stitching layer, a convolutional layer, and a random deactivation layer;

[0158] The final image features are input into the size transformation layer for processing to obtain the size-adjusted image features; the size-adjusted image features are then combined with the intermediate image features through a stitching layer to obtain the first image stitching features; the first image stitching features are then extracted through a convolutional layer and regularized through a random deactivation layer to obtain the second image stitching features.

[0159] The text encoding layer is constructed based on a recurrent neural network architecture, including: a text encoder and two independent text decoders, wherein the text encoder includes an embedding layer and a first recurrent neural network unit (RNN); the two independent text decoders each have a recurrent neural network unit (RNN), namely a second recurrent neural network unit and a third recurrent neural network unit; the text regularization layer is constructed based on a random deactivation layer.

[0160] The text encoding layer processing procedure is as follows:

[0161] Obtain the question code; process the question code through the embedding layer to obtain a dense word vector sequence; use the dense word vector sequence as input, process it sequentially through the first recurrent neural network unit according to time steps, and fuse context information to generate a context semantic vector, which serves as a fixed-dimensional sentence vector output by the encoder;

[0162] The fixed-dimensional sentence vectors generated by the encoder are used as initial hidden states and input into the second and third recurrent neural network units, respectively. The second recurrent neural network unit extracts the target sentence vector (next sentence features) using forward autoregression, and the third recurrent neural network unit extracts the backward target sentence vector (previous sentence features) using backward autoregression. The model is trained unsupervised by maximizing the generation probability of adjacent sentences. Finally, the target sentence vector and the backward target sentence vector are combined and output as the sentence vector from the text encoding layer.

[0163] like Figure 5 The diagram shows the network structure of the cross-modal fusion module of the present invention. The cross-modal fusion layer includes a first linear layer and a second linear layer, a first normalization layer and a second normalization layer, a multi-head attention module, a splicing layer, a dynamic gating module, a gated weighting layer, and a feedforward network. The dynamic gating module, in the processing order, includes: a linear layer, an activation function GELU, a linear layer, and an activation function Sigmoid. The feedforward network, in the processing order, includes: a linear layer, an activation function GELU, a linear layer, and a normalization layer.

[0164] The processing procedure for the cross-modal fusion layer is as follows:

[0165] The process involves: acquiring second image stitching features, processing them sequentially through a first linear layer and a first normalization layer to obtain normalized image mapping features; acquiring regularized sentence vectors, processing them sequentially through a second linear layer and a second normalization layer to obtain normalized text mapping features with the same dimension as the normalized image mapping features; inputting the normalized image mapping features and normalized text mapping features into a multi-head attention module to obtain image-text interaction features; stitching the image-text interaction features and normalized text mapping features together to obtain image-text stitching features; processing the image-text stitching features through a dynamic gating module to obtain gating weights; inputting the image-text interaction features, normalized text mapping features, and gating weights together into a gating weighting layer to obtain initial fusion features; subjecting the initial fusion features and normalized text mapping features to a first residual normalization process to obtain intermediate fusion features; processing the intermediate fusion features through a feedforward network to obtain feedforward enhancement features; subjecting the feedforward enhancement features and intermediate fusion features to a second residual normalization process to obtain a fusion feature vector; and outputting the fusion feature vector from the cross-modal fusion layer. By setting up a multi-head attention module, a dynamic gating module, and a gating weighting layer in the cross-modal fusion module, deep interaction between image encoding and text encoding is achieved, enabling the initial visual question answering model to answer questions related to underground target attributes more accurately in complex backgrounds.

[0166] The answer decoding layer is constructed based on a multilayer perceptron network with integrated Dropout regularization, and includes the following layers in the order of processing: a first random deactivation layer, a first linear layer, an activation function Tanh, a second random deactivation layer, and a second linear layer.

[0167] The fused feature vector is processed sequentially through the first random deactivation layer, the first linear layer, the activation function Tanh, the second random deactivation layer, and the second linear layer to obtain the answer encoding, which is then output from the answer decoding layer.

[0168] S4. Train the initial B-scan image interpretation question-answering model using the training set to obtain the B-scan image interpretation question-answering model; during training, use the following loss function to train only the core sub-network of B-scan image interpretation question-answering:

[0169] In the specific implementation, the constructed scenario included varying numbers of holes and cracks. The training set size was 800, the test set size was 80, a 500MHz dipole antenna was used, the signal was a Ricker wave, and the scenario size was 4m * 1.2m. The relative permittivity of the cracks and holes was set to 1, the relative permittivity of the background medium was set to 6, and the conductivity was set to 0.01 S / m. Irregularly shaped holes and cracks were randomly generated.

[0170] The following loss function is used for training:

[0171]

[0172] In the formula, For the total loss function, As weight, The loss function for the answer. As weight, The inversion loss function is... As weight, The Dice loss function;

[0173]

[0174] In the formula, Batch size represents the number of question-answer pairs input at once when calculating the loss; Here are the index numbers of the question-answer pairs in the sample; there are a total of [number missing]. A question and answer pair, The correct category score is a numerical value representing the probability of belonging to that category. The number of candidate answers, For the first The first question and answer, the first The original score of the nth candidate answer is expressed as the numerical value of the nth answer. The first question and answer, the first The probability value of each candidate answer output. This is the index number of the candidate answer;

[0175]

[0176] In the formula For the sample size, Let be the predicted probability vector for the nth sample. Let n be the true label vector of the nth sample. It is a smoothing factor;

[0177]

[0178] In the formula, For the sample size, The prediction result for the nth sample. Let be the true label of the nth sample.

[0179] Adam was used as the optimizer for the training process. The initial learning rate was set to 0.00001, the batch size was set to 16, and the training was conducted for 150 rounds to obtain the weights of the trained model.

[0180] S5. The user inputs the actual collected ground-penetrating radar B-scan image and the question into the B-scan image interpretation question-answering model to obtain the question auxiliary image and the corresponding natural language answer; including the following steps:

[0181] The actual B-scan images and questions are acquired. The front-end processing module processes the actual B-scan images and questions to obtain the corresponding two-channel B-scan images and question codes, respectively.

[0182] The two-channel B-scan image and the question code are input into the B-scan image interpretation question-answering core subnetwork for processing to obtain the two-channel inversion image and the answer code. The question can be any of the five levels of questions.

[0183] The two-channel inversion image and the answer encoding are input into the back-end output module for processing to obtain the B-scan inversion image and the answer. The B-scan inversion image is a question-assisting image, and the answer is a natural language answer.

[0184] like Figure 6 As shown, the model input and output diagrams are presented when the input ground-penetrating radar data contains different numbers of cavities and cracks. From top to bottom, the scenarios are two cavities, one cavity and one crack, and two cavities and one crack. From left to right, the original B-scan data, the actual target dielectric constant mapping diagram, the model input question text and the output predicted answer text, and the electromagnetic inversion prediction image are presented. This realizes electromagnetic inversion and multiple multi-level interactions based on ground-penetrating radar B-scan images.

[0185] The effects of the method of the present invention will be illustrated below with reference to an embodiment:

[0186] To quantitatively analyze the performance of this method, different network models were trained using the same training dataset and their performance was evaluated on the same test set. Answer accuracy was measured by mean intersection over union (mIou), and inversion accuracy was measured by overall accuracy (OA). The experimental results are shown in the table below.

[0187] Among them, Swin-UNet is a scheme proposed by Cao et al. in the paper "Swin-Unet: Unet-like PureTransformer for Medical Image Segmentation" in 2022;

[0188] The U-Net scheme was proposed by Olaf et al. in their 2015 paper "U-Net: Convolutional Networks for Biomedical Image Segmentation".

[0189] HRNet-W32 is a configuration scheme in the HRNet series of networks. The HRNet scheme was proposed by Sun et al. in the paper "Deep High-Resolution Representation Learning for Human PoseEstimation" in 2019. The default configuration in the paper is called HRNet-W32.

[0190] The HRNet-W48 scheme is a wider and larger capacity version of the HRNet series of networks, with stronger performance but also greater computational overhead; experiments and analyses of HRNet networks with different widths were conducted in the paper "High-Resolution Representations for Labeling Pixels and Regions".

[0191]

[0192] The proposed method significantly outperforms other methods on OA. When BERT is replaced with a text encoding network, mIou reaches its highest value, differing from the proposed method by only 0.0014, both far exceeding other methods. However, the Skip-Thoughts method used in the proposed method has far fewer parameters and computational cost than BERT. With a performance difference of only 0.0014, the proposed method demonstrates a better balance between efficiency and accuracy.

[0193] To more intuitively reveal the performance of the proposed method in predicting answers across different categories, a confusion matrix was further used for fine-grained accuracy analysis. The confusion matrix uses rows to represent the true categories and columns to represent the predicted categories. Values ​​on the main diagonal reflect the proportion of correct predictions for each category, while off-diagonal elements show misclassifications. The results show that the proposed method performs excellently in determining the number, location, and depth of targets, but is prone to confusion in determining crack width.

[0194] The interactive question-and-answer method for B-scan images provided by this invention realizes end-to-end ground-penetrating radar visual question answering. Users only need to input the acquired B-scan image and the question they wish to ask to obtain the corresponding B-scan inversion image and answer from the model. Furthermore, users can continue to ask new questions based on the inversion image. The model supports multi-round, continuous, and interactive question answering and exhibits good generalization ability.

[0195] The interactive question-and-answer system for B-scan images disclosed in this invention is characterized in that it comprises: a B-scan image acquisition module, a disease information question-and-answer pair generation module, an initial B-scan image interpretation question-and-answer model construction module, an initial B-scan image interpretation question-and-answer model training module, and a B-scan image interpretation question-and-answer model application module; wherein the B-scan image acquisition module, disease information question-and-answer pair generation module, initial B-scan image interpretation question-and-answer model construction module, initial B-scan image interpretation question-and-answer model training module, and B-scan image interpretation question-and-answer model application module are connected in series.

[0196] The B-scan image acquisition module is used to acquire ground-penetrating radar B-scan images containing at least two categories of disease targets and upload the data to the disease information question-and-answer pair generation module.

[0197] The disease information question-answer pair generation module is used to parse the B-scan image obtained in step S1 based on the received data, obtain the disease information in the B-scan image, and generate question-answer pairs including existence features, quantity features, attribute features and relation features based on the disease information to build a training set, and upload the data to the initial B-scan image interpretation question-answering model building module;

[0198] The initial B-scan image interpretation question answering model building module is used to build an initial B-scan image interpretation question answering model based on the received data. It includes a front-end processing module, a B-scan image interpretation question answering core sub-network, and a back-end output module.

[0199] A front-end processing module is constructed based on a multi-channel image enhancement strategy to process ground-penetrating radar B-scan images and question-answer pairs, so as to construct the input features for training the core sub-network of B-scan image interpretation and question answering.

[0200] A B-scan image interpretation question answering core sub-network is constructed based on a deep feature modeling and fusion mechanism to extract and fuse input features, thereby obtaining parsed image data and answer text data as intermediate representations.

[0201] Based on the multi-channel feature element-wise addition and dictionary decoding mechanism, a back-end output module is constructed to reconstruct the parsed image data and answer text data respectively, and generate the predictive question auxiliary image and the predictive natural language answer, which serve as the final output of the initial B-scan image interpretation question answering model;

[0202] The data is then uploaded to the initial B-scan image interpretation question-answering model training module.

[0203] The initial B-scan image interpretation question answering model training module is used to train the initial B-scan image interpretation question answering model using the training set based on the received data, to obtain the B-scan image interpretation question answering model, and upload the data to the B-scan image interpretation question answering model application module;

[0204] The B-scan image interpretation question-and-answer model application module is used to process data received from the user. The user inputs the actual collected ground-penetrating radar B-scan image and the question into the B-scan image interpretation question-and-answer model to obtain the question-assisting image and the corresponding natural language answer, thus completing the interactive question-and-answer process for the B-scan image.

Claims

1. An interactive question-and-answer method for B-scan images, characterized in that, Includes the following steps: S1. Acquire ground-penetrating radar B-scan images containing at least two categories of diseased targets; S2. Analyze the B-scan image obtained in step S1 to obtain the disease information in the B-scan image, and generate question-answer pairs including existence features, quantity features, attribute features and relationship features based on the disease information to construct a training set; S3. Construct the initial B-scan image interpretation question answering model, including the front-end processing module, the B-scan image interpretation question answering core sub-network, and the back-end output module; A front-end processing module is constructed based on a multi-channel image enhancement strategy to process ground-penetrating radar B-scan images and question-answer pairs, so as to construct the input features for training the core sub-network of B-scan image interpretation and question answering. A B-scan image interpretation question answering core sub-network is constructed based on a deep feature modeling and fusion mechanism to extract and fuse input features, thereby obtaining parsed image data and answer text data as intermediate representations. Based on the multi-channel feature element-wise addition and dictionary decoding mechanism, a back-end output module is constructed to reconstruct the parsed image data and answer text data respectively, and generate the predictive question auxiliary image and the predictive natural language answer, which serve as the final output of the initial B-scan image interpretation question answering model; S4. Train the initial B-scan image interpretation question answering model using the training set to obtain the B-scan image interpretation question answering model; S5. The user inputs the actual collected ground-penetrating radar B-scan image and the question into the B-scan image interpretation question-and-answer model to obtain the question auxiliary image and the corresponding natural language answer, thus completing the interactive question-and-answer for the B-scan image; The front-end processing module described in step S3 performs the following processing steps on the B-scan image in sequence: The B-scan image is subtracted from the background data and then resized to obtain the adjusted B-scan image. The adjusted B-scan image is then normalized and histogram equalized to obtain the normalized B-scan image. Finally, the adjusted B-scan image and the normalized B-scan image data are stitched together to generate a two-channel B-scan image. The B-scan image is mapped to a dielectric constant map, and background clutter is suppressed. The dielectric constant map after background clutter suppression is separated according to the type of disease target to obtain a separated dielectric constant map containing only void disease targets and another separated dielectric constant map containing only crack disease targets. The two separated dielectric constant maps are resized and then stitched together to form a two-channel dielectric constant image. The front-end processing module performs the following processing steps on the question-and-answer set in sequence: Construct a dictionary containing commonly used terms and phrases in the field of ground penetrating radar; encode the questions in the question-and-answer set according to the dictionary to obtain question codes; encode the answers in the question-and-answer set according to the dictionary to obtain answer tag codes; The two-channel B-scan image, the two-channel dielectric constant image, the question code, and the answer label code are used as input features for training the B-scan image interpretation question-answering core sub-network; The B-scan image interpretation question answering core sub-network described in step S3 includes: an image encoding layer, an image stitching regularization layer, a text encoding layer, a text regularization layer, a cross-modal fusion layer, and an answer decoding layer; The output of the image encoding layer is used as the input of the image stitching regularization layer, the output of the text encoding layer is used as the input of the text regularization layer, the output of the image stitching regularization layer and the output of the text regularization layer are used together as the input of the cross-modal fusion layer, and the output of the cross-modal fusion layer is used as the input of the answer decoding layer. The processing procedure of the B-scan image interpretation question answering core subnetwork is as follows: The two-channel B-scan image is input into the image coding layer and processed to obtain intermediate image features, final image features, and the two-channel inverted image; the intermediate image features and the final image features are input into the image stitching regularization layer and processed to obtain the second image stitching features; The question encoding is input into the text encoding layer and processed to obtain the statement vector; the statement vector is input into the text regularization layer and processed to obtain the regularized statement vector. The second image stitching features and the regularized sentence vector are input into the cross-modal fusion layer and processed to obtain the fused feature vector; the fused feature vector is then input into the answer decoding layer and processed to obtain the answer encoding. The B-scan image interpretation question-answering core subnetwork outputs a two-channel inverted image and answer encoding.

2. The interactive question-and-answer method for B-scan images according to claim 1, characterized in that, Step S2 specifically includes the following steps: A1. Analyze the B-scan image to obtain information about the disease targets in the image, including: the relative permittivity, burial depth, location, size, shape and number of the disease targets; A2. For any one of the various disease targets, generate a first-level question-and-answer pair to ask and answer whether the disease target of the current category exists; if so, generate the corresponding answer and proceed to step A3. If not, for the disease target of the current category that does not exist, set the answers to all its questions to the default answers, and check whether all categories have completed the traversal of steps A2 to A5. If yes, proceed to step A6; otherwise, continue to loop through steps A2 to A5. A3. For existing disease targets of the current category, generate a second-level question-and-answer pair to inquire about their physical attributes, perform corresponding checks based on the questions asked, and then generate corresponding answers based on the check results; the physical attributes include the location, shape, and size of the disease target; Determine whether all samples in the current category have completed the traversal of step A3. If yes, proceed to step A4; otherwise, continue to loop through step A3. A4. If all samples in the current category have completed the traversal in step A3, count the number of samples in the current category; then generate a third-level question-and-answer pair based on the statistical results, which is used to ask and answer the question about the number of samples in the current category. A5. If the number of samples in the current category is greater than one, then for the current category, check the geometric relationship between each disease target in the same category of samples, and then generate a fourth-level question-and-answer pair based on the check results, which is used to ask and answer questions about the geometric relationship between each disease target in the same category of samples; the geometric relationship includes spatial relationship, scale difference and morphological association; If the number of samples in the current category is less than or equal to one, then for the current category, set the default answer to the question of the geometric relationship between each disease target in the same category of samples; Check if all categories have completed the traversal of steps A2 to A5. If yes, proceed to step A6; otherwise, continue to loop through steps A2 to A5. A6. If all categories have completed the traversal of steps A2 to A5, count the number of categories; if the number of categories is greater than one, generate a fifth-level question-and-answer pair to inquire about the geometric relationship between different categories of disease targets, perform corresponding checks according to the questions, and generate corresponding answers based on the check results. If the number of disease target categories is less than or equal to one, then set the default answer to the geometric relationship question between different disease target categories.

3. The interactive question-and-answer method for B-scan images according to claim 2, characterized in that, The image encoding layer includes an image encoder, an image decoder, and an image projection module; The image encoder includes: a first image encoding module to a fifteenth image encoding module. The first and second image coding modules have the same structure, and each module includes, in sequence, a 3x3 convolutional layer, a batch normalization layer, and a ReLU activation function layer. The third, sixth, ninth, and twelfth image coding modules have the same structure, and each module includes a residual module and a hole space pyramid pooling layer connected in series. The fourth, seventh, tenth, and thirteenth image coding modules have the same structure, and each module includes a max pooling layer. The fifteenth image coding module includes a hollow space pyramid pooling layer; The fifth and eighth image coding modules have the same structure. Each module includes a two-branch pyramid convolutional module, a batch normalization layer, a ReLU activation function layer, a multi-scale depth convolution with spatially selective feature calibration, a batch normalization layer, and a ReLU activation function layer connected in sequence. The eleventh and fourteenth image coding modules have the same structure. Each module includes a four-branch pyramid convolutional module, a batch normalization layer, a ReLU activation function layer, a multi-scale depth convolution with spatially selective feature calibration, a batch normalization layer, and a ReLU activation function layer connected in sequence. The image decoder includes: a first image decoding module to a seventh image decoding module. Among them, the first, third, fifth and seventh image decoding modules have the same structure, and each module includes a two-dimensional transposed convolutional layer; The second, fourth, and sixth image decoding modules have the same structure. Each module includes a four-branch pyramid convolutional module, a batch normalization layer, a ReLU activation function layer, a multi-scale depth convolution with spatially selective feature calibration, a batch normalization layer, and a ReLU activation function layer connected in sequence. The first to third image encoding modules are connected in series. The output of the previous image encoding module is used as the input of the next image encoding module. The output of the third image encoding module is connected to the output of the seventh image decoding module and then used as the input of the image projection module. Meanwhile, the output of the third image encoding module also serves as the input of the fourth image encoding module; the fourth to sixth image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the sixth image encoding module is connected to the output of the fifth image decoding module and then serves as the input of the sixth image decoding module; the output of the sixth image decoding module serves as the input of the seventh image decoding module. Meanwhile, the output of the sixth image encoding module also serves as the input of the seventh image encoding module; the seventh to ninth image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the ninth image encoding module is connected to the output of the third image decoding module and then serves as the input of the fourth image decoding module; the output of the fourth image decoding module serves as the input of the fifth image decoding module. Meanwhile, the output of the ninth image encoding module also serves as the input of the tenth image encoding module; the tenth to twelfth image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the twelfth image encoding module is connected to the output of the first image decoding module and then serves as the input of the second image decoding module; the output of the second image decoding module serves as the input of the third image decoding module. Meanwhile, the output of the twelfth image encoding module also serves as the input of the thirteenth image encoding module; the thirteenth to fifteenth image encoding modules are connected in series, with the output of the previous image encoding module serving as the input of the next image encoding module; the output of the fifteenth image encoding module serves as the input of the first image decoding module; The image projection module includes: a first projection submodule and a second projection submodule connected in series. The structure of the first projection submodule is the same as that of the fifth image encoding module. The structure of the second projection submodule includes a 1x1 convolutional layer and a Sigmoid activation function layer connected in series. The processing procedure of the image coding layer is as follows: Two-channel B-scan images are acquired, features are extracted and downsampled step by step through the image encoding module, and intermediate image features are output from the fifteenth image encoding module. The intermediate image features are input into the first image decoding module. The decoding features are upsampled step by step by the first to seventh image decoding modules. The feature maps corresponding to each stage of the encoder are fused with the decoding features through skip connections. Finally, the final image features are output from the seventh image decoding module. The final image features are input into the image projection module and processed to obtain a two-channel inversion image corresponding to the size of the two-channel B-scan image. The image coding layer outputs a two-channel inverted image; The image stitching regularization layer includes: a size transformation layer, a stitching layer, a convolutional layer, and a random deactivation layer; The final image features are input into the size transformation layer for processing to obtain the size-adjusted image features; the size-adjusted image features are then combined with the intermediate image features through a stitching layer to obtain the first image stitching features; the first image stitching features are then extracted through a convolutional layer and regularized through a random deactivation layer to obtain the second image stitching features.

4. The interactive question-and-answer method for B-scan images according to claim 3, characterized in that, The text encoding layer is constructed based on the recurrent neural network architecture Skip-Thoughts; the text regularization layer is constructed based on a random deactivation layer.

5. An interactive question-and-answer method for B-scan images according to claim 4, characterized in that, The cross-modal fusion layer includes a first linear layer and a second linear layer, a first normalization layer and a second normalization layer, a multi-head attention module, a splicing layer, a dynamic gating module, a gated weighting layer, and a feedforward network; wherein, the dynamic gating module includes, in the order of processing, a linear layer, an activation function GELU, a linear layer, and an activation function Sigmoid; the feedforward network includes, in the order of processing, a linear layer, an activation function GELU, a linear layer, and a normalization layer; The processing procedure for the cross-modal fusion layer is as follows: The process involves: acquiring second image stitching features, processing them sequentially through a first linear layer and a first normalization layer to obtain normalized image mapping features; acquiring regularized sentence vectors, processing them sequentially through a second linear layer and a second normalization layer to obtain normalized text mapping features with the same dimension as the normalized image mapping features; inputting the normalized image mapping features and normalized text mapping features into a multi-head attention module to obtain image-text interaction features; stitching the image-text interaction features and normalized text mapping features together to obtain image-text stitching features; processing the image-text stitching features through a dynamic gating module to obtain gating weights; inputting the image-text interaction features, normalized text mapping features, and gating weights together into a gating weighting layer to obtain initial fusion features; subjecting the initial fusion features and normalized text mapping features to a first residual normalization process to obtain intermediate fusion features; processing the intermediate fusion features through a feedforward network to obtain feedforward enhancement features; subjecting the feedforward enhancement features and intermediate fusion features to a second residual normalization process to obtain a fusion feature vector; and outputting the fusion feature vector from the cross-modal fusion layer. The answer decoding layer is constructed from a multilayer perceptron network.

6. An interactive question-and-answer method for B-scan images according to claim 5, characterized in that, The training described in step S4 includes the following steps: During training, the following loss function is used to train only the core sub-network for B-scan image interpretation and question answering: In the formula, For the total loss function, As weight, The loss function for the answer. As weight, The inversion loss function is... As weight, The Dice loss function; In the formula, For batch size, This refers to the index number of the question-answer pairs in the sample. Correct category score, The number of candidate answers, For the first The first question and answer, the first The raw scores of the candidate answers, This is the index number of the candidate answer; In the formula For the sample size, Let be the predicted probability vector for the nth sample. Let n be the true label vector of the nth sample. It is a smoothing factor; In the formula, For the sample size, The prediction result for the nth sample. Let be the true label of the nth sample.

7. An interactive question-and-answer method for B-scan images according to claim 6, characterized in that, Step S5 includes the following steps: The actual B-scan images and questions are acquired. The front-end processing module processes the actual B-scan images and questions to obtain the corresponding two-channel B-scan images and question codes, respectively. The two-channel B-scan image and the question code are input into the B-scan image interpretation question-answering core subnetwork for processing to obtain the two-channel inversion image and the answer code. The question can be any of the five levels of questions. The two-channel inversion image and the answer encoding are input into the back-end output module for processing to obtain the B-scan inversion image and the answer. The B-scan inversion image is a question-assisting image, and the answer is a natural language answer.

8. An interactive question-and-answer system for B-scan images, characterized in that, The interactive question-and-answer system for B-scan images includes: a B-scan image acquisition module, a disease information question-and-answer pair generation module, an initial B-scan image interpretation question-and-answer model construction module, an initial B-scan image interpretation question-and-answer model training module, and a B-scan image interpretation question-and-answer model application module; the B-scan image acquisition module, disease information question-and-answer pair generation module, initial B-scan image interpretation question-and-answer model construction module, initial B-scan image interpretation question-and-answer model training module, and B-scan image interpretation question-and-answer model application module are connected in series. The B-scan image acquisition module is used to acquire ground-penetrating radar B-scan images containing at least two categories of disease targets and upload the data to the disease information question-and-answer pair generation module. The disease information question-answer pair generation module is used to parse the B-scan image obtained in step S1 based on the received data, obtain the disease information in the B-scan image, and generate question-answer pairs including existence features, quantity features, attribute features and relation features based on the disease information to build a training set, and upload the data to the initial B-scan image interpretation question-answering model building module; The initial B-scan image interpretation question answering model building module is used to build an initial B-scan image interpretation question answering model based on the received data. It includes a front-end processing module, a B-scan image interpretation question answering core sub-network, and a back-end output module. A front-end processing module is constructed based on a multi-channel image enhancement strategy to process ground-penetrating radar B-scan images and question-answer pairs, so as to construct the input features for training the core sub-network of B-scan image interpretation and question answering. A B-scan image interpretation question answering core sub-network is constructed based on a deep feature modeling and fusion mechanism to extract and fuse input features, thereby obtaining parsed image data and answer text data as intermediate representations. Based on the multi-channel feature element-wise addition and dictionary decoding mechanism, a back-end output module is constructed to reconstruct the parsed image data and answer text data respectively, and generate the predictive question auxiliary image and the predictive natural language answer, which serve as the final output of the initial B-scan image interpretation question answering model; The front-end processing module performs the following processing steps on the B-scan image in sequence: The B-scan image is subtracted from the background data and then resized to obtain the adjusted B-scan image. The adjusted B-scan image is then normalized and histogram equalized to obtain the normalized B-scan image. Finally, the adjusted B-scan image and the normalized B-scan image data are stitched together to generate a two-channel B-scan image. The B-scan image is mapped to a dielectric constant map, and background clutter is suppressed. The dielectric constant map after background clutter suppression is separated according to the type of disease target to obtain a separated dielectric constant map containing only void disease targets and another separated dielectric constant map containing only crack disease targets. The two separated dielectric constant maps are resized and then stitched together to form a two-channel dielectric constant image. The front-end processing module performs the following processing steps on the question-and-answer set in sequence: Construct a dictionary containing commonly used terms and phrases in the field of ground penetrating radar; encode the questions in the question-and-answer set according to the dictionary to obtain question codes; encode the answers in the question-and-answer set according to the dictionary to obtain answer tag codes; The two-channel B-scan image, the two-channel dielectric constant image, the question code, and the answer label code are used as input features for training the B-scan image interpretation question-answering core sub-network; The B-scan image interpretation question answering core sub-network includes: an image encoding layer, an image stitching regularization layer, a text encoding layer, a text regularization layer, a cross-modal fusion layer, and an answer decoding layer; The output of the image encoding layer is used as the input of the image stitching regularization layer, the output of the text encoding layer is used as the input of the text regularization layer, the output of the image stitching regularization layer and the output of the text regularization layer are used together as the input of the cross-modal fusion layer, and the output of the cross-modal fusion layer is used as the input of the answer decoding layer. The processing procedure of the B-scan image interpretation question answering core subnetwork is as follows: The two-channel B-scan image is input into the image coding layer and processed to obtain intermediate image features, final image features, and the two-channel inverted image; the intermediate image features and the final image features are input into the image stitching regularization layer and processed to obtain the second image stitching features; The question encoding is input into the text encoding layer and processed to obtain the statement vector; the statement vector is input into the text regularization layer and processed to obtain the regularized statement vector. The second image stitching features and the regularized sentence vector are input into the cross-modal fusion layer and processed to obtain the fused feature vector; the fused feature vector is then input into the answer decoding layer and processed to obtain the answer encoding. The B-scan image interpretation question-answering core subnetwork outputs a two-channel inverted image and the answer encoding. The data is then uploaded to the initial B-scan image interpretation question-answering model training module. The initial B-scan image interpretation question answering model training module is used to train the initial B-scan image interpretation question answering model using the training set based on the received data, to obtain the B-scan image interpretation question answering model, and upload the data to the B-scan image interpretation question answering model application module; The B-scan image interpretation question-and-answer model application module is used to process data received from the user. The user inputs the actual collected ground-penetrating radar B-scan image and the question into the B-scan image interpretation question-and-answer model to obtain the question-assisting image and the corresponding natural language answer, thus completing the interactive question-and-answer process for the B-scan image.

Images