Feature recognition-based detection report generation method, device and equipment

Abstract

The application provides a feature recognition-based detection report generation method, device and equipment, and belongs to the field of industrial nondestructive testing. The method comprises the following steps: acquiring a target nondestructive testing image; inputting the target nondestructive testing image into a detection report generation multi-modal large model to obtain a target detection report output by the detection report generation multi-modal large model, wherein the detection report generation multi-modal large model comprises an image semantic extraction model for acquiring a target global image text feature of the target nondestructive testing image, an object detection model for acquiring a target local image feature and a target defect detection result of the target nondestructive testing image, and a report generation model for generating the target detection report according to the target defect detection result and a target fusion feature, and the target fusion feature comprises the target global image text feature and the target local image feature. Not only can a personalized report with image-text fusion be generated, but also the accuracy of the generated detection report can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of industrial nondestructive testing, and particularly relates to a detection report generation method, device and equipment based on feature recognition. BACKGROUND

[0002] With the improvement of the intelligent level of industrial manufacturing, the role of nondestructive testing (NDT) technology in product quality control and defect identification is increasingly important. In particular, in the production process of key components such as aviation components, technical welds, and composite materials, detection methods such as X-ray images are widely used to characterize and judge internal defects. The current nondestructive testing intelligent system mainly relies on deep learning models to identify and classify defects in images. In the industrial detection scene, not only is it required to identify defects in images, but also to generate diagnostic recommendations or reports based on images. Traditional methods usually rely on fixed templates or rule bases, lack deep semantic matching capabilities between image semantics and report content, and are difficult to achieve accurate and personalized automatic report generation. SUMMARY

[0003] The present application provides a detection report generation method, device and equipment based on feature recognition, to realize the generation of a personalized report with text and images, and to improve the accuracy of the generated detection report.

[0004] The present application provides a detection report generation method based on feature recognition, comprising:

[0005] obtaining a defect detection result image for a target object;

[0006] inputting the defect detection result image into a detection report generation multi-modal large model to obtain a detection result output by the detection report generation multi-modal large model;

[0007] The detection report generation multi-modal large model comprises an image semantic extraction model for obtaining a target global image text feature of the target nondestructive testing image, an object detection model for obtaining a target local image feature and a target defect detection result of the target nondestructive testing image, and a report generation model for generating the target detection report according to the target defect detection result and a target fusion feature, wherein the target fusion feature comprises the target global image text feature and the target local image feature.

[0008] According to the feature recognition-based detection report generation method provided in the application, the report generation model is obtained by training in the following manner: a reference non-destructive testing image is obtained from training data; the reference non-destructive testing image is input into the image semantic extraction model to obtain a reference global image text feature; the reference non-destructive testing image is input into the object detection model to obtain a reference local image feature and a reference defect detection result; a reference fusion feature is obtained according to the reference global image text feature and the reference local image feature; and model parameters of the language large model are updated according to the reference fusion feature and the reference defect detection result to obtain the report generation model.

[0009] According to the feature recognition-based detection report generation method provided in the application, the object detection model comprises a YOLO model, and the reference non-destructive testing image is input into the object detection model to obtain a reference local image feature and a reference defect detection result, which comprises: the reference non-destructive testing image is input into the YOLO model to extract an initial local image feature from an intermediate convolutional layer of the YOLO model; a defect detection result output by the YOLO model is obtained, and the initial local image feature is visually marked according to the defect detection result to obtain the reference local image feature.

[0010] According to the feature recognition-based detection report generation method provided in the application, the reference fusion feature is obtained according to the reference global image text feature and the reference local image feature, which comprises: a transformer feature is obtained by performing transformer coding on the reference global image text feature; and the reference fusion feature is obtained according to the reference local image feature and the transformer feature.

[0011] According to the feature recognition-based detection report generation method provided in the application, the reference fusion feature is obtained according to the reference local image feature and the transformer feature, which comprises: an initial fusion feature is obtained according to the reference local image feature and the transformer feature; at least one diagnostic report text with the highest similarity is obtained by matching the initial fusion feature with semantic features in a historical defect diagnosis report library; semantic information in the at least one diagnostic report text is extracted to obtain a report feature; and the reference fusion feature is obtained according to the initial fusion feature and the report feature.

[0012] According to the feature recognition-based detection report generation method provided in the application, the model parameters of the language large model are updated according to the reference fusion feature and the reference defect detection result, and the report generation model is obtained, which comprises: obtaining a projection mapping matrix of a language token corresponding to an image and the language large model; obtaining a target token of the reference fusion feature according to the projection mapping matrix; inputting the target token and the reference defect detection result into the language large model to obtain a reference detection report; updating the model parameters of the language large model according to the reference detection report to obtain the report generation model.

[0013] According to the feature recognition-based detection report generation method provided in the application, the model parameters of the language large model are updated according to the reference detection report, and the report generation model is obtained, which comprises: updating the model parameters of the language large model according to the reference detection report through a balanced focal loss function to obtain the report generation model.

[0014] The application further provides a feature recognition-based detection report generation device, comprising:

[0015] The acquisition unit is configured to acquire a target nondestructive testing image.

[0016] The generation unit is configured to input the target nondestructive testing image into a detection report generation multi-modal large model to obtain a target detection report output by the detection report generation multi-modal large model.

[0017] The detection report generation multi-modal large model comprises an image semantic extraction model configured to acquire a target global image text feature of the target nondestructive testing image, an object detection model configured to acquire a target local image feature and a target defect detection result of the target nondestructive testing image, and a report generation model configured to generate the target detection report according to the target defect detection result and a target fusion feature, wherein the target fusion feature comprises the target global image text feature and the target local image feature.

[0018] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the feature recognition-based detection report generation method according to any one of the above when executing the computer program.

[0019] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the feature recognition-based detection report generation method according to any one of the above.

[0020] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the feature recognition-based detection report generation method according to any one of the above.

[0021] The feature recognition-based detection report generation method, device and equipment provided by the application first acquire a target nondestructive testing image, and then input the target nondestructive testing image into a detection report generation multi-modal large model to obtain a target detection report output by the detection report generation multi-modal large model, wherein the detection report generation multi-modal large model comprises an image semantic extraction model for acquiring a target global image text feature of the target nondestructive testing image, an object detection model for acquiring a target local image feature and a target defect detection result of the target nondestructive testing image, and a report generation model for generating the target detection report according to the target defect detection result and a target fusion feature, wherein the target fusion feature comprises the target global image text feature and the target local image feature.

[0022] In this way, personalized detection reports can be generated for different nondestructive testing images, and the accuracy of the generated detection reports can be improved. Moreover, the application breaks through the limitation of traditional reliance on visual features by utilizing the multi-modal semantic alignment capability of images and texts, realizes collaborative defect diagnosis of images and texts, and has strong practicability, good expandability, and significant practical value and industry application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0024] Figure 1 is one of the flowcharts of the feature recognition-based detection report generation method provided by the application.

[0025] Figure 2 is another flowchart of the feature recognition-based detection report generation method provided by the application.

[0026] Figure 3 is a functional unit composition block diagram of the feature recognition-based detection report generation device provided by the application.

[0027] Figure 4 is a structural schematic diagram of the electronic device provided by the application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0030] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] Traditional methods for generating inspection reports for nondestructive testing images typically rely on fixed templates or rule bases, lacking the ability to perform deep semantic matching between image semantics and report content, making it difficult to achieve accurate and personalized automatic report generation.

[0032] To address the aforementioned issues, this application provides a method, apparatus, and device for generating detection reports based on feature recognition. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0033] Please see Figure 1 , Figure 1 This is one of the flowcharts of a detection report generation method based on feature recognition provided in this application. The detection report generation method based on feature recognition includes the following steps.

[0034] S101, acquire the target non-destructive testing image.

[0035] The target loss detection image is an NDT image of a target object obtained based on an NDT technique. The target object can be an industrial part or material. The NDT technique includes X-ray imaging, ultrasonic detection, infrared thermal imaging, magnetic powder detection, etc. These NDT techniques can effectively identify defects existing in the interior or surface of an object without any form of damage to the object. Through subsequent detailed analysis of the image, cracks, corrosion, pores, foreign matter, etc. that may exist in the object during manufacturing or use can be revealed, which may affect the structural integrity and performance of the object.

[0036] S102, input the target NDT image into a detection report generation multi-modal large model to obtain a target detection report output by the detection report generation multi-modal large model.

[0037] The detection report generation multi-modal large model includes an image semantic extraction model for obtaining a target global image text feature of the target NDT image, an object detection model for obtaining a target local image feature and a target defect detection result of the target NDT image, and a report generation model for generating the target detection report according to the target defect detection result and a target fusion feature, the target fusion feature including the target global image text feature and the target local image feature. The target detection report is a structured text result (such as defect name, severity, recommended treatment measures, etc.).

[0038] In a specific implementation, a natural language generation indicator such as BLEU and ROUGE can also be introduced in the report generation task to comprehensively evaluate the quality of the report generation model output. In the defect detection classification task, the defect detection capability of the object detection model can also be evaluated through evaluation indicators such as accuracy, recall rate, and F1 score. Then based on the above evaluation results, the model architecture, loss function, data augmentation strategy, etc. are iteratively optimized to ensure the stability and generalization ability of the model in actual industrial applications.

[0039] It can be seen that in the embodiment, first, a target nondestructive testing image is acquired, and then the target nondestructive testing image is input into a detection report generation multi-modal large model to obtain a target detection report output by the detection report generation multi-modal large model. The detection report generation multi-modal large model includes an image semantic extraction model for acquiring a target global image text feature of the target nondestructive testing image, an object detection model for acquiring a target local image feature and a target defect detection result of the target nondestructive testing image, and a report generation model for generating the target detection report according to the target defect detection result and a target fusion feature. The target fusion feature includes the target global image text feature and the target local image feature. In this way, personalized detection reports can be generated for different nondestructive testing images, and the accuracy of the generated detection reports can be improved. Moreover, the application utilizes the multi-modal semantic alignment capability of images and texts, breaks through the limitation of traditional reliance on visual features, realizes collaborative defect diagnosis of images and texts, and has a clear system structure, strong modularity, and significant practical value and industry application prospect.

[0040] In one possible embodiment, the report generation model is obtained by training based on the following manner: a reference nondestructive testing image is acquired from training data; the reference nondestructive testing image is input into the image semantic extraction model to obtain a reference global image text feature; the reference nondestructive testing image is input into the object detection model to obtain a reference local image feature and a reference defect detection result; a reference fusion feature is acquired according to the reference global image text feature and the reference local image feature; and model parameters of a language large model are updated according to the reference fusion feature and the reference defect detection result to obtain the report generation model.

[0041] In training, first, a multi-modal data set containing nondestructive testing images of industrial parts and corresponding diagnostic labels or text reports (i.e., defect detection results) can be collected and constructed. Then, positive and negative sample statistical information is recorded for each defect type to facilitate subsequent evaluation and weighted processing. Finally, the data set is divided into a training set, a validation set and a test set according to a suitable proportion (for example, 7:2:1) to ensure the scientificity of model training and evaluation. The model is trained by using reference nondestructive testing images of the training set.

[0042] In this solution, the object detection model can be used to detect defect regions in the reference non-destructive testing image. Then, the defect bounding box results output by the object detection model are visually annotated, and information such as defect location, type, and confidence is recorded. That is, the reference image features include visually annotated defect bounding boxes, and the reference defect detection results include information such as defect location, type, and confidence. The image semantic extraction model can be a contrastive language-image pretraining (CLIP) CLIP, a bootstrapping language-image pretraining (BLIP), or a similar model.

[0043] As can be seen, in this embodiment, the language large model is fine-tuned by fusing the features and defect detection results to obtain a report generation module for generating a detection report. This can achieve deep semantic matching between image semantics and detection reports, and improve the accuracy and intelligence level of report generation. Specifically, the report generation can not only accurately understand the local defect features in the image, but also combine global semantic information to generate a detection report highly related to the image content. This deep semantic matching enables the report generation model to automatically extract valuable information from the image and express it clearly in natural language, avoiding subjective errors and information omissions in the manual analysis process.

[0044] In addition, it can also improve the readability and logicality of the report, making the report content more consistent with the standards and requirements of the professional field. Whether it is the specific location and type of defects, or the severity and possible impact of defects, the report generation module can accurately and clearly describe it to provide scientific basis for subsequent decision-making and operation. Ultimately, through this deep semantic matching, the report generation model not only improves the generation efficiency of the detection report, but also significantly improves the quality of the report, making the workflow in the non-destructive testing field more intelligent and automated, reducing manual intervention, and improving overall work efficiency and accuracy.

[0045] In one possible embodiment, the object detection model includes a YOLO model, and inputting the reference non-destructive testing image into the object detection model to obtain reference local image features and reference defect detection results includes: inputting the reference non-destructive testing image into the YOLO model to extract initial local image features from an intermediate convolutional layer of the YOLO model; obtaining defect detection results output by the YOLO model, and visually marking the initial local image features according to the defect detection results to obtain the reference local image features.

[0046] The reference nondestructive testing image can be input into the YOLO model, the YOLO model can realize the defect detection, classification and positioning tasks in the reference nondestructive testing image, obtain the defect detection result, and extract the reference local image features in the intermediate layer of the CNN. The YOLO model can not only complete the traditional object detection task, but also process complex image features in the nondestructive testing scene, especially when precise positioning of defects is required, and has strong ability. Specifically, the intermediate convolutional layer of the YOLO model contains rich local feature information, which is very important for subsequent analysis. The features extracted by the intermediate convolutional layer are more fine-grained than the classification results of the final output layer, and can capture detailed information such as texture, shape and edge in the image.

[0047] In a specific implementation, the defect detection result includes information such as defect position, type and confidence. The image features extracted from the CNN can be visually marked based on the defect detection result to obtain reference local image features used to generate fusion features. This can help the subsequent report generation model better understand and analyze the nature, position and type of defects in the image.

[0048] As can be seen, in this embodiment, by fusing the intermediate convolutional layer output of the YOLO model with the global image semantic features, the report generation model can more accurately identify and interpret the detailed information of the defects, thereby generating a more accurate detection report. This process not only improves the accuracy of defect detection, but also greatly enhances the logicality and professionalism of the report, so that the final report is more in line with the actual detection requirements and can provide scientific basis for subsequent decision-making and optimization.

[0049] In one possible embodiment, the reference fusion features are obtained according to the reference global image text features and the reference local image features, comprising: performing transformer encoding on the reference global image text features to obtain transformer features; and obtaining the reference fusion features according to the reference local image features and the transformer features.

[0050] The transformer encoder can utilize its self-attention mechanism to effectively capture long-term dependencies between different parts of the image when processing image text features, while comprehensively analyzing text descriptions in different regions. This process can combine the global information of the reference nondestructive testing image (such as the overall structure of the image, the environmental background and the general defect description) with the semantics of the text features, thereby generating more rich and detailed image semantic representations.

[0051] Next, the reference fusion features are further extracted by combining the reference local image features with the encoded features obtained through the Transformer encoder. The reference local image features come from the local features extracted by the YOLO model in the early stage, which focus on the defect location and specific detail information in the image. By combining these local features with the global image text features, the fusion features not only retain the local detail information, but also strengthen the global semantic understanding of the defect, ensuring that the relationship between each detail in the image and the global information is fully considered when generating the detection report.

[0052] As can be seen, in this embodiment, based on the fusion of encoded features and local image features, the subsequent report generation model can more accurately analyze the specific situation of the defects in the image, and generate a detection report with higher precision and interpretability by combining image semantics and text description. Through this fusion method, the system can better integrate the complementary information of images and texts when processing complex image content, thereby improving the overall performance and realizing more intelligent automatic detection and report generation.

[0053] In one possible embodiment, the reference fusion features are obtained according to the reference local image features and the transformer features, including: obtaining initial fusion features according to the reference local image features and the transformer features; matching the initial fusion features with semantic features in a historical defect diagnosis report library to obtain at least one diagnostic report text with the highest similarity; extracting semantic information in the at least one diagnostic report text to obtain report features; and obtaining the reference fusion features according to the initial fusion features and the report features.

[0054] Wherein, when performing feature fusion, splicing or weighted fusion can be used. As shown in Figure 2 As shown in the figure, first, the reference non-destructive testing image is input into the semantic image extraction model to extract global image semantic features, and then multi-modal large model feature encoding is performed to obtain transformer features. At the same time, CNN features are extracted based on the YOLO model to obtain local image features, and defect identification and labeling are performed to obtain defect detection results. Then the transformer features and the local image features are fused into picture features (i.e. initial fusion features), and then the extracted picture features are used. Then CLIP is used to match the similarity of the semantic features in the historical defect diagnosis report library, select K most similar diagnostic report texts, and extract the semantic information therein as report features. Then the picture features and the report features are fused to obtain the reference fusion features. Finally, the report language large model is fine-tuned based on the reference fusion features and the defect detection results, i.e. the report generation multi-modal large model is fine-tuned, to generate a detection report.

[0055] It can be seen that, in the embodiment, the diagnostic report text generation fusion feature is used to fine-tune the language large model to obtain the report generation model, which overcomes the problem that the traditional report generation method mainly performs defect identification and analysis based on image features, fails to fully utilize the complementary relationship between text and image, and leads to difficulty in simulating the comprehensive judgment ability of artificial, thereby affecting the accuracy and readability of automatic report generation. The accuracy and efficiency of report generation are improved, especially in handling complex scenarios, the visual information of the image and the semantic information of the text can be combined to make the report more comprehensive and accurate. In addition, the report generation model not only improves the reliability of the diagnostic results, but also can automatically adjust the generated content according to different detection needs, improves the user experience and the applicability of the model, especially in the automatic detection and analysis of multiple complex defects, it can better simulate the thinking process of artificial, so that the final report is more in line with professional standards and actual application needs.

[0056] In one possible embodiment, the updating of the model parameters of the language large model according to the reference fusion feature and the reference defect detection result to obtain the report generation model comprises: obtaining a projection mapping matrix of the image feature and the language token corresponding to the language large model; obtaining a target token of the reference fusion feature according to the projection mapping matrix; inputting the target token and the reference defect detection result into the language large model to obtain a reference detection report; updating the model parameters of the language large model according to the reference detection report to obtain the report generation model.

[0057] The projection mapping matrix of the image and the language token of the language large model can be trained to project the fusion feature into a token format acceptable to the language large model. In a specific implementation, the process of training the projection mapping matrix of the image feature and the language large model involves aligning the feature spaces of the image feature and the language model, so that the image feature can be effectively combined with the text token. First, a mapping mechanism can be designed to extract high-dimensional features of the image using a deep learning model such as a convolutional neural network (CNN) or other visual feature extraction network. Then, the neural network is used to learn to map these visual features to the feature space of the language model, i.e., a projection matrix is obtained by training, which can convert the image features into a token format that meets the input requirements of the language large model.

[0058] The key of this projection process is to ensure that the visual information in the image features can be understood by the language model and combined with the text content. Through backpropagation and optimization algorithms, the model can continuously adjust the projection matrix so that the semantic features of the image are aligned with the text features in the same space, thereby improving the accuracy and interpretability of the generated report. Finally, the trained projection mapping matrix can achieve efficient conversion between image features and language tokens, so that the language large model can more accurately understand the image content and generate a more semantically deep detection report.

[0059] In this embodiment, by fusing the feature and image projection mapping strategy, the multi-report generation model can better adapt to the industrial detection semantic task, and the function of automatically generating a structured diagnostic report based on defect images is realized for the first time.

[0060] In one possible embodiment, updating the model parameters of the language large model according to the reference detection report to obtain the report generation model comprises: updating the model parameters of the language large model according to the reference detection report by using an equilibrium focal loss function to obtain the report generation model.

[0061] In this scheme, the equilibrium focal loss (EFL) function is introduced to fine-tune the language large model, which can solve the problem of imbalance between difficult and easy samples in the model when processing data. The equilibrium focal loss function gives higher weight to difficult samples, so that the model can better focus on these difficult features, while reducing the overfitting of easy samples.

[0062] As can be seen, in this way, the model can more accurately identify and process complex defect features when generating reports, improving the quality and accuracy of the reports. This fine-tuning process is optimized through the backpropagation mechanism to the loss function, so that the model gradually adapts to specific tasks during the training process, thereby achieving more accurate automatic report generation.

[0063] Next, a detection report generation device based on feature recognition provided by the present application will be described. The detection report generation device based on feature recognition described below corresponds to the detection report generation method based on feature recognition described above.

[0064] Please refer to Figure 3The application discloses a feature recognition-based detection report generation device 300, which comprises an acquisition unit 301 configured to acquire a target nondestructive testing image; and a generation unit 302 configured to input the target nondestructive testing image into a detection report generation multi-modal large model to obtain a target detection report output by the detection report generation multi-modal large model; wherein the detection report generation multi-modal large model comprises an image semantic extraction model configured to acquire a target global image text feature of the target nondestructive testing image, an object detection model configured to acquire a target local image feature and a target defect detection result of the target nondestructive testing image, and a report generation model configured to generate the target detection report according to the target defect detection result and a target fusion feature comprising the target global image text feature and the target local image feature.

[0065] In one possible embodiment, the feature recognition-based detection report generation device 300 further comprises a training unit configured to train the report generation model by: acquiring a reference nondestructive testing image from training data; inputting the reference nondestructive testing image into the image semantic extraction model to obtain a reference global image text feature; inputting the reference nondestructive testing image into the object detection model to obtain a reference local image feature and a reference defect detection result; acquiring a reference fusion feature according to the reference global image text feature and the reference local image feature; and updating model parameters of a language large model according to the reference fusion feature and the reference defect detection result to obtain the report generation model.

[0066] In one possible embodiment, the object detection model comprises a YOLO model, and the inputting of the reference nondestructive testing image into the object detection model to obtain the reference local image feature and the reference defect detection result comprises: inputting the reference nondestructive testing image into the YOLO model to extract an initial local image feature from an intermediate convolutional layer of the YOLO model; acquiring a defect detection result output by the YOLO model; and performing visual marking on the initial local image feature according to the defect detection result to obtain the reference local image feature.

[0067] In one possible embodiment, in the acquiring of the reference fusion feature according to the reference global image text feature and the reference local image feature, the training unit is specifically configured to: perform transformer coding on the reference global image text feature to obtain a transformer feature; and acquire the reference fusion feature according to the reference local image feature and the transformer feature.

[0068] In a possible embodiment, in the aspect of obtaining the reference fusion feature according to the reference local image feature and the transformer feature, the training unit is specifically configured to: obtain an initial fusion feature according to the reference local image feature and the transformer feature; match the initial fusion feature with semantic features in a historical defect diagnosis report library to obtain at least one diagnostic report text with the highest similarity; extract semantic information in the at least one diagnostic report text to obtain a report feature; and obtain the reference fusion feature according to the initial fusion feature and the report feature.

[0069] In a possible embodiment, in the aspect of updating the model parameters of the language large model according to the reference fusion feature and the reference defect detection result to obtain the report generation model, the training unit is specifically configured to: obtain a projection mapping matrix of a language token corresponding to the image and the language large model; obtain a target token of the reference fusion feature according to the projection mapping matrix; input the target token and the reference defect detection result into the language large model to obtain a reference detection report; and update the model parameters of the language large model according to the reference detection report to obtain the report generation model.

[0070] In a possible embodiment, in the aspect of updating the model parameters of the language large model according to the reference detection report to obtain the report generation model, the training unit is specifically configured to: update the model parameters of the language large model according to the reference detection report through a balanced focal loss function to obtain the report generation model.

[0071] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of an electronic device provided in the present application. As shown in Figure 4As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can invoke the logic instructions in the memory 430 to execute a feature recognition-based detection report generation method, which includes: obtaining a target non-destructive testing image; inputting the target non-destructive testing image into a detection report generation multi-modal large model to obtain a target detection report output by the detection report generation multi-modal large model; wherein the detection report generation multi-modal large model includes an image semantic extraction model for obtaining a target global image text feature of the target non-destructive testing image, an object detection model for obtaining a target local image feature and a target defect detection result of the target non-destructive testing image, and a report generation model for generating the target detection report according to the target defect detection result and a target fusion feature, the target fusion feature including the target global image text feature and the target local image feature.

[0072] In addition, the logic instructions in the memory 430 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0073] In another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a feature recognition-based detection report generation method provided by each of the above methods, the method comprising: obtaining a target non-destructive testing image; inputting the target non-destructive testing image into a detection report generation multi-modal large model to obtain a target detection report output by the detection report generation multi-modal large model; wherein the detection report generation multi-modal large model comprises an image semantic extraction model for obtaining a target global image text feature of the target non-destructive testing image, an object detection model for obtaining a target local image feature and a target defect detection result of the target non-destructive testing image, and a report generation model for generating the target detection report according to the target defect detection result and a target fusion feature, the target fusion feature comprising the target global image text feature and the target local image feature.

[0074] In yet another aspect, the present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements a feature recognition-based detection report generation method, the method comprising: obtaining a target non-destructive testing image; inputting the target non-destructive testing image into a detection report generation multi-modal large model to obtain a target detection report output by the detection report generation multi-modal large model; wherein the detection report generation multi-modal large model comprises an image semantic extraction model for obtaining a target global image text feature of the target non-destructive testing image, an object detection model for obtaining a target local image feature and a target defect detection result of the target non-destructive testing image, and a report generation model for generating the target detection report according to the target defect detection result and a target fusion feature, the target fusion feature comprising the target global image text feature and the target local image feature.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0076] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0077] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A detection report generation method based on feature recognition, characterized in that, include: Acquire the target non-destructive testing image; The target non-destructive testing image is input into the detection report to generate a multimodal large model, and the target detection report output by the multimodal large model is obtained. The multimodal large model for generating the detection report includes an image semantic extraction model for obtaining the target global image text features of the target nondestructive testing image, an object detection model for obtaining the target local image features and target defect detection results of the target nondestructive testing image, and a report generation model for generating the target detection report based on the target defect detection results and target fusion features, wherein the target fusion features include the target global image text features and the target local image features; The report generation model is trained in the following manner: Obtain reference non-destructive testing images from the training data; The reference non-destructive testing image is input into the image semantic extraction model to obtain the reference global image text features; The reference nondestructive testing image is input into the object detection model to obtain reference local image features and reference defect detection results. The reference local image features include defect boxes with visual annotations. A reference fusion feature is obtained based on the reference global image text features and the reference local image features; The model parameters of the language large model are updated based on the reference fusion features and the reference defect detection results to obtain the report generation model.

2. The method according to claim 1, characterized in that, The object detection model includes a YOLO model. The step of inputting the reference nondestructive testing image into the object detection model to obtain reference local image features and reference defect detection results includes: The reference nondestructive testing image is input into the YOLO model, and initial local image features are extracted from the intermediate convolutional layers of the YOLO model. Obtain the defect detection results from the YOLO model, and visualize and label the initial local image features based on the defect detection results to obtain the reference local image features.

3. The method according to claim 1, characterized in that, The step of obtaining reference fusion features based on the reference global image text features and the reference local image features includes: The text features of the reference global image are encoded using transformer to obtain transformer features; The reference fusion feature is obtained based on the reference local image features and the transformer features.

4. The method according to claim 3, characterized in that, The step of obtaining the reference fusion feature based on the reference local image features and the transformer features includes: Initial fusion features are obtained based on the reference local image features and the transformer features; The initial fusion features are matched with the semantic features in the historical defect diagnosis report database to obtain at least one diagnosis report text with the highest similarity. Extract semantic information from the text of at least one diagnostic report to obtain report features; The reference fusion feature is obtained based on the initial fusion feature and the reported feature.

5. The method according to claim 1, characterized in that, The step of updating the model parameters of the large language model based on the reference fusion features and the reference defect detection results to obtain the report generation model includes: Obtain the projection mapping matrix between the image and the language tokens corresponding to the large language model; The target token of the reference fusion feature is obtained based on the projection mapping matrix; The target token and the reference defect detection result are input into the language big model to obtain a reference detection report; The model parameters of the language large model are updated based on the reference detection report to obtain the report generation model.

6. The method according to claim 5, characterized in that, The step of updating the model parameters of the language large model based on the reference detection report to obtain the report generation model includes: The report generation model is obtained by updating the model parameters of the large language model using the balanced focus loss function based on the reference detection report.

7. A detection report generation device based on feature recognition, characterized in that, include: The acquisition unit is used to acquire the target non-destructive testing image; The generation unit is used to input the target non-destructive testing image into the detection report to generate a multimodal large model, and obtain the target detection report output by the detection report generation multimodal large model; The multimodal large model for generating the detection report includes an image semantic extraction model for obtaining the target global image text features of the target nondestructive testing image, an object detection model for obtaining the target local image features and target defect detection results of the target nondestructive testing image, and a report generation model for generating the target detection report based on the target defect detection results and target fusion features, wherein the target fusion features include the target global image text features and the target local image features; The training unit is specifically used to train the report generation model in the following manner: obtaining a reference non-destructive testing image from the training data; inputting the reference non-destructive testing image into the image semantic extraction model to obtain reference global image text features; inputting the reference non-destructive testing image into the object detection model to obtain reference local image features and reference defect detection results, wherein the reference local image features include visually labeled defect boxes; obtaining reference fusion features based on the reference global image text features and the reference local image features; and updating the model parameters of the large language model based on the reference fusion features and the reference defect detection results to obtain the report generation model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the feature recognition-based detection report generation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the feature recognition-based detection report generation method as described in any one of claims 1 to 6.

Images