Detection report generation method, device and equipment based on feature recognition

Abstract

The invention provides a detection report generation method, device and equipment based on feature recognition, and belongs to the field of industrial nondestructive detection. The method comprises the following steps: acquiring a target nondestructive testing image; inputting the target nondestructive detection image into the detection report to generate a multi-modal large model, and obtaining a target detection report output by the detection report to generate the multi-modal large model, the detection report generation multi-modal large model comprises an image semantic extraction model used for acquiring target global image text features of a target nondestructive detection image, and an object detection model used for acquiring target local image features of the target nondestructive detection image and a target defect detection result; and a report generation model used for generating a target detection report according to the target defect detection result and the target fusion feature, wherein the target fusion feature comprises a target global image text feature and a target local image feature. The image-text fusion personalized report can be generated, and the accuracy of the generated detection report can be improved.

Description

Technical Field

[0001] The present application relates to the field of industrial non-destructive testing, and specifically relates to a method, device and equipment for generating a test report based on feature recognition. Background Art

[0002] With the increasing intelligence of industrial manufacturing, nondestructive testing (NDT) technology is playing an increasingly important role in product quality control and defect identification. In particular, in the production of key components such as aviation structures, technical welds, and composite materials, X-ray imaging and other testing methods are widely used to characterize and identify internal defects. Current intelligent nondestructive testing systems primarily rely on deep learning models to identify and classify defects in images. In industrial inspection scenarios, it is not only necessary to identify defects in images but also to generate diagnostic recommendations or reports based on the images. Traditional methods typically rely on fixed templates or rule bases and lack the ability to deeply match image semantics with report content, making it difficult to achieve accurate and personalized automatic report generation. Summary of the Invention

[0003] The present application provides a method, apparatus and device for generating a test report based on feature recognition, which is used to generate a personalized report with integrated graphics and text, while improving the accuracy of the generated test report.

[0004] This application provides a method for generating a test report based on feature recognition, including: Obtain defect detection result images for the target object; Inputting the defect detection result image into the detection report generation multimodal large model to obtain the detection result output by the detection report generation multimodal large model; Among them, the multimodal large model for generating the inspection report includes an image semantic extraction model for obtaining the target global image text features of the target non-destructive inspection image, an object detection model for obtaining the target local image features and target defect detection results of the target non-destructive inspection image, and a report generation model for generating the target inspection report based on the target defect detection results and target fusion features, wherein the target fusion features include the target global image text features and the target local image features.

[0005] According to the feature recognition-based inspection report generation method provided in the present application, the report generation model is trained based on the following method: obtaining a reference non-destructive inspection image from training data; inputting the reference non-destructive inspection image into the image semantic extraction model to obtain a reference global image text feature; inputting the reference non-destructive inspection image into the object detection model to obtain a reference local image feature and a reference defect detection result; obtaining a reference fusion feature based on the reference global image text feature and the reference local image feature; and updating the model parameters of the language large model based on the reference fusion feature and the reference defect detection result to obtain the report generation model.

[0006] According to the feature recognition-based inspection report generation method provided in the present application, the object detection model includes a YOLO model, and the step of inputting the reference nondestructive testing image into the object detection model to obtain reference local image features and reference defect detection results includes: inputting the reference nondestructive testing image into the YOLO model, extracting initial local image features from the intermediate convolutional layer of the YOLO model; obtaining the defect detection result from the YOLO model, and visually marking the initial local image features according to the defect detection result to obtain the reference local image features.

[0007] According to the feature recognition-based detection report generation method provided in the present application, the reference fusion feature is obtained based on the reference global image text feature and the reference local image feature, including: transformer encoding the reference global image text feature to obtain transformer feature; and obtaining the reference fusion feature based on the reference local image feature and the transformer feature.

[0008] According to the feature recognition-based inspection report generation method provided in the present application, the reference fusion feature is obtained based on the reference local image feature and the transformer feature, including: obtaining an initial fusion feature based on the reference local image feature and the transformer feature; matching the initial fusion feature with the semantic features in the historical defect diagnosis report library to obtain at least one diagnosis report text with the highest similarity; extracting the semantic information in the at least one diagnosis report text to obtain a report feature; and obtaining the reference fusion feature based on the initial fusion feature and the report feature.

[0009] According to the feature recognition-based detection report generation method provided in the present application, the model parameters of the language large model are updated according to the reference fusion feature and the reference defect detection result to obtain the report generation model, including: obtaining the projection mapping matrix of the language token corresponding to the image and the language large model; obtaining the target token of the reference fusion feature according to the projection mapping matrix; inputting the target token and the reference defect detection result into the language large model to obtain a reference detection report; and updating the model parameters of the language large model according to the reference detection report to obtain the report generation model.

[0010] According to the feature recognition-based detection report generation method provided in the present application, the model parameters of the large language model are updated according to the reference detection report to obtain the report generation model, including: updating the model parameters of the large language model through the balanced focus loss function according to the reference detection report to obtain the report generation model.

[0011] This application also provides a detection report generation device based on feature recognition, comprising: An acquisition unit, used for acquiring a target nondestructive testing image; a generating unit, configured to input the target nondestructive testing image into a test report generating multimodal large model, and obtain a target test report output by the test report generating multimodal large model; Among them, the multimodal large model for generating the inspection report includes an image semantic extraction model for obtaining the target global image text features of the target non-destructive inspection image, an object detection model for obtaining the target local image features and target defect detection results of the target non-destructive inspection image, and a report generation model for generating the target inspection report based on the target defect detection results and target fusion features, wherein the target fusion features include the target global image text features and the target local image features.

[0012] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating a detection report based on feature recognition as described above is implemented.

[0013] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for generating a detection report based on feature recognition as described above is implemented.

[0014] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for generating a detection report based on feature recognition.

[0015] The feature recognition-based inspection report generation method, apparatus, and device provided in the present application first obtain a target nondestructive inspection image, then input the target nondestructive inspection image into a large multimodal model for inspection report generation, and obtain a target inspection report output by the large multimodal model for inspection report generation, wherein the large multimodal model for inspection report generation includes an image semantic extraction model for obtaining target global image text features of the target nondestructive inspection image, an object detection model for obtaining target local image features and target defect detection results of the target nondestructive inspection image, and a report generation model for generating the target inspection report based on the target defect detection results and target fusion features, wherein the target fusion features include the target global image text features and the target local image features.

[0016] This allows for the generation of personalized inspection reports for different non-destructive inspection images, while also improving the accuracy of the generated inspection reports. Furthermore, this application utilizes the multimodal semantic alignment capabilities of images and text, breaking through the limitations of traditional methods that rely solely on visual features, and enabling collaborative defect diagnosis using images and text. Furthermore, the full-process modular design supports the integrated deployment of inspection, analysis, and report generation, making it highly practical and scalable, making it easy to promote and deploy in different industrial scenarios. It possesses significant practical value and industry application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of a method for generating a test report based on feature recognition provided in this application.

[0019] Figure 2 This is the second flow chart of a method for generating a test report based on feature recognition provided in this application.

[0020] Figure 3 This is a block diagram of the functional units of a detection report generation device based on feature recognition provided by this application.

[0021] Figure 4 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0023] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0024] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0025] Traditional methods usually rely on fixed templates or rule libraries when generating inspection reports for non-destructive testing images. They lack the ability to deeply match image semantics with report content, making it difficult to achieve accurate and personalized automatic report generation.

[0026] In response to the above problems, the present application provides a method, device and equipment for generating a test report based on feature recognition. The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0027] See also Figure 1 , Figure 1 This is one of the flow charts of a method for generating a test report based on feature recognition provided in this application. The method for generating a test report based on feature recognition includes the following steps.

[0028] S101, acquiring a target nondestructive testing image.

[0029] The target loss detection image is a non-destructive testing image of the target object acquired using non-destructive testing technology. The target object can be an industrial part or material. Non-destructive testing technologies include X-ray imaging, ultrasonic testing, infrared thermal imaging, and magnetic particle testing. These non-destructive testing technologies can effectively identify defects within or on the surface of an object without any form of damage to the object. Subsequent detailed analysis of the image can reveal cracks, corrosion, pores, foreign matter, and other problems that may exist during the manufacturing or use process of the object, which may affect the object's structural integrity and performance.

[0030] S102: Input the target nondestructive testing image into a test report generation multimodal large model to obtain a target test report output by the test report generation multimodal large model.

[0031] The multimodal large model for generating inspection reports includes an image semantic extraction model for acquiring global text features of the target NDT image; an object detection model for acquiring local image features and defect detection results; and a report generation model for generating the target inspection report based on the defect detection results and target fusion features. The target fusion features include global text features and local image features. The target inspection report is a structured text result (e.g., defect name, severity, recommended treatment measures, etc.).

[0032] In specific implementations, natural language generation metrics such as BLEU and ROUGE can be introduced into report generation tasks to comprehensively evaluate the quality of the report generation model's output. In defect detection and classification tasks, the object detection model's defect detection capabilities can be assessed using metrics such as accuracy, recall, and F1 score. Based on these evaluation results, iterative optimization of the model architecture, loss function, data augmentation strategy, and other aspects can be performed to ensure the model's stability and generalization capabilities in real-world industrial applications.

[0033] As can be seen, in this embodiment, the target nondestructive testing image is first obtained, and then the target nondestructive testing image is input into the test report generation multimodal large model to obtain the target detection report output by the test report generation multimodal large model. The test report generation multimodal large model includes an image semantic extraction model for obtaining the target global image text features of the target nondestructive testing image, an object detection model for obtaining the target local image features and target defect detection results of the target nondestructive testing image, and a report generation model for generating the target detection report based on the target defect detection results and target fusion features. The target fusion features include the target global image text features and the target local image features. In this way, personalized test reports can be generated for different nondestructive testing images, while also improving the accuracy of the generated test reports. In addition, this application utilizes the multimodal semantic alignment capability of images and text, breaking through the limitations of traditional reliance on visual features alone, and realizing image-text collaborative defect diagnosis. The system has a clear structure and strong modularity, making it easy to promote and deploy in different industrial scenarios, and has significant practical value and industry application prospects.

[0034] In a possible embodiment, the report generation model is trained based on the following method: obtaining a reference nondestructive testing image from training data; inputting the reference nondestructive testing image into the image semantic extraction model to obtain reference global image text features; inputting the reference nondestructive testing image into the object detection model to obtain reference local image features and reference defect detection results; obtaining reference fusion features based on the reference global image text features and the reference local image features; and updating model parameters of a large language model based on the reference fusion features and the reference defect detection results to obtain the report generation model.

[0035] During training, a multimodal dataset consisting of nondestructive inspection images of industrial parts and their corresponding diagnostic labels or text reports (i.e., defect detection results) can be collected and constructed. Statistics of positive and negative samples are then recorded for each defect type to facilitate subsequent evaluation and weighting. Finally, the dataset is divided into training, validation, and test sets in an appropriate ratio (e.g., 7:2:1) to ensure scientifically sound model training and evaluation. Model training is performed using reference nondestructive inspection images from the training set.

[0036] In this solution, an object detection model can be used to detect defect areas in a reference nondestructive testing image. The defect boxes output by the object detection model are then visually annotated, and information such as the defect location, type, and confidence level is recorded. That is, the reference image features include visually annotated defect boxes, and the reference defect detection results include information such as the defect location, type, and confidence level. The image semantic extraction model can be contrastive language-image pretraining (CLIP), bootstrapping language-image pretraining (BLIP), or similar models.

[0037] It can be seen that in this embodiment, by fine-tuning the language model by fusing features and defect detection results, a report generation module that is ultimately used to generate a test report is obtained, which can achieve deep semantic matching between image semantics and test reports, thereby improving the accuracy and intelligence level of report generation. Specifically, report generation can not only accurately understand the local defect features in the image, but also combine global semantic information to generate a test report that is highly relevant to the image content. This deep semantic matching enables the report generation model to automatically extract valuable information from the image and express it clearly in the form of natural language, avoiding subjective errors and information omissions in the manual analysis process.

[0038] Furthermore, it improves the readability and logic of reports, making them more consistent with the standards and requirements of the professional field. The report generation module accurately and clearly describes the specific location and type of defects, as well as their severity and potential impact, providing a scientific basis for subsequent decision-making and operations. Ultimately, through this deep semantic matching, the report generation model not only improves the efficiency of test report generation but also significantly enhances the quality of reports, making workflows in the field of nondestructive testing more intelligent and automated, reducing manual intervention and improving overall work efficiency and accuracy.

[0039] In a possible embodiment, the object detection model includes a YOLO model, and inputting the reference nondestructive testing image into the object detection model to obtain reference local image features and reference defect detection results includes: inputting the reference nondestructive testing image into the YOLO model, extracting initial local image features from an intermediate convolutional layer of the YOLO model; obtaining a defect detection result from the YOLO model, and visually marking the initial local image features according to the defect detection result to obtain the reference local image features.

[0040] Reference nondestructive testing images can be input into the YOLO model, which then performs defect detection, classification, and localization tasks in the nondestructive testing images, obtains defect detection results, and extracts reference local image features in the CNN intermediate layer. The YOLO model is not only capable of completing traditional object detection tasks, but is also capable of processing complex image features in nondestructive testing scenarios, demonstrating its powerful capabilities when precise defect location is required. Specifically, the intermediate convolutional layers of the YOLO model contain rich local feature information, which is critical for subsequent analysis. The features extracted by the intermediate convolutional layers are more fine-grained than the classification results of the final output layer, and can capture detailed information in the image, such as texture, shape, and edges.

[0041] In specific implementations, defect detection results include information such as defect location, type, and confidence level. Image features extracted from the CNN can be visually labeled based on the defect detection results to obtain reference local image features for generating fusion features. This helps subsequent report generation models better understand and analyze the nature, location, and type of defects in the image.

[0042] As can be seen, in this embodiment, by fusing the output of the YOLO model's intermediate convolutional layer with global image semantic features, the report generation model can more accurately identify and interpret detailed defect information, thereby generating a more precise inspection report. This process not only improves defect detection accuracy but also greatly enhances the report's logic and professionalism, making the final report more consistent with actual inspection requirements and providing a scientific basis for subsequent decision-making and optimization.

[0043] In a possible embodiment, obtaining a reference fusion feature based on the reference global image text feature and the reference local image feature includes: performing transformer encoding on the reference global image text feature to obtain a transformer feature; and obtaining the reference fusion feature based on the reference local image feature and the transformer feature.

[0044] This can be done through a transformer encoder. The transformer encoder leverages its self-attention mechanism to effectively capture long-term dependencies between image components when processing image text features, while also comprehensively analyzing text descriptions of different regions. This process combines the global information of the reference NDT image (e.g., the overall image structure, environmental context, and general defect description) with the semantics of the text features, generating a richer and more detailed image semantic representation.

[0045] Next, the reference local image features are combined with the encoded features obtained through the Transformer encoder to further extract reference fused features. These reference local image features are derived from the local features extracted by the previous YOLO model, focusing on the defect location and specific details in the image. By combining these local features with the global image context features, the fused features not only preserve the local details but also strengthen the global semantic understanding of the defect, ensuring that the relationship between every detail in the image and the global information is fully considered when generating the inspection report.

[0046] As can be seen, in this embodiment, the fusion of coded features and local image features enables the subsequent report generation model to more accurately analyze the specific defects in the image. It also combines image semantics and text descriptions to generate inspection reports with higher accuracy and interpretability. This fusion approach allows the system to better integrate the complementary information of images and text when processing complex image content, thereby improving overall performance and enabling more intelligent automated inspection and report generation.

[0047] In a possible embodiment, obtaining the reference fusion feature based on the reference local image feature and the transformer feature includes: obtaining an initial fusion feature based on the reference local image feature and the transformer feature; matching the initial fusion feature with the semantic features in a historical defect diagnosis report library to obtain at least one diagnosis report text with the highest similarity; extracting semantic information from the at least one diagnosis report text to obtain a report feature; and obtaining the reference fusion feature based on the initial fusion feature and the report feature.

[0048] Among them, when performing feature fusion, splicing or weighted fusion can be used. Figure 2 As shown, reference nondestructive inspection images are first fed into a semantic image extraction model to extract global image semantic features. This is then followed by feature encoding using a large multimodal model to produce transformer features. CNN features are also extracted using the YOLO model to obtain local image features, and defects are identified and labeled to produce defect detection results. The transformer features and local image features are then fused into image features (i.e., initial fused features). These extracted image features are then used to perform similarity matching with semantic features from a library of historical defect diagnosis reports using CLIP. K most similar diagnosis report texts are selected, and their semantic information is extracted as report features. The image features and report features are then fused to produce reference fused features. Finally, the report language large model is fine-tuned based on the reference fused features and the defect detection results. This fine-tuning of the report generation multimodal large model generates the inspection report.

[0049] It can be seen that in this embodiment, the diagnostic report text is used to generate fusion features, thereby fine-tuning the language model to obtain a report generation model, which overcomes the problem that the traditional report generation method mainly relies on image features for defect identification and analysis, fails to fully utilize the complementary relationship between text and image, resulting in the model being difficult to simulate human comprehensive judgment ability, thereby affecting the accuracy and readability of automatic report generation. The accuracy and efficiency of report generation are improved, especially when dealing with complex scenarios, by combining the visual information of the image with the semantic information of the text, making the report more comprehensive and accurate. In addition, the report generation model not only improves the reliability of the diagnostic results, but also can automatically adjust the generated content according to different detection requirements, improves the user experience and the applicability of the model, especially in the automatic detection and analysis of various complex defects, can better simulate the human thinking process, so that the final report is more in line with professional standards and actual application needs.

[0050] In a possible embodiment, the updating of the model parameters of the language large model according to the reference fusion feature and the reference defect detection result to obtain the report generation model includes: obtaining a projection mapping matrix of the language tokens corresponding to the image features and the language large model; obtaining the target token of the reference fusion feature according to the projection mapping matrix; inputting the target token and the reference defect detection result into the language large model to obtain a reference detection report; and updating the model parameters of the language large model according to the reference detection report to obtain the report generation model.

[0051] Among them, the projection mapping matrix between the image and the language tokens of the large language model can be trained to project the fused features into a token format acceptable to the large language model. In specific implementation, the process of training the projection mapping matrix between image features and the large language model involves aligning the image features with the feature space of the language model so that the image features can be effectively combined with the text tokens. First, a mapping mechanism can be designed to extract the high-dimensional features of the image using a deep learning model such as a convolutional neural network (CNN) or other visual feature extraction network. Then, a neural network is used to learn to map these visual features to the feature space of the language model. That is, a projection matrix is ​​obtained through training, which can convert the image features into a token format that meets the input requirements of the large language model.

[0052] The key to this projection process is ensuring that the visual information contained in image features can be understood by the language model and integrated with the textual content. Through backpropagation and optimization algorithms, the model continuously adjusts the projection matrix, aligning the semantic features of the image with the textual features in the same space, thereby improving the accuracy and interpretability of generated reports. Ultimately, the trained projection mapping matrix enables efficient conversion between image features and language tokens, enabling the large language model to more accurately understand image content and generate detection reports with greater semantic depth.

[0053] Yes, in this embodiment, by fusing features and image projection mapping strategies, the multi-report generation model can be better adapted to industrial inspection semantic tasks, and for the first time realizes the function of automatically generating structured diagnostic reports based on defect images.

[0054] In a possible embodiment, updating the model parameters of the large language model according to the reference detection report to obtain the report generation model includes: updating the model parameters of the large language model through the balanced focus loss function according to the reference detection report to obtain the report generation model.

[0055] This solution introduces the Equilibrium Focal Loss (EFL) function to fine-tune large language models, addressing the imbalance between difficult-to-classify and easy-to-classify examples when processing data. By assigning higher weights to difficult-to-classify examples, the EFL function allows the model to better focus on these difficult-to-learn features while mitigating overfitting to easy-to-classify examples.

[0056] This approach allows the model to more accurately identify and process complex defect characteristics when generating reports, improving both report quality and accuracy. This fine-tuning process optimizes the loss function through backpropagation, allowing the model to gradually adapt to specific tasks during training, thereby achieving more accurate automatic report generation.

[0057] The following describes a detection report generation device based on feature recognition provided by the present application. The detection report generation device based on feature recognition described below corresponds to the detection report generation method based on feature recognition described above.

[0058] See also Figure 3The inspection report generation device 300 based on feature recognition is characterized by comprising: an acquisition unit 301 for acquiring a target non-destructive inspection image; a generation unit 302 for inputting the target non-destructive inspection image into a multimodal large model for generating an inspection report, and obtaining a target inspection report output by the multimodal large model for generating an inspection report; wherein the multimodal large model for generating an inspection report comprises an image semantic extraction model for acquiring target global image text features of the target non-destructive inspection image, an object detection model for acquiring target local image features and target defect detection results of the target non-destructive inspection image, and a report generation model for generating the target inspection report according to the target defect detection results and target fusion features, wherein the target fusion features comprise the target global image text features and the target local image features.

[0059] In a possible embodiment, the feature recognition-based inspection report generation device 300 further includes a training unit, which is specifically used to train a report generation model in the following manner: obtaining a reference non-destructive inspection image from training data; inputting the reference non-destructive inspection image into the image semantic extraction model to obtain a reference global image text feature; inputting the reference non-destructive inspection image into the object detection model to obtain a reference local image feature and a reference defect detection result; obtaining a reference fusion feature based on the reference global image text feature and the reference local image feature; and updating the model parameters of the language model based on the reference fusion feature and the reference defect detection result to obtain the report generation model.

[0060] In a possible embodiment, the object detection model includes a YOLO model, and inputting the reference nondestructive testing image into the object detection model to obtain reference local image features and reference defect detection results includes: inputting the reference nondestructive testing image into the YOLO model, extracting initial local image features from an intermediate convolutional layer of the YOLO model; obtaining a defect detection result from the YOLO model, and visually marking the initial local image features according to the defect detection result to obtain the reference local image features.

[0061] In a possible embodiment, in terms of obtaining reference fusion features based on the reference global image text features and the reference local image features, the training unit is specifically used to: perform transformer encoding on the reference global image text features to obtain transformer features; and obtain the reference fusion features based on the reference local image features and the transformer features.

[0062] In a possible embodiment, in terms of obtaining the reference fusion feature based on the reference local image feature and the transformer feature, the training unit is specifically used to: obtain an initial fusion feature based on the reference local image feature and the transformer feature; match the initial fusion feature with the semantic features in the historical defect diagnosis report library to obtain at least one diagnosis report text with the highest similarity; extract the semantic information in the at least one diagnosis report text to obtain a report feature; and obtain the reference fusion feature based on the initial fusion feature and the report feature.

[0063] In one possible embodiment, in terms of updating the model parameters of the language large model based on the reference fusion feature and the reference defect detection result to obtain the report generation model, the training unit is specifically used to: obtain the projection mapping matrix of the language token corresponding to the image and the language large model; obtain the target token of the reference fusion feature based on the projection mapping matrix; input the target token and the reference defect detection result into the language large model to obtain a reference detection report; and update the model parameters of the language large model based on the reference detection report to obtain the report generation model.

[0064] In one possible embodiment, in terms of updating the model parameters of the large language model according to the reference detection report to obtain the report generation model, the training unit is specifically used to: update the model parameters of the large language model through the balanced focus loss function according to the reference detection report to obtain the report generation model.

[0065] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of the electronic device provided by this application. Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440. The processor 410, the communications interface 420, and the memory 430 communicate with each other via the communications bus 440. The processor 410 may invoke logic instructions in the memory 430 to execute a method for generating an inspection report based on feature recognition. The method includes: acquiring a nondestructive inspection image of a target; inputting the nondestructive inspection image of the target into a large multimodal model for generating an inspection report, and obtaining a target inspection report output by the large multimodal model for generating an inspection report. The large multimodal model for generating an inspection report includes an image semantic extraction model for acquiring target global image text features of the target nondestructive inspection image; an object detection model for acquiring target local image features and target defect detection results of the target nondestructive inspection image; and a report generation model for generating the target inspection report based on the target defect detection results and target fusion features, wherein the target fusion features include the target global image text features and the target local image features.

[0066] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0067] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the feature recognition-based inspection report generation method provided by the above-mentioned methods, the method comprising: obtaining a target non-destructive inspection image; inputting the target non-destructive inspection image into a multimodal large model for generating an inspection report, and obtaining a target inspection report output by the multimodal large model for generating an inspection report; wherein the multimodal large model for generating an inspection report includes an image semantic extraction model for obtaining target global image text features of the target non-destructive inspection image, an object detection model for obtaining target local image features and target defect detection results of the target non-destructive inspection image, and a report generation model for generating the target detection report based on the target defect detection results and target fusion features, wherein the target fusion features include the target global image text features and the target local image features.

[0068] On the other hand, the present application also provides a computer program product, including a computer program, which is executed by a processor to generate a test report based on feature recognition, the method including: obtaining a target non-destructive testing image; inputting the target non-destructive testing image into a test report generation multimodal large model to obtain a target detection report output by the test report generation multimodal large model; wherein the test report generation multimodal large model includes an image semantic extraction model for obtaining target global image text features of the target non-destructive testing image, an object detection model for obtaining target local image features and target defect detection results of the target non-destructive testing image, and a report generation model for generating the target detection report based on the target defect detection results and target fusion features, the target fusion features including the target global image text features and the target local image features.

[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0070] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for generating a test report based on feature recognition, characterized in that: include: Acquire target nondestructive testing images; Inputting the target nondestructive testing image into the test report generation multimodal large model to obtain the target detection report output by the test report generation multimodal large model; Among them, the multimodal large model for generating the inspection report includes an image semantic extraction model for obtaining the target global image text features of the target non-destructive inspection image, an object detection model for obtaining the target local image features and target defect detection results of the target non-destructive inspection image, and a report generation model for generating the target inspection report based on the target defect detection results and target fusion features, wherein the target fusion features include the target global image text features and the target local image features.

2. The method according to claim 1, characterized in that The report generation model is trained based on the following method: Obtain reference NDT images from training data; Inputting the reference nondestructive testing image into the image semantic extraction model to obtain reference global image text features; Inputting the reference nondestructive testing image into the object detection model to obtain reference local image features and reference defect detection results; Acquire a reference fusion feature according to the reference global image text feature and the reference local image feature; The model parameters of the language large model are updated according to the reference fusion features and the reference defect detection results to obtain the report generation model.

3. The method according to claim 2, characterized in that The object detection model includes a YOLO model, and inputting the reference nondestructive testing image into the object detection model to obtain reference local image features and reference defect detection results includes: Inputting the reference nondestructive testing image into the YOLO model, and extracting initial local image features from the middle convolutional layer of the YOLO model; Obtain the defect detection result from the YOLO model, and visually mark the initial local image features according to the defect detection result to obtain the reference local image features.

4. The method according to claim 2, characterized in that The obtaining of a reference fusion feature according to the reference global image text feature and the reference local image feature includes: Performing transformer encoding on the reference global image text feature to obtain a transformer feature; The reference fusion feature is obtained according to the reference local image feature and the transformer feature.

5. The method according to claim 4, characterized in that The obtaining the reference fusion feature according to the reference local image feature and the transformer feature includes: Acquire initial fusion features according to the reference local image features and the transformer features; Matching the initial fusion features with semantic features in a historical defect diagnosis report library to obtain at least one diagnosis report text with the highest similarity; Extracting semantic information from the at least one diagnostic report text to obtain report features; The reference fusion feature is obtained according to the initial fusion feature and the report feature.

6. The method according to claim 2, characterized in that The updating of the model parameters of the language large model according to the reference fusion feature and the reference defect detection result to obtain the report generation model includes: Obtaining a projection mapping matrix of the image and the language tokens corresponding to the large language model; Obtaining a target token of the reference fusion feature according to the projection mapping matrix; Inputting the target token and the reference defect detection result into the language macro model to obtain a reference detection report; The model parameters of the large language model are updated according to the reference detection report to obtain the report generation model.

7. The method according to claim 6, characterized in that The updating of the model parameters of the large language model according to the reference detection report to obtain the report generation model includes: According to the reference detection report, the model parameters of the language large model are updated through the balanced focus loss function to obtain the report generation model.

8. A detection report generating device based on feature recognition, characterized in that: include: An acquisition unit, used for acquiring a target nondestructive testing image; a generating unit, configured to input the target nondestructive testing image into a test report generating multimodal large model, and obtain a target test report output by the test report generating multimodal large model; Among them, the multimodal large model for generating the inspection report includes an image semantic extraction model for obtaining the target global image text features of the target non-destructive inspection image, an object detection model for obtaining the target local image features and target defect detection results of the target non-destructive inspection image, and a report generation model for generating the target inspection report based on the target defect detection results and target fusion features, wherein the target fusion features include the target global image text features and the target local image features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for generating a detection report based on feature recognition as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a detection report based on feature recognition as described in any one of claims 1 to 7 is implemented.

Images