A method and device for generating a graphic report and a storage medium

Abstract

The application provides a kind of graphic report generation method, device and storage medium, comprising: according to file type, signal analysis data is classified, and pre-processing is carried out, to determine the image data and text data of signal analysis data;Determine the first image processing model according to image data, and the image data is processed using the first image processing model, to determine the first image feature vector;Determine the first text processing model according to text data, and the first image feature vector and text data are processed using the first text processing model, to generate the first literal description;Using retrieval enhancement generation model, the first image feature vector and the first literal description are fused, to generate the first graphic report of signal analysis data.The application generates graphic report by using retrieval enhancement generation model and multi-model cooperative processing technology, which can provide detailed context explanation and logical association, thereby improving the generation efficiency and professionalism of graphic report.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and particularly relates to a method and device for generating a picture-text report and a storage medium. BACKGROUND

[0002] In the technical field of data analysis, especially in the scene of a multi-modal data processing system combining images and texts, more and more application scenarios require efficient and intelligent picture-text generation methods. Traditional picture-text report generation methods for signal analysis data usually rely on manual input of texts and require manual interpretation of image data. Such methods not only have the problem of low efficiency, but also are easily disturbed by human factors, and the quality of the generated picture-text report is poor. SUMMARY

[0003] The present application provides a method and device for generating a picture-text report and a storage medium to solve the problem of low efficiency and poor quality of the existing picture-text report generation for signal analysis data.

[0004] Based on the above problems, in a first aspect, the present application provides a method for generating a picture-text report, comprising:

[0005] classifying and preprocessing signal analysis data according to file types to determine image data and text data of the signal analysis data;

[0006] determining a first image processing model according to the image data and processing the image data using the first image processing model to determine a first image feature vector;

[0007] determining a first text processing model according to the text data and processing the first image feature vector and the text data using the first text processing model to generate a first textual description;

[0008] fusing the first image feature vector and the first textual description using a retrieval enhancement generation model to generate a first picture-text report of the signal analysis data.

[0009] In combination with the first aspect, in a possible implementation, the method for determining a first image processing model according to the image data comprises:

[0010] determining an image data type according to image metadata of the image data or using a lightweight classification model to classify the image data;

[0011] determining a first image processing model according to the image data type and a matching relationship between the image data type and an image processing model.

[0012] With reference to the first aspect, in a possible implementation manner, the method further includes: determining the first image processing model in the following manner:

[0013] reading a performance margin of the computing system;

[0014] determining the first image processing model according to the performance margin of the computing system.

[0015] With reference to the first aspect, in a possible implementation manner, the first image feature vector includes: target detection region attributes and anomaly detection region attributes.

[0016] The target detection region attributes include: target detection region types, target detection region positions, target detection region sizes and target detection region labels.

[0017] The anomaly detection region attributes include: anomaly detection region types, anomaly detection region positions, anomaly detection region sizes and anomaly detection region labels.

[0018] With reference to the first aspect, in a possible implementation manner, the determining the first text processing model according to the text data includes:

[0019] determining text processing model selection elements according to the text data; wherein the text processing model selection elements include: text types, text complexities and target output styles;

[0020] determining the first text processing model according to the text processing model selection elements.

[0021] With reference to the first aspect, in a possible implementation manner, the fusing the first image feature vector and the first text description by using the retrieval enhancement generation model to generate the first image-text report of the signal analysis data includes:

[0022] constructing a retrieval knowledge base;

[0023] taking the first image feature vector as retrieval information, retrieving the retrieval knowledge base by using the retrieval enhancement generation model to determine a first retrieval result;

[0024] inputting the first retrieval result and the first text description into the first text processing model to generate the first image-text report of the signal analysis data.

[0025] With reference to the first aspect, in a possible implementation manner, the method further includes: determining a value of top-k in the first retrieval result according to an image data type.

[0026] With reference to the first aspect, in a possible implementation manner, the method further includes:

[0027] Adjust the relevance of the image and the text in the first picture-text report, and generate a second picture-text report of the signal analysis data, comprising:

[0028] Adopt an image understanding model to perform semantic regression on the first picture-text report, and determine a consistency evaluation result;

[0029] In the case where the consistency evaluation result is lower than a preset score, a semantic reasoning model is used to determine the missing content of the text in the first picture-text report;

[0030] The text missing content is used as retrieval information, and a retrieval enhanced generation model is used to retrieve the retrieval knowledge base to determine a second retrieval result;

[0031] The second retrieval result and the first picture-text report are input into a first text processing model to generate a second picture-text report of the signal analysis data.

[0032] In a second aspect, a picture-text report generation device is provided, comprising:

[0033] A preprocessing module is configured to classify and preprocess signal analysis data according to file types, and determine image data and text data of the signal analysis data;

[0034] An image processing module is configured to determine a first image processing model according to the image data, and process the image data by using the first image processing model to determine a first image feature vector;

[0035] A text processing module is configured to determine a first text processing model according to the text data, and process the first image feature vector and the text data by using the first text processing model to generate a first textual description;

[0036] A fusion module is configured to fuse the first image feature vector and the first textual description by using a retrieval enhanced generation model to generate a first picture-text report of the signal analysis data.

[0037] In a third aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the picture-text report generation method according to the first aspect or any possible implementation manner of the first aspect.

[0038] The beneficial effects of the present application include:

[0039] The present invention provides a method, apparatus, and storage medium for generating graphic reports, comprising: classifying signal analysis data according to file type and preprocessing it to determine image data and text data of the signal analysis data; determining a first image processing model based on the image data and processing the image data using the first image processing model to determine a first image feature vector; determining a first text processing model based on the text data and processing the first image feature vector and text data using the first text processing model to generate a first text description; and using a retrieval-enhanced generation model to fuse the first image feature vector and the first text description to generate a first graphic report of the signal analysis data. The graphic report generation method provided by the present invention, by combining the first image processing model and the first text processing model, and employing a retrieval-enhanced generation model and multi-model collaborative processing technology, can more accurately handle the association between image data and text data, deepen contextual understanding, and generate graphic reports that can provide detailed contextual explanations and logical connections, enabling precise fusion of images and text, thereby improving the generation efficiency and professionalism of graphic reports and achieving an efficient and intelligent graphic report generation method. Attached Figure Description

[0040] Figure 1 A flowchart of the graphic report generation method provided by the present invention;

[0041] Figure 2 This is a structural diagram of the graphic report generation device provided by the present invention. Detailed Implementation

[0042] This invention provides a method, apparatus, and storage medium for generating graphic reports. Preferred embodiments of the invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the invention. Furthermore, the embodiments and features described herein can be combined with each other unless otherwise specified.

[0043] This invention provides a method for generating graphic reports, such as... Figure 1 As shown, it includes:

[0044] S101. Classify the signal analysis data according to the file type and perform preprocessing to determine the image data and text data of the signal analysis data;

[0045] S102. Determine a first image processing model based on the image data, and use the first image processing model to process the image data to determine the first image feature vector;

[0046] S103. Determine a first text processing model based on the text data, and use the first text processing model to process the first image feature vector and text data to generate a first text description;

[0047] S104, adopt the retrieval enhancement generation model to fuse the first image feature vector and the first text description, and generate a first image-text report of the signal analysis data.

[0048] The present application is in the technical field of data analysis, particularly in the field of multi-modal data processing system combining image and text. Signal analysis data can refer to feature extraction, analysis and interpretation of various signals to reveal the information contained in various signals. Traditional image-text report generation methods of signal analysis data usually rely on manual input of text to explain image data. This method not only has the problem of low efficiency, but also is easily disturbed by human factors, and the quality of the generated image-text report is poor. With the evolution of artificial intelligence technology, especially the development of image recognition, natural language processing (NLP) and multi-modal learning, more and more systems try to combine image and text, however, existing artificial intelligence technology still faces many technical challenges in the field of image-text report generation for signal analysis data, especially in the aspects of accurate fusion of image and text content, context understanding and processing efficiency.

[0049] The present application can include image and text in signal analysis data, for example, image can include frequency spectrum, waveform, grayscale, color, heat map, etc.; text can include analysis report, technical specification, etc. According to the file type of signal analysis data, such as file extension, image and text are classified, for example,.jpg,.png represents image,.txt,.docx represents text. Preprocess the image, for example, through denoising, normalization, size adjustment, so that the image can adapt to the input of the subsequent first image processing model, so as to obtain the image data of the signal analysis data. Preprocess the text, for example, perform word segmentation, stop word removal, stem extraction, etc. on the text to ensure that the text content can be accurately understood by the first text processing model, and obtain the text data. Image data can be represented as: D image ={I1, I2, …, I n}, where I1, I2, …, I n represent each preprocessed image. Text data can be represented as: D text ={T1, T2, …, T n}, where T1, T2, …, T nrepresents each pre-processed text. According to the image data and the text data, the first image processing model is used to process the image data. The first image processing model can be selected according to the type of image data. For example, if the image data belongs to medical imaging, U-Net, SegFormer, and SAM model can be used for image segmentation, and YOLOv8 model can be used for target detection to extract the type and size of the target detection region from the image data, and generate the label of the target detection region. The formula for determining the first image processing model is as follows: M image =Y OLO or Segformer. Thus, the first image feature vector is obtained. The formula for the first image feature vector is as follows: F image =M image (D image ), wherein M image represents the first image processing model. The first text processing model is used to process the first image feature vector and the text data to generate the first textual description. The first text processing model can be determined according to the characteristics of the text data and the model. For example, the BioGPT (Generative Pre-trained Transformer for Biomedical Text Generation and Mining) is suitable for the medical field, and in the case of a large number of medical field terms in the text data, the BioGPT model can be selected as the first text processing model. The formula for determining the first text processing model is as follows: M text =LLaMA or GPT; wherein LLaMA (Large Language Model Application) and GPT (Generative Pre-trained Transformer) represent optional first text processing models. The first text processing model is used to generate the first textual description related to the image content according to the first image feature vector and the text data. The first textual description can include a textual description of the image content and a textual description of the combination of the image content and the text data. The formula for the first textual description is as follows: R text =M text (F image , D text ), wherein M text represents the first text processing model.

[0050] Further, a retrieval-augmented generation (RAG) model is used to deeply integrate the first image feature vector and the first textual description, ensuring high consistency and context association of the image-text content. The RAG module uses the first image feature vector as retrieval information to ensure that the generated textual description is closely related to the details of the image content, guiding the first text processing model to generate an accurate first image-text report. The formula for generating the first image-text report is: R fusion =RAG(F image , R text ), where RAG represents the use of a retrieval-augmented generation model to deeply integrate the first image feature vector and the first textual description. The first image-text report can include various output formats such as HTML, PDF, etc., and supports viewing and downloading.

[0051] By combining the first image processing model and the first text processing model, using the retrieval-augmented generation model and multi-model collaborative processing technology, the present application can more accurately process the association of image data and text data, deepen the understanding of the context, and generate an image-text report that can provide detailed contextual explanations and logical associations, enabling accurate fusion of images and text, thereby improving the efficiency and professionalism of image-text report generation and achieving an efficient and intelligent image-text generation method.

[0052] In another embodiment of the present application, the first image processing model is determined based on the image data, comprising:

[0053] Step 1: Classify the image data based on the image metadata of the image data or using a lightweight classification model to determine the image data type;

[0054] Step 2: Determine the first image processing model based on the image data type, the matching relationship between the image data type and the image processing model.

[0055] The application first classifies the image data according to the image metadata or the lightweight classification model, determines the image data type, selects the most suitable first image processing model according to the matching relationship between the preset image data type and the image processing model, and realizes the accurate matching of the model and the image data. For step 1, the image metadata can be the descriptive information in the image data. The image metadata can include: image file name, path label, exchangeable image file format (EXIF, Exchangeable Image File Format) information. The image file name includes the naming information of the image. The path label includes the storage path of the image. The EXIF information includes shooting device parameters, shooting time information, shooting location information, content label, etc. The shooting device parameters can include resolution, color depth, format (JPG / PNG, etc.), medical CT layer thickness, industrial camera exposure time; the content label can include the category identifier generated by manual annotation or file naming rule, such as “lung CT” and “gear defect image”. The image data type can be determined by analyzing the key fields in the image metadata of the image data, for example, the image data type is determined as medical image data when the “DICOM” format identifier is included in the image metadata of the image data and the content label is “brain”. A lightweight classification model is obtained by using a neural network with less parameters and low computational complexity (such as MobileNet, ShuffleNet) and focusing on coarse-grained classification of image types after lightweight training. The image data is classified by using the lightweight classification model. The lightweight classification model can extract basic image distribution features such as outline, texture, CT gray scale, X-ray radiation pattern, and industrial edge intensity through a shallow convolutional layer, output classification probability, for example, 60% probability is “industrial detection image”, 30% is “education image”, and the highest probability category is selected as the image data type. The image data type can represent the field of the image, such as medical image, industrial detection image, education image, and general vision image. For step 2, a mapping table of “image data type and image processing model matching relationship” is established, for example, generated based on historical experimental data or industry experience, for example, as shown in Table 1:

[0056]

[0057] Table 1

[0058] According to the image data type, the image data type and the image processing model matching relationship, the first image processing model is determined. If the corresponding image processing model includes multiple, a filtering parameter can be added, for example, the processing speed priority selects YOLOv8, and the accuracy priority selects Faster R-CNN, so as to determine the first image processing model. According to the image data type, the first image processing model is customized and selected, for example, medical images and industrial detection features are different, and different models are needed, and the adaptive model can improve the feature extraction accuracy. Different types of images need specific model structures (such as the convolution kernel size of CNN and attention mechanism) for key features (such as the lesion texture of medical images and the defect contour of industrial images), and the first image processing model can strengthen the feature extraction ability. Without manual debugging or traversing all models, the classification and matching rules are automatically determined, the dependence on manual experience and the trial and error cost are reduced, and the standardized image processing scene is suitable.

[0059] In another embodiment of the application, the first image processing model is determined in the following way:

[0060] Step 1, read the performance margin of the computing system;

[0061] Step 2, according to the performance margin of the computing system, the first image processing model is determined.

[0062] The application first reads the performance margin of the computing system, such as CPU / GPU computing power, memory capacity, etc., and then matches the corresponding image processing model according to the specific indicators of the performance margin of the computing system, such as the percentage of residual computing power, the idle memory capacity, to realize the dynamic adaptation of model complexity and system resources. For step 1, the performance margin of the computing system can be obtained by calling the computing system API. The performance margin of the computing system includes: CPU idle rate, GPU idle rate, and physical memory idle capacity. For step 2, according to the performance margin of the computing system, a first image processing model is determined, for example, when real-time processing is required or when running on a resource-limited computing system, a lighter image processing model (such as MobileNet) is used, which has higher computing efficiency and is suitable for edge computing applications. According to the performance margin of the real-time computing system, the first image processing model is selected, which avoids the waste of resources caused by "large model small task" (such as using ResNet to process simple image classification) or the performance bottleneck caused by "small model large task" (such as using MobileNet to process high-resolution medical images). In a resource-constrained situation (such as when the performance margin is lower than a threshold), a lightweight model is automatically selected to prevent system overload and crash, especially suitable for edge computing devices or multi-task parallel scenarios, to ensure system stability. High-performance systems can call complex models to improve accuracy, and low-performance systems can use lightweight models to ensure real-time performance. For example, when the GPU idle rate is greater than 70%, the Transformer model is selected, and when it is less than 30%, the CNN lightweight variant is switched to, which can improve processing efficiency. And without manual intervention in model deployment, the resource state is automatically adapted, reducing the workload of manual configuration, and supporting automatic management of large-scale distributed computing environments.

[0063] In another embodiment of the application, the first image feature vector includes: target detection region attributes and anomaly detection region attributes.

[0064] The target detection region attributes include: target detection region type, target detection region position, target detection region size, and target detection region label.

[0065] The anomaly detection region attributes include: anomaly detection region type, anomaly detection region position, anomaly detection region size, and anomaly detection region label.

[0066] The first image feature vector can be a structured description of key regions in image data, including target detection region attributes and anomaly detection region attributes. The target detection region attributes are used to describe the characteristics of normal targets in the image (type, location, size, label), and the anomaly detection region attributes focus on the feature description of abnormal regions, which together constitute the semantic feature representation of the image and provide structured data support for subsequent analysis. The target detection region type can describe the physical category or functional attribute of the detected target. For example, "gear", "bolt", "circuit board" in industrial detection, "lung", "liver", "tumor" in medical images. The target detection region location represents the spatial coordinates of the target in the image, usually represented by a bounding box (Bounding Box), for example (x1, y1, x2, y2) represents the position coordinates of the upper left corner and the lower right corner. The target detection region size represents the geometric size of the target region. The target detection region label can refer to the standardized coding or text description of the target type, which needs to comply with the preset label system, for example, industrial detection image label: {"gear":"GEAR_001", "bolt":"BOLT_002"}; medical image label: {"lung":"LUNG_01"}. The target detection region label can ensure that the detection results of different images and different image processing models can be uniformly stored and retrieved, for example, all gear images can be batch extracted through the label "GEAR_001". The target detection region attribute represents "objects that should exist", while the anomaly detection region attribute represents "defects that should not exist on the object", for example, the target detection region type in the target detection region attribute is "circuit board", and the anomaly detection region type in the anomaly detection region attribute is "virtual welding of solder joints on the circuit board". The anomaly detection region attribute can be applied to the generation of image-text reports. For example, in the process of generating image-text reports for medical images, the anomaly detection region attribute can assist in generating diagnostic explanations, such as locating tumors, bleeding, lesions, and organ abnormalities. An example of the generated image-text report is "a high-density shadow appears in the lower lobe of the left lung, with irregular edges, suspected to be a nodular lesion". In the process of generating image-text reports for industrial detection, the anomaly detection region attribute can assist in generating defect detection reports, such as marking surface cracks, scratches, foreign objects, or structural omissions, etc. An example of the generated image-text report is "a linear crack appears in the upper left corner of the aluminum part, about 2.1mm long and 0.3mm deep". In the process of generating image-text reports for educational images, the anomaly detection region attribute can be used to mark key areas in the teaching materials (such as cell structure abnormalities, building structure errors, etc.), and automatically generate explanations or comments. In the process of generating image-text reports for general vision, the anomaly detection region attribute can be used to identify intruders, illegal parking, and abnormal fire sources, and an example of the generated image-text report is "unauthorized personnel have been detected in this area for more than 30 seconds".The anomaly detection region can include: a region where a difference is expected to exist, an anomaly region identified by a model prediction result, a region highly coinciding with a manually annotated anomaly set, a model prediction low confidence region, etc. For the region where a difference is expected to exist, the statistical distribution of the normal sample is significantly different, such as gray scale, texture, shape deviation, and asymmetric region in brain image, discontinuous copper wire in PCB. For the anomaly region identified by the model prediction result, for example, the pixels or regions in the image data are detected by the classifier and segmentation model, and the parts with defects or abnormalities are marked. The YOLOv8 model is used to mark the bounding box of specific defects such as cracks. For the region highly coinciding with the manually annotated anomaly set, for example, the ground truth annotated by radiologists or quality control personnel is compared to confirm those highly coinciding regions. For the model prediction low confidence region, a model uncertainty analysis method such as Monte Carlo Dropout (MC Dropout) or model ensemble is used to identify the model low confidence region. The anomaly detection region type can describe the specific morphology or cause of the anomaly phenomenon, for example: "crack", "wear", "deformation", "missing part" in industrial detection; "nodule", "hemorrhage", "fibrosis" in medical images. The position of the anomaly detection region is consistent with the position of the target detection region, for example, in a gear image, the anomaly position is (200, 250, 220, 270), indicating the defect at the gear tooth tip. The size of the anomaly detection region represents the geometric size of the anomaly region, for example, the length of the crack (such as 10 mm), the depth of the wear (such as 0.5 mm); the proportion of the anomaly area to the target area (such as the virtual welding area of the welding spot accounts for 20% of the total welding spot area). The label of the anomaly detection region represents the standardized coding of the anomaly type, which can include severity or processing suggestions, for example, industrial detection image label: {"crack-mild": "CRACK_01", "crack-severe": "CRACK_02"}; medical image label: {"benign nodule": "NODULE_BENIGN", "malignant nodule": "NODULE_MALIGNANT"}. The formula for determining the properties of the anomaly detection region is: R. anomalous = M detect (F image ), wherein M detect represents the detection module of the first image processing model. The first image processing model is used to process the image data, and the determined first image feature vector includes the target detection region attribute and the anomaly detection region attribute, so that the model can simultaneously process the "target recognition" and "anomaly detection" tasks, for example, in medical images, both organ position (target detection) and lesion region (anomaly detection) are labeled, realizing composite analysis.

[0067] In yet another embodiment of the present application, the first text processing model is determined according to the text data, comprising:

[0068] Step 1, determining text processing model selection elements according to the text data; wherein the text processing model selection elements include: text type, text complexity and target output style;

[0069] Step 2, determining the first text processing model according to the text processing model selection elements.

[0070] In the present application, the text processing model selection elements of text type, text complexity and target output style are extracted from the text data, and then the corresponding first text processing model is matched according to the text processing model selection elements. For step 1, the text type can refer to the field and genre category to which the text content belongs. For example, abstract, teaching script, medical interpretation, etc. It can be determined by text keyword recognition, topic model analysis or pre-defined text classifier. For example, if the words such as “lung” and “tumor” frequently appear in the text, it can be determined as medical interpretation. The text complexity can measure the complexity of the text in terms of semantics, syntax, logical structure, etc. and the degree of association between images and text. It is analyzed by using natural language processing technology. For example, the complexity of the proportion of long sentences, the number of low-frequency words, the relationship between sentences, and the number of images associated with the text in the text is calculated. The readability index of the text (such as Flesch-Kincaid readability score) is used to quantify the complexity, and the lower the score, the more difficult the text is to understand, and the higher the complexity. The target output style can refer to the language style and expression characteristics that the user expects the generated text to present, such as professional terms, easy-to-understand, colloquial, teaching-oriented, scientific style, etc. The target output style can be explicitly specified by user input or inferred based on the text purpose and audience group. For step 2, the first text processing model is determined according to the text processing model selection elements. A corresponding table of “text processing model selection elements and first text processing model” is constructed, for example, as shown in Table 2:

[0071]

[0072] Table 2

[0073] The most suitable first text processing model under different combinations of text processing model selection elements can be determined based on experiments and practical application experience. When there are multiple candidate first text processing models, further screening can be performed according to other factors, such as the computing resource requirement, processing speed, diversity of generated text, and the like, of the first text processing model. For example, on a device with limited resources, a lightweight model with less parameters and faster running speed is preferred; if the diversity and innovation of the text are pursued, a model with stronger generation capability is selected. Selecting a suitable first text processing model according to the text processing model selection elements can ensure that the model is adapted to the subject and structural characteristics of the text, for example, when processing an academic paper, a model good at logical reasoning and professional term expression is selected, which can improve the accuracy of text processing.

[0074] In yet another embodiment of the present application, a retrieval enhancement generation model is used to fuse the first image feature vector and the first textual description to generate a first graphic-text report of the signal analysis data, comprising:

[0075] Step 1, constructing a retrieval knowledge base;

[0076] Step 2, using the first image feature vector as retrieval information, using the retrieval enhancement generation model to retrieve the retrieval knowledge base to determine a first retrieval result;

[0077] Step 3, inputting the first retrieval result and the first textual description into a first text processing model to generate a first graphic-text report of the signal analysis data.

[0078] The present application firstly constructs a retrieval knowledge base including signal analysis data field knowledge; then uses the first image feature vector as the retrieval condition, and uses the retrieval enhancement generation model to match the relevant information from the retrieval knowledge base to obtain the first retrieval result; finally, the first retrieval result and the first text description are input into the first text processing model to generate a structured first image-text report, realizing the deep combination of field knowledge, image data and text data. For step 1, the retrieval knowledge base can integrate the standard specifications, historical cases and other authoritative knowledge in the signal analysis data field, avoiding the model generating incorrect conclusions only based on the image feature vector and the text data. The retrieval knowledge base can include: typical image data abnormal samples, expert report fragments containing annotation information, standard sentence templates, domain terminology ontology library, and multi-modal comparison samples. The typical image data abnormal sample can be an image sample in the image data set that has obvious differences from normal images. These differences may be due to errors or interference factors in the data collection process, or abnormal conditions reflected by the image itself. The expert report fragment containing annotation information can be a report content fragment that has been reviewed and annotated by experts. The annotation information usually includes classification, key marking, explanation and other information of the report content. The standard sentence template can be a pre-designed sentence structure template that meets the requirements of a specific field or purpose. These templates are usually used to generate standardized text content to ensure consistency and standardization of information expression. The domain terminology ontology library can refer to a terminology system constructed for a specific domain, including professional vocabulary, concepts and their relationships within the domain. The domain terminology ontology library not only lists the terms, but also defines the hierarchical structure and relationship between the terms. The multi-modal comparison sample can be a data sample that includes multiple modalities (such as text and image), and these modalities have corresponding relationships. Multi-modal comparison samples are used to study the association and consistency between different modalities of data. For step 2, the first image feature vector is converted into a semantic vector through a pre-trained cross-modal model, such as a contrastive language-image pre-training model (CLIP). The semantic vector is used as retrieval information, and a retrieval enhancement generation model is used to retrieve the retrieval knowledge base. According to the semantic similarity, the top-k is searched to obtain the first retrieval result. For step 3, the first retrieval result and the first text description are input into the first text processing model. The first text description includes feature interpretation based on image data and text data, and the first retrieval result combines background knowledge from image data and retrieval knowledge base, so that the first image feature vector and the first text description are fused. Through the first text processing model, a first image-text report of signal analysis data is generated.For example, for image data being CT images, the abnormal detection area attribute of the first image feature vector is "left lower lobe high-density area with a diameter of 1.4 cm", and the first textual description is "a high-density area with a diameter of 1.4 cm appears in the left lower lobe, with blurred edges". Taking the first image feature vector as the retrieval information, the retrieval enhanced generation model is used to retrieve the retrieval knowledge base, and it is concluded that the first retrieval result includes "a 1.2 cm diameter shadow appears in the left upper lobe, and further CT is recommended" and "CT discovers a ground glass opacity in the right middle lobe, which is an early lesion". The first retrieval result and the first textual description are input into the first text processing model to generate the first image-text report of the signal analysis data, which is displayed as "according to the CT images, a high-density shadow is observed in the left lower lobe, with a range of about 1.4 cm and blurred edges, and further imaging examination is recommended to exclude early lesions". The retrieval enhanced generation model not only fuses the first image feature vector and the first textual description, but also ensures that the generated text content is highly consistent with the image content through the enhancement mechanism of the retrieval knowledge base. The retrieval enhanced generation model guides the first text processing model through the first image feature vector, so that the generated first image-text report can be dynamically adjusted based on the content and context of the image, thereby generating more accurate descriptions. Through the retrieval enhancement of the first image feature vector, the context consistency between the image and the textual description is ensured. The generated text is no longer a simple description of the image content, but is intelligently generated according to the details and context in the image. The retrieval enhanced generation model can dynamically adjust the content of the generated text according to the image content through intelligent reasoning and enhancement, so that the first image-text report is more consistent with the actual scene requirements, and the accuracy, logic and professionalism are improved.

[0079] In another embodiment of the present application, the value of top-k in the first retrieval result is determined according to the type of image data.

[0080] The present application determines the value of top-k (i.e. returns the top-k search results) of the first search result according to the image data type (such as medical images, industrial detection). The value of top-k indicates that the search enhanced generation model searches the search knowledge base, and selects the most relevant k results from a large number of candidate data according to the similarity measure. Image data types can include: medical images, industrial detection images, educational images, general visual images. Medical images can refer to images generated by various imaging techniques for medical diagnosis, treatment and research, which can reflect the structure and function of the human body. Industrial detection images can refer to images generated by various imaging techniques for detecting product quality, equipment status and production process in industrial production processes. Educational images can refer to images used for educational and teaching purposes, including textbook illustrations, teaching demonstration images, experimental images, etc. These images help students better understand and master knowledge. General visual images can refer to images commonly seen in daily life that are not specific to a particular field. These images can be used for various visual tasks such as image recognition, classification, retrieval, etc. Different image data types adapt to different top-k values due to differences in domain knowledge complexity and feature diversity, achieving adaptive optimization of the number of search results. The setting of the top-k value can optimize accuracy and coverage. For example, for medical images, the value of top-k is recommended to be 3 to 5 to improve accuracy, and for industrial detection images, educational images, and general visual images, the value of top-k is recommended to be 5 to 10 to improve coverage.

[0081] In another embodiment of the present application, the method further comprises:

[0082] Adjusting the correlation between images and text in the first graphic report to generate a second graphic report of signal analysis data, comprising:

[0083] Step 1: using an image understanding model to perform semantic regression on the first graphic report to determine a consistency evaluation result;

[0084] Step 2: in the case where the consistency evaluation result is lower than a preset score, using a semantic reasoning model to determine the missing content of the text in the first graphic report;

[0085] Step 3: using the missing content of the text as search information, using a search enhanced generation model to search the search knowledge base to determine a second search result;

[0086] Step 4: inputting the second search result and the first graphic report into a first text processing model to generate a second graphic report of signal analysis data.

[0087] This invention employs a retrieval-enhanced generative model. Through a feedback mechanism and inference process, it adjusts the correlation between images and text to optimize the first image-text report and generate a second image-text report based on signal analysis data. The formula for generating the second image-text report is expressed as: R enhanced =Inference(R fusion ); where R fusion This represents the first image-text report; Inference represents the feedback mechanism and reasoning process. For step 1 above, an image understanding model is used to perform semantic back-reference on the first image-text report, determining whether the text descriptions in the report correctly correspond to the image content, and thus determining the consistency assessment result. For image data D... image Using an image understanding model, image embedding vectors are generated. Regarding the textual description R of the first graphic report fusion Extracted text embedding vectors The formula for the consistency assessment results is described as follows:

[0088]

[0089] in, Cosine similarity is used to measure the semantic consistency between an image and text. For example, the CLIP and BLIP-2 models are used to perform semantic back-matching on the first image-text report. By calculating the similarity of semantic features between the image and text, a quantitative score for image-text consistency is given. For example, if the peak signal frequency shown in the image matches the frequency data mentioned in the text highly, the consistency score is high; conversely, if the image shows an abnormal region but is not mentioned in the text, the score is low. The final output is the consistency evaluation result, which reflects the closeness of the correlation between the image and text content.

[0090] Regarding step 2 above, if the consistency evaluation result is lower than the preset score, the semantic matching of the first image-text report is insufficient or the content logic is broken. A semantic reasoning model is then used to determine the missing text in the first image-text report. For example, masked language modeling is performed on the text description of the first image-text report to determine if any words are missing. This is then automatically generated or completed in the first image-text report. If there are missing parts in the text, such as "images show increased lung density, [MASK]", the semantic reasoning model will determine the missing content based on the corresponding image context and the trained parameters. Predict the most likely missing words (e.g., "left lung," "nodule," "size," etc.). The formula for missing text content is described as follows:

[0091]

[0092] in, This indicates missing text content. a vocabulary, i.e. a set of all possible words that the semantic inference model can predict. the context of the image corresponding to the given first image-text report and the semantic inference model model parameters the probability of the vocabulary appearing. to find the vocabulary that maximizes the probability .

[0093] For example, key region detection is performed on the image data to determine the region that is not mentioned in the textual description. For example, a template / knowledge guided language model is used to ask questions such as "Does this CT image mention a left-sided lesion?" to determine the missing content in the text.

[0094] For step 3 above, the missing text content is used as retrieval information, and a retrieval enhancement generation model is used to retrieve the retrieval knowledge base again to improve the professionalism of the missing text content, and a second retrieval result is determined. The formula description of the second retrieval result is:

[0095]

[0096] wherein, is the retrieval knowledge base; is the vector representation of the text in the retrieval knowledge base; is the vector representation of the missing text content.

[0097] For step 4 above, the second retrieval result is integrated with the text content in the first image-text report, and is inserted or supplemented to the appropriate position in logical order to make the text content more complete and logical. Ensure that the correspondence between the image and the newly added text content is clear. The integrated data is input into the first text processing model, and the model optimizes the language, sorts out the logic and formats the text, such as adjusting the fluency of the sentence, adding transitional sentences, standardizing the expression of professional terms, etc. Finally, a second image-text report is generated that includes complete image-text information and is closely related, presenting the signal analysis data in a clear, accurate and professional manner. The formula description of the generated second image-text report is:

[0098]

[0099] wherein, is the context of the image corresponding to the second image-text report, is the first text processing model, is the vector representation of the second retrieval result; The first text processing model parameter is processed. Illustratively, the image data is a lung CT, and the textual description of the first image report is "the image shows increased density in the lung, and re-examination is recommended". By using the image understanding model to perform semantic response on the first image report, the first image report does not mention "left lung" or "clear lesion size", and the consistency evaluation result is lower than the preset score. The CLIP model indicates that there is a nodular abnormality in the middle lobe of the left lung, and the semantic reasoning model is used to ask "is this a nodule? How big is it?" to determine the missing content of the text. The missing content of the text is used as retrieval information, and a retrieval enhanced generation model is used to retrieve a retrieval knowledge base to determine a second retrieval result; the second retrieval result and the first image report are input into the first text processing model to generate a second image report of the signal analysis data. The second image report is optimized to generate "a high-density nodule with a diameter of about 1.4 cm appears in the middle lobe of the left lung, and further examination is recommended to exclude early lesions". Through consistency evaluation and missing content supplement, it is ensured that the signal features displayed in the image in the second image report and the text description are mutually echoed, avoiding the case that the image displays abnormal waveforms but the text does not explain, so that the reader can more clearly and accurately understand the signal analysis result. The coherent and complete second image report reduces the difficulty for the reader to understand the signal analysis data, reduces the confusion caused by information gaps or contradictions, and improves the readability and practicality of the report.

[0100] Based on the same inventive concept, the present application also provides an image report generation device. Since the principles of the problems solved by these devices are similar to the aforementioned image report generation method, the implementation of the device can be referred to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0101] The present application provides an image report generation device, as shown in Figure 2 , comprising:

[0102] The preprocessing module 201 is used for classifying and preprocessing the signal analysis data according to the file type, determining the image data and the text data of the signal analysis data;

[0103] The image processing module 202 is used for determining a first image processing model according to the image data, and processing the image data by using the first image processing model to determine a first image feature vector;

[0104] The text processing module 203 is used for determining a first text processing model according to the text data, and processing the first image feature vector and the text data by using the first text processing model to generate a first textual description;

[0105] The fusion module 204 is used for fusing the first image feature vector and the first textual description by using a retrieval enhanced generation model to generate a first image report of the signal analysis data.

[0106] In another embodiment of the present application, the image processing module 202 is configured to determine the image data type by classifying the image data according to image metadata of the image data or using a lightweight classification model.

[0107] The first image processing model is determined according to the image data type and a matching relationship between the image data type and an image processing model.

[0108] In another embodiment of the present application, the image processing module 202 is further configured to determine the first image processing model in the following manner:

[0109] The performance margin of the computing system is read;

[0110] The first image processing model is determined according to the performance margin of the computing system.

[0111] In another embodiment of the present application, the first image feature vector includes target detection region attributes and abnormal detection region attributes.

[0112] The target detection region attributes include target detection region types, target detection region positions, target detection region sizes and target detection region labels.

[0113] The abnormal detection region attributes include abnormal detection region types, abnormal detection region positions, abnormal detection region sizes and abnormal detection region labels.

[0114] In another embodiment of the present application, the text processing module 203 is configured to determine text processing model selection elements according to the text data, wherein the text processing model selection elements include text types, text complexities and target output styles.

[0115] The first text processing model is determined according to the text processing model selection elements.

[0116] In another embodiment of the present application, the fusion module 204 is configured to construct a retrieval knowledge base.

[0117] The first image feature vector is used as retrieval information, the retrieval knowledge base is retrieved using a retrieval enhancement generation model, and a first retrieval result is determined.

[0118] The first retrieval result and the first textual description are input into the first text processing model, and a first image-text report of the signal analysis data is generated.

[0119] In another embodiment of the present application, the fusion module 204 is further configured to determine the value of top-k in the first retrieval result according to the image data type.

[0120] In still another embodiment of the present application, the fusion module 204 is further configured to:

[0121] adjust the correlation degree of the image and the text in the first picture-text report, generate a second picture-text report of the signal analysis data, comprising:

[0122] perform semantic regression on the first picture-text report using an image understanding model to determine a consistency evaluation result;

[0123] in a case where the consistency evaluation result is lower than a preset score, determine a text missing content in the first picture-text report using a semantic reasoning model;

[0124] use the text missing content as retrieval information, perform retrieval on the retrieval knowledge base using a retrieval enhancement generation model to determine a second retrieval result;

[0125] input the second retrieval result and the first picture-text report into a first text processing model to generate a second picture-text report of the signal analysis data.

[0126] Based on the same inventive concept, the present application provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to perform the steps of the picture-text report generation method according to any one of the above embodiments.

[0127] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by hardware, or by means of software and necessary general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0128] Those skilled in the art can understand that the drawings are only schematic of a preferred embodiment, and the modules or flows in the drawings are not necessarily required for implementing the present application.

[0129] Those skilled in the art can understand that the modules in the device in the embodiments can be distributed in the device in the embodiments according to the description of the embodiments, or can be changed and located in one or more devices different from the present embodiment. The modules of the above embodiments can be combined into one module, or can be further split into a plurality of sub-modules.

[0130] The above embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0131] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method of generating a graphic report, characterized by, The method comprises the following steps: classifying and preprocessing signal analysis data according to file types to determine image data and text data of the signal analysis data; determining a first image processing model according to the image data and processing the image data using the first image processing model to determine a first image feature vector; determining a first text processing model according to the text data and processing the first image feature vector and the text data using the first text processing model to generate a first textual description; fusing the first image feature vector and the first textual description using a retrieval enhancement generation model to generate a first image-text report of the signal analysis data; adjusting the correlation between images and texts in the first image-text report to generate a second image-text report of the signal analysis data, comprising: performing semantic back-to-text on the first image-text report using an image understanding model to determine a consistency evaluation result; the formula of the consistency evaluation result is: ； wherein, is a cosine similarity; is an image embedding vector obtained by processing an image in the first image-text report using an image understanding model; is a text embedding vector obtained by processing text in the first image-text report. in the case that the consistency evaluation result is lower than a preset score, determining text missing content in the first image-text report using a semantic reasoning model, comprising: determining text missing content in the first image-text report using a semantic reasoning model according to the context corresponding to the image in the first image-text report; the formula of the text missing content is: ； wherein, is a text missing content; is a vocabulary; is a vocabulary predicted by a semantic inference model; is a context corresponding to the image in the first image-text report; is a semantic inference model parameter; is a context corresponding to the image in the first image-text report and a semantic inference model parameter , the probability of the vocabulary appearing; is the vocabulary that maximizes the probability ; taking the text missing content as retrieval information, retrieving a retrieval knowledge base using a retrieval enhancement generation model to determine a second retrieval result; the formula of the second retrieval result is: ； wherein, to retrieve a knowledge base; to retrieve a vector representation of a text of a knowledge base; to retrieve a vector representation of a text missing content; inputting the second retrieval result and the first image-text report into the first text processing model to generate a second image-text report of the signal analysis data; the formula of the second image-text report is: ； wherein, is a context corresponding to the image in the second image-text report, is a first text processing model, is a vector representation of the second search result; is a first text processing model parameter.

2. The method of claim 1, wherein, the method further comprises the following steps: determining a first image processing model according to the image data, comprising: classifying the image data according to image metadata of the image data or using a lightweight classification model to determine an image data type; 3. The method of claim 1, wherein, determining the first image processing model according to the image data type, a matching relationship between the image data type and an image processing model. The method further comprises the following steps for determining the first image processing model: reading the performance margin of a computing system; 4. The method of claim 1, wherein, determining the first image processing model according to the performance margin of the computing system. The first image feature vector comprises target detection region attributes and abnormal detection region attributes; the target detection region attributes comprise target detection region types, target detection region positions, target detection region sizes and target detection region labels; 5. The method of claim 1, wherein, the abnormal detection region attributes comprise abnormal detection region types, abnormal detection region positions, abnormal detection region sizes and abnormal detection region labels. The method further comprises the following steps for determining the first text processing model according to the text data: determining text processing model selection elements according to the text data; wherein the text processing model selection elements comprise text types, text complexities and target output styles; determining the first text processing model according to the text processing model selection elements.

6. The method of claim 1, wherein, The retrieval enhancement generation model is used for fusing the first image feature vector and the first textual description to generate the first picture-text report of the signal analysis data. A retrieval knowledge base is constructed. The first image feature vector is used as retrieval information to retrieve the retrieval knowledge base by using the retrieval enhancement generation model to determine a first retrieval result. The first retrieval result and the first textual description are input into a first text processing model to generate the first picture-text report of the signal analysis data.

7. The method of claim 6, wherein, The method further includes determining the value of top-k in the first retrieval result according to the image data type.

8. A graphic report generating apparatus characterized by comprising: The method includes: A preprocessing module is configured to classify and preprocess the signal analysis data according to the file type to determine image data and text data of the signal analysis data. An image processing module is configured to determine a first image processing model according to the image data and process the image data by using the first image processing model to determine a first image feature vector. A text processing module is configured to determine a first text processing model according to the text data and process the first image feature vector and the text data by using the first text processing model to generate a first textual description. A fusion module is configured to fuse the first image feature vector and the first textual description by using a retrieval enhancement generation model to generate a first picture-text report of the signal analysis data. The fusion module is further configured to: Adjust the correlation between the image and the text in the first picture-text report to generate a second picture-text report of the signal analysis data, including: Determine a consistency evaluation result by using an image understanding model to perform semantic regression on the first picture-text report; the formula of the consistency evaluation result is: ； wherein, is a cosine similarity; is an image embedding vector obtained by processing an image in the first image-text report using an image understanding model; is a text embedding vector obtained by processing text in the first image-text report. In a case where the consistency evaluation result is lower than a preset score, determine the text missing content in the first picture-text report by using a semantic reasoning model, including: Determine the text missing content in the first picture-text report by using the semantic reasoning model according to the context corresponding to the image in the first picture-text report; the formula of the text missing content is: ； wherein, is a text missing content; is a vocabulary; is a vocabulary predicted by a semantic inference model; is a context corresponding to the image in the first image-text report; is a semantic inference model parameter; is a context corresponding to the image in the first image-text report and a semantic inference model parameter , the probability of the vocabulary appearing; is the vocabulary that maximizes the probability ; Use the text missing content as retrieval information to retrieve the retrieval knowledge base by using the retrieval enhancement generation model to determine a second retrieval result; the formula of the second retrieval result is: ； wherein, to retrieve a knowledge base; to retrieve a vector representation of a text of a knowledge base; to retrieve a vector representation of a text missing content; Input the second retrieval result and the first picture-text report into the first text processing model to generate a second picture-text report of the signal analysis data; the formula of the second picture-text report is: ； wherein, is a context corresponding to the image in the second image-text report, is a first text processing model, is a vector representation of the second search result; is a first text processing model parameter.

9. A computer-readable storage medium, characterized in that, The computer program stored on the computer readable storage medium is run by the processor to execute the steps of the picture-text report generation method according to any one of claims 1 to 7.

Images