Model-based report conclusion generation method and device, computer equipment and medium

Abstract

The invention discloses a model-based report conclusion generation method and device, computer equipment and a medium. The method comprises the following steps: acquiring a report picture of a to-be-confirmed report conclusion; inputting the report picture to be subjected to report conclusion confirmation into the preset generation model, and adding a preset prompt text into the preset generation model to obtain a target report conclusion output by the model; the preset prompt text is selected according to requirements of different use scenes, and the preset generation model is of a structure with picture recognition and semantic understanding at the same time. The semantic content of the document and the image recognition result can be understood at the same time through the set model structure; various different report contents can be identified through a set model structure, a report conclusion can be directly generated based on the report, and the method is not sensitive to problem setting; a report conclusion can be reasoned through a set model structure, and abnormal data in a current report can be identified; therefore, an accurate report conclusion can be quickly obtained.

Description

Technical Field

[0001] The present invention relates to the field of information extraction, and in particular to a report conclusion generation method, device, computer equipment and medium based on a model. Background Art

[0002] At present, in many fields, users urgently need intelligent systems to efficiently solve complex problems, especially when processing documents such as user reports, they can quickly extract key information and give accurate conclusions; however, the existing visual question answering system VQA has limitations. It can only simply understand and answer questions based on images, but cannot understand and answer the content of the document text; reading comprehension technology can ask questions based on the content and questions of the text, but is very sensitive to questions and requires high accuracy in the question statement; text summarization technology is a semantic summary, and cannot summarize based on the results, and cannot infer whether there are abnormalities in the current data in the report; in summary, the existing technology cannot quickly obtain accurate report conclusions based on user reports; Therefore, those skilled in the art are in urgent need of finding a new technical solution to solve the above problems. Summary of the invention

[0003] Based on this, a model-based report conclusion generation method, device, computer equipment and medium are provided to solve the problem that the prior art cannot quickly obtain accurate report conclusions in document report conclusion recognition.

[0004] A method for generating report conclusions based on a model, comprising: Get the report image to be confirmed. After the report image to be confirmed is input into the preset generation model, a preset prompt text is added to the preset generation model to obtain the target report conclusion output by the model; the preset prompt text is selected according to the requirements of different usage scenarios, and the preset generation model is a structure with both image recognition and semantic understanding.

[0005] A report conclusion generating device based on a model, comprising: The first acquisition module is used to acquire the report image to be confirmed. The output module is used to input the report image to be confirmed into the preset generation model, and at the same time add the preset prompt text to the preset generation model to obtain the target report conclusion output by the model; the preset prompt text is selected according to the requirements of different usage scenarios, and the preset generation model is a structure with both image recognition and semantic understanding.

[0006] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned model-based report conclusion generation method when executing the computer program.

[0007] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the model-based report conclusion generation method is implemented.

[0008] The above-mentioned model-based report conclusion generation method, device, computer equipment and storage medium obtain a report image of the report conclusion to be confirmed; after the report image of the report conclusion to be confirmed is input into the preset generation model, a preset prompt text is added to the preset generation model to obtain the target report conclusion output by the model; the preset prompt text is selected according to the needs of different usage scenarios, and the preset generation model is a structure with both image recognition and semantic understanding. The model structure set in this application can simultaneously understand the semantic content of the document and the image recognition results; the model structure set can identify various report contents, and can directly generate report conclusions based on the report, which is not sensitive to the problem setting; the model structure set can infer the report conclusions, and can identify abnormal data in the current report; in summary, accurate report conclusions can be quickly obtained through a specially set model structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0010] Figure 1 is a schematic diagram of an application environment of a model-based report conclusion generation method in one embodiment of the present invention; Figure 2 is a flow chart of a method for generating report conclusions based on a model in one embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a report conclusion generating device based on a model in one embodiment of the present invention; Figure 4 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] The model-based report conclusion generation method provided by the present invention can be applied in the following aspects: Figure 1 In an application environment, a client communicates with a server via a network. The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0013] In one embodiment, if Figure 2 As shown in the figure, a model-based report conclusion generation method is provided, which is applied in Figure 1 The server in the example is used as an example to illustrate the process, including the following steps: S10, obtaining a report image to be confirmed for report conclusion.

[0014] It can be understood that report images can come from images in a variety of report application scenarios, such as medical reports, academic reports, and investigation reports, among which report images are a type of data recorded in the report, such as images of examinations in medical reports, images of statistical situations in academic reports, on-site images in investigation reports, etc.

[0015] S20, after inputting the report picture to be confirmed into the preset generation model, a preset prompt text is added to the preset generation model to obtain the target report conclusion output by the model; the preset prompt text is selected according to the requirements of different usage scenarios, and the preset generation model is a structure with both image recognition and semantic understanding.

[0016] Understandably, the preset generation model may be composed of two model structures, and the data processed by the two model structures may be different, including a recognition model structure for processing image-related data and a semantic understanding model structure for processing text data, such as using LayoutLMv3 (multimodal basic model) to process image-related data and using LM (text large model / basic model) to process text data. At the same time, the two model structures are connected, so that the data between the two model structures has a transmission function; the preset prompt text is the text content required in the target report conclusion output by the model, such as "the conclusion of the above examination is" or "the report conclusion is", etc. Furthermore, in different usage scenarios, the specific content of the preset prompt text may be different, such as in the medical report mentioned above, the specific content of the preset prompt text may be "the medical report conclusion is", in the academic report mentioned above, the specific content of the preset prompt text may be "the academic report conclusion is", and in the investigation report mentioned above, the specific content of the preset prompt text may be "the investigation report conclusion is", etc.; the target report conclusion is the text data of the report conclusion plus the preset prompt text and the report conclusion obtained from the report image.

[0017] In the embodiment of steps S10 to S20, the set model structure can be used to simultaneously understand the document semantic content and image recognition results; the set model structure can be used to identify various report contents, and report conclusions can be directly generated based on the report, and are not sensitive to problem settings; the set model structure can be used to infer report conclusions and identify abnormal data in the current report.

[0018] Furthermore, before inputting the report picture to be confirmed into the preset generation model, the method further includes: Get training reports and training reminder texts; After segmenting the training report according to a preset processing method, a picture containing the report conclusion in the training report is obtained; After processing the image containing the report conclusion through the first model structure and processing the training prompt text and the report conclusion through the second model structure, confirming that the results output by the two model structures meet the preset processing results, the preset generation model is obtained.

[0019] Understandably, the training report may be a pre-prepared report content, and the report content in different usage scenarios has been associated with a preset training prompt text, wherein the preset training prompt text is consistent with the content of the preset prompt text mentioned above, and the specific difference is that the preset prompt text is the text in the model usage state, while the preset training prompt text is the text in the model training state; the preset processing method specifically refers to the image with text conclusions segmented after the report is recognized, such as segmentation based on image recognition technology (including OCR recognition technology and deep learning model, 1. Use OCR technology to extract text content from the report.

[0020] Apply image analysis algorithms (such as edge detection, region segmentation, etc.) to identify specific areas in the report (such as the conclusion section is usually located at the end of the report and has a clear title), and combine the text content and location information to determine the image range of the conclusion section; 2. Train a convolutional neural network (CNN) or object detection model to identify the conclusion section in the report, including the location and bounding box of the conclusion section. The model can learn the unique visual features of the conclusion section. Apply the trained model to predict the new report and extract the image of the conclusion part), segmentation based on NLP technology (including text classification and keyword extraction and matching, 1. Treat the report text as a series of sentences or paragraphs, use text classification models (such as BERT, RoBERTa, etc.) to predict whether each sentence or paragraph belongs to the conclusion part, and extract the sentences or paragraphs marked as conclusions based on the classification results; 2. Use NLP tools (such as Jieba word segmentation, TextRank, etc.) to segment the report text and extract keywords, match the preset keyword list (such as "conclusion", "summary", etc.), determine the sentences or paragraphs containing these keywords, and extract Take out these sentences or paragraphs as the conclusion part), combine image recognition and NLP hybrid segmentation (including multimodal fusion model and end-to-end model, 1. For reports with both text and images, you can combine image recognition and NLP technology to extract the conclusion. First, use OCR technology to extract text content from the image, then apply NLP technology to identify the conclusion part in the text, and finally extract the image area containing the conclusion part based on the position information of the text content in the image; 2. Train an end-to-end deep learning model that can process text and image data at the same time. The model can learn the association between text and image and automatically segment the conclusion part from the image); The first model structure and the second model structure are consistent with the above-mentioned contents, and there are differences between the two in processing data. The preset generation model composed of the two model structures can be understood as a fusion of multimodal structures. After the encoding and decoding processing is completed by the first model structure and the second model structure, the output results corresponding to the two models are output as a model output result, and the gap between the result and the target output result is determined by the result, so as to continuously adjust the parameters of the entire model structure until the output of the model structure output result that meets the target output result is output; This embodiment is a training process of a preset generation model. By specially processing the conclusion image and the training prompt text in the training report respectively (using the first and second model structures), the key information in both can be captured and understood more carefully, which helps to improve the accuracy and precision of the model when processing similar tasks; the preset processing method segments the training report, especially extracting the image containing the conclusion, which helps to reduce unnecessary data interference, so that the model can focus more on learning key information, so as to accurately identify reports in different situations and directly obtain the report conclusion corresponding to the document report; it involves the processing of multiple data types (images and texts), and performs feature extraction and learning through two independent model structures. The preset generation model finally generated can better understand and process different types of data, thereby enhancing the reasoning ability of report conclusions under different reports and filtering out abnormal data that is not related to the report conclusions.

[0021] Furthermore, the segmentation process of the training report according to the preset processing method to obtain the picture containing the report conclusion in the training report includes: Identify various common structural data in the training report through a preset recognition model; Determine first conclusion text data and table data corresponding to the first picture data from the various common structure data; Converting the table data into second picture data having second conclusion text data; The first image data of the first conclusion text data and the second image data of the second conclusion text data are used as the images containing the report conclusion in the training report.

[0022] Understandably, there may be a variety of data in the training report, including table data, image data, text data, digital data, etc. The preset recognition model is specifically used to recognize the above-mentioned multiple data. Since the present embodiment recognizes and processes multiple data, the present embodiment may adopt the above-mentioned multimodal fusion model and end-to-end model. Specifically, firstly, a special image OCR recognition model is used to recognize whether the data corresponding to the area is table data or image data, and the report conclusion will appear below the table area. At this time, after recognizing the image data, the text below the table area can be recognized to determine that the current area is a table area. Finally, a text fragment recognition model is trained to recognize the conclusion text content therein, and the text content of the report conclusion part can be obtained. In addition, the table data is converted into a second picture data, that is, the area corresponding to the table data is used as a sub-image, and the second conclusion text data in the table data can be obtained by recognizing the sub-image. In this way, recognizing the second conclusion text data in the table data through the picture data will be more accurate in text extraction. This embodiment uses a preset recognition model to identify common structural data in training reports. Automated recognition greatly improves processing efficiency and reduces the need for manual intervention. It can further analyze and determine which data is conclusive, especially text and table data associated with the first image data, which helps to accurately extract key information. For tabular data, converting it into image form is conducive to the extraction of text in the image data, and integrating images containing conclusive information together is conducive to the subsequent training and processing of data by the preset generation model.

[0023] Further, the processing of the picture containing the conclusion by the first model structure includes: The picture containing the report conclusion is partially or completely selected through the first model structure to obtain the conclusion text corresponding to the picture containing the report conclusion, and the conclusion is used as the result output by the text model structure.

[0024] Understandably, the text and image masking process of the first model structure is to partially or completely select the pictures containing the conclusions, including randomly masking a certain length of tokenspan (used to refer to a continuous segment or subsequence in the text), and using the text and image context for prediction (helping the model to learn the bidirectional representation of the text modality), using blocking and masking strategies to randomly mask some patches in the image (small non-overlapping patches in the image), and converting the maskedpatches into discrete tokens for prediction through imagetokenizer (image tokenizer) (helping the model to learn the representation of the image modality), and learning the alignment relationship between text data and image data by predicting whether the image data corresponding to the text data is masked (helping the model to understand the association between text and image); This embodiment can obtain a report conclusion that conforms to the output result of the model by partially or completely selecting the pictures containing the conclusion, that is, the result is closer to the user model requirement.

[0025] Further, the processing of the training prompt text by the second model structure includes: Determine the dependency relationship and context information of the training prompt text and the report conclusion through the second model structure; After determining the association relationship between the text basic unit and other text basic units through the dependency relationship and the context information, determining the selection probability of each text basic unit as the next text basic unit; The text basic unit whose selection probability meets the preset selection probability is used as the result of the model structure output.

[0026] Understandably, the multi-layer neural network in the second model structure (such as the layer based on the Transformer architecture) can receive the embedded representation of n input tokens (text basic units) to determine the dependency and contextual information between words in the text; in each layer of the second model, the self-attention mechanism is used to calculate the relationship between each token (text basic unit) and other tokens (other text basic units) (the output data processed by the self-attention mechanism is passed to the feedforward neural network layer for further nonlinear transformation and feature extraction), allowing the model to consider the context of the entire input sequence when processing the current token (text basic unit). When generating text, the second model structure calculates a probability distribution for each possible next token (other text basic unit), and the probability reflects the selection probability of each token (text basic unit) as the next token (other text basic unit) in the current context; the preset selection probability can be the highest selection probability, or it can be a randomly generated probability value; In addition, in this embodiment, the output token selected by selection probability will be added to the input sequence, and the second model will repeat all the above-mentioned steps to generate the next token. The iterative process in this embodiment will continue until a complete sentence or text is generated, thereby ensuring that the report conclusions are complete and usable.

[0027] Furthermore, after processing the picture containing the conclusion by the first model structure and processing the training prompt text by the second model structure, it also includes: Fusing the image containing the report conclusion, the training prompt text and the report conclusion to obtain fused data; After performing position encoding processing on the fused data, an encoding result is obtained; After position decoding processing is performed on the encoding result, two model structure output results are obtained.

[0028] Understandably, the purpose of fusion is to determine the initial input of the model, and the input data is the fusion data of multimodal input; the encoder can use the above-mentioned first model structure for encoding processing, and through its pre-trained multimodal representation ability, encode the input fusion data into a high-quality hidden state representation, which not only captures the information of the text content, but also contains the layout, context and image data of the document. The encoding result output by the encoder is a series of hidden state vectors, which will be used as the input of the decoder; the decoder can use the above-mentioned second model structure for decoding processing. The decoder accepts the output of the encoder (i.e., the hidden state vector) as the initial input. In addition, it can also accept an optional starting token or prefix text as the starting point for generating text (such as a preset prompt text). The decoder uses the ability of the generative language model to gradually generate the target text based on the output of the encoder and possible starting token / prefix text. In the generation process, the decoder will consider the hidden state of the encoder and the generated text content to generate text related to the document content and in line with the language specification. The output result of the decoder is the generated text sequence, that is, the result of the output of the two model structures; In this embodiment, the encoder pre-trained by the first model structure can be introduced as the basis of understanding, without the need for a large amount of data training. At the same time, it can be connected to the language model decoder to obtain reasoning and generation capabilities, which is more flexible than simple text summarization technology.

[0029] Furthermore, the confirmation that the output results of the two model structures meet the preset processing results includes: The loss value between the results output by the two model structures and the preset processing results is calculated by a preset loss function. When the loss value meets the preset loss value, it is confirmed that the results output by the two model structures meet the preset processing results.

[0030] Understandably, the loss function can be selected from functions such as cross entropy loss and negative log-likelihood loss. The specific process is to first define an appropriate training goal (i.e., preset processing result), such as maximizing the log-likelihood probability of the generated text, minimizing the error between the generated text and the true label, etc., and use optimization algorithms such as gradient descent to update the parameters of the model structure to minimize the loss function, and optimize the performance of the model structure by adjusting hyperparameters such as learning rate, batch size, and training rounds. This embodiment can speed up the training process. Since the network receives accurate information, it can usually learn the correct parameters faster and reduce error accumulation, making the entire model structure training process more stable.

[0031] In summary, the above provides a model-based report conclusion generation method, which obtains a report image of a report conclusion to be confirmed; after the report image of the report conclusion to be confirmed is input into the preset generation model, a preset prompt text is added to the preset generation model to obtain the target report conclusion output by the model; the preset prompt text is selected according to the needs of different usage scenarios, and the preset generation model is a structure with both image recognition and semantic understanding. The set model structure can simultaneously understand the semantic content of the document and the image recognition results; the set model structure can identify various report contents, and can directly generate report conclusions based on the report, which is not sensitive to the problem setting; the set model structure can infer the report conclusions, and can identify abnormal data in the current report. In summary, accurate report conclusions can be quickly obtained through a specially set model structure.

[0032] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0033] In one embodiment, a model-based report conclusion generating device is provided, and the model-based report conclusion generating device corresponds one-to-one with the model-based report conclusion generating method in the above embodiment. Figure 3 As shown, the model-based report conclusion generating device includes a first acquisition module 11 and an output module 12. Each functional module is described in detail as follows: The first acquisition module 11 is used to acquire a report image to be confirmed on the report conclusion; The output module 12 is used to input the report image to be confirmed into the preset generation model, and at the same time add the preset prompt text to the preset generation model to obtain the target report conclusion output by the model; the preset prompt text is selected according to the requirements of different usage scenarios, and the preset generation model is a structure with both image recognition and semantic understanding.

[0034] Furthermore, the model-based report conclusion generating device also includes: The second acquisition module is used to acquire the training report and training prompt text; A segmentation module, used to segment the training report according to a preset processing method to obtain a picture containing the report conclusion in the training report; A confirmation module is used to process the image containing the report conclusion through the first model structure, and after processing the training prompt text and the report conclusion through the second model structure, confirm that the results output by the two model structures meet the preset processing results to obtain the preset generation model.

[0035] Furthermore, the segmentation module includes: A recognition unit, used to recognize various common structure data in the training report through a preset recognition model; A first determining unit, used to determine first conclusion text data and table data corresponding to the first picture data from the various common structure data; A conversion unit, used for converting the table data into second picture data having second conclusion text data; As a unit, it is used to use the first picture data of the first conclusion text data and the second picture data of the second conclusion text data as the picture containing the report conclusion in the training report.

[0036] Furthermore, the confirmation module includes: An output unit is used to partially or completely select the picture containing the report conclusion through the first model structure, obtain the conclusion text corresponding to the picture containing the report conclusion, and use the conclusion as the result of the text model structure output.

[0037] Furthermore, the confirmation module includes: A second determining unit, configured to determine the dependency relationship and context information between the training prompt text and the report conclusion through the second model structure; A third determining unit is used to determine the probability of each text basic unit being selected as the next text basic unit after determining the association relationship between the text basic unit and other text basic units through the dependency relationship and the context information; The conforming unit is used to output the text basic unit whose selection probability conforms to the preset selection probability as the result of the model structure output.

[0038] Furthermore, the model-based report conclusion generating device also includes: A fusion module, used for fusing the picture containing the report conclusion, the training prompt text and the report conclusion to obtain fused data; An encoding module, used for performing position encoding processing on the fused data to obtain an encoding result; The decoding module is used to perform position decoding processing on the encoding result to obtain the output results of two model structures.

[0039] Furthermore, the confirmation module includes: The confirmation unit is used to calculate the loss value between the results output by the two model structures and the preset processing results through a preset loss function, and when the loss value meets the preset loss value, confirm that the results output by the two model structures meet the preset processing results.

[0040] For the specific limitations of the model-based report conclusion generation device, please refer to the limitations of the model-based report conclusion generation method above, which will not be repeated here. The various modules in the above-mentioned model-based report conclusion generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0041] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the model-based report conclusion generation method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model-based report conclusion generation method is implemented.

[0042] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the model-based report conclusion generation method in the above embodiment are implemented, such as Figure 2 Alternatively, when the processor executes the computer program, the functions of each module / unit of the report conclusion generating device based on the model in the above embodiment are realized, for example Figure 3 The functions of modules 11 to 13 are shown in Figure 1. To avoid repetition, they will not be described here.

[0043] In one embodiment, a computer-readable storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, the steps of the model-based report conclusion generation method in the above embodiment are implemented, such as Figure 2 Alternatively, when the computer program is executed by the processor, the functions of each module / unit of the report conclusion generating device based on the model in the above embodiment are realized, for example Figure 3 The functions of modules 11 to 13 are shown in Figure 1. To avoid repetition, they will not be described here.

[0044] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0045] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0046] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A model-based report conclusion generation method, characterized in that: include: Obtain the report image to be used to confirm the report conclusion; After the report picture to be confirmed is input into the preset generation model, a preset prompt text is added to the preset generation model to obtain the target report conclusion output by the model; The preset prompt text is selected according to the requirements of different usage scenarios, and the preset generation model is a structure with both image recognition and semantic understanding.

2. The model-based report conclusion generation method according to claim 1, characterized in that: Before inputting the report picture to be confirmed into the preset generation model, the method further includes: Get training reports and training reminder texts; After segmenting the training report according to a preset processing method, a picture containing the report conclusion in the training report is obtained; After processing the image containing the report conclusion through the first model structure and processing the training prompt text and the report conclusion through the second model structure, confirming that the results output by the two model structures meet the preset processing results, the preset generation model is obtained.

3. The model-based report conclusion generation method according to claim 2, characterized in that: The segmentation process of the training report according to the preset processing method to obtain the picture containing the report conclusion in the training report includes: Identify various common structural data in the training report through a preset recognition model; Determine first conclusion text data and table data corresponding to the first picture data from the various common structure data; Converting the table data into second picture data having second conclusion text data; The first image data of the first conclusion text data and the second image data of the second conclusion text data are used as the images containing the report conclusion in the training report.

4. The model-based report conclusion generation method according to claim 2, characterized in that: The step of processing the picture containing the conclusion by using the first model structure includes: The picture containing the report conclusion is partially or completely selected through the first model structure to obtain the conclusion text corresponding to the picture containing the report conclusion, and the conclusion is used as the result output by the text model structure.

5. The model-based report conclusion generation method according to claim 2, characterized in that: The processing of the training prompt text by the second model structure includes: Determine the dependency relationship and context information of the training prompt text and the report conclusion through the second model structure; After determining the association relationship between the text basic unit and other text basic units through the dependency relationship and the context information, determining the selection probability of each text basic unit as the next text basic unit; The text basic unit whose selection probability meets the preset selection probability is used as the result of the model structure output.

6. The model-based report conclusion generation method according to claim 2, characterized in that: After the picture containing the conclusion is processed by the first model structure and the training prompt text is processed by the second model structure, the method further includes: Fusing the picture containing the report conclusion, the training prompt text and the report conclusion to obtain fused data; After performing position encoding processing on the fused data, an encoding result is obtained; After position decoding processing is performed on the encoding result, two model structure output results are obtained.

7. The model-based report conclusion generation method according to claim 6, characterized in that: The confirmation that the output results of the two model structures meet the preset processing results includes: The loss value between the results output by the two model structures and the preset processing results is calculated by a preset loss function. When the loss value meets the preset loss value, it is confirmed that the results output by the two model structures meet the preset processing results.

8. A report conclusion generating device based on a model, characterized in that: include: The first acquisition module is used to acquire the report image to be confirmed. An output module is used to input the report picture to be confirmed into a preset generation model, and at the same time add a preset prompt text to the preset generation model to obtain a target report conclusion output by the model; The preset prompt text is selected according to the requirements of different usage scenarios, and the preset generation model is a structure with both image recognition and semantic understanding.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the model-based report conclusion generating method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model-based report conclusion generating method according to any one of claims 1 to 7 is implemented.

Images