Material report automatic generation system and method

Abstract

The invention relates to the field of data processing, and discloses a material report automatic generation system and method.The system comprises a chart recognition module, a text recognition module, a formula recognition module, a client template analysis module and a material report generation module, multi-scale local chart extraction is carried out on pre-obtained client materials based on a Transform-CNN cascade model, and a material report is generated; obtaining chart information of the customer materials; performing semantic association confrontation training on the customer material based on a multi-modal joint pre-training strategy to obtain text information of the customer material; the topological relation activation module is used for performing topological relation activation on customer materials based on a dynamic characteristic cross attention mechanism to obtain formula information of the customer materials; the NLP analysis module is used for performing NLP analysis on a pre-acquired client template to obtain a report requirement of the client template; and the material report module is used for filling the chart data, the text data and the formula data into a pre-acquired client template based on the report requirements so as to obtain a material report of the client material. According to the invention, the efficiency of automatically generating the material report can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a material report automatic generation system and method. Background Technology

[0002] In the field of data processing, the generation of material reports is of great significance for material research and quality control in various industries. Currently, the state of material report generation technology is not optimistic, with efficiency being a key factor restricting its development.

[0003] In terms of speed, existing technologies, whether manual or partially automated, have significant shortcomings. Manually, staff need to identify, extract, and input information such as charts, text, and formulas from client materials one by one—a tedious and time-consuming process that struggles to meet the demand for rapid processing of large volumes of materials. Even with automation tools, the system's information processing efficiency remains low when faced with complex and diverse client materials. When processing materials containing multiple complex charts and large amounts of text, existing technologies often require a considerable amount of time to complete information extraction and preliminary processing, severely impacting the overall progress of report generation.

[0004] In terms of effectiveness, existing technologies struggle to guarantee accuracy and completeness. Manual processing is prone to errors and omissions in information extraction due to individual differences and work fatigue. Automated technologies, on the other hand, show a significant drop in accuracy when dealing with complex chart structures, semantically ambiguous text, and formulas with intricate logical relationships. Existing technologies often fail to accurately identify and parse data with special formats in complex charts, implicit semantic relationships in text, and combinations of multiple symbols in formulas. This results in material reports containing errors and missing key information, failing to provide reliable data for subsequent decision-making and research. Summary of the Invention

[0005] This invention provides a material report automatic generation system and method, the main purpose of which is to solve the problem of low efficiency in automatic material report generation.

[0006] To achieve the above objectives, the present invention provides an automatic material report generation system, characterized in that the system includes a chart recognition module, a text recognition module, a formula recognition module, a customer template analysis module, and a material report generation module, wherein:

[0007] The chart recognition module is used to perform multi-scale local chart extraction on the pre-acquired customer materials based on the Transformer-CNN cascaded model to obtain the chart information of the customer materials.

[0008] The text recognition module is used to perform semantic association adversarial training on the customer material based on a multimodal joint pre-training strategy to obtain the text information of the customer material;

[0009] The formula recognition module is used to activate the topological relationship of the customer material based on the dynamic feature cross-attention mechanism to obtain the formula information of the customer material.

[0010] The customer template analysis module is used to perform NLP parsing on the pre-acquired customer templates to obtain the reporting requirements of the customer templates;

[0011] The material report generation module is used to fill the chart data, text data and formula data into the pre-acquired customer template based on the report requirements to obtain the material report of the customer material.

[0012] In a preferred embodiment, when the chart recognition module performs multi-scale local chart extraction on the pre-acquired customer materials based on a Transformer-CNN cascaded model to obtain the chart information of the customer materials, it is specifically used for:

[0013] The pre-acquired customer material is cleaned of noise using a Transformer-CNN cascaded model to obtain the standard material of the customer material.

[0014] Multi-scale local feature extraction is performed on the standard material to obtain the target feature map of the standard material;

[0015] The target feature map is graphically encoded using a locality-sensitive hash attention mechanism to obtain the feature chart data of the customer material.

[0016] The feature chart data is decoded in parallel by multiple heads to obtain the chart information of the customer material.

[0017] In a preferred embodiment, when the chart recognition module performs multi-scale local feature extraction on the standard material to obtain the target feature map of the standard material, it is specifically used for:

[0018] Multi-level convolutional feature extraction is performed on the standard material to obtain the initial feature map of the customer material;

[0019] Generate a multi-scale feature map of the customer's material based on the standard material;

[0020] Based on the multi-scale feature map, feature adversarial training is performed on the initial feature map to obtain the target feature map of the customer material.

[0021] In a preferred embodiment, the text recognition module, when performing semantic association adversarial training on the customer material based on a multimodal joint pre-training strategy to obtain the text information of the customer material, is specifically used for:

[0022] A graphic training dataset of the customer's materials is constructed based on pre-acquired historical material data;

[0023] Based on the image and text training dataset, the customer material is encoded using a dual-tower encoding method to obtain the image-text tower of the customer material.

[0024] Semantic space adversarial training is performed on the image-text tower and the standard material to obtain text-image data;

[0025] Cross-modal knowledge distillation is performed on the text-image data to obtain the text information of the customer material.

[0026] In a preferred embodiment, when the text recognition module performs cross-modal knowledge distillation on the text-image data to obtain the text information of the customer material, it is specifically used for:

[0027] Modality-specific pruning is performed on the text data in the text-image data to obtain the core text data of the text data;

[0028] Text knowledge is extracted from the image data in the text-image data to obtain the image-textualized data of the image data.

[0029] Text-aware decoding is performed on the core text data and the image textual data to obtain the text information of the customer material.

[0030] In a preferred embodiment, when the formula recognition module performs topological relationship activation on the customer material based on a dynamic feature cross-attention mechanism to obtain the formula information of the customer material, it is specifically used for:

[0031] Based on the dynamic feature cross-attention mechanism, multi-scale visual feature extraction is performed on the customer material to obtain the image features of the customer material;

[0032] Based on the topological inference head, differential calculations are performed on the image features to obtain the symbolic data of the customer material;

[0033] The symbol data is fused at two levels to obtain the formula information of the customer material.

[0034] In a preferred embodiment, when the formula recognition module performs differential calculations on the image features based on the topological inference head to obtain the customer material symbol data, it is specifically used for:

[0035] The symbol positions of the image features are obtained by performing symbol recognition on the image features based on the topological inference head;

[0036] Symbolic features are extracted from the customer materials to obtain a symbolic feature map of the customer materials;

[0037] Based on the symbol location and the symbol feature map, differential calculations are performed on the customer material to obtain the customer material symbol data.

[0038] In a preferred embodiment, when the formula recognition module performs a differential calculation on the customer material based on the symbol position and the symbol feature map to obtain the customer material symbol data, it is specifically used for:

[0039] The symbol type of the customer data is identified by performing symbol type recognition;

[0040] Based on the symbol type, the symbol feature map, and the symbol position, a differentiation calculation is performed on the customer material to obtain the symbol data of the customer material, wherein the formula for the differentiation calculation is:

[0041] (s ij (For terminology symbols)

[0042] s ij =-||q i -k j ||(s ij (for numerical symbols)

[0043] (s ij (for operators)

[0044] In the formula, s ij For symbolic data, q i Let k be the vector at the position of the i-th symbol. j Let be the vector of the i-th symbol feature map, where i is the symbol position number, j is the symbol feature map number, T is the vector transpose flag, μ is the strength of the control direction constraint, and (i→j) is the logical flow direction in the customer material.

[0045] In a preferred embodiment, when the customer template analysis module performs NLP parsing on the pre-acquired customer template to obtain the report requirements of the customer template, it is specifically used for:

[0046] The physical structure of the pre-acquired customer template is analyzed to obtain the template structure of the customer template;

[0047] Logical rules are extracted from the customer template to obtain the template rules for the customer template;

[0048] The template rules and template structure are hierarchically encoded to obtain the reporting requirements of the customer template.

[0049] To address the above problems, the present invention also provides a method for automatically generating material reports, the method comprising:

[0050] S1. Based on the Transformer-CNN cascaded model, multi-scale local chart extraction is performed on the pre-acquired customer materials to obtain the chart information of the customer materials;

[0051] S2. Based on a multimodal joint pre-training strategy, perform semantic association adversarial training on the customer material to obtain the text information of the customer material;

[0052] S3, used to activate the topological relationship of the customer material based on the dynamic feature cross-attention mechanism, and obtain the formula information of the customer material;

[0053] S4. Used to perform NLP parsing on the pre-acquired customer template to obtain the report requirements of the customer template;

[0054] S5. To fill the chart data, text data and formula data into the pre-acquired customer template based on the report requirements, so as to obtain the material report of the customer material.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. This system rapidly acquires chart information through a Transformer-CNN cascaded model, a multimodal joint pre-training strategy, and a dynamic feature cross-attention mechanism, avoiding the tedious process of manually identifying charts one by one. Simultaneously, it utilizes historical material data to construct a text and image training dataset, performing dual-tower encoding and semantic space adversarial training to quickly acquire text information. Then, it models the topological relationships of client materials through a topological relationship activation mechanism, parsing out formula information. Finally, this enables the system to extract information from a large amount of material in a short time, significantly improving report generation speed.

[0057] 2. During processing, the system accurately extracts chart information through a locality-sensitive hashing attention mechanism and multi-head parallel decoding, ensuring the integrity and correctness of the chart data. Simultaneously, it employs cross-modal knowledge distillation technology to process text-image data, obtaining accurate text information and avoiding errors caused by semantic comprehension biases. Then, it performs differentiated calculations based on a topological inference head and performs two-level fusion of symbolic data to accurately parse formula information. Finally, it performs NLP parsing on the pre-acquired customer template to accurately obtain report requirements, accurately filling the customer template with chart data, text data, and formula data to generate a high-quality material report. Attached Figure Description

[0058] Figure 1 A system architecture diagram of an automatic material report generation system provided in an embodiment of the present invention;

[0059] Figure 2 This is a flowchart illustrating an embodiment of the automatic material report generation method provided by the present invention.

[0060] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments belong to some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “said” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0063] Depending on the context, the word "if" or "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0064] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0065] In practice, the server-side equipment deployed by the material report auto-generation system may consist of one or more devices. The aforementioned material report auto-generation system can be implemented as: a business instance, a virtual machine, or a hardware device. For example, the material report auto-generation system can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, the material report auto-generation system can be understood as software deployed on a cloud node to provide automatic material report generation to various user terminals. Alternatively, the material report auto-generation system can also be implemented as a virtual machine deployed on one or more devices in a cloud node. This virtual machine contains application software for managing various user terminals. Alternatively, the material report auto-generation system can also be implemented as a server composed of numerous identical or different types of hardware devices, with one or more hardware devices configured to provide automatic material report generation to various user terminals.

[0066] In terms of implementation, the automatic material report generation system and the user client are mutually compatible. That is, if the automatic material report generation system is implemented as an application installed on a cloud service platform, then the user client is implemented as a client that establishes a communication connection with the application; or if the automatic material report generation system is implemented as a website, then the user client is implemented as a webpage; or if the automatic material report generation system is implemented as a cloud service platform, then the user client is implemented as a mini-program in an instant messaging application.

[0067] like Figure 1 The figure shown is a system architecture diagram of an automatic material report generation system provided in an embodiment of the present invention.

[0068] The material report automatic generation system 100 of this invention can be set up in a cloud server. In terms of implementation, it can be one or more service devices, or it can be installed as an application in the cloud (e.g., a mobile service operator's server, server cluster, etc.), or it can be developed into a website. Depending on the functions implemented, the material report automatic generation system 100 may include a chart recognition module 101, a text recognition module 102, a formula recognition module 103, a customer template analysis module 104, and a material report generation module 105. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0069] In this embodiment of the invention, each of the above-mentioned modules in the automatic material report generation system can be implemented independently and can call other modules. Here, "calling" can be understood as one module connecting to multiple modules of another type and providing corresponding services to those connected modules. In the automatic material report generation system provided by this embodiment of the invention, the applicability of the system architecture can be adjusted by adding modules and directly calling them without modifying the program code, achieving cluster-based horizontal expansion to quickly and flexibly expand the automatic material report generation system. In practical applications, the above modules can be set in the same device or different devices, or they can be set in a virtual device, such as a service instance in a cloud server.

[0070] The following describes the components and workflow of the automatic material report generation system using specific embodiments:

[0071] The chart recognition module 101 is used to perform multi-scale local chart extraction on the pre-acquired customer materials based on the Transformer-CNN cascaded model to obtain the chart information of the customer materials.

[0072] In this embodiment of the invention, when the chart recognition module performs multi-scale local chart extraction on the pre-acquired customer materials based on the Transformer-CNN cascade model to obtain the chart information of the customer materials, it is specifically used for:

[0073] The pre-acquired customer material is cleaned of noise using a Transformer-CNN cascaded model to obtain the standard material of the customer material.

[0074] Multi-scale local feature extraction is performed on the standard material to obtain the target feature map of the standard material;

[0075] The target feature map is graphically encoded using a locality-sensitive hash attention mechanism to obtain the feature chart data of the customer material.

[0076] The feature chart data is decoded in parallel by multiple heads to obtain the chart information of the customer material.

[0077] When the chart recognition module performs multi-scale local feature extraction on the standard material to obtain the target feature map of the standard material, it is specifically used for:

[0078] Multi-level convolutional feature extraction is performed on the standard material to obtain the initial feature map of the customer material;

[0079] Generate a multi-scale feature map of the customer's material based on the standard material;

[0080] Based on the multi-scale feature map, feature adversarial training is performed on the initial feature map to obtain the target feature map of the customer material.

[0081] Specifically, the Transformer-CNN cascaded model is an artificial intelligence model that integrates the advantages of Transformer and CNN. It combines the powerful global feature capture and long sequence processing capabilities of Transformer with the excellent local feature extraction capabilities of CNN. In the material report automatic generation system, it is mainly used to extract multi-scale local charts from customer materials. Through operations such as noise cleaning, multi-scale local feature extraction, chart encoding, and multi-head parallel decoding of customer materials, it accurately obtains chart information, effectively improving the accuracy and efficiency of chart information extraction in the material report generation process, and playing a key role in the entire material report automatic generation process.

[0082] Specifically, the Locality Sensitive Hash (LSH) attention mechanism transforms the target feature map into more structured and easier-to-process feature chart data. It utilizes the characteristics of the LSH algorithm to quickly identify and focus on similar local feature regions in the target feature map, and assigns weights through the attention mechanism to highlight key chart feature parts and suppress irrelevant information. This achieves more efficient and accurate encoding of chart information, laying the foundation for accurately obtaining chart information of customer materials and ensuring the accuracy and efficiency of material report generation.

[0083] Specifically, the multi-head parallel decoding involves decoding feature chart data from multiple different perspectives or subspaces. This method differs from a single decoding method. It can simultaneously capture different aspects of features and information in the data, thereby providing a more comprehensive and accurate analysis of the data. By having multiple heads work in parallel, different feature information can be processed in parallel, avoiding the problem of missing some key information that may occur with a single decoding method, thus improving the accuracy and efficiency of decoding.

[0084] Specifically, the characteristics of the Transformer-CNN cascade model are used to perform a comprehensive scan of the pre-acquired customer materials.

[0085] Furthermore, the model will identify and remove noise in the material, which may be interference information generated during data acquisition, transmission, and other processes.

[0086] Furthermore, through a series of algorithmic processing steps, the noisy original customer materials are transformed into clean and standardized materials, providing a high-quality data foundation for subsequent feature extraction work.

[0087] Furthermore, for the cleaned standard materials, the model first performs multi-level convolutional feature extraction, by sliding convolutional kernels of different sizes on the material data to capture basic features at various scales and form an initial feature map.

[0088] Furthermore, multi-scale feature maps are generated based on standard materials to extract material features from different resolutions.

[0089] Furthermore, the initial feature map is trained using multi-scale feature maps. During the training process, the features are continuously optimized to suppress redundant and interfering information, ultimately resulting in a target feature map that accurately reflects the key features of the chart.

[0090] Furthermore, a Locality Sensitive Hash (LSH) attention mechanism is employed to process the target feature map. This mechanism uses the LSH algorithm to quickly locate similar local feature regions in the target feature map, and then uses the attention mechanism to assign weights to different feature regions, highlighting key chart features and weakening irrelevant information. After this processing, the target feature map is encoded into structured customer material feature chart data that is easy to process subsequently; this data contains the key information of the chart.

[0091] Furthermore, the obtained feature chart data is subjected to multi-head parallel decoding. Multiple decoding heads process the data simultaneously from different angles and subspaces, with each head focusing on different feature parts of the data and parsing the data in parallel.

[0092] Furthermore, this method comprehensively captures all kinds of information in the data, avoiding the omission of key information due to a single decoding perspective.

[0093] Furthermore, by integrating the processing results of each decoding head, the feature chart data is transformed into intuitive and accurate customer material chart information, providing crucial data support for the generation of material reports.

[0094] Specifically, for the standard material after noise removal, convolutional neural network (CNN) operations are applied. In this process, convolutional kernels of different sizes are used to slide across the standard material data in a layered manner.

[0095] Furthermore, each convolutional kernel scans the data from different scales and angles to capture basic features such as lines, textures, and shapes.

[0096] Furthermore, as convolutional layers are continuously stacked, these basic features are gradually combined and abstracted.

[0097] Furthermore, after multiple convolutional operations, the features extracted from all convolutional layers are integrated to obtain the initial feature map of the customer's material. This map initially contains some feature information of the customer's material chart, but further optimization is still needed.

[0098] Furthermore, based on standard materials, multi-scale feature maps are generated through various technical means.

[0099] Furthermore, a common approach is to utilize the principle of image pyramids to perform different degrees of downsampling and upsampling operations on standard materials.

[0100] Furthermore, downsampling can reduce the resolution of an image, allowing the model to observe the overall characteristics of the material from a macroscopic perspective; upsampling, on the other hand, restores some detailed information based on a low-resolution image.

[0101] Furthermore, convolutional operations on standard materials can be performed using convolutional kernels of different sizes at different scales to obtain features from different receptive fields.

[0102] Furthermore, by combining these operations, a series of feature maps with different resolutions and receptive fields are obtained, namely multi-scale feature maps.

[0103] Furthermore, these multi-scale feature maps reflect the characteristics of the client's material charts from different levels, providing rich information for subsequent feature adversarial training.

[0104] In summary, the Transformer-CNN cascade model integrates the powerful global information capture capability of Transformer and the excellent local feature extraction capability of CNN.

[0105] In summary, by cleaning noise from customer materials and removing interfering information, the accuracy of subsequent data processing is ensured.

[0106] In summary, during the multi-scale local feature extraction process, the model utilizes multi-level convolutional feature extraction, multi-scale feature map generation, and feature adversarial training to comprehensively capture chart features from different scales and levels. It can accurately identify various elements in the chart, effectively avoid feature omissions or misjudgments, and thus obtain target feature maps that accurately reflect the content of the client's material charts, laying the foundation for generating accurate chart information.

[0107] In summary, the locality-sensitive hashing attention mechanism and multi-head parallel decoding technology employed in this model significantly improve the speed of chart information processing.

[0108] In summary, the Locality Sensitive Hash Attention (LSA) mechanism can quickly locate key feature regions in the target feature map and encode them efficiently, reducing unnecessary computational resource consumption; while multi-head parallel decoding allows feature map data to be processed from multiple perspectives simultaneously, realizing parallel parsing of information and greatly shortening the decoding time.

[0109] In summary, these technologies work together to enable the system to extract and convert customer materials into chart information in a short time, accelerating the overall process of material report generation and meeting the demand for efficient material report generation in practical applications.

[0110] In general, in real-world applications, the format, content, and quality of client materials vary considerably. The Transformer-CNN cascade model, with its multi-scale processing and powerful feature learning capabilities, can flexibly handle a wide range of complex client materials.

[0111] In summary, regardless of whether the materials have complex chart structures and large amounts of data, or contain noise and have non-standard formats, the model can effectively extract key chart information from complex backgrounds and interference information through multi-scale local chart extraction, thereby enhancing the adaptability and robustness of the material report automatic generation system to different types of customer materials.

[0112] In summary, the generated multi-scale feature maps are used as a reference for adversarial feature training against the initial feature maps.

[0113] In summary, during the training process, an adversarial training model is constructed, and the goal of the model is to allow the initial feature map to continuously optimize its feature representation through learning from multi-scale feature maps.

[0114] In summary, by comparing the differences between two feature maps, calculating the loss function, and then using the backpropagation algorithm to adjust the model parameters, the initial feature map gradually incorporates more accurate and key graph features.

[0115] In general, this process weakens features that are not obvious in the multi-scale feature map or are irrelevant to the key information of the chart, and highlights the features in the multi-scale feature map that reflect the essence of the chart.

[0116] In summary, after multiple iterations of training, when the loss function converges to a stable state, the optimized initial feature map obtained is the target feature map of the customer's material. This map can more accurately reflect the characteristics of the customer's material chart, providing higher quality feature data for subsequent chart encoding and decoding.

[0117] The text recognition module 102 is used to perform semantic association adversarial training on the customer material based on a multimodal joint pre-training strategy to obtain the text information of the customer material;

[0118] In this embodiment of the invention, when the text recognition module performs semantic association adversarial training on the customer material based on a multimodal joint pre-training strategy to obtain the text information of the customer material, it is specifically used for:

[0119] A graphic training dataset of the customer's materials is constructed based on pre-acquired historical material data;

[0120] Based on the image and text training dataset, the customer material is encoded using a dual-tower encoding method to obtain the image-text tower of the customer material.

[0121] Semantic space adversarial training is performed on the image-text tower and the standard material to obtain text-image data;

[0122] Cross-modal knowledge distillation is performed on the text-image data to obtain the text information of the customer material.

[0123] When the text recognition module performs cross-modal knowledge distillation on the text-image data to obtain the text information of the customer material, it is specifically used for:

[0124] Modality-specific pruning is performed on the text data in the text-image data to obtain the core text data of the text data;

[0125] Text knowledge is extracted from the image data in the text-image data to obtain the image-textualized data of the image data.

[0126] Text-aware decoding is performed on the core text data and the image textual data to obtain the text information of the customer material.

[0127] Specifically, historical material data is collected in advance, which includes a rich variety of customer material samples, potentially covering different types and formats of text and image information.

[0128] Furthermore, these historical data materials are screened and preprocessed.

[0129] Furthermore, remove data that is of poor quality, incomplete, or has low relevance to the current customer's materials.

[0130] Furthermore, the preprocessing stage includes operations such as cropping, scaling, and normalizing the images to ensure the consistency and standardization of the image data; cleaning the text to remove special characters, stop words, etc., and performing natural language processing operations such as word segmentation and part-of-speech tagging.

[0131] Furthermore, the processed images and corresponding text are paired to form image-text pairs.

[0132] Furthermore, these image-text pairs are integrated to construct an image-text training dataset suitable for customer materials, which will provide sufficient and high-quality data support for subsequent model training.

[0133] Furthermore, the client's materials are processed using a dual-tower encoding model, which contains two independent encoders, one for processing image data and the other for processing text data.

[0134] Furthermore, in terms of image encoders, images from the image and text training dataset are input into pre-trained image feature extraction models such as convolutional neural networks (CNNs). Through operations such as convolution and pooling, deep features of the images are extracted, and the image data is mapped into a high-dimensional feature space.

[0135] Furthermore, in terms of text encoders, the corresponding text data is input into text feature extraction models such as recurrent neural networks (RNN), long short-term memory networks (LSTM), or Transformers to perform semantic analysis and feature extraction on the text, thus mapping the text data to a high-dimensional feature space.

[0136] Furthermore, the two encoders work independently, but are optimized during training based on the relationships between the images and text.

[0137] Furthermore, two feature vectors representing the image and text features are obtained respectively. These two feature vectors constitute the image-text tower of the client material, which preserves the feature information of the image and text respectively and the potential correlation between them.

[0138] Furthermore, image-text towers and standard materials are introduced into the semantic space adversarial training process. In this process, an adversarial training model is constructed, which consists of a generator and a discriminator.

[0139] Furthermore, the generator's task is to generate text-image data that is semantically consistent with the standard material based on the image-text tower, while the discriminator is responsible for determining whether the generated data truly comes from the standard material.

[0140] Furthermore, at the start of training, the text-image data generated by the generator may differ significantly from the standard material, and the discriminator can accurately distinguish between them.

[0141] Furthermore, as training progresses, the generator continuously learns the semantic features of the image-text pyramid and standard materials, adjusting its parameters to generate text-image data that more closely resembles the standard materials. The discriminator is also continuously optimized to improve its discriminative ability.

[0142] Furthermore, through continuous adversarial training, the generator and discriminator reach a relatively balanced state. At this point, the text-image data generated by the generator is highly similar to the standard material in semantic space, thus obtaining text-image data that meets the requirements.

[0143] Furthermore, the goal of cross-modal knowledge distillation is to extract the knowledge contained in text-image data from the joint representation of images and text, and transform it into plain text information.

[0144] Furthermore, select one teacher model and one student model.

[0145] Furthermore, the teacher model is typically a complex and high-performance multimodal model that can fully understand cross-modal information in text-image data.

[0146] Furthermore, the student model is a relatively simple text model used to learn and receive knowledge from the teacher model.

[0147] Furthermore, during the knowledge distillation process, text-image data is input into the teacher model, which generates a soft label based on the feature information of the image and text. This soft label contains the semantic association information between the image and text.

[0148] Furthermore, the text portion of the same text-image data is input into the student model, allowing the student model to learn the soft labels generated by the teacher model.

[0149] Furthermore, by minimizing the difference between the student model's output and the soft label, the student model gradually learns cross-modal knowledge in the text-image data and transforms it into its own text representation.

[0150] Furthermore, after multiple iterations of training, the student model is able to accurately extract textual information from customer materials from text-image data, completing the process of cross-modal knowledge distillation.

[0151] Specifically, in this step, the text data in the text-image data is analyzed in depth first.

[0152] Furthermore, modality-specific pruning is based on the structural, semantic, and syntactic features of text data. Specific algorithms identify redundant parts of the text data that contribute little to the overall semantics or are not closely related to the image.

[0153] For example, some frequently occurring but semantically vague function words, or modifying phrases that are irrelevant in the current context.

[0154] Furthermore, these identified parts are removed from the text data, retaining only those parts that express key information, are closely related to the image, and play a core role in understanding the overall content. This yields the core text data, making the text information more concise and crucial.

[0155] Furthermore, for image data within text-image data, specialized image analysis and text conversion techniques are employed to extract textual knowledge.

[0156] Furthermore, computer vision technologies such as object detection and image recognition are used to identify various elements in images, such as objects, scenes, and symbols.

[0157] Furthermore, these identified elements are converted into corresponding text descriptions.

[0158] For example, it can identify an apple in an image as the text word "apple" and a street in a scene as "street".

[0159] Furthermore, it analyzes the relationships between image elements and expresses them in text form, such as "apple on the table" describing the positional relationship between the apple and the table.

[0160] Furthermore, by integrating the text information derived from image elements and their relationships, we obtain image-textualized data of the image data, thus presenting the image information in text form, which facilitates subsequent processing together with text data.

[0161] Furthermore, the core text data and image textualization data obtained in the first two steps are input into the text-aware decoding model.

[0162] Furthermore, based on natural language processing technology, the model is able to understand the semantic and contextual information of text data.

[0163] Furthermore, the model first fuses the core text data and image textual data, organically combining the information from both and fully considering their correlation and complementarity.

[0164] Furthermore, the fused data is analyzed and transformed through a series of decoding operations, such as deep learning-based neural network decoding algorithms.

[0165] Furthermore, during the decoding process, the model transforms the data into fluent, coherent, and logical text information based on the language model and semantic understanding rules.

[0166] Furthermore, the final output is the integrated text information of the client's materials, which combines the key content of the original text data and the textual knowledge contained in the image data, and can more comprehensively and accurately reflect the textual information of the client's materials.

[0167] In summary, the multimodal joint pre-training strategy integrates multiple modalities of information from the client's materials, such as text and images.

[0168] In summary, this approach allows for a comprehensive understanding of the client's materials from different perspectives, uncovering potential semantic connections within them.

[0169] In summary, during adversarial training, the model continuously learns and optimizes, enabling it to more accurately grasp the true semantics of the text, avoid misinterpretation or omission of textual information, and thus obtain more accurate and complete customer material textual information.

[0170] For example, when customer materials include product images and related text descriptions, a multimodal joint pre-training strategy can combine the product's appearance features in the images with product parameters in the text to accurately extract key product descriptions as text information.

[0171] In summary, the model was trained using a graphic training dataset constructed from historical material data, exposing it to a large number of different types of customer material samples.

[0172] In summary, during this process, the model learns the commonalities and differences among various materials, enhancing its adaptability to different customer materials.

[0173] In summary, semantic association adversarial training further optimizes the model, enabling it to accurately perform semantic association analysis and obtain effective textual information when faced with new and unseen customer materials, thereby improving the model's generalization performance.

[0174] Overall, this means the system can better handle a wide variety of complex customer materials, expanding the application scope of the automatic material report generation system.

[0175] In summary, text information obtained through semantic association adversarial training can be more closely integrated with the actual needs and application scenarios of customer materials.

[0176] In summary, because the model considers the correlation between text and other modal information during training, the extracted text information not only contains basic textual content, but also contains deep semantics related to other information, making the text information more targeted and practical.

[0177] In summary, when generating material reports, this high-quality text information can more accurately describe the properties and characteristics of materials, providing more valuable references for report users and improving the quality and usability of material reports.

[0178] The formula recognition module 103 is used to activate the topological relationship of the customer material based on the dynamic feature cross-attention mechanism to obtain the formula information of the customer material.

[0179] In this embodiment of the invention, when the formula recognition module performs topological relationship activation on the customer material based on a dynamic feature cross-attention mechanism to obtain the formula information of the customer material, it is specifically used for:

[0180] Based on the dynamic feature cross-attention mechanism, multi-scale visual feature extraction is performed on the customer material to obtain the image features of the customer material;

[0181] Based on the topological inference head, differential calculations are performed on the image features to obtain the symbolic data of the customer material;

[0182] The symbol data is fused at two levels to obtain the formula information of the customer material.

[0183] When the formula recognition module performs differential calculations on the image features based on the topological inference head to obtain the customer material symbol data, it is specifically used for:

[0184] The symbol positions of the image features are obtained by performing symbol recognition on the image features based on the topological inference head;

[0185] Symbolic features are extracted from the customer materials to obtain a symbolic feature map of the customer materials;

[0186] Based on the symbol location and the symbol feature map, differential calculations are performed on the customer material to obtain the customer material symbol data.

[0187] When the formula recognition module performs differential calculations on the customer material based on the symbol position and the symbol feature map to obtain the customer material symbol data, it is specifically used for:

[0188] The symbol type of the customer data is identified by performing symbol type recognition;

[0189] Based on the symbol type, the symbol feature map, and the symbol position, a differentiation calculation is performed on the customer material to obtain the symbol data of the customer material, wherein the formula for the differentiation calculation is:

[0190] (s ij(For terminology symbols)

[0191] s ij =-||q i -k j ||(s ij (for numerical symbols)

[0192] (s ij (for operators)

[0193] In the formula, s ij For symbolic data, q i Let k be the vector at the position of the i-th symbol. j Let be the vector of the i-th symbol feature map, where i is the symbol position number, j is the symbol feature map number, T is the vector transpose flag, μ is the strength of the control direction constraint, and (i→j) is the logical flow direction in the customer material.

[0194] Specifically, the topology inference head is an algorithm component with specific functions. Its core function is to extract multi-scale visual features from customer materials based on a dynamic feature cross-attention mechanism, process the obtained image features, and thereby mine the formula-related symbol data in the customer materials, laying the foundation for generating accurate formula information in the future.

[0195] Specifically, the dynamic feature cross-attention mechanism is a technique in the field of deep learning used to handle feature interactions and model data dependencies. It inputs each feature into the model and then automatically learns the relationships between features through the structure of the model.

[0196] Specifically, image data of customer materials is input into a model with a dynamic feature cross-attention mechanism.

[0197] Furthermore, the model divides the image into multiple scales, from the macroscopic overall image scale to the progressively refined local detail scale.

[0198] Furthermore, at each scale, the dynamic feature cross-attention mechanism begins to take effect. It automatically analyzes the correlation between different feature channels, dynamically cross-combines features at each scale, and assigns attention weights to different feature cross-coordination based on the content of the input image.

[0199] Furthermore, in this way, the rich visual features contained in the image at various scales are extracted comprehensively and meticulously, ultimately forming a feature set that can represent the client's material image.

[0200] Furthermore, the image features of the customer materials obtained in the previous step are input into the topology inference head.

[0201] Furthermore, the topology inference head will perform in-depth differential calculations on these image features. It will analyze the differences in spatial structure, feature distribution, and other aspects of image features, and through computational operations, convert the information contained in the image features into a symbolic expression.

[0202] Furthermore, these symbolic data can abstract and represent key information in customer material images from a specific logical and structural perspective, laying the foundation for subsequent mining of the material's inherent laws and knowledge.

[0203] Furthermore, after obtaining the symbolic data, the process enters a two-stage fusion phase.

[0204] Furthermore, the first-level fusion will first integrate the symbol data according to certain rules or feature attributes, such as grouping and merging symbols with similar semantics or structures, and initially mining the shallow relationships between symbols.

[0205] Furthermore, a second level of fusion is conducted. This stage involves a more in-depth analysis of the symbolic data after the first level of fusion. Using more complex algorithms and logical rules, the symbolic data from different groups are deeply fused in an attempt to extract regular patterns and relationships.

[0206] Furthermore, after two-level fusion, the formula information corresponding to the customer's materials was successfully obtained from the symbolic data. This formula information can be used to explain certain characteristics and patterns of the customer's materials or serve as an important basis for subsequent decision-making and analysis.

[0207] Specifically, the extracted image features are input into the topology inference head.

[0208] Furthermore, the topology inference head will perform part-by-part analysis of image features based on its internally predefined symbol recognition rules and pattern matching algorithms.

[0209] Furthermore, it attempts to identify various symbols contained in the image features. During the identification process, it simultaneously records the position information of each identified symbol in the original image feature space.

[0210] Furthermore, by traversing the entire image features, a list containing all identified symbols and their corresponding locations is finally generated. The symbol locations recorded in this list will provide crucial localization information for subsequent processing.

[0211] Furthermore, a specially designed symbol feature extraction algorithm is used to process the client's materials. This algorithm focuses on symbol-related information from the client's materials, and extracts feature vectors that can characterize the symbol's properties by analyzing various attributes such as color, texture, and shape of the client's materials.

[0212] Furthermore, these extracted symbolic feature vectors are organized according to a certain spatial layout to form a two-dimensional or multi-dimensional matrix, which is the symbolic feature map.

[0213] Furthermore, the symbol feature map comprehensively and systematically describes the feature information of various symbols in the client's materials, providing a rich data foundation for further analysis of the internal relationships and characteristics of the symbols.

[0214] Furthermore, the previously acquired symbol location information is combined with the generated symbol feature map to perform differential calculations.

[0215] Furthermore, based on the symbol position, the feature information of the corresponding symbol position is accurately extracted from the symbol feature map.

[0216] Furthermore, a series of differentiated computational operators are used to perform operations on these extracted symbolic features. These operations focus on the differences between symbolic features at different locations, such as the magnitude of change in feature values ​​and the angle between feature vectors.

[0217] Furthermore, by quantifying and analyzing these differences, symbolic data of the customer's materials is generated. This symbolic data contains key information such as the relative relationships and degree of difference between symbols in the customer's materials, which helps to deeply understand the internal structure and characteristics of the customer's materials.

[0218] Specifically, the calculation method for the terminology symbol formula is as follows: first, calculate vector q respectively. i and k j dot multiplied by q i ·k j Then calculate their norms ||q| respectively. i ||and||k j Finally, divide the dot product by the product of the two norms to obtain s. ij The value of is usually between [-1, 1].

[0219] Furthermore, the terminology formula calculates the vector q. i and k j Cosine similarity. It measures the similarity of two vectors in terms of direction. The closer the value is to 1, the more similar the two vectors are in direction; the closer the value is to -1, the more opposite the directions are; a value of 0 indicates that the two vectors are orthogonal (perpendicular). In fields such as natural language processing, it is often used to measure the semantic similarity between word vectors or text vectors.

[0220] Specifically, the formula for calculating the numerical sign is as follows: first calculate q. i and k j The difference vector q i -k jThen calculate the norm of the difference vector ||q. i -k j ||, and finally take its opposite to get s. ij .

[0221] Furthermore, the numerical symbol calculation formula reflects the degree of difference between two vectors; the smaller the value, the closer the two vectors are and the smaller the difference; the larger the value, the greater the difference between the two vectors. In tasks such as clustering and classification, it can be used to measure the distance between samples, help determine the similarity between samples, and then perform clustering or classification operations.

[0222] Specifically, the calculation formula for the operator is as follows: first, calculate vector q. i transpose of vector k j product This is a common matrix multiplication operation, yielding a scalar value. Then, (i→j) is calculated, where (i→j) represents the logical flow direction in the customer material, and μ is the strength of the control direction constraint. These two results are then added together to obtain s. ij .

[0223] Furthermore, in the formula for calculating the operator, the vector inner product part can reflect the vector q. i and k j The similarity, and (i→j) represents the influence of the logical flow factors in the customer materials on the result, thus more comprehensively characterizing q. i and k j The relationship between vectors is used in scenarios where multiple factors need to be considered to measure the relationship between vectors.

[0224] In general, the formulas in customer materials often have complex topological structures. The dynamic feature cross-attention mechanism can extract visual features from customer materials at multiple scales and fully capture the spatial positional relationships and logical connections between various symbols and elements in the formula.

[0225] In summary, by processing these features through the topological inference head, differential computing is used to accurately identify different types of symbols (term symbols, numerical symbols, operation symbols, etc.) and their interrelationships, thereby clearly parsing out the complex formula structure and transforming it into formula information that the system can understand and process, providing accurate formula content support for material reports.

[0226] In general, formulas are presented in various forms in customer materials, and there may be differences in layout, variations in symbols, etc.

[0227] In summary, the dynamic nature of this mechanism makes it highly adaptable, allowing it to dynamically adjust the focus and processing methods for features based on the characteristics of different customer materials.

[0228] In summary, regardless of the form in which the formula appears, the formula information can be accurately extracted by activating the topological relationship. This ensures that the automatic material report generation system can stably and accurately handle formula-related content when facing various complex situations, thereby improving the system's versatility and robustness.

[0229] In summary, formula information is an important component of material reports, and accurate formula information is crucial for the scientific validity and reliability of the reports.

[0230] In summary, the formula information obtained through the dynamic feature cross-attention mechanism can completely and accurately reflect the formula content in the client's materials. This information, combined with the chart and text information extracted by the system, is populated into the material report according to the client's template requirements, making the material report more complete and accurate. This provides more valuable reference for relevant personnel and enhances the practicality and credibility of the material report in actual applications.

[0231] The customer template analysis module 104 is used to perform NLP parsing on the pre-acquired customer template to obtain the report requirements of the customer template.

[0232] In this embodiment of the invention, when the customer template analysis module performs NLP parsing on the pre-acquired customer template to obtain the report requirements of the customer template, it is specifically used for:

[0233] The physical structure of the pre-acquired customer template is analyzed to obtain the template structure of the customer template;

[0234] Logical rules are extracted from the customer template to obtain the template rules for the customer template;

[0235] The template rules and template structure are hierarchically encoded to obtain the reporting requirements of the customer template.

[0236] Specifically, the pre-acquired customer template files are read, and the corresponding parsing tool is used according to the file format (such as PDF, Word, HTML, etc.).

[0237] For example, if it's a PDF file, you can use libraries like PyPDF2; if it's a Word file, the python-docx library is a good choice.

[0238] Furthermore, a detailed analysis was conducted on the physical elements of the template, such as page layout, paragraph division, font styles, and table structure.

[0239] Furthermore, it identifies the titles, body text, images, tables, and other content of each section, and determines their positional relationships and hierarchical structure.

[0240] Furthermore, this information is organized into a tree structure or graph structure that is easy to understand and process, thereby obtaining the template structure of the customer template.

[0241] For example, by using the title as the parent node, the body paragraphs below it as child nodes, and tables as independent child nodes, the physical structure of the template can be clearly demonstrated.

[0242] Furthermore, we conduct in-depth analysis of the text content of the customer templates and use natural language processing technology to identify the hidden logical rules within them.

[0243] Further text preprocessing is performed, including noise removal, word segmentation, and part-of-speech tagging. Then, rule-matching algorithms or machine learning models are used to extract key logical information from the text, such as conditional judgments, order requirements, and data constraints.

[0244] For example, a template might specify that "if data item A is greater than 100, then data item B must be empty," which is a typical logical rule.

[0245] Furthermore, all the extracted logical rules are organized and categorized to form a rule set, which is the template rule for the customer template.

[0246] Furthermore, add necessary metadata to each rule, such as the rule's scope of application and triggering conditions, for subsequent use and management.

[0247] Furthermore, by comprehensively considering the template rules and template structure obtained above, a suitable hierarchical coding scheme is designed.

[0248] Furthermore, based on the hierarchical relationship of the template structure, a unique code is assigned to each structural element, and the level of the code must correspond to the hierarchy of the structure.

[0249] For example, in a tree-structured template, the root node can be represented by a first-level code, its child nodes by a second-level code, and so on.

[0250] Furthermore, the template rules are associated with the corresponding structural elements, and each rule is also assigned a code based on the associated structural element encoding.

[0251] Furthermore, the correspondence between rules and structures, as well as their hierarchical position within the entire template, can be clearly represented through encoding.

[0252] Furthermore, the coded template rules and template structure information are integrated to form a complete report requirements document, which details the structure and rules to be followed when generating reports based on customer templates.

[0253] In summary, through NLP analysis, the system is able to gain a deep understanding of the specific needs contained in the customer's template.

[0254] In general, client templates may contain information on various aspects such as format requirements, key content, and data presentation methods. NLP analysis can accurately extract these key points and transform them into specific report requirements.

[0255] Overall, this makes the material report generation module more targeted in filling in data, and the generated material reports can closely meet customer expectations, avoiding problems such as content mismatch and format confusion, and greatly improving the practicality and usability of the reports.

[0256] In general, templates provided by different clients may vary significantly in terms of structure, language, and reporting requirements.

[0257] In summary, the application of NLP parsing technology enables the system to process various templates. No matter how complex or diverse the format and content of a client's template may be, the system can extract valid reporting requirements from it through NLP parsing.

[0258] In summary, this versatility and flexibility allows the automatic material report generation system to be widely applicable to the needs of different customers, eliminating the need for extensive customization for each new customer template and reducing the system's usage and maintenance costs.

[0259] In summary, the explicit reporting requirements provide clear guidance for filling in chart, text, and formula data.

[0260] In summary, based on these requirements, the material report generation module can quickly and accurately fill the corresponding data into the designated locations in the customer template, avoiding blind attempts and incorrect data entry, effectively reducing the time and computing resource consumption required for data entry, and greatly improving the efficiency of material report generation.

[0261] Overall, this is of great significance for scenarios that require processing large amounts of customer materials and templates, as it can significantly improve work efficiency and meet the timeliness requirements of actual business operations.

[0262] The material report generation module 105 is used to fill the chart data, text data and formula data into the pre-acquired customer template based on the report requirements to obtain the material report of the customer material.

[0263] Specifically, after filling in the chart data, text data, and formula data, the entire customer template is fully integrated and checked.

[0264] Furthermore, check the overall layout to see if the data in each section is completely filled in, if the positions are correct, and if the layout of charts, text, and formulas is coordinated and aesthetically pleasing.

[0265] Furthermore, double-check the accuracy of the data, including data points in charts, key information in text, and formula calculation results. Also, check the consistency of the report format, such as page numbering, header and footer content, etc.

[0266] Furthermore, after careful inspection and necessary adjustments, a complete, accurate, and customer-compliant material quality report is finally generated.

[0267] Reference Figure 2 The diagram shown is a flowchart illustrating an automatic material report generation method according to an embodiment of the present invention. In this embodiment, the automatic material report generation method includes:

[0268] S1. Based on the Transformer-CNN cascaded model, multi-scale local chart extraction is performed on the pre-acquired customer materials to obtain the chart information of the customer materials;

[0269] S2. Based on a multimodal joint pre-training strategy, perform semantic association adversarial training on the customer material to obtain the text information of the customer material;

[0270] S3, used to activate the topological relationship of the customer material based on the dynamic feature cross-attention mechanism, and obtain the formula information of the customer material;

[0271] S4. Used to perform NLP parsing on the pre-acquired customer template to obtain the report requirements of the customer template;

[0272] S5. To fill the chart data, text data and formula data into the pre-acquired customer template based on the report requirements, so as to obtain the material report of the customer material.

[0273] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0274] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0275] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A material report automatic generation system, characterized in that, The system includes a chart recognition module, a text recognition module, a formula recognition module, a customer template analysis module, and a material report generation module, wherein: The chart recognition module is used to perform multi-scale local chart extraction on the pre-acquired customer materials based on the Transformer-CNN cascaded model to obtain the chart information of the customer materials. The text recognition module is used to perform semantic association adversarial training on the customer material based on a multimodal joint pre-training strategy to obtain the text information of the customer material; The formula recognition module is used to activate the topological relationship of the customer material based on the dynamic feature cross-attention mechanism to obtain the formula information of the customer material. The customer template analysis module is used to perform NLP parsing on the pre-acquired customer templates to obtain the reporting requirements of the customer templates; The material report generation module is used to fill the chart data, text data and formula data into the pre-acquired customer template based on the report requirements to obtain the material report of the customer material.

2. The material report automatic generation system as described in claim 1, characterized in that, When the chart recognition module performs multi-scale local chart extraction on the pre-acquired customer materials based on the Transformer-CNN cascade model to obtain the chart information of the customer materials, it is specifically used for: The pre-acquired customer material is cleaned of noise using a Transformer-CNN cascaded model to obtain the standard material of the customer material. Multi-scale local feature extraction is performed on the standard material to obtain the target feature map of the standard material; The target feature map is graphically encoded using a locality-sensitive hash attention mechanism to obtain the feature chart data of the customer material. The feature chart data is decoded in parallel by multiple heads to obtain the chart information of the customer material.

3. The material report automatic generation system as described in claim 2, characterized in that, When the chart recognition module performs multi-scale local feature extraction on the standard material to obtain the target feature map of the standard material, it is specifically used for: Multi-level convolutional feature extraction is performed on the standard material to obtain the initial feature map of the customer material; Generate a multi-scale feature map of the customer's material based on the standard material; Based on the multi-scale feature map, feature adversarial training is performed on the initial feature map to obtain the target feature map of the customer material.

4. The material report automatic generation system as described in claim 1, characterized in that, When the text recognition module performs semantic association adversarial training on the customer material based on a multimodal joint pre-training strategy to obtain the text information of the customer material, it is specifically used for: A graphic training dataset of the customer's materials is constructed based on pre-acquired historical material data; Based on the image and text training dataset, the customer material is encoded using a dual-tower encoding method to obtain the image-text tower of the customer material. Semantic space adversarial training is performed on the image-text tower and the standard material to obtain text-image data; Cross-modal knowledge distillation is performed on the text-image data to obtain the text information of the customer material.

5. The material report automatic generation system as described in claim 4, characterized in that, When the text recognition module performs cross-modal knowledge distillation on the text-image data to obtain the text information of the customer material, it is specifically used for: Modality-specific pruning is performed on the text data in the text-image data to obtain the core text data of the text data; Text knowledge is extracted from the image data in the text-image data to obtain the image-textualized data of the image data. Text-aware decoding is performed on the core text data and the image textual data to obtain the text information of the customer material.

6. The material report automatic generation system as described in claim 1, characterized in that, When the formula recognition module performs topological relationship activation on the customer material based on a dynamic feature cross-attention mechanism to obtain the formula information of the customer material, it is specifically used for: Based on the dynamic feature cross-attention mechanism, multi-scale visual feature extraction is performed on the customer material to obtain the image features of the customer material; Based on the topological inference head, differential calculations are performed on the image features to obtain the symbolic data of the customer material; The symbol data is fused at two levels to obtain the formula information of the customer material.

7. The material report automatic generation system as described in claim 6, characterized in that, When the formula recognition module performs differential calculations on the image features based on the topological inference head to obtain the customer material symbol data, it is specifically used for: The symbol positions of the image features are obtained by performing symbol recognition on the image features based on the topological inference head; Symbolic features are extracted from the customer materials to obtain a symbolic feature map of the customer materials; Based on the symbol location and the symbol feature map, differential calculations are performed on the customer material to obtain the customer material symbol data.

8. The material report automatic generation system as described in claim 7, characterized in that, When the formula recognition module performs differential calculations on the customer material based on the symbol position and the symbol feature map to obtain the customer material symbol data, it is specifically used for: The symbol type of the customer data is identified by performing symbol type recognition; Based on the symbol type, the symbol feature map, and the symbol position, a differentiation calculation is performed on the customer material to obtain the symbol data of the customer material, wherein the formula for the differentiation calculation is: (s ij (For terminology symbols) s ij =-||q i -k j ||(s ij (for numerical symbols) (s ij (for operators) In the formula, s ij For symbolic data, q i Let k be the vector at the position of the i-th symbol. j Let be the vector of the i-th symbol feature map, where i is the symbol position number, j is the symbol feature map number, T is the vector transpose flag, μ is the strength of the control direction constraint, and (i→j) is the logical flow direction in the customer material.

9. The material report automatic generation system as described in claim 1, characterized in that, When the customer template analysis module performs NLP parsing on the pre-acquired customer template to obtain the report requirements of the customer template, it is specifically used for: The physical structure of the pre-acquired customer template is analyzed to obtain the template structure of the customer template; Logical rules are extracted from the customer template to obtain the template rules for the customer template; The template rules and template structure are hierarchically encoded to obtain the reporting requirements of the customer template.

10. A method for automatically generating material reports, characterized in that, The method includes: S1. Based on the Transformer-CNN cascaded model, multi-scale local chart extraction is performed on the pre-acquired customer materials to obtain the chart information of the customer materials; S2. Based on a multimodal joint pre-training strategy, perform semantic association adversarial training on the customer material to obtain the text information of the customer material; S3, used to activate the topological relationship of the customer material based on the dynamic feature cross-attention mechanism, and obtain the formula information of the customer material; S4. Used to perform NLP parsing on the pre-acquired customer template to obtain the report requirements of the customer template; S5. To fill the chart data, text data and formula data into the pre-acquired customer template based on the report requirements, so as to obtain the material report of the customer material.

Images