Data processing method and device, equipment, medium and program product

Abstract

The invention provides a data processing method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring original data of at least two different modes of an object to be audited; extracting spatial perception feature representation from the original data of the image modality, extracting semantic understanding feature representation from the original data of the text modality, and extracting relation perception feature representation from the original data of the structured modality; calculating a semantic alignment weight between the spatial perception feature representation and the semantic understanding feature representation through a position perception attention mechanism, and performing fusion according to the semantic alignment weight to generate a collaborative feature representation; the relation perception feature representation is used as a query vector, the collaborative feature representation is used as a key vector and a value vector, the relevancy between the query vector and the key vector is calculated through multi-head cross attention, and a fusion feature representation is generated according to the relevancy and the value vector; and inputting the fusion feature representation into an auditing decision model to output an auditing result for the to-be-audited object.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence or cloud computing technology, and more specifically to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the increasing complexity of data processing needs, automated review of data from diverse sources and formats (i.e., multimodal data) has become a key technological aspect in many fields, including finance, insurance, content security, and industrial quality inspection. Traditional data review methods primarily rely on manual review. When processing massive amounts of heterogeneous data, manual review is inefficient, leading to prolonged processing cycles and failing to meet the real-time and high-throughput requirements of modern businesses. Furthermore, due to human subjectivity, review standards are difficult to standardize and are easily influenced by subjective factors, resulting in misjudgments and information omissions. In addition, traditional data review often relies on single-modality automated identification. While this improves efficiency to some extent, its inherent limitations lead to incomplete information capture, making it difficult to guarantee accuracy and stability when dealing with complex and ever-changing data. For example, text modalities may be difficult to understand due to the abundance of technical terminology, while image modalities may miss key information due to limited focus areas. Moreover, commonly used evaluation metrics cannot meet diverse recognition needs. Summary of the Invention

[0003] In view of the above problems, embodiments of this disclosure provide a data processing method, apparatus, device, medium, and program product.

[0004] This disclosure provides a data processing method, comprising: acquiring raw data of at least two different modalities of an object to be reviewed, the different modalities including image modality, text modality, and structured modality; extracting spatially perceived feature representations from the raw data of the image modality, extracting semantically understood feature representations from the raw data of the text modality, and extracting relation-aware feature representations from the raw data of the structured modality; calculating semantic alignment weights between the spatially perceived feature representations and the semantically understood feature representations through a position-aware attention mechanism, and fusing them according to the semantic alignment weights to generate a collaborative feature representation; using the relation-aware feature representation as a query vector, and the collaborative feature representation as a key vector and a value vector, calculating the relevance between the query vector and the key vector through multi-head cross-attention, and generating a fused feature representation according to the relevance and the value vector; and inputting the fused feature representation into a review decision model to output a review result for the object to be reviewed.

[0005] According to embodiments of this disclosure, obtaining raw data of at least two different modalities of an object to be audited includes: obtaining an original image, an original text file, and a structured data table associated with the object to be audited; performing image preprocessing operations on the original image to obtain raw data of the image modality; performing text preprocessing operations on the original text file to obtain raw data of the text modality; and performing data cleaning and formatting operations on the structured data table to obtain raw data of the structured modality.

[0006] According to embodiments of this disclosure, extracting spatially aware feature representations from the raw data of the image modality, extracting semantically understanding feature representations from the raw data of the text modality, and extracting relation-aware feature representations from the raw data of the structured modality includes: extracting spatially aware feature representations from the raw data of the image modality using a convolutional neural network model; extracting semantically understanding feature representations from the raw data of the text modality using a pre-trained language model; and extracting relation-aware feature representations from the raw data of the structured modality using an embedding layer or encoder, wherein the embedding layer or encoder is used to map discrete structured fields into continuous target vectors.

[0007] According to embodiments of this disclosure, based on an initial feature representation, a semantic alignment weight is calculated between a spatially perceived feature representation and a semantically understood feature representation using a location-aware attention mechanism. This includes: determining a first target modality and a second target modality to be interacted with, wherein the first target modality and the second target modality are different modalities selected from an image modality, a text modality, and a structured modality; transforming the initial feature representation of the first target modality through a first linear projection layer to generate a query vector; transforming the initial feature representation of the second target modality through a second linear projection layer to generate a key vector; generating a learnable location encoding matrix associated with the first target modality and the second target modality; and calculating a first attention weight matrix that integrates spatial / semantic location information based on the query vector, the key vector, and the learnable location encoding matrix, wherein the first attention weight matrix includes semantic alignment weights.

[0008] According to embodiments of this disclosure, the step of using the relation-aware feature representation as a query vector, the collaborative feature representation as a key vector and a value vector, calculating the relevance between the query vector and the key vector through multi-head cross-attention, and generating a fused feature representation based on the relevance and the value vector includes: transforming the relation-aware feature representation into a multi-head query vector group through a third linear projection layer; transforming the collaborative feature representation into a multi-head key vector group through a fourth linear projection layer, and into a multi-head value vector group through a fifth linear projection layer; for each attention head, calculating the target attention weight between the query vector subgroup and the key vector subgroup of each attention head in parallel, and using the target attention weight to perform a weighted summation on the corresponding value vector subgroup to generate the output of the attention head; concatenating the outputs of all attention heads and fusing them through a linear projection layer to generate the fused feature representation.

[0009] According to embodiments of this disclosure, after obtaining raw data of at least two different modalities of the object to be audited, the data processing method further includes: generating a unique target identifier and a corresponding cryptographic hash value for each type of raw data; storing the target identifier and the cryptographic hash value as an associated data pair in a distributed ledger; wherein the audit result includes the target identifier of at least a portion of the raw data associated with the result.

[0010] According to embodiments of this disclosure, the fused feature representation is input into an audit decision model to output an audit result for the object to be audited, including: outputting an interpretability report based on the audit decision model; the interpretability report includes a quantified value of the contribution of the initial feature representation of each modality to the audit result in the fused feature representation.

[0011] According to embodiments of this disclosure, the raw data of the image modality includes medical invoice images, the raw data of the text modality includes medical record text, and the raw data of the structured modality includes a detailed list of medical expenses. Extracting spatially perceptual feature representations from the raw data of the image modality includes extracting visual layout features related to invoice type, amount area, and seal position in the medical invoice image. Extracting semantically understanding feature representations from the raw data of the text modality includes extracting semantic features of medical entities related to disease diagnosis, treatment measures, and drug names in the medical record text. Extracting relationally perceptual feature representations from the raw data of the structured modality includes extracting price associations, quantity relationships, and medical insurance catalog matching features among the various chargeable items in the detailed list of medical expenses.

[0012] This disclosure also provides a data processing apparatus, comprising: an acquisition module for acquiring raw data of at least two different modalities of an object to be reviewed, the different modalities including image modality, text modality, and structured modality; an extraction module for extracting spatially perceived feature representations from the raw data of the image modality, extracting semantically understood feature representations from the raw data of the text modality, and extracting relation-aware feature representations from the raw data of the structured modality; a first fusion module for calculating semantic alignment weights between the spatially perceived feature representations and the semantically understood feature representations through a position-aware attention mechanism, and fusing them according to the semantic alignment weights to generate a collaborative feature representation; a second fusion module for using the relation-aware feature representation as a query vector, and the collaborative feature representation as a key vector and a value vector, calculating the relevance between the query vector and the key vector through multi-head cross-attention, and generating a fused feature representation according to the relevance and the value vector; and an review module for inputting the fused feature representation into a review decision model to output a review result for the object to be reviewed.

[0013] In another aspect, this disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the data processing method described above.

[0014] Another aspect of this disclosure provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the data processing method described above.

[0015] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0016] According to the data processing methods, apparatus, devices, media, and program products provided in this disclosure, by comprehensively processing raw information from multiple modalities such as images, text, and structured data, the multi-dimensional features and related details of the object to be reviewed are fully captured, significantly reducing the risk of information omissions and misjudgments caused by partial or isolated information analysis. Furthermore, by dynamically modeling the semantic relationships and importance between different modal information using a cross-modal attention mechanism, adaptive weighted fusion of multi-modal features is achieved, enabling the fused feature representation to more accurately and robustly reflect the overall state of the object, thereby improving the accuracy and reliability of subsequent review decisions. This method significantly improves the efficiency of data review and shortens the business processing cycle through automated, end-to-end processing. Simultaneously, based on objective analysis of the review decision model, it effectively avoids the subjectivity and consistency differences present in manual review, ensuring the standardization of the review process and the reproducibility of the results. Attached Figure Description

[0017] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustrations depict application scenarios of data processing methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure.

[0019] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 3 A flowchart illustrating a data processing method based on multimodal feature fusion according to an embodiment of the present disclosure is shown schematically.

[0021] Figure 4 A flowchart illustrating a feature extraction method for different modal data according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 5 A flowchart illustrating multimodal feature fusion according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 6 A flowchart illustrating the processing of an audit decision model according to an embodiment of the present disclosure is shown schematically.

[0024] Figure 7 A schematic block diagram of a data processing apparatus according to embodiments of the present disclosure is shown; and

[0025] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0026] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] It should be noted that the data processing methods and apparatus specified in this disclosure can be used in the financial field in the fields of artificial intelligence or cloud computing technology, or in any field other than the financial field. The application fields of the data processing methods and apparatus disclosed in this disclosure are not limited.

[0031] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0032] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, and corresponding measures have been taken to ensure that they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0033] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this disclosure all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0034] This disclosure provides a data processing method, apparatus, device, medium, and program product.

[0035] Figure 1 The illustrations depict application scenarios of data processing methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure.

[0036] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0037] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to submit multimodal data, configure review strategies, trigger automated review processes, and receive corresponding review results and explanatory reports. For example, users with different business roles (such as insurance claims adjusters, financial risk control specialists, and content moderators) can use their respective terminal devices to perform operations corresponding to their responsibilities on a unified multimodal data review platform, such as submitting multimodal materials to be reviewed, viewing automated review results, analyzing explanatory reports, or adjusting review model parameters.

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server providing various services, such as a backend processing server that automatically processes multimodal raw data submitted by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend processing server can respond to the review requests submitted by the terminal devices, and through the deployed multimodal feature extraction model, cross-modal attention fusion model, and review decision model, perform collaborative analysis and comprehensive judgment on the input images, text, and structured data, and return the generated review results, confidence assessments, and interpretability reports to the corresponding user's terminal device in real time.

[0040] It should be noted that the data processing method provided in this embodiment can generally be executed by server 105. Correspondingly, the data processing device provided in this embodiment can generally be located in server 105. The data processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data processing device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] The following will be based on Figure 1 The described scene, through Figure 2 and Figure 6 The data processing method of the embodiments of this disclosure will be described in detail.

[0043] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0044] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S250.

[0045] In operation S210, raw data of the object to be audited in at least two different modalities are obtained. The different modalities include image modality, text modality and structured modality.

[0046] In embodiments of this disclosure, user consent or authorization may be obtained before acquiring raw data of at least two different modalities of the object to be reviewed. For example, before operation S210, a request may be sent to the user to acquire raw data of at least two different modalities of the object to be reviewed. If the user consents or authorizes the acquisition of raw data of at least two different modalities of the object to be reviewed, operation S210 is performed.

[0047] Image modalities include visual information such as medical invoice images, medical record pictures, and vehicle damage photos; text modalities include unstructured text such as medical record descriptions, claim descriptions, and insurance terms; and structured modalities include data with fixed formats such as tabular data, time series, and expense details.

[0048] In operation S220, spatially perceptual feature representations are extracted from the raw data of the image modality, semantically perceptual feature representations are extracted from the raw data of the text modality, and relation-perceptual feature representations are extracted from the raw data of the structured modality.

[0049] Raw data from different modalities are input into pre-trained feature extraction models corresponding to each modality. For example, convolutional neural networks are used to process image data, natural language processing models are used to process text data, and specific encoders are used to process structured data. The feature extraction models can extract high-dimensional, semantically rich initial feature representations from the raw data respectively.

[0050] In operation S230, the semantic alignment weight between the spatially perceived feature representation and the semantically understood feature representation is calculated through a position-aware attention mechanism, and the two are fused according to the semantic alignment weight to generate a collaborative feature representation.

[0051] The first-level cross-modal attention fusion is performed. The extracted spatially perceptual feature representation and semantically understood feature representation are input into the attention calculation process. This process, by introducing a learnable positional encoding bias, explicitly models the implicit alignment relationship between the text sequence position and the image spatial position when calculating the attention weights between text semantics and image spatial features, thereby quantifying the semantic matching degree between key entities in the text description and the corresponding visual regions in the image. Based on the calculated semantic alignment attention weights, the two features are fused to generate an image-text collaborative feature representation. This representation deeply integrates textual and image information; for example, it associates the description of "right anterior lobe nodule" in medical record text with the corresponding region features in CT images, achieving semantic alignment and complementarity between "what is seen" and "what is described."

[0052] In operation S240, the relation-aware feature representation is used as the query vector, and the collaborative feature representation is used as the key vector and value vector. The correlation between the query vector and the key vector is calculated through multi-head cross attention, and a fused feature representation is generated based on the correlation and the value vector.

[0053] A second-level cross-modal attention fusion is performed to achieve attention to multimodal context in structured data. The relation-aware feature representation is used as the query vector, and the previously generated image-text collaborative feature representation is used as the key and value vectors, respectively, for multi-head cross-attention computation. By parallel computation of attention distributions in multiple subspaces, the model can focus on the multimodal context most relevant to structured information such as cost details and compliance rules in the image-text collaborative features from different semantic dimensions. For example, the system can automatically learn to associate and verify the "ultrasound examination fee" item in the medical expense list with the "ultrasound examination" description in the report text and the corresponding examination record in the imaging data. Finally, the value vectors are weighted and fused according to the calculated attention weights to generate a final fused feature representation that comprehensively reflects the multi-dimensional information of the object to be reviewed.

[0054] In operation S250, the fused feature representation is input into the audit decision model to output the audit result for the object to be audited.

[0055] In the embodiments of this disclosure, a corresponding operation entry point can be provided to the user, allowing the user to choose to agree to or reject the automated decision result. That is, before executing the processing / decision for the audit result of the object to be audited, the user's instruction to agree or reject the processing / decision can be obtained through the corresponding operation entry point. If the user agrees to the processing / decision, the processing / decision for the audit result of the object to be audited is executed, i.e., step S240 is executed. If the user refuses to process / decision, the expert decision-making process is entered.

[0056] The fused feature representation is input into the review decision model (e.g., a fully connected neural network classifier), which can infer based on the fused global information and output an automated review result for the object to be reviewed, such as "review passed", "review rejected" or "manual review required" and its confidence level.

[0057] Taking auto insurance claims as an example, in traditional claims review processes, applicants need to submit a large amount of textual materials and images in various formats as evidence. Claims adjusters need to spend a significant amount of time manually reviewing these documents, extracting and verifying information item by item. This method is not only inefficient, leading to longer claims processing times and increased waiting times for applicants, but it is also prone to oversights during lengthy review processes, such as missing information or misjudgments. Furthermore, due to the inherent subjectivity of manual review, differences in the experience and abilities of claims adjusters can lead to inconsistent review standards, affecting the fairness and accuracy of claims processing.

[0058] In view of this, in order to address the problems of low efficiency and error-proneness in traditional insurance claims review and the limitations of single-modality recognition methods, the data processing method provided in the embodiments of this disclosure can be applied.

[0059] Figure 3 A flowchart illustrating a data processing method based on multimodal feature fusion according to an embodiment of the present disclosure is shown.

[0060] like Figure 3 As shown, firstly, the system acquires multi-source raw data, including vehicle damage photos (image modality), accident description text (text modality), and insurance policies and historical claims records (structured modality), submitted by the complainant, and performs standardized cleaning and preprocessing. Then, parallel feature extraction models are used to extract image features (identifying damage location and extent), text features (understanding the accident process and liability statements), and structured features (encoding insurance information and risk characteristics). Next, a cross-modal attention fusion model can be used to dynamically calculate the semantic association weights between features from different modalities (e.g., comparing the collision location described in the text with the actual damage area in the image), achieving complementary and deep fusion of multi-dimensional information. Finally, the generated unified fused feature representation is input into the review decision model for comprehensive evaluation, outputting a claims review conclusion (e.g., automatic approval, manual review required, or fraud risk warning).

[0061] Understandably, by comprehensively processing information from multiple modalities such as images, text, and structured data, it is possible to capture a wide range of details about the audited object, significantly reducing the risk of information omissions and misjudgments, and improving the accuracy of the audit. Simultaneously, calculating the correlation weights between the initial feature representations of different modalities to cross-validate information consistency can effectively improve the accuracy and reliability of the audit. Furthermore, by introducing an audit decision model, the audit content can be automatically learned and identified, avoiding the subjectivity and inconsistencies inherent in manual audits, and ensuring the standardization of the audit process and the reproducibility of the results.

[0062] According to embodiments of this disclosure, obtaining raw data of at least two different modalities of an object to be audited includes: obtaining an original image, an original text file, and a structured data table associated with the object to be audited; performing image preprocessing operations on the original image to obtain raw data of the image modality; performing text preprocessing operations on the original text file to obtain raw data of the text modality; and performing data cleaning and formatting operations on the structured data table to obtain raw data of the structured modality.

[0063] For example, in a car insurance claim scenario, the original images may include accident scene photos, vehicle damage photos, etc.; the original text files may include claim application forms, relevant supporting documents, etc.; and the structured data tables may be vehicle information tables, claim record tables, etc. In a medical claim scenario, the original images may be medical invoice images, the original text files may be medical record texts, and the structured data tables may be medical expense detail tables.

[0064] After data collection is completed, to ensure data security and immutability, the collected data can be stored on a blockchain network. Blockchain's distributed storage and encryption technologies effectively prevent data tampering or leakage, providing reliable protection for subsequent data processing and analysis.

[0065] Preprocessing the original image can eliminate image noise, correct geometric distortions, or unify image size, thereby improving the robustness of subsequent feature extraction. For example, Gaussian filtering can be used to effectively remove noise interference from the image and improve its purity; histogram equalization can be used to enhance image contrast and make details in the image more clearly visible; binarization can also be performed to convert the image into a black and white binary image, facilitating subsequent feature extraction and analysis, ultimately obtaining high-quality raw data of image modalities.

[0066] Text preprocessing of raw text files can standardize text format, remove irrelevant symbols, or perform word segmentation, facilitating effective encoding by language models. Taking medical record text as an example, after obtaining medical record text containing key information such as diagnosis results and treatment plans, text cleaning can be performed to remove irrelevant characters and special symbols; word segmentation can be performed to divide the text into independent word units according to certain rules; part-of-speech tagging can be performed to label the part of speech of each word, such as noun, verb, etc.; and semantic analysis can also be performed to gain a deeper understanding of the meaning of the text, thereby obtaining accurate and standardized raw text modal data.

[0067] Data cleaning and formatting operations on structured data tables can handle missing and outlier values, and convert heterogeneous data fields into unified numerical or vector representations, constructing structured features that can be used for model calculations. Taking a medical expense detail table as an example, the data cleaning process can check for missing values ​​and fill or delete them as needed; check the accuracy of the data and correct erroneous data values; and check for duplicate data and remove duplicate records. Data formatting operations organize the data according to a unified format, such as a unified date format or a unified amount format, ensuring data consistency and standardization, ultimately obtaining the original data in a structured modality.

[0068] Understandably, by performing a series of preprocessing operations on the data, noise, inconsistencies, and non-standard format issues in the original data of each modality are effectively eliminated, thereby transforming the original information into high-quality, well-organized modal data suitable for model processing.

[0069] According to embodiments of this disclosure, extracting spatially aware feature representations from the raw data of the image modality, extracting semantically understanding feature representations from the raw data of the text modality, and extracting relation-aware feature representations from the raw data of the structured modality includes: extracting spatially aware feature representations from the raw data of the image modality using a convolutional neural network model; extracting semantically understanding feature representations from the raw data of the text modality using a pre-trained language model; and extracting relation-aware feature representations from the raw data of the structured modality using an embedding layer or encoder, wherein the embedding layer or encoder is used to map discrete structured fields into continuous target vectors.

[0070] Figure 4 A flowchart illustrating a feature extraction method for different modal data according to an embodiment of the present disclosure is shown schematically.

[0071] like Figure 4 As shown, for the preprocessed raw data, the feature extraction model corresponding to each modality can be used to extract the initial feature representation from the raw data of each modality in parallel, so as to perform multimodal feature fusion based on the initial feature representation.

[0072] For raw image data (such as photos of vehicle damage or medical images), convolutional neural network models can be used for processing. This model, through its multi-layer convolution and pooling operations, automatically learns and extracts local and global visual features from the image, such as identifying damaged areas, the authenticity of invoices and stamps, or pathological markers. Ultimately, the model outputs a fixed-dimensional dense vector, which is the initial feature representation of the image (i.e., the spatially perceptual feature representation) (denoted as ). For example, a four-dimensional feature vector can be represented as... =[0.1,0.2,0.3,0.4], where the numerical values ​​encode key visual semantic information of the image.

[0073] For raw text-based data (such as accident descriptions and diagnostic reports), a pre-trained language model can be used for processing. This model can deeply understand the contextual semantics, syntactic structure, and entity relationships of the text. By encoding the text, the model can output a vector containing rich semantic information, namely the initial feature representation of the text (i.e., the semantic understanding feature representation) (denoted as...). For example, a corresponding feature vector can be represented as: =[0.5,0.6,0.7,0.8], which encodes the core content of the text.

[0074] For raw data in structured modalities (such as information extracted from databases or tables, or data identified and structured from images using OCR (Optical Character Recognition) technology), processing can be performed using an embedding layer or a specific encoder to map discrete, categorical, or numerical structured fields (such as expense item codes, amount values, and policy category codes) into a continuous vector space. The embedding layer or encoder can learn a distributed representation of each field or value, outputting a structured initial feature representation (i.e., a relation-aware feature representation) (denoted as ). For example, for a set of cost data, its feature vector can be represented as... =[5000,1500,3000,1000,10500], these values ​​can be used directly as features or after normalization.

[0075] Understandably, matching a suitable feature extraction model to each modality of data ensures the depth and effectiveness of feature extraction. Furthermore, the feature extraction model can automatically extract and analyze raw data, reducing manual intervention and thus improving the consistency and accuracy of the review process.

[0076] According to embodiments of this disclosure, based on an initial feature representation, a semantic alignment weight is calculated between a spatially perceived feature representation and a semantically understood feature representation using a location-aware attention mechanism. This includes: determining a first target modality and a second target modality to be interacted with, wherein the first target modality and the second target modality are different modalities selected from an image modality, a text modality, and a structured modality; transforming the initial feature representation of the first target modality through a first linear projection layer to generate a query vector; transforming the initial feature representation of the second target modality through a second linear projection layer to generate a key vector; generating a learnable location encoding matrix associated with the first target modality and the second target modality; and calculating a first attention weight matrix that integrates spatial / semantic location information based on the query vector, the key vector, and the learnable location encoding matrix, wherein the first attention weight matrix includes semantic alignment weights.

[0077] Figure 5 A flowchart illustrating multimodal feature fusion according to an embodiment of the present disclosure is shown schematically.

[0078] like Figure 5 As shown, based on the extracted initial feature representations of each modality, namely spatial perception feature representation, semantic understanding feature representation and relation perception feature representation, the dynamic correlation weights (i.e. attention weights) between different modal features can be calculated through a cross-modal attention fusion model, and then weighted fusion can be performed to obtain the fused feature representation.

[0079] First, different modalities can be selected from image, text, and structured modalities to determine the pair of modalities that need to be interacted with. In this embodiment, the image modality is the first target modality (query), and the text modality is the second target modality (query recipient). Then, the spatially perceived feature representation of the image modality is transformed through a first linear projection layer to generate a query vector (Query, Q), and the semantic understanding feature representation of the text modality is transformed through a second linear projection layer to generate a key vector (Key, K). Simultaneously, a learnable positional encoding bias matrix P is introduced, which encodes the correspondence between the coordinates of the image feature grid and the positions of the text feature sequence. Subsequently, the first-stage attention weight matrix A1 is calculated based on the query vector Q, the key vector K, and the positional bias matrix P, and its calculation method can be expressed as follows:

[0080]

[0081] Where, d K It is the dimension of the key vector K, and the Softmax function is used to normalize the weights.

[0082] Each element of the attention weight matrix A1 quantifies the semantic correlation strength (i.e., semantic alignment weight) between features at a spatial location in the image (represented by Q) and features at a sequential location in the text (represented by K), for example, associating a specific region in a CT image with statements in a diagnostic report describing the lesion in that region. Finally, the semantic understanding feature representation of the text (as a value vector V1) is weighted and summed using the weight matrix A1 to generate an image-text collaborative feature representation.

[0083] Understandably, by modeling the interaction between modalities as attention computation at the feature element level, it is possible to overcome the limitations of overall modal alignment and achieve more accurate semantic association matching.

[0084] According to embodiments of this disclosure, the step of using the relation-aware feature representation as a query vector, the collaborative feature representation as a key vector and a value vector, calculating the relevance between the query vector and the key vector through multi-head cross-attention, and generating a fused feature representation based on the relevance and the value vector includes: transforming the relation-aware feature representation into a multi-head query vector group through a third linear projection layer; transforming the collaborative feature representation into a multi-head key vector group through a fourth linear projection layer, and into a multi-head value vector group through a fifth linear projection layer; for each attention head, calculating the target attention weight between the query vector subgroup and the key vector subgroup of each attention head in parallel, and using the target attention weight to perform a weighted summation on the corresponding value vector subgroup to generate the output of the attention head; concatenating the outputs of all attention heads and fusing them through a linear projection layer to generate the fused feature representation.

[0085] The query is represented by a structured modality relation-aware feature representation, and the key and value are represented by image-text collaborative feature representations. First, the query, key, and value vectors are transformed into corresponding multi-head vector groups through different linear projection layers. Then, for each attention head, the attention weights between the query subgroup and the key subgroup corresponding to that head are calculated in parallel, and these weights are used to perform a weighted summation of the value vector subgroup corresponding to that head, generating the output of that head. Finally, the outputs of all attention heads are concatenated and fused through a linear projection layer to generate the final fused feature representation.

[0086] The above process enables context-aware cross-modal information focusing guided by structured rules. Its core lies in using a multi-head attention mechanism to allow structured features to retrieve the most relevant parts of the image-text collaborative features in parallel from multiple different semantic subspaces (such as cost compliance, treatment plan matching, etc.). The resulting fused feature representation integrates visual evidence, textual description, and structured rules, providing a unified representation that has been verified and focused from multiple perspectives, offering robust and interpretable input for subsequent review decisions. Specifically, the two-stage hierarchical architecture—using an intermediate semantic alignment layer of "image-text collaboration" followed by "structured-multimodal" attention focusing—significantly enhances the discriminative power and interpretability of the final features.

[0087] In other embodiments, the technical effect of the above-described modal-based interaction fusion logic can be achieved through a more direct mathematical form. That is, the system can learn a dynamic fusion weight vector for each of the image, text, and structured data (the elements of which are the weights of the image features). Weights of text features Weights of structured features Each fusion weight vector can be derived through the cross-modal attention mechanism described above to reflect the importance of each modal feature dimension in the global multimodal context.

[0088] The initial feature representations of images, text, and structured data (which may have different dimensions) are transformed into a feature space of the same dimension using an appropriate mapping function (such as a linear layer), resulting in dimension-aligned feature vectors for each modality (f). i f t and f s Then, the respective feature vectors are element-wise weighted using the corresponding fusion weight vector (i.e., ⊙, which represents element-wise multiplication).

[0089] Weight vector The eigenvectors can be represented as:

[0090] , , ,

[0091] Final fusion feature representation It can be calculated directly using the following formula:

[0092]

[0093] Specifically, the calculation process is as follows:

[0094]

[0095] Understandably, using structured features as queries (such as review rules) and performing multi-head attention retrieval on a "knowledge base" composed of image and text collaborative features can automatically highlight the evidence fragments most relevant to the current rule from the image and text context. Furthermore, it can generate unified decision features that have undergone rule alignment and multi-perspective verification, rather than simply stacking features. In addition, the generated attention weights can be directly visualized, clearly showing which parts of the image, text, and structured data the model uses to make judgments, greatly improving the credibility and transparency of the review results.

[0096] According to embodiments of this disclosure, after obtaining raw data of at least two different modalities of the object to be audited, the data processing method further includes: generating a unique target identifier and a corresponding cryptographic hash value for each type of raw data; storing the target identifier and the cryptographic hash value as an associated data pair in a distributed ledger; wherein the audit result includes the target identifier of at least a portion of the raw data associated with the result.

[0097] After obtaining the raw data, a unique target identifier (such as a globally unique identifier or a sequence number generated by business rules) can be generated for each modality of raw data (e.g., raw images, raw text files, and structured data tables). Simultaneously, a cryptographic hash function can be used to calculate the complete content of each type of raw data, generating its corresponding cryptographic hash value. This hash value serves as a "digital fingerprint" of the data content, exhibiting strong collision resistance; any minor modification to the raw data will cause unpredictable changes to its hash value.

[0098] Each "target identifier" and its corresponding "cryptographic hash value" are treated as an indivisible linked data pair, submitted, and permanently stored in the distributed ledger. The distributed ledger is preferably a blockchain, whose consensus mechanism ensures that once a record is confirmed by the network, it cannot be unilaterally tampered with or deleted by a single node, thus providing the highest level of trust anchor for data integrity and timestamps.

[0099] When generating review results in subsequent processes (such as "claim approved" or "requires manual review"), these results can be linked to the key original data upon which the conclusion was based. Specifically, the output review results can explicitly include target identifiers for at least part of the original data used. For example, at the end of a claim rejection report, target identifiers for vehicle damage photos and claim report text, which serve as key evidence, can be attached.

[0100] Understandably, storing the hash value of the original data on an immutable distributed ledger provides objective evidence for subsequent audits to verify whether the data has been tampered with. Furthermore, by utilizing the target identifier carried in the audit results, the original evidence used to generate that conclusion can be precisely located.

[0101] According to embodiments of this disclosure, the fused feature representation is input into an audit decision model to output an audit result for the object to be audited, including: outputting an interpretability report based on the audit decision model; the interpretability report includes a quantified value of the contribution of the initial feature representation of each modality to the audit result in the fused feature representation.

[0102] Figure 6 A flowchart illustrating the processing of an audit decision model according to an embodiment of the present disclosure is shown.

[0103] like Figure 6 As shown, multimodal features are fused, meaning the fused feature representation is input into a pre-trained review decision model (such as a deep learning model). The input layer of the model architecture receives the fused feature representation rich in multimodal information, and then the hidden layers (composed of multiple neural networks, such as alternating convolutional and pooling layers) extract more abstract and higher-level discriminative feature representations. Based on the high-level features extracted by the hidden layers, the output layer outputs the final structured and information-rich review result.

[0104] The review results can include specific and actionable review conclusions, such as: the approved reasonable compensation amount (e.g., 8,000 yuan); identified unreasonable expenses and their specific amounts (e.g., drug expenses of 1,500 yuan, reason: exceeding the scope of medical insurance reimbursement); review status (e.g., automatic approval, recommended rejection, transferred to manual review) and its confidence level. Furthermore, by calculating the activation intensity of neurons related to each modality feature in the model, using gradient backpropagation methods or attribution analysis with attention weights, the review results can also include an interpretable report, which can include inverse analysis of the fused feature representation. Specifically, the report can quantify and display the contribution of the initial feature representations of images, text, and structured modalities to the final review result in this decision-making process. Contribution can be expressed as a percentage, weighted score, or heatmap. For example, the report could show: "In this review decision, the contribution of the medical invoice image is 45%, the contribution of the medical record text description is 35%, and the contribution of the structured expense list is 20%." For images, you can also highlight the areas that have the greatest impact on decision-making (such as a key number on an invoice).

[0105] After generating the review results and interpretability report, both can be simultaneously output to the claims processing system. The structured results can be directly accessed by the system, providing a basis for claims decisions and driving automated claims payments, denial notifications, or the creation of manual review workflows, significantly improving processing efficiency. Simultaneously, an interpretability report containing contribution analysis (e.g., explanations of specific reasons for unreasonable items) can be provided to the user (applicant or customer service personnel) along with the claims conclusion. This transparent review process improves the overall user experience and enhances user satisfaction with insurance services.

[0106] Understandably, by introducing an interpretable report generation mechanism, not only are accurate audit conclusions output, but the decision-making process is made more transparent and the attribution of evidence is visualized.

[0107] In some exemplary embodiments, the raw data of the image modality includes a medical invoice image, the raw data of the text modality includes medical record text, and the raw data of the structured modality includes a detailed list of medical expenses. Extracting spatially perceptual feature representations from the raw data of the image modality includes extracting visual layout features related to invoice type, amount area, and seal position in the medical invoice image. Extracting semantically perceptual feature representations from the raw data of the text modality includes extracting semantic features of medical entities related to disease diagnosis, treatment measures, and drug names in the medical record text. Extracting relationally perceptual feature representations from the raw data of the structured modality includes extracting price associations, quantity relationships, and medical insurance catalog matching features among the various chargeable items in the detailed list of medical expenses.

[0108] The method provided in this disclosure can be applied to medical expense reimbursement review scenarios. The raw data of the image modality includes images of receipts such as medical invoices and expense lists; the raw data of the text modality includes medical document texts such as medical records and diagnostic reports; and the raw data of the structured modality includes a medical expense detail table containing items, unit prices, quantities, amounts, and medical insurance codes.

[0109] Correspondingly, the spatially-aware feature representations extracted from the raw image modality data mainly capture layout information such as the overall format of the invoice image, key numerical regions, and the position of the seal. The semantically-understanding feature representations extracted from the raw text modality data mainly capture core entities in medical documents, such as disease names, treatment methods, and drug information, along with their contextual semantics. The relation-aware feature representations extracted from the raw structured modality data mainly capture structured association features such as the price-quantity relationships between items in the expense details table and the matching rules with the medical insurance catalog.

[0110] By extracting these modal features that are highly relevant to the review scenario, an accurate and interpretable information foundation is provided for subsequent multimodal fusion and intelligent decision-making.

[0111] Based on the above data processing method, this disclosure also provides a data processing apparatus. The following will be combined with... Figure 7 The device is described in detail.

[0112] Figure 7 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0113] like Figure 7 As shown, the data processing device 700 of this embodiment includes an acquisition module 710, an extraction module 720, a first fusion module 730, a second fusion module 740, and an audit module 750.

[0114] The acquisition module 710 is used to acquire raw data of the object to be reviewed in at least two different modalities, including image modality, text modality, and structured modality. In one embodiment, the acquisition module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0115] The extraction module 720 is used to extract spatially aware feature representations from the raw data of the image modality, semantically understanding feature representations from the raw data of the text modality, and relation-aware feature representations from the raw data of the structured modality. In one embodiment, the extraction module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0116] The first fusion module 730 is used to calculate the semantic alignment weight between the spatially perceived feature representation and the semantically understood feature representation through a position-aware attention mechanism, and to fuse them according to the semantic alignment weight to generate a collaborative feature representation. Furthermore, it performs weighted fusion of the initial feature representation according to the relevance weight to generate a fused feature representation. In one embodiment, the first fusion module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0117] The second fusion module 740 is used to calculate the relevance between the query vector and the key vector using the relation-aware feature representation as the query vector and the collaborative feature representation as the key vector and value vector, and to generate a fused feature representation based on the relevance and the value vector. In one embodiment, the second fusion module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0118] The review module 750 is used to input the fused feature representation into the review decision model to output the review result for the object to be reviewed. In one embodiment, the review module 750 can be used to perform the operation S250 described above, which will not be repeated here.

[0119] According to embodiments of this disclosure, the acquisition module 710 can be specifically used to: acquire the original image, the original text file, and the structured data table associated with the object to be reviewed; perform image preprocessing operations on the original image to obtain the original data of the image modality; perform text preprocessing operations on the original text file to obtain the original data of the text modality; and perform data cleaning and formatting operations on the structured data table to obtain the original data of the structured modality.

[0120] According to embodiments of this disclosure, the extraction module 720 can be specifically used to: extract spatially perceptual feature representations from the raw data of the image modality through a convolutional neural network model; extract semantically perceptual feature representations from the raw data of the text modality through a pre-trained language model; and extract relation-aware feature representations from the raw data of the structured modality through an embedding layer or encoder, wherein the embedding layer or encoder is used to map discrete structured fields into continuous target vectors.

[0121] According to embodiments of this disclosure, the first fusion module 730 can be specifically used to: determine a first target modality and a second target modality to be interacted with, wherein the first target modality and the second target modality are different modalities selected from image modality, text modality and structured modality; transform the initial feature representation of the first target modality through a first linear projection layer to generate a query vector; transform the initial feature representation of the second target modality through a second linear projection layer to generate a key vector; generate a learnable positional encoding matrix related to the first target modality and the second target modality; and calculate a first attention weight matrix that fuses spatial / semantic positional information based on the query vector, the key vector and the learnable positional encoding matrix, wherein the first attention weight matrix includes semantic alignment weights.

[0122] According to embodiments of this disclosure, the second fusion module 740 can be specifically used to: transform the relation-aware feature representation into a multi-head query vector group through a third linear projection layer; transform the collaborative feature representation into a multi-head key vector group through a fourth linear projection layer, and into a multi-head value vector group through a fifth linear projection layer; for each attention head, calculate in parallel the target attention weight between the query vector subgroup and the key vector subgroup of each attention head, and use the target attention weight to perform a weighted summation on the corresponding value vector subgroup to generate the output of the attention head; concatenate the outputs of all attention heads, and fuse them through a linear projection layer to generate the fused feature representation.

[0123] According to embodiments of this disclosure, after obtaining raw data of at least two different modalities of the object to be audited, the acquisition module 710 can also be used to: generate a unique target identifier and a corresponding cryptographic hash value for each type of raw data; store the target identifier and the cryptographic hash value as an associated data pair in a distributed ledger; wherein the audit result includes the target identifier of at least a portion of the raw data associated with the result.

[0124] According to embodiments of this disclosure, the audit module 750 can also be used to: output an interpretability report based on the audit decision model; the interpretability report includes a quantified value of the contribution of the initial feature representation of each modality to the audit result in the fused feature representation.

[0125] According to embodiments of this disclosure, the original data of the image modality includes a medical invoice image, the original data of the text modality includes medical record text, and the original data of the structured modality includes a detailed list of medical expenses. The extraction module 720 can be used to: extract visual layout features of the medical invoice image regarding invoice type, amount area, and seal position; extract medical entity semantic features of the medical record text regarding disease diagnosis, treatment measures, and drug names; and extract price associations, quantity relationships, and medical insurance catalog matching features among the various charge items in the detailed list of medical expenses.

[0126] According to embodiments of this disclosure, any multiple modules among the acquisition module 710, extraction module 720, first fusion module 730, second fusion module 740, and review module 750 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 710, extraction module 720, first fusion module 730, second fusion module 740, and review module 750 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 710, extraction module 720, first fusion module 730, second fusion module 740 and audit module 750 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0127] Figure 8A block diagram of an electronic device for a data processing method according to an embodiment of the present disclosure is shown schematically.

[0128] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0129] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0130] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0131] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0132] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include the memory (ROM) 802 and / or random access memory (RAM) 803 described above and / or one or more memories other than memory (ROM) 802 and random access memory (RAM) 803.

[0133] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data processing methods provided in the embodiments of this disclosure.

[0134] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0135] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0136] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0137] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0140] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data processing method, characterized in that, The method includes: Obtain raw data of the object to be reviewed in at least two different modalities, including image modality, text modality, and structured modality; Spatial-aware feature representations are extracted from the raw data of the image modality, semantic-understanding feature representations are extracted from the raw data of the text modality, and relation-aware feature representations are extracted from the raw data of the structured modality. The semantic alignment weight between spatially perceived feature representation and semantically understood feature representation is calculated using a position-aware attention mechanism, and then fused according to the semantic alignment weight to generate a collaborative feature representation. Using the relation-aware feature representation as the query vector and the collaborative feature representation as the key vector and value vector, the relevance between the query vector and the key vector is calculated through multi-head cross attention, and a fused feature representation is generated based on the relevance and the value vector. The fused feature representation is input into the audit decision model to output the audit result for the object to be audited.

2. The method according to claim 1, characterized in that, The acquisition of raw data in at least two different modalities of the object to be audited includes: Obtain the original image, original text file, and structured data table associated with the object to be reviewed; The original image is subjected to image preprocessing operations to obtain the original data of the image modality; The original text file is subjected to text preprocessing operations to obtain the original data of the text modality; The structured data table is cleaned and formatted to obtain the original data of the structured modality.

3. The method according to claim 1 or 2, characterized in that, The extraction of spatially aware feature representations from the raw data of the image modality, the extraction of semantically understanding feature representations from the raw data of the text modality, and the extraction of relation-aware feature representations from the raw data of the structured modality include: Spatial-aware feature representations are extracted from raw data of image modalities using a convolutional neural network model; Semantic understanding feature representations are extracted from raw data of text modalities using pre-trained language models; Relationship-aware feature representations are extracted from raw data of structured modalities through an embedding layer or encoder, which maps discrete structured fields to continuous target vectors.

4. The method according to claim 1 or 2, characterized in that, The semantic alignment weights between spatially perceived feature representations and semantically understood feature representations are calculated using a position-aware attention mechanism, including: A first target modality and a second target modality to be interacted with are determined, wherein the first target modality and the second target modality are different modalities selected from image modality, text modality and structured modality; The initial feature representation of the first target modality is transformed through the first linear projection layer to generate a query vector; The initial feature representation of the second target modality is transformed through the second linear projection layer to generate a key vector; Generate a learnable positional encoding matrix associated with the first target mode and the second target mode; Based on the query vector, the key vector, and the learnable position encoding matrix, a first attention weight matrix that integrates spatial / semantic position information is calculated. The first attention weight matrix includes semantic alignment weights.

5. The method according to claim 1, characterized in that, The process of using the relation-aware feature representation as the query vector, the collaborative feature representation as the key vector and value vector, calculating the relevance between the query vector and the key vector through multi-head cross-attention, and generating a fused feature representation based on the relevance and the value vector includes: The relation-aware feature representation is transformed into a multi-head query vector group through a third linear projection layer; The collaborative feature representation is transformed into a multi-head key vector group through the fourth linear projection layer, and then transformed into a multi-head value vector group through the fifth linear projection layer; For each attention head, the target attention weights between the query vector subgroup and the key vector subgroup of each attention head are calculated in parallel, and the corresponding value vector subgroups are weighted and summed using the target attention weights to generate the output of the attention head; The outputs of all attention heads are concatenated and fused through a linear projection layer to generate the fused feature representation.

6. The method according to claim 1, characterized in that, After obtaining raw data of at least two different modalities of the object to be audited, the method further includes: Generate a unique target identifier and a corresponding cryptographic hash value for each type of raw data; The target identifier and the cryptographic hash value are stored as an associated data pair in the distributed ledger; The audit result includes the target identifier of at least a portion of the original data associated with the generation of the result.

7. The method according to claim 1 or 6, characterized in that, The step of inputting the fused feature representation into the review decision model to output the review result for the object to be reviewed includes: An interpretable report is output based on the aforementioned audit decision model; The interpretability report includes a metric value for the contribution of the initial feature representation of each modality to the audit result in the fused feature representation.

8. The method according to claim 1, characterized in that, The raw data of the image modality includes medical invoice images, the raw data of the text modality includes medical record text, and the raw data of the structured modality includes a detailed list of medical expenses. The extraction of spatially aware feature representations from the raw data of the image modality includes: Extract visual layout features from the medical invoice image regarding invoice type, amount area, and seal position; The extraction of semantic understanding feature representations from the raw data of the text modality includes: Extract the semantic features of medical entities related to disease diagnosis, treatment measures, and drug names from the medical record text; The extraction of relation-aware feature representations from the raw data of the structured modality includes: Extract the price correlation, quantity relationship and medical insurance catalog matching features among the various charge items in the medical expense details table.

9. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire raw data of the object to be reviewed in at least two different modalities, including image modality, text modality and structured modality; The extraction module is used to extract spatially aware feature representations from the raw data of the image modality, extract semantic understanding feature representations from the raw data of the text modality, and extract relation-aware feature representations from the raw data of the structured modality. The first fusion module is used to calculate the semantic alignment weight between spatially perceived feature representation and semantically understood feature representation through a position-aware attention mechanism, and to fuse them according to the semantic alignment weight to generate a collaborative feature representation; The second fusion module is used to use the relation-aware feature representation as a query vector, the collaborative feature representation as a key vector and a value vector, calculate the relevance between the query vector and the key vector through multi-head cross attention, and generate a fused feature representation based on the relevance and the value vector. The review module is used to input the fused feature representation into the review decision model to output the review result for the object to be reviewed.

10. An electronic device, comprising: One or more processors; Storage device for storing one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images