Multi-modal data verification system and method in medical scientific research

Abstract

The invention belongs to the field of medical data processing. The invention provides a multi-modal data verification system and method in medical scientific research. The method comprises the following steps: extracting multi-modal data from different storage media; respectively analyzing and processing the extracted multi-modal data, respectively establishing a feature data model of each modal data and establishing a feature index; mapping the feature data models of the modal data to a unified vector space, performing data fusion according to the relationship of feature indexes of the feature data models, and generating and storing a wide model with multi-dimensional indexes; and according to a verification rule in a data processing process, slicing the wide model according to the feature index, and extracting a data feature information part corresponding to the verification rule to complete data verification. The method has the beneficial effects that all data of different modalities are subjected to homogenized fusion, so that verification for data of the same modality is not needed any more, and the verification result is more accurate.

Description

Technical Field

[0001] This application belongs to the field of medical data processing, and in particular relates to a multimodal data verification system and method for medical research. Background Technology

[0002] Medical research typically requires the collection, analysis, and processing of large amounts of data. However, this data usually originates from different medical systems and exists in various modalities, such as medical record text information, media files such as images from examinations and tests, and evaluation report files. Data validation is an important part of data quality control. Therefore, after collecting this data in the data center, it is necessary to validate the data separately.

[0003] Multimodal data is typically stored in a distributed manner on its own suitable media. For example, structured data is stored in a database, and files are stored in a file system. For the validation of multimodal data, the current industry practice is to perform independent validation for different modalities of data. However, the validation of medical research data often requires validation rules that need to be applied to multiple different modalities simultaneously. For example, it may be necessary to determine whether structured data meets the conditions based on a certain feature of an image file. Existing validation methods cannot perform unified validation of data from different modalities, which may lead to data errors and thus biased research results. Summary of the Invention

[0004] This application addresses the problem in existing technologies that cannot uniformly verify data from different modalities when processing multimodal medical research data, and provides a multimodal data verification system and method for medical research.

[0005] Firstly, this application provides a multimodal data verification system for medical research, including a data acquisition module, a data processing module, a data fusion module, and a data verification module.

[0006] The data acquisition module is used to extract multimodal data from different storage media;

[0007] The data processing module is used to analyze and process the extracted multimodal data respectively, establish feature data models for each modality and establish feature indexes. The feature data model refers to a data model established based on the correlation relationship of the set feature information.

[0008] The data fusion module is used to map the feature data models of each modality into a unified vector space, perform data fusion according to the relationship of the feature indexes of each feature data model, generate a wide model with multi-dimensional indexes and save it. The relationship of the feature indexes of each feature data model is obtained according to the set correlation relationship of feature information.

[0009] The data verification module is used to slice the wide model according to the feature index based on the verification rules of the data processing process, and extract the data feature information corresponding to the verification rules to complete the data verification.

[0010] In some embodiments, the multimodal data includes text data, image files, and structured data.

[0011] In some embodiments, the data processing module includes a text processing unit, a multimedia processing unit, and a structured data processing unit;

[0012] The text processing unit is used to process text data, extract the required information, convert the extracted required information into feature information according to the set correlation relationship of feature information, and establish a feature index to obtain a feature data model.

[0013] The multimedia processing unit is used to perform image and / or video processing on multimedia files to obtain feature description data therein, and to convert the obtained feature description data into a characterizable feature data model according to the set correlation relationship of feature information, and to establish a feature index.

[0014] The structured data processing unit is used to process structured data, extract corresponding attribute values ​​as feature information, establish feature indexes, and obtain feature data models.

[0015] In some embodiments, the text processing unit processes text data, including processing the text data using natural language analysis.

[0016] The required information includes at least keywords and topics.

[0017] In some embodiments, the multimedia processing unit performs image and / or video processing on the multimedia file, including: acquiring feature description data of the multimedia file using image recognition.

[0018] Secondly, this application provides a method for multimodal data validation in medical research, comprising the following steps:

[0019] Extracting multimodal data from different storage media;

[0020] The extracted multimodal data are analyzed and processed separately, and feature data models and feature indexes are established for each modality. The feature data model refers to a data model established based on the correlation relationship of the set feature information.

[0021] The feature data models of each modality are mapped to a unified vector space. Data fusion is performed according to the relationship of the feature indexes of each feature data model to generate a wide model with multi-dimensional indexes and save it. The relationship of the feature indexes of each feature data model is obtained according to the association relationship of the set feature information.

[0022] Based on the verification rules of the data processing procedure, the wide model is sliced ​​according to the feature index, and the data feature information corresponding to the verification rules is extracted to complete the data verification.

[0023] In some embodiments, the multimodal data includes text data, image files, and structured data.

[0024] In some embodiments, the step of analyzing and processing the extracted multimodal data, establishing feature data models for each modality and establishing feature indexes includes:

[0025] The text data is processed and the required information is extracted. Based on the established correlation between the feature information, the extracted information is converted into feature information and a feature index is established to obtain the feature data model.

[0026] Perform image and / or video processing on multimedia files to obtain feature description data, and convert the obtained feature description data into a representative feature data model based on the established correlation relationship of feature information, and establish a feature index;

[0027] The structured data is processed to extract the corresponding attribute values ​​as feature information, and a feature index is established to obtain the feature data model.

[0028] In some embodiments, the processing of text data includes: processing the text data using natural language analysis.

[0029] The required information includes at least keywords and topics.

[0030] In some embodiments, the image and / or video processing of the multimedia file includes: acquiring feature description data of the multimedia file using image recognition.

[0031] Thirdly, this application also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the multimodal data verification method in medical research as described above.

[0032] Fourthly, this application also provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal data verification method in medical research as described above.

[0033] The beneficial effects of this application are as follows: This application indirectly establishes a correlation between various modal data through the feature data model and the feature information contained therein, so that all data of different modalities are homogeneously fused. Therefore, it is no longer a verification of data of the same modality, thus the verification results are more accurate. Attached Figure Description

[0034] Figure 1 A schematic system block diagram of a multimodal data verification system for medical research in this application embodiment.

[0035] Figure 2 A schematic system block diagram of a multimodal data verification system in medical research in another embodiment of this application.

[0036] Figure 3 A schematic flowchart of a multimodal data verification method in medical research in this application embodiment.

[0037] Figure 4 A schematic flowchart of a multimodal data verification method in medical research in another embodiment of this application. Detailed Implementation

[0038] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0039] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0040] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0041] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0042] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0043] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0044] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0045] See Figure 1 In a first aspect, embodiments of this application provide a multimodal data verification system for medical research, including a data acquisition module, a data processing module, a data fusion module, and a data verification module.

[0046] The data acquisition module is used to extract multimodal data from different storage media.

[0047] The data processing module is used to analyze and process the extracted multimodal data, establish feature data models for each modality, and create feature indexes. Here, the feature data model refers to the data model established based on the defined correlation relationships of feature information.

[0048] The data fusion module is used to map the feature data models of each modality into a unified vector space, perform data fusion according to the relationship of the feature indexes of each feature data model, generate a wide model with multi-dimensional indexes and save it. Here, the relationship of the feature indexes of each feature data model is obtained according to the set correlation relationship of feature information.

[0049] The data verification module is used to slice the wide model according to the feature index based on the verification rules of the data processing process, and extract the data feature information corresponding to the verification rules to complete the data verification.

[0050] It is understood that the above embodiments establish feature data models for each modality data by setting the correlation relationship of feature information, and then map the feature data models of each modality data into a unified vector space. Data fusion is completed by the relationship of feature indexes of each feature data model obtained according to the correlation relationship of the set feature information, which indirectly makes the modality data achieve homogeneous fusion. Therefore, the verification is no longer for the same modality data, so the verification results are more accurate.

[0051] In some embodiments, multimodal data may include text data, image files, and structured data, etc.

[0052] It is understandable that multimodal data in medical research includes text files such as assessment reports and medical records, multimedia files such as test images, and structured data. Since the processing methods for each modality of data are different, this embodiment focuses on the most representative text data, image files, and structured data.

[0053] See Figure 2 In some embodiments, the data processing module may include a text processing unit, a multimedia processing unit, and a structured data processing unit.

[0054] The text processing unit is used to process text data, extract the required information, convert the extracted information into feature information according to the set correlation of feature information, and establish a feature index to obtain a feature data model.

[0055] The multimedia processing unit is used to perform image and / or video processing on multimedia files to obtain feature description data, and to convert the obtained feature description data into a characterizable feature data model according to the set correlation relationship of feature information, and to establish a feature index.

[0056] The structured data processing unit is used to process structured data, extract corresponding attribute values ​​as feature information, establish feature indexes, and obtain feature data models.

[0057] It is understood that the above embodiments propose a scheme for implementing the data processing module, which describes how to process text data, image files and structured data to obtain feature data models and feature indexes.

[0058] In addition, in the text processing unit, natural language processing (NLP) can be used to process text data, which is existing technology and will not be detailed here. Its purpose is to analyze the meaning of the text, extracting only the data relevant to data validation and feature data association, and eliminating other redundant items. The required information includes at least keywords and themes.

[0059] In the multimedia processing unit, when performing image and / or video processing on multimedia files, existing technologies such as image recognition can be used to acquire feature description data of the multimedia files. The purpose is to extract the information that needs to be verified from the multimedia files and transform it into the same structure and indexing method as other data for subsequent fusion.

[0060] See Figure 3 Secondly, this application provides a method for multimodal data verification in medical research, comprising the following steps:

[0061] Extracting multimodal data from different storage media;

[0062] The extracted multimodal data are analyzed and processed separately, and feature data models and feature indexes are established for each modality. The feature data model refers to a data model established based on the correlation relationship of the set feature information.

[0063] The feature data models of each modality are mapped to a unified vector space. Data fusion is performed according to the relationship of the feature indexes of each feature data model to generate a wide model with multi-dimensional indexes and save it. The relationship of the feature indexes of each feature data model is obtained according to the association relationship of the set feature information.

[0064] Based on the verification rules of the data processing procedure, the wide model is sliced ​​according to the feature index, and the data feature information corresponding to the verification rules is extracted to complete the data verification.

[0065] It is understood that the above embodiments establish feature data models for each modality data by setting the correlation relationship of feature information, and then map the feature data models of each modality data into a unified vector space. Data fusion is completed by the relationship of feature indexes of each feature data model obtained according to the correlation relationship of the set feature information, which indirectly makes the modality data achieve homogeneous fusion. Therefore, the verification is no longer for the same modality data, so the verification results are more accurate.

[0066] In some embodiments, multimodal data may include text data, image files, and structured data, etc.

[0067] It is understandable that multimodal data in medical research includes text files such as assessment reports and medical records, multimedia files such as test images, and structured data. Since the processing methods for each modality of data are different, this embodiment focuses on the most representative text data, image files, and structured data.

[0068] See Figure 4 In some embodiments, the process of analyzing and processing the extracted multimodal data separately, establishing feature data models for each modality and creating feature indexes, may include:

[0069] The text data is processed and the required information is extracted. Based on the established correlation between the feature information, the extracted information is converted into feature information and a feature index is established to obtain the feature data model.

[0070] Perform image and / or video processing on multimedia files to obtain feature description data, and convert the obtained feature description data into a representative feature data model based on the established correlation relationship of feature information, and establish a feature index;

[0071] The structured data is processed to extract the corresponding attribute values ​​as feature information, and a feature index is established to obtain the feature data model.

[0072] It is understood that the above embodiments propose a scheme to analyze and process the extracted multimodal data separately, establish feature data models for each modality of data and establish feature indexes, and respectively describe how to process text data, image files and structured data to obtain feature data models and feature indexes.

[0073] Alternatively, natural language processing (NLP) can be used to process text data; this is an existing technology and will not be detailed here. Its purpose is to analyze the meaning of the text, extracting only data relevant to data validation and feature association, while eliminating redundant information. The required information includes at least keywords and themes.

[0074] When performing image and / or video processing on multimedia files, existing technologies such as image recognition can be used to obtain feature description data. The purpose is to extract the information that needs to be verified from the multimedia file and transform it into the same structure and indexing method as other data for subsequent fusion.

[0075] Thirdly, embodiments of this application also provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the multimodal case data processing method as described above.

[0076] Fourthly, embodiments of this application also provide an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the multimodal case data processing method described above. The above embodiments only illustrate one or more implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0077] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0078] It should be noted that the information interaction and execution process between the above-mentioned devices / units / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0080] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0081] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0086] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multimodal data validation system for medical research, characterized in that, It includes a data acquisition module, a data processing module, a data fusion module, and a data verification module. The data acquisition module is used to extract multimodal data from different storage media; The data processing module is used to analyze and process the extracted multimodal data respectively, establish feature data models for each modality and establish feature indexes. The feature data model refers to a data model established based on the correlation relationship of the set feature information. The data fusion module is used to map the feature data models of each modality into a unified vector space, perform data fusion according to the relationship of the feature indexes of each feature data model, generate a wide model with multi-dimensional indexes and save it. The relationship of the feature indexes of each feature data model is obtained according to the set correlation relationship of feature information. The data verification module is used to slice the wide model according to the feature index based on the verification rules of the data processing process, and extract the data feature information corresponding to the verification rules to complete the data verification.

2. The multimodal data verification system for medical research according to claim 1, characterized in that, The multimodal data includes text data, image files, and structured data.

3. The multimodal data verification system for medical research according to claim 2, characterized in that, The data processing module includes a text processing unit, a multimedia processing unit, and a structured data processing unit. The text processing unit is used to process text data, extract the required information, convert the extracted required information into feature information according to the set correlation relationship of feature information, and establish a feature index to obtain a feature data model. The multimedia processing unit is used to perform image and / or video processing on multimedia files to obtain feature description data therein, and to convert the obtained feature description data into a characterizable feature data model according to the set correlation relationship of feature information, and to establish a feature index. The structured data processing unit is used to process structured data, extract corresponding attribute values ​​as feature information, establish feature indexes, and obtain feature data models.

4. The multimodal data verification system for medical research according to claim 3, characterized in that, The text processing unit processes text data, including processing text data using natural language analysis. The required information includes at least keywords and topics.

5. The multimodal data verification system for medical research according to any one of claims 3-4, characterized in that, The multimedia processing unit performs image and / or video processing on multimedia files, including: acquiring feature description data of multimedia files using image recognition.

6. A multimodal data validation method for medical research, characterized in that, Includes the following steps: Extracting multimodal data from different storage media; The extracted multimodal data are analyzed and processed separately, and feature data models and feature indexes are established for each modality. The feature data model refers to a data model established based on the correlation relationship of the set feature information. The feature data models of each modality are mapped to a unified vector space. Data fusion is performed according to the relationship of the feature indexes of each feature data model to generate a wide model with multi-dimensional indexes and save it. The relationship of the feature indexes of each feature data model is obtained according to the association relationship of the set feature information. Based on the verification rules of the data processing procedure, the wide model is sliced ​​according to the feature index, and the data feature information corresponding to the verification rules is extracted to complete the data verification.

7. The multimodal data validation method in medical research according to claim 6, characterized in that, The multimodal data includes text data, image files, and structured data.

8. The multimodal data validation method in medical research according to claim 7, characterized in that, The process of analyzing and processing the extracted multimodal data, establishing feature data models for each modality, and creating feature indexes includes: The text data is processed and the required information is extracted. Based on the established correlation between the feature information, the extracted information is converted into feature information and a feature index is established to obtain the feature data model. Perform image and / or video processing on multimedia files to obtain feature description data, and convert the obtained feature description data into a representative feature data model based on the established correlation relationship of feature information, and establish a feature index; The structured data is processed to extract the corresponding attribute values ​​as feature information, and a feature index is established to obtain the feature data model.

9. The multimodal data verification method in medical research according to claim 8, characterized in that, The text data processing includes: processing the text data using natural language analysis methods; The required information includes at least keywords and topics.

10. The method for multimodal data validation in medical research according to any one of claims 8-9, characterized in that, The image and / or video processing of the multimedia file includes: acquiring feature description data of the multimedia file using image recognition.

Images