Multi-modal case data processing system and method

Abstract

The invention belongs to the field of medical data processing. The invention provides a multi-modal case data processing system and method.The multi-modal case data processing system comprises a data obtaining module, a data processing module and a data merging module, and the data obtaining module is used for obtaining multi-modal data in real time; the data processing module is used for analyzing and processing the acquired multi-modal data in real time, marking time attributes for the multi-modal data after analysis and processing are completed, and independently caching the data marked with the time attributes; and the data merging module is used for merging the data with the same time attribute. The time attributes are added into the data, and the data are merged according to the time attributes, so that the time nodes of all the data are ensured to be consistent, the synchronization of the multi-modal medical record data is achieved, and the accuracy of scientific research analysis results can be ensured.

Description

Technical Field

[0001] This application belongs to the field of medical data processing, and in particular relates to a multimodal case data processing system and method. Background Technology

[0002] Medical research projects typically require the collection and analysis of large amounts of patient medical record data. This data usually originates from different clinical systems across multiple hospitals, and these systems store medical record data in different ways. Some systems use structured storage, allowing for direct data extraction, while others use unstructured data, requiring analysis before extraction. Some hospitals even use paper-based medical records. Furthermore, medical records often contain a significant amount of image data from laboratory tests and examinations. Data centers face the challenge of handling these diverse data formats and varying collection and analysis methods when extracting medical record data.

[0003] The current industry practice is to process image files separately, using technologies such as OCR, to convert them into structured data storage, and then integrate them with other data. However, for medical research projects, it is usually necessary to collect and analyze data in real time from many hospital systems. This requires a real-time streaming process and the timeliness of the data source. Current industry preprocessing methods cannot ensure that these pre-processed image data and ordinary real-time data are at the same time point, thus affecting the final research analysis results. Summary of the Invention

[0004] This application addresses the problem in the prior art that image data and ordinary real-time data are not synchronized when processing multimodal medical record data, and provides a multimodal medical record data processing system and method.

[0005] Firstly, this application provides a multimodal case data processing system, including a data acquisition module, a data processing module, and a data merging module.

[0006] The data acquisition module is used to acquire multimodal data in real time;

[0007] The data processing module is used to analyze and process the acquired multimodal data in real time, and mark the time attributes of the data after the analysis and processing are completed, and then cache the data marked with time attributes separately.

[0008] The data merging module is used to merge data with the same time attribute.

[0009] In some embodiments, the multimodal data includes image files and general structured data.

[0010] In some embodiments, the data processing module includes an image processing unit and a general structured data processing unit;

[0011] The image processing unit is used to perform image processing on the acquired medical record image files, obtain the image feature data therein, and cache it separately after marking the time attribute.

[0012] The ordinary structured data processing unit is used to perform real-time streaming processing on the acquired ordinary structured data, and to mark the time attribute of any ordinary structured data after processing is completed, and to cache it separately.

[0013] In some embodiments, the time attribute is the acquisition time of the corresponding data.

[0014] In some embodiments, the data merging module includes a synchronization merging unit and a data iteration unit.

[0015] The synchronous merging unit is used to merge data with the same time attribute;

[0016] The data iteration unit is used to compare the merged data with the currently processed ordinary structured data, and upgrade the merged data to a version that conforms to the currently processed ordinary structured data.

[0017] In some embodiments, the data merging module also pre-sets the logical relationship between image feature data and ordinary structured data;

[0018] In the synchronous merging unit, when merging data with the same time attribute, it is done according to the association logic relationship;

[0019] In the data iteration unit, when comparing the merged data with the currently processed ordinary structured data, it is done according to the association logic relationship, and the merged data is upgraded to a version that conforms to the currently processed ordinary structured data according to the association logic relationship.

[0020] Secondly, this application provides a method for processing multimodal case data, including the following steps:

[0021] Real-time acquisition of multimodal data;

[0022] The acquired multimodal data is analyzed and processed in real time, and time attributes are marked on the data after the analysis and processing are completed. The data marked with time attributes are then cached separately.

[0023] Merge data with the same time attribute.

[0024] In some embodiments, the multimodal data includes image files and general structured data.

[0025] In some embodiments, the real-time analysis and processing of the acquired multimodal data, the marking of time attributes after the analysis and processing, and the separate caching of each time-attributed data include:

[0026] The acquired medical record image files are processed to obtain image feature data, which are then labeled with time attributes and cached separately.

[0027] The system performs real-time streaming processing on the acquired ordinary structured data, and marks the time attribute of each ordinary structured data after processing is completed, and caches it separately.

[0028] In some embodiments, the time attribute is the acquisition time of the corresponding data.

[0029] In some embodiments, the following steps are also included:

[0030] The merged data is compared with the currently processed ordinary structured data, and the merged data is upgraded to a version that conforms to the currently processed ordinary structured data.

[0031] In some embodiments, the logical relationship between image feature data and ordinary structured data is also preset in advance;

[0032] When merging data with the same time attribute, it is done according to the aforementioned logical relationship;

[0033] When comparing the merged data with the currently processed ordinary structured data, it is done according to the aforementioned association logic relationship, and the merged data is also upgraded to a version that conforms to the currently processed ordinary structured data based on the aforementioned association logic relationship.

[0034] Thirdly, this application also provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the multimodal case data processing method described above.

[0035] Fourthly, this application also provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal case data processing method described above.

[0036] The beneficial effects of this application are as follows: by adding time attributes to each data and merging the data according to the time attributes, this application ensures that the time nodes of all data are consistent, achieves the synchronization of multimodal medical record data, and can guarantee the accuracy of scientific research analysis results. Attached Figure Description

[0037] Figure 1A schematic system block diagram of the multimodal case data processing system in this application embodiment.

[0038] Figure 2 A schematic system block diagram of a multimodal case data processing system in another embodiment of this application.

[0039] Figure 3 A schematic flowchart of the multimodal case data processing method in the embodiments of this application.

[0040] Figure 4 A schematic flowchart of a multimodal case data processing method in another embodiment of this application. Detailed Implementation

[0041] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0042] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0043] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0044] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0045] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0046] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0048] See Figure 1 In a first aspect, embodiments of this application provide a multimodal case data processing system, including a data acquisition module, a data processing module, and a data merging module.

[0049] The data acquisition module is used to acquire multimodal data in real time.

[0050] The data processing module is used to analyze and process the acquired multimodal data in real time, and after the analysis and processing are completed, it marks the time attributes of the data and then caches the data marked with time attributes separately.

[0051] The data merging module is used to merge data with the same time attribute.

[0052] It is understandable that the above embodiments, by marking the time attribute of the data after the data analysis and processing is completed and caching it separately, enable the slower-processed data to be traced and merged with the data with the same time attribute after the analysis and processing is completed, ensuring that the time nodes of all data are consistent, achieving the synchronization of multimodal medical record data, and ensuring the accuracy of scientific research analysis results.

[0053] In some embodiments, multimodal data may include image files and general structured data.

[0054] It is understandable that multimodal medical record data refers to medical record data in multiple modes, including image files obtained from scanning paper medical records and ordinary structured data, as well as audio data. However, the processing speed of image files is much slower than that of ordinary structured data, and it is also the easiest to cause data asynchrony. Therefore, this embodiment separately proposes image files and ordinary structured data.

[0055] See Figure 2 In some embodiments, the data processing module may include an image processing unit and a general structured data processing unit.

[0056] The image processing unit is used to process the acquired medical record image files, obtain the image feature data, and cache them separately after marking them with time attributes.

[0057] The ordinary structured data processing unit is used to perform real-time streaming processing on the acquired ordinary structured data, and to mark the time attribute of any ordinary structured data after processing is completed, and to cache it separately.

[0058] It is understood that in the above embodiments, image files and ordinary structured data are processed separately and marked with time attributes respectively, so that the streaming processing of ordinary structured data will not be slowed down due to the slow processing speed of image files.

[0059] In addition, the image processing can be any existing image processing method, such as image text recognition, as long as it can obtain image feature data.

[0060] In some embodiments, the time attribute preferably includes the acquisition time of the corresponding data.

[0061] It is understandable that the time attribute should include the time when the corresponding data was acquired; if it is the time when the corresponding data was processed, then the purpose of this application cannot be achieved.

[0062] See Figure 2 In some embodiments, the data merging module may include a synchronous merging unit and a data iteration unit;

[0063] Among them, the synchronous merging unit is used to merge data with the same time attribute;

[0064] The data iteration unit is used to compare the merged data with the currently processed ordinary structured data and upgrade the merged data to a version that conforms to the currently processed ordinary structured data.

[0065] It is understandable that, since the processing speed of some data such as image files is relatively slow, new data may exist in the ordinary structured data of the streaming process after the processing is completed. Therefore, in this embodiment, a data iteration unit is added to upgrade the merged data to a version that conforms to the currently processed ordinary structured data.

[0066] In addition, since the processing speed of certain data such as image files is relatively slow, it is possible to choose to cache faster data (such as ordinary structured data) that has been processed and corresponds to the time attribute of the image file only when some slower data such as image files is being processed, thereby saving cache resources.

[0067] In some embodiments, the data merging module may also pre-set the logical relationship between image feature data and ordinary structured data;

[0068] In the synchronous merging unit, when merging data with the same time attribute, it is done according to the aforementioned association logic relationship;

[0069] In the data iteration unit, when comparing the merged data with the currently processed ordinary structured data, it is done according to the aforementioned association logic relationship, and the merged data is also upgraded to a version that conforms to the currently processed ordinary structured data according to the aforementioned association logic relationship.

[0070] It is understandable that some of the image files in paper medical records contain handwritten image feature data, which are not categorized and / or not written according to standard (such as abbreviated disease names). During image recognition, there may be a problem in distinguishing the relationship between image feature data and ordinary structured data. Therefore, adding the logical relationship between image feature data and ordinary structured data can effectively improve the efficiency of data merging and iteration.

[0071] See Figure 3 Secondly, this application provides a streaming method for multimodal case data, comprising the following steps:

[0072] Real-time acquisition of multimodal data;

[0073] The acquired multimodal data is analyzed and processed in real time, and time attributes are marked on the data after the analysis and processing are completed. The data marked with time attributes are then cached separately.

[0074] Merge data with the same time attribute.

[0075] It is understandable that the above embodiments, by marking the time attribute of the data after the data analysis and processing is completed and caching it separately, enable the slower-processed data to be traced and merged with the data with the same time attribute after the analysis and processing is completed, ensuring that the time nodes of all data are consistent, achieving the synchronization of multimodal medical record data, and ensuring the accuracy of scientific research analysis results.

[0076] In some embodiments, multimodal data may include image files and general structured data.

[0077] It is understandable that multimodal medical record data refers to medical record data in multiple modes, including image files obtained from scanning paper medical records and ordinary structured data, as well as audio data. However, the processing speed of image files is much slower than that of ordinary structured data, and it is also the easiest to cause data asynchrony. Therefore, this embodiment separately proposes image files and ordinary structured data.

[0078] See Figure 4 In some embodiments, the acquired multimodal data is analyzed and processed in real time, and time attributes are marked after the analysis and processing are completed. The data with marked time attributes are then cached separately. This may include:

[0079] The acquired medical record image files are subjected to image recognition to obtain image feature data, which are then labeled with time attributes and cached separately.

[0080] The system performs real-time streaming processing on the acquired ordinary structured data, and marks the time attribute of each ordinary structured data after processing is completed, and caches it separately.

[0081] It is understood that in the above embodiments, image files and ordinary structured data are processed separately and marked with time attributes respectively, so that the streaming processing of ordinary structured data will not be slowed down due to the slow processing speed of image files.

[0082] In addition, the image processing can be any existing image processing method, such as image text recognition, as long as it can obtain image feature data.

[0083] In some embodiments, the time attribute here preferably includes the acquisition time of the corresponding data.

[0084] It is understandable that the time attribute should include the time when the corresponding data was acquired; if it is the time when the corresponding data was processed, then the purpose of this application cannot be achieved.

[0085] See Figure 4 In some embodiments, the following steps may also be included:

[0086] The merged data is compared with the currently processed ordinary structured data, and the merged data is upgraded to a version that conforms to the currently processed ordinary structured data.

[0087] It is understandable that, since the processing speed of some data such as image files is relatively slow, new data may exist in the ordinary structured data of the streaming process after the processing is completed. Therefore, in this embodiment, a data iteration unit is added to upgrade the merged data to a version that conforms to the currently processed ordinary structured data.

[0088] In addition, since the processing speed of certain data such as image files is relatively slow, it is possible to choose to cache faster data (such as ordinary structured data) that has been processed and corresponds to the time attribute of the image file only when some slower data such as image files is being processed, thereby saving cache resources.

[0089] In some embodiments, the logical relationship between image feature data and ordinary structured data is also preset in advance;

[0090] When comparing the merged data with the currently processed ordinary structured data, it is done according to the aforementioned association logic relationship, and the merged data is also upgraded to a version that conforms to the currently processed ordinary structured data based on the aforementioned association logic relationship.

[0091] It is understandable that some of the image files in paper medical records contain handwritten image feature data, which are not categorized and / or not written according to standard (such as abbreviated disease names). During image recognition, there may be a problem in distinguishing the relationship between image feature data and ordinary structured data. Therefore, adding the logical relationship between image feature data and ordinary structured data can effectively improve the efficiency of data merging and iteration.

[0092] Thirdly, embodiments of this application also provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the multimodal case data processing method as described above.

[0093] Fourthly, embodiments of this application also provide an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the multimodal case data processing method described above. The above embodiments only illustrate one or more implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0094] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0095] It should be noted that the information interaction and execution process between the above-mentioned devices / units / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0098] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0099] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0100] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0103] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multimodal case data processing system, characterized in that, It includes a data acquisition module, a data processing module, and a data merging module. The data acquisition module is used to acquire multimodal data in real time; The data processing module is used to analyze and process the acquired multimodal data in real time, and mark the time attributes of the data after the analysis and processing are completed, and then cache the data marked with time attributes separately. The data merging module is used to merge data with the same time attribute.

2. The multimodal case data processing system according to claim 1, characterized in that, The multimodal data includes image files and general structured data; The data processing module includes an image processing unit and a general structured data processing unit; The image processing unit is used to perform image processing on the acquired medical record image files, obtain the image feature data therein, and cache it separately after marking the time attribute. The ordinary structured data processing unit is used to perform real-time streaming processing on the acquired ordinary structured data, and to mark the time attribute of any ordinary structured data after processing is completed, and to cache it separately.

3. The multimodal case data processing system according to claim 1, characterized in that, The time attribute refers to the time when the corresponding data was acquired.

4. The multimodal case data processing system according to any one of claims 1-3, characterized in that, The data merging module includes a synchronization merging unit and a data iteration unit. The synchronous merging unit is used to merge data with the same time attribute; The data iteration unit is used to compare the merged data with the currently processed ordinary structured data, and upgrade the merged data to a version that conforms to the currently processed ordinary structured data.

5. The multimodal case data processing system according to claim 4, wherein the data merging module further includes a pre-set logical relationship between image feature data and ordinary structured data; In the synchronous merging unit, when merging data with the same time attribute, it is done according to the association logic relationship; In the data iteration unit, when comparing the merged data with the currently processed ordinary structured data, it is done according to the association logic relationship, and the merged data is upgraded to a version that conforms to the currently processed ordinary structured data according to the association logic relationship.

6. A method for processing multimodal case data, characterized in that, Includes the following steps: Real-time acquisition of multimodal data; The acquired multimodal data is analyzed and processed in real time, and time attributes are marked on the data after the analysis and processing are completed. The data marked with time attributes are then cached separately. Merge data with the same time attribute.

7. The multimodal case data processing method according to claim 6, characterized in that, The multimodal data includes image files and general structured data; The process involves real-time analysis and processing of the acquired multimodal data, marking each data point with a time attribute after analysis and processing, and then separately caching each time-attributed data point. This includes: The acquired medical record image files are processed to obtain image feature data, which are then labeled with time attributes and cached separately. The system performs real-time streaming processing on the acquired ordinary structured data, and marks the time attribute of each ordinary structured data after processing is completed, and caches it separately.

8. The multimodal case data processing method according to claim 6, characterized in that, The time attribute refers to the time when the corresponding data was acquired.

9. The multimodal case data processing method according to any one of claims 6-8, characterized in that, It also includes the following steps: The merged data is compared with the currently processed ordinary structured data, and the merged data is upgraded to a version that conforms to the currently processed ordinary structured data.

10. The multimodal case data processing method according to claim 9, characterized in that, It also pre-sets the logical relationship between image feature data and ordinary structured data; When merging data with the same time attribute, it is done according to the aforementioned logical relationship; when comparing the merged data with the currently processed ordinary structured data, it is also done according to the aforementioned logical relationship, and the merged data is upgraded to a version that conforms to the currently processed ordinary structured data according to the aforementioned logical relationship.

Images