Multi-modal data fusion acquisition method and system based on VR emergency scene

Abstract

The invention discloses a multi-modal data fusion acquisition method and system based on a VR emergency scene, and mainly relates to the technical field of data processing. Comprising the following steps: arranging a multi-modal scene sensing equipment set, and carrying out multi-modal data acquisition on a target user in a preset window by utilizing the multi-modal scene sensing equipment set to obtain a multi-modal data sequence set; performing semantic memory breakpoint recognition on the instruction text sequence to obtain a semantic memory breakpoint set and an instruction operation identifier set; splitting the multi-modal data sequence set to obtain a plurality of multi-modal data sub-sequence sets; and fusing the plurality of multi-modal data sub-sequence sets to obtain a multi-modal data fusion acquisition result set. The method and the device have the beneficial effects that the technical problem of low fusion reliability of the acquired data caused by lack of deep alignment of the multi-modal data in the VR emergency scene in the prior art is solved, and the technical effect of improving the fusion and acquisition quality of the multi-modal data is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-modal data fusion collection method and system based on a VR emergency scene. BACKGROUND

[0002] Virtual Reality Technology (VR for short) is a computer simulation system that can create and experience a virtual world. It uses a computer to generate a simulated environment and provides visual, auditory, tactile and other sensory stimuli to the user. The user interacts with this environment (and the virtual objects and characters in it) in real time through a head-mounted display, gestures (data gloves), body movements (data clothes) and natural language, etc., and experiences a sense of immersion[1]. Although virtual reality technology can simulate a realistic three-dimensional world, the virtual environment created by it is basically separated from the real world. The mixed reality (MR) technology, such as augmented reality (AR) technology, can better combine the virtual world and the real world.

[0003] With the development of VR technology and immersive interaction technology, VR scenes are increasingly applied to the field of medical training, especially in first aid skill training. Traditional first aid training methods rely on offline operation and video teaching, which cannot objectively and real-time quantify the operation details, instruction response and action execution process of the trainees. In addition, due to the existence of different multi-modal data in the scene, the instructions and actions often cannot be reliably aligned, which leads to the inability to effectively compare with the standard first aid template, affecting the feedback quality. Therefore, there is a technical problem in the prior art that the multi-modal data under the VR emergency scene is not deeply aligned, resulting in low reliability of data fusion collection. SUMMARY

[0004] The present application provides a multi-modal data fusion collection method and system based on a VR emergency scene, which is used to solve the technical problem in the prior art that the multi-modal data under the VR emergency scene is not deeply aligned, resulting in low reliability of data fusion collection.

[0005] In view of the above problems, the present application provides a multi-modal data fusion collection method and system based on a VR emergency scene.

[0006] In a first aspect of the present application, a multi-modal data fusion collection method based on a VR emergency scene is provided, the method comprising: A multimodal scene perception device set is deployed, and the multimodal scene perception device set is used to collect multimodal data of the target user within a preset window to obtain a multimodal data sequence set; instruction text sequences are extracted from the multimodal data sequence set, and semantic memory breakpoints are identified in the instruction text sequences to obtain a semantic memory breakpoint set and an instruction operation identifier set, wherein the semantic memory breakpoints and instruction operation identifiers correspond one-to-one; the multimodal data sequence set is split according to the semantic memory breakpoint set to obtain multiple multimodal data subsequence sets; and the multiple multimodal data subsequence sets are fused together with the instruction operation identifier set to obtain a multimodal data fusion acquisition result set.

[0007] Preferably, a multimodal scene perception device set is deployed, and the multimodal scene perception device set is used to collect multimodal data of the target user within a preset window to obtain a multimodal data sequence set. This includes: deploying optical acquisition devices within the target motion capture area, wherein the optical acquisition devices include an infrared camera and a VR interactive controller; fixing an IMU inertial measurement unit to the target user's forearm and first aid chest mold, and wearing a head-mounted microphone on the target user's head, wherein the head-mounted microphone includes a real-time speech recognition engine for transcribing speech into timestamped instruction text; and performing unified clock and spatial coordinate processing on the infrared camera, VR interactive controller, IMU inertial measurement unit, and head-mounted microphone to obtain the deployed multimodal scene perception device set.

[0008] Preferably, the method further includes: obtaining a data acquisition start command, and based on the data acquisition start command, initiating the multimodal scene perception device set to perform multimodal data acquisition within a preset window; wherein, an infrared camera is used to acquire the body posture trajectory of the target user within the preset window to obtain a body posture trajectory sequence; a VR interactive controller is used to acquire the speed, force, and angle of the target user's hand pressing and breathing actions within the preset window to obtain a hand operation posture sequence; an IMU inertial measurement unit is used to acquire the target user's motion acceleration and angular velocity within the preset window to obtain a motion velocity sequence; a headset microphone is used to acquire the target user's voice input within the preset window, and a real-time speech recognition engine is used to convert the command text with a timestamp to obtain a command text sequence; the body posture trajectory sequence, hand operation posture sequence, motion velocity sequence, and command text sequence are summarized to obtain the multimodal data sequence set.

[0009] Preferably, extracting instruction text sequences from a multimodal data sequence set, and performing semantic memory breakpoint identification on the instruction text sequences to obtain a semantic memory breakpoint set and an instruction operation identifier set, includes: extracting a first instruction text and a second instruction text from the instruction text sequence; performing explicit and implicit semantic memory association judgment on the first instruction text and the second instruction text to obtain a first judgment result; if the first judgment result indicates an association, adding the first instruction text and the second instruction text into an initially empty memory unit to obtain a first memory unit; using the first memory unit to perform explicit and implicit semantic memory association judgment on a third instruction text in the instruction text sequence to obtain a second judgment result; if the second judgment result indicates an association, adding the third instruction text into a memory unit. First memory unit, then obtain second memory unit; if the second judgment result is no association, then the second instruction text is used as a semantic memory breakpoint and added to the semantic memory breakpoint set, and a memory unit reset instruction is obtained. Based on the memory unit reset instruction, the third instruction text is added to the initially empty memory unit, and the second memory unit is obtained; and so on, based on the second memory unit, explicit and implicit semantic memory association judgment is performed on the fourth instruction text in the instruction text sequence, and the third memory unit is obtained according to the judgment result. The explicit and implicit semantic memory association judgment of the next instruction text is executed until the end of the instruction text sequence is reached, and a semantic memory breakpoint set is obtained; based on the semantic memory breakpoint set, instruction operation recognition is performed on the instruction text sequence to obtain an instruction operation identifier set.

[0010] Preferably, if the first judgment result is that there is no association, the first instruction text is added to the semantic memory breakpoint set as a semantic memory breakpoint, and the second instruction text is added to the initially empty memory unit to obtain the first memory unit.

[0011] Preferably, the explicit and implicit semantic memory association judgment is performed on the first instruction text and the second instruction text to obtain a first judgment result, including: extracting keywords from the first instruction text and the second instruction text according to preset keyword types to obtain a first keyword set and a second keyword set, wherein the preset keyword types include action type, operation object, quantity information and temporal information; performing explicit semantic memory association analysis on the first keyword set and the second keyword set to obtain explicit semantic memory association results; performing implicit action intent association analysis on the first instruction text and the second instruction text to obtain implicit semantic memory association results; and determining whether the explicit semantic memory association results and the implicit semantic memory association results are both related. If yes, the first judgment result is that there is a relationship; if no, the first judgment result is that there is no relationship.

[0012] Preferably, the explicit semantic memory association analysis is performed on the first keyword set and the second keyword set to obtain explicit semantic memory association results, including: performing independent explicit semantic memory association screening on the first keyword set and the second keyword set to obtain independent screening explicit semantic memory association coefficients; performing overall explicit independent semantic memory association analysis on the first keyword set and the second keyword set to obtain overall screening explicit semantic memory association coefficients; performing weighted analysis on the independent screening explicit semantic memory association coefficients and the overall screening explicit semantic memory association coefficients to obtain explicit semantic memory association coefficients; when the explicit semantic memory association coefficient is greater than or equal to a preset coefficient threshold, the explicit semantic memory association result is associated; when the explicit semantic memory association coefficient is less than the preset coefficient threshold, the explicit semantic memory association result is not associated.

[0013] Preferably, implicit action intent association analysis is performed on the first instruction text and the second instruction text to obtain implicit semantic memory association results, including: calculating the timestamp difference between the first instruction text and the second instruction text, determining whether the timestamp difference is less than or equal to a preset time window, if not, the implicit semantic memory association result is not associated; if so, implicit action intent association analysis is performed on the first instruction text and the second instruction text from the perspective of the association between the preceding and following emergency procedures to determine the implicit semantic memory association result.

[0014] Preferably, the plurality of multimodal data subsequence sets are fused together with the instruction operation identifier set to obtain a multimodal data fusion acquisition result set, including: extracting a first multimodal data subsequence set and a first instruction operation identifier from the plurality of multimodal data subsequence sets and the instruction operation identifier set; performing motion compensation and correction on the first multimodal data subsequence set using a temporal graph convolutional network to obtain a first corrected multimodal data subsequence set; retrieving a first standard first aid operation video based on the first instruction operation identifier, using the first standard first aid operation video as a template, adaptively fusing the first corrected multimodal data subsequence set to obtain a first multimodal data fusion acquisition result, and adding the first multimodal data fusion acquisition result to the multimodal data fusion acquisition result set.

[0015] A second aspect of this application provides a multimodal data fusion and acquisition system based on VR emergency rescue scenarios, the system comprising: A multimodal data acquisition module is used to deploy a set of multimodal scene sensing devices, and use the set of multimodal scene sensing devices to acquire multimodal data of the target user within a preset window to obtain a set of multimodal data sequences. The semantic memory breakpoint recognition module is used to extract instruction text sequences from a multimodal data sequence set, perform semantic memory breakpoint recognition on the instruction text sequences, and obtain a semantic memory breakpoint set and an instruction operation identifier set, wherein the semantic memory breakpoints correspond one-to-one with the instruction operation identifiers; The splitting module is used to split the multimodal data sequence set according to the semantic memory breakpoint set to obtain multiple multimodal data sub-sequence sets; The data fusion module is used to combine the instruction operation identifier set to fuse the multiple multimodal data subsequence sets respectively, so as to obtain a multimodal data fusion acquisition result set.

[0016] One or more technical solutions provided in this application have at least the following technical effects or advantages: This application deploys a set of multimodal scene perception devices to collect multimodal data from a target user within a preset window, obtaining a set of multimodal data sequences. Then, it extracts instruction text sequences from these sequences and performs semantic memory breakpoint identification on the instruction text sequences, obtaining a set of semantic memory breakpoints and a set of instruction operation identifiers, where each semantic memory breakpoint corresponds one-to-one with an instruction operation identifier. Based on the set of semantic memory breakpoints, the multimodal data sequence set is split into multiple sets of multimodal data sub-sequences. These sub-sequences are then fused together with the instruction operation identifier set to obtain a set of multimodal data fusion acquisition results. This achieves the technical effect of improving the reliability of multimodal data fusion and the reliable synchronization between instructions and actions in VR emergency rescue scenarios. Attached Figure Description

[0017] Appendix Figure 1 This is a schematic diagram of the multimodal data fusion and acquisition method based on VR emergency rescue scenarios provided in an embodiment of the present invention.

[0018] Appendix Figure 2 This is a schematic diagram of the structure of a multimodal data fusion and acquisition system based on VR emergency rescue scenarios provided in an embodiment of the present invention.

[0019] The markings shown in the attached figure: Multimodal data acquisition module 11, semantic memory breakpoint recognition module 12, splitting module 13, data fusion module 14. Detailed Implementation

[0020] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims. It should be noted that the terms "comprising" and "having" are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices.

[0021] Example 1, as shown in the appendix Figure 1 As shown, this application provides a multimodal data fusion acquisition method based on VR emergency rescue scenarios, wherein the method includes: Step S100: Deploy a set of multimodal scene sensing devices, and use the set of multimodal scene sensing devices to collect multimodal data from the target user within a preset window to obtain a set of multimodal data sequences; Furthermore, a set of multimodal scene sensing devices is deployed, and the set of multimodal scene sensing devices is used to collect multimodal data of the target user within a preset window to obtain a set of multimodal data sequences. In this embodiment, step S100 further includes: Optical acquisition devices are deployed within the target motion capture area, including an infrared camera and a VR interactive controller. An IMU (Inertial Measurement Unit) is fixed to the target user's forearm and first aid chest mold, and a head-mounted microphone is worn on the target user's head. The head-mounted microphone includes a real-time speech recognition engine for transcribing speech into timestamped instruction text. The infrared camera, VR interactive controller, IMU inertial measurement unit, and head-mounted microphone are processed with unified clock and spatial coordinates to obtain a set of multimodal scene perception devices that have been deployed.

[0022] Furthermore, step S100 in this embodiment of the application also includes: Obtain a data acquisition start command, and based on the data acquisition start command, start the multimodal scene perception device set to perform multimodal data acquisition within a preset window; Among them, an infrared camera is used to collect the body posture trajectory of the target user within a preset window to obtain a body posture trajectory sequence. The speed, force, and angle of the target user's hand pressing and breathing movements within a preset window are collected using a VR interactive controller to obtain a sequence of hand operation postures. The IMU (Inertial Measurement Unit) is used to collect the acceleration and angular velocity of the target user's movements within a preset window to obtain a sequence of movement velocities. The system uses a headset microphone to collect the voice input of the target user within a preset window, and then uses a real-time speech recognition engine to convert the command text with timestamps to obtain a command text sequence. The multimodal data sequence set is obtained by summarizing the body posture trajectory sequence, hand operation posture sequence, action speed sequence, and instruction text sequence.

[0023] In one possible embodiment, the target user is a user undergoing VR first aid scenario training via VR devices. The multimodal scene perception device set is a collection of various types of sensing devices used for synchronous perception. In a VR scenario, to analyze the target user's first aid operations, it is necessary to first collect data on the target user's actions. Therefore, by appropriately deploying the multimodal scene perception device set, multi-dimensional data collection of the target user's actions can be performed within a pre-defined window (e.g., 15 minutes) set by those skilled in the art, thereby obtaining a multimodal data sequence set reflecting the operational situation.

[0024] The target motion capture area is the area within which the target user can move during VR first aid scenario drills. Optical acquisition devices are deployed to continuously monitor the target user's movements and postures. For example, an infrared camera, capturing data at a high frame rate of 60fps, can completely record the movement trajectories of the user's joints, such as the shoulder and elbow joints, avoiding interruptions in movement trajectories due to low frame rates. The VR interactive controller has low latency characteristics, enabling real-time capture of the force applied by the hand during compressions. Preferably, there are no obstructions within this area. Furthermore, the IMU (Inertial Measurement Unit) uses a 6-axis sensor with a sampling rate of 100Hz, including a 3-axis accelerometer and a 3-axis gyroscope. Two IMUs are fixed to the user's left and right forearms respectively using Velcro, specifically 10cm from the wrist joint, and one IMU is fixed to the precordial region corresponding to the heart position on the first aid chest mold. This captures the changes in chest mold compressions and the target user's compression movements.

[0025] Furthermore, the headset microphone employs noise-canceling technology with a pickup distance of 0-50cm. Utilizing an embedded real-time speech recognition engine, it can perform real-time speech recognition and text conversion. Timestamped instructions, such as "2:10 start pressing; 2:20 stop breathing," prevent time discrepancies during subsequent data fusion analysis. Preferably, the real-time speech recognition engine can be implemented using iFlytek's speech recognition API.

[0026] When the target user presses a button on the VR controller, a data acquisition start command is triggered. The infrared camera acquires the coordinates of 30 body joint points per frame, the VR controller acquires 50 sets of force, speed, and angle data per second, the IMU acquires 100 sets of acceleration and angular velocity data per second, and the headset microphone converts 10 timestamped text segments per second. The acquired data is then aggregated to a local server to obtain the multimodal data sequence set.

[0027] The infrared camera's 30 key point coordinates can completely reconstruct the user's body posture, such as whether the torso leans forward or the knees bend during compressions, thus determining whether the body posture meets the requirements for emergency operations. At the same time, the VR controller's force data, such as the required compression force of 30-50N for CPR, can directly determine whether the operation force is up to standard. Meanwhile, the IMU's acceleration data can help analyze the stability of the movement. For example, if the acceleration fluctuation range during compression exceeds ±2g, it indicates that the movement is unstable. The timestamped text commands can serve as the core data for subsequent semantic breakpoint recognition, linking the command content with the logical relationship of the action execution, and avoiding the separation of data belonging to the same operation content.

[0028] Step S200: Extract instruction text sequences from the multimodal data sequence set, perform semantic memory breakpoint identification on the instruction text sequences, and obtain a semantic memory breakpoint set and an instruction operation identifier set, wherein the semantic memory breakpoints correspond one-to-one with the instruction operation identifiers; Furthermore, the instruction text sequence is extracted from the multimodal data sequence set, and semantic memory breakpoint identification is performed on the instruction text sequence to obtain a semantic memory breakpoint set and an instruction operation identifier set. Step S200 in this embodiment further includes: Extract the first instruction text and the second instruction text from the instruction text sequence; Perform explicit and implicit semantic memory association judgment on the first instruction text and the second instruction text to obtain a first judgment result. If the first judgment result is that there is an association, add the first instruction text and the second instruction text into the initially empty memory unit to obtain the first memory unit. The first memory unit is used to perform explicit and implicit semantic memory association judgment on the third instruction text in the instruction text sequence to obtain a second judgment result. If the second judgment result is that there is an association, the third instruction text is added to the first memory unit to obtain the second memory unit. If the second judgment result is that there is no association, the second instruction text is used as a semantic memory breakpoint and added to the semantic memory breakpoint set. A memory unit reset instruction is obtained, and the third instruction text is added to the initially empty memory unit based on the memory unit reset instruction to obtain the second memory unit. Similarly, based on the second memory unit, the explicit and implicit semantic memory association judgment is performed on the fourth instruction text in the instruction text sequence, and the third memory unit is obtained according to the judgment result. The explicit and implicit semantic memory association judgment of the next instruction text is then performed until the end of the instruction text sequence is reached, and the semantic memory breakpoint set is obtained. Based on the semantic memory breakpoint set, the instruction text sequence is used to identify instruction operations and obtain an instruction operation identifier set.

[0029] Furthermore, if the first judgment result is that there is no association, the first instruction text is added to the semantic memory breakpoint set as a semantic memory breakpoint, and the second instruction text is added to the initially empty memory unit to obtain the first memory unit.

[0030] It should be noted that the instruction text sequence is a continuous instruction text stream output by the real-time speech recognition engine, containing multiple voice commands issued by the target user during VR emergency rescue scenario training, such as starting chest compressions, calling 120, and using an AED. Therefore, semantic memory breakpoints can be identified by performing adjacency semantic analysis on the instruction text sequence. These semantic memory breakpoints are logical boundaries identified through continuous semantic analysis, used to distinguish different task stages or operational process nodes in the instructions issued by the target user. In other words, semantic memory breakpoints reflect the switching points of the user's semantic intent.

[0031] In one possible embodiment, a first instruction text and a second instruction text are extracted from the instruction text sequence. Then, the semantic relationship between the two is analyzed to determine whether they belong to the same operation stage. Preferably, during the semantic relationship comparison process, it is necessary to determine whether there is a correlation between the two based on explicit similarity comparison and implicit time and step associations, thereby determining whether the first instruction text and the second instruction text belong to the same operation stage.

[0032] In current multimodal data fusion processes, simply comparing time similarity can easily lead to two commands belonging to different operations being grouped into the same fusion time period due to operational delays, resulting in a failure to grasp the actual operations of the target user. However, by performing explicit and implicit semantic memory association on the command text sequence, different commands can be reliably distinguished, thereby improving the reliability of subsequent multimodal data fusion.

[0033] Preferably, the first judgment result has two possibilities: the first is that there is a correlation, and the second is that there is no correlation. In the two cases, the first instruction text and the second instruction text are processed differently.

[0034] When the first judgment result indicates a correlation, a temporary semantic container is obtained to store instructions belonging to the same logical stage, thereby ensuring semantic contextual coherence; that is, an initially empty memory unit. Furthermore, since the first instruction text and the second instruction text are related, such as the first instruction text being "start pressing" and the second instruction text being "continue pressing," both instruction texts are added to the memory unit to obtain the first memory unit.

[0035] When the first judgment result is that there is no correlation, such as the first instruction text being "press to end" and the second instruction text being "open AED", the operation between the two instruction texts is not a continuous operation of the same operation item, indicating that a semantic switch has occurred at the second instruction text. At this time, the instruction text preceding the second instruction text, that is, the first instruction text, is taken as a semantic memory breakpoint and added to the semantic memory breakpoint set. The second instruction text is then added to the initially empty memory unit for subsequent semantic memory analysis, that is, to obtain the first memory unit.

[0036] Based on the same principle as the explicit and implicit semantic memory association judgment for the first and second instruction texts, the explicit and implicit semantic memory association judgment is performed on the instruction text stored in the first memory unit and the third instruction text to obtain a second judgment result. If the second judgment result indicates that there is an association, it means that the target user's operation between the instruction text stored in the first memory unit and the third instruction text is continuous. In order to ensure semantic coherence, the third instruction text is added to the first memory unit to form the second memory unit.

[0037] If the second judgment result is no association, it indicates that the semantic stage switched at the third instruction text. At this time, the instruction text preceding the third instruction text, i.e., the second instruction text, is taken as the breakpoint of the previous semantic memory segment and added to the semantic memory breakpoint set. A memory unit reset instruction is then obtained, which resets the previously obtained first memory unit, resulting in an initially empty memory unit. The third instruction text is then added to this initially empty memory unit, resulting in the second memory unit. The two processing methods for the second judgment result of association being present or absent are as described above. Both methods will obtain the second memory unit, but the data stored in the second memory unit will differ. This process continues until the end of the instruction text sequence is reached, resulting in the completed semantic memory breakpoint set.

[0038] Furthermore, the first instruction text and the second instruction text are subjected to explicit and implicit semantic memory association judgment to obtain a first judgment result. Step S200 of this application embodiment also includes: The first instruction text and the second instruction text are extracted according to the preset keyword types to obtain the first keyword set and the second keyword set. The preset keyword types include action type, operation object, quantity information and timing information. Explicit semantic memory association analysis is performed on the first keyword set and the second keyword set to obtain explicit semantic memory association results; Implicit action intent association analysis is performed on the first instruction text and the second instruction text to obtain implicit semantic memory association results; Determine whether both the explicit semantic memory association results and the implicit semantic memory association results are associated; if so, the first determination result is that an association exists. If not, the first judgment result is that there is no association.

[0039] In one possible embodiment, natural language processing techniques are used to analyze whether there is a semantic continuity or repetition relationship between the instructions corresponding to the first and second instruction texts from both explicit and implicit perspectives. The preset keyword type is a keyword type pre-defined by those skilled in the art to assist semantic recognition, including action type, operation object, quantity information, and timing information. Specifically, the action describes the action type of the instruction, such as compression or ventilation; the operation object is the target object of the instruction, such as a patient, chest, or AED; the quantity information is the number of times the corresponding action is performed, such as 30 times; and the timing information describes the stage of the corresponding action, such as start, continue, or stop.

[0040] Preferably, explicit semantic association analysis is based on direct comparison analysis of keywords, while implicit semantic association analysis focuses on the rationality of association in the sequence of actions and the association between the implicit action intentions in the preceding and following emergency procedures. Both explicit and implicit semantic memory association results include both associated and unassociated results.

[0041] When both the explicit semantic memory association result and the implicit semantic memory association result are associated, it indicates that the first instruction text and the second instruction text are associated, whether from the perspective of surface intuitive semantics or implicit association. At this time, the first judgment result is that there is an association.

[0042] When the explicit semantic memory association result is associated, while the implicit semantic memory association result is not associated, it indicates that although the first and second instruction texts are similar in terms of surface-level intuitive semantics, they are not associated in terms of implicit association. For example, consider the instructions for applying pressure to stop bleeding and applying pressure to fix the dressing in wound dressing. The first instruction text is "Apply pressure to stop bleeding for 3 minutes," with keywords including the intention to apply pressure, the object of the operation being the wound, and the quantity information "3 minutes." The second instruction text is "Apply pressure to fix the gauze dressing," with keywords including the intention to apply pressure and the object of the operation being the gauze. In this case, because the second instruction text lacks quantity information, the analysis of the explicit semantic memory association result shows no association between the intention and the quantity information. Figure 1 Both the instruction and the object of the operation pertain to the compression action in trauma management, indicating a high degree of semantic similarity between the two instruction texts. However, the implicit intention of compression in the first instruction text is to stop bleeding, while the implicit intention in the second instruction text is to secure the gauze to prevent it from falling off. These two implicit intentions differ, therefore, the first and second instruction texts are not related. Explicit association relies on keyword similarity, while implicit association relies on the correctness of the intention and the smoothness of the process. When the results of these two methods conflict, it indicates that although the two instruction texts have superficial similarities, their purposes have switched. This effectively avoids misjudgments of semantic breakpoints caused by coincidental keyword similarities, ensuring that subsequent data segmentation accurately corresponds to different stages of emergency care.

[0043] When the explicit semantic memory association result is not associated, while the implicit semantic memory association result is associated, and when both the explicit and implicit semantic memory association results are not associated, the intuitive semantics are different, meaning the continuity of the operation is broken. Even if they are different operations within the same process, they need to be determined as not associated; that is, the first judgment result is not associated. This avoids the erroneous fusion of action data from different stages, such as the erroneous fusion of the bending-over observation action during assessment and the device retrieval action during defibrillation preparation.

[0044] Furthermore, explicit semantic memory association analysis is performed on the first keyword set and the second keyword set to obtain explicit semantic memory association results. Step S200 in this embodiment further includes: Independent explicit semantic memory association filtering is performed on the first keyword set and the second keyword set to obtain the independent explicit semantic memory association coefficient; An overall explicit independent semantic memory association analysis was performed on the first keyword set and the second keyword set to obtain the overall explicit semantic memory association coefficient for screening. The explicit semantic memory association coefficient is obtained by performing a weighted analysis on the independent screening explicit semantic memory association coefficient and the overall screening explicit semantic memory association coefficient. When the explicit semantic memory association coefficient is greater than or equal to the preset coefficient threshold, the explicit semantic memory association result is an association; When the explicit semantic memory association coefficient is less than the preset coefficient threshold, the explicit semantic memory association result is no association.

[0045] In one possible embodiment, keywords of the same type in the first keyword set and the second keyword set are compared using a cosine similarity function according to a preset keyword type, resulting in a set of similarity calculation results. The similarity calculation result closest to the mean of the similarity calculation result set is extracted and used as the independent selection coefficient for explicit semantic memory association.

[0046] According to the weight values ​​corresponding to different keyword types pre-defined by those skilled in the art, the similarity calculation result set is subjected to weighted analysis to obtain the overall explicit semantic memory association coefficient. The overall explicit semantic memory association coefficient reflects the overall similarity between two keyword sets; the larger the coefficient, the higher the corresponding association.

[0047] Then, the independent screening explicit semantic memory association coefficient and the overall screening explicit semantic memory association coefficient are weighted and analyzed according to preset independent weight values ​​and overall weight values ​​to obtain the explicit semantic memory association coefficient. The explicit semantic memory association coefficient is compared with a preset coefficient threshold set by those skilled in the art. If the explicit semantic memory association coefficient is greater than or equal to the preset coefficient threshold, the explicit semantic memory association result is considered associated. If the explicit semantic memory association coefficient is less than the preset coefficient threshold, the explicit semantic memory association result is considered not associated.

[0048] Furthermore, implicit action intent association analysis is performed on the first instruction text and the second instruction text to obtain implicit semantic memory association results. Step S200 in this embodiment of the application also includes: Calculate the timestamp difference between the first instruction text and the second instruction text, and determine whether the timestamp difference is less than or equal to a preset time window. If not, the implicit semantic memory association result is no association. If so, then from the perspective of the correlation between the first and second instruction texts in the emergency rescue process, an implicit action intent correlation analysis is performed to determine the implicit semantic memory correlation result.

[0049] In one possible embodiment, the timestamp information of the two instruction texts is first read, and their time interval is calculated. If the time difference between the two instructions exceeds a preset time window (i.e., a preset time period) set by those skilled in the art, and the trainee does not issue any further instructions for an extended period, it is considered that the instruction has logically deviated from the current operational flow and is therefore deemed unrelated, and no further analysis is performed. If the time difference is less than or equal to the preset time window, the implicit action intent of the two instruction texts is further analyzed from the perspective of the correlation between the preceding and following emergency procedures, i.e., it is determined whether there is a reasonable sequential relationship or logical connection between them in the emergency task sequence. For example, if the first instruction is to use an AED and the second instruction is to place the patch, and the time interval between the two is less than or equal to the preset time window, then the latter is considered a continuation of the former and belongs to an implicit association.

[0050] By employing both temporal logic and operational flow constraints, the system enhances its ability to recognize implicit continuity in instruction text, effectively distinguishing between instruction phase transitions and the continuation of the same operation, thereby reducing semantic breakpoint misjudgments. This provides a stable semantic foundation for subsequent instruction-driven data fusion and action template comparison.

[0051] Step S300: Based on the semantic memory breakpoint set, split the multimodal data sequence set to obtain multiple multimodal data subsequence sets; In one possible embodiment, a preset window is divided into multiple time periods, using the timestamp corresponding to each semantic memory breakpoint in the semantic memory breakpoint set as the dividing node. Then, based on these multiple time periods, multimodal data subsequences belonging to the same time period in the multimodal sequence set are aggregated to obtain the multiple multimodal data subsequence sets. This achieves the goal of providing reliable data subsequences for subsequent multimodal data fusion, thereby improving the reliability of the fusion process.

[0052] Step S400: Combine the instruction operation identifier set and fuse the multiple multimodal data subsequence sets respectively to obtain a multimodal data fusion acquisition result set.

[0053] Furthermore, by combining the instruction operation identifier set, the multiple multimodal data subsequence sets are fused respectively to obtain a multimodal data fusion acquisition result set. Step S400 in this embodiment further includes: Extract the first multimodal data subsequence set and the first instruction operation identifier from the plurality of multimodal data subsequence sets and instruction operation identifier sets; A temporal graph convolutional network is used to perform action compensation and correction on the first multimodal data subsequence set to obtain a first corrected multimodal data subsequence set. Based on the first instruction operation identifier, the first standard emergency operation video is retrieved. Using the first standard emergency operation video as a template, the first corrected multimodal data subsequence set is adaptively fused to obtain the first multimodal data fusion acquisition result, and the first multimodal data fusion acquisition result is added to the multimodal data fusion acquisition result set.

[0054] In one possible embodiment, the first digit of the first multimodal data subsequence set and the first instruction operation identifier refer to any one of the multiple multimodal data subsequence sets and any one of the instruction operation identifier sets. Multiple sample sets of the first multimodal data subsequence and multiple sample sets of corrected multimodal data subsequence are used as training data samples. The framework constructed based on the temporal graph convolutional network is trained under supervised supervision based on the training data samples until training converges, obtaining the trained temporal graph convolutional network. The temporal graph convolutional network is then used to perform action compensation and correction on the first multimodal data subsequence set to obtain the first corrected multimodal data subsequence set.

[0055] Then, based on the first instruction operation identifier, a first standard first aid operation video is retrieved. This first standard first aid operation video is a pre-recorded standardized operation video conforming to first aid guidelines, such as the AHA cardiopulmonary resuscitation guidelines, containing key action nodes with timestamps, such as the moment when the compression depth reaches 5cm and the ventilation angle reaches 30°, serving as a reference template for data fusion. Multiple sample standard first aid operation videos, multiple sample corrected multimodal data subsequence sets, and corresponding multiple sample multimodal data fusion acquisition results are acquired as a sample set. The sample set is divided into a training set and a validation set, preferably in a 3:2 ratio. The training set is used to supervise the training of the framework constructed based on a feedforward neural network. After training, multiple sample standard first aid operation videos and multiple sample corrected multimodal data subsequence sets from the validation set are used as input to obtain multiple output multimodal data fusion acquisition results. The overall similarity between the multiple sample multimodal data fusion acquisition results and the multiple output multimodal data fusion acquisition results is compared. If the overall similarity is greater than or equal to a preset threshold, the validation is passed, and the trained adaptive fusion device is obtained.

[0056] An adaptive fusion processor is used to analyze the first standard emergency medical procedure video and the first corrected multimodal data subsequence set to obtain the first multimodal data fusion acquisition result, and the first multimodal data fusion acquisition result is added to the multimodal data fusion acquisition result set.

[0057] Example 2, based on the same inventive concept as the multimodal data fusion acquisition method for VR emergency rescue scenarios in the foregoing examples, as shown in the appendix. Figure 2As shown, this application provides a multimodal data fusion acquisition system based on VR emergency rescue scenarios. The system and method embodiments in this application are based on the same inventive concept. The system includes:

[0058] The multimodal data acquisition module 11 is used to deploy a set of multimodal scene sensing devices and use the set of multimodal scene sensing devices to acquire multimodal data of the target user within a preset window to obtain a set of multimodal data sequences. The semantic memory breakpoint recognition module 12 is used to extract instruction text sequences from a multimodal data sequence set, perform semantic memory breakpoint recognition on the instruction text sequences, and obtain a semantic memory breakpoint set and an instruction operation identifier set, wherein the semantic memory breakpoints correspond one-to-one with the instruction operation identifiers; The splitting module 13 is used to split the multimodal data sequence set according to the semantic memory breakpoint set to obtain multiple multimodal data sub-sequence sets; The data fusion module 14 is used to combine the instruction operation identifier set to fuse the multiple multimodal data subsequence sets respectively to obtain a multimodal data fusion acquisition result set.

[0059] Furthermore, the multimodal data acquisition module 11 is also used to perform the following steps: Optical acquisition devices are deployed within the target motion capture area, including an infrared camera and a VR interactive controller. An IMU (Inertial Measurement Unit) is fixed to the target user's forearm and first aid chest mold, and a head-mounted microphone is worn on the target user's head. The head-mounted microphone includes a real-time speech recognition engine for transcribing speech into timestamped instruction text. The infrared camera, VR interactive controller, IMU inertial measurement unit, and head-mounted microphone are processed with unified clock and spatial coordinates to obtain a set of multimodal scene perception devices that have been deployed.

[0060] Furthermore, the multimodal data acquisition module 11 is also used to perform the following steps: Obtain a data acquisition start command, and based on the data acquisition start command, start the multimodal scene perception device set to perform multimodal data acquisition within a preset window; Among them, an infrared camera is used to collect the body posture trajectory of the target user within a preset window to obtain a body posture trajectory sequence. The speed, force, and angle of the target user's hand pressing and breathing movements within a preset window are collected using a VR interactive controller to obtain a sequence of hand operation postures. The IMU (Inertial Measurement Unit) is used to collect the acceleration and angular velocity of the target user's movements within a preset window to obtain a sequence of movement velocities. The system uses a headset microphone to collect the voice input of the target user within a preset window, and then uses a real-time speech recognition engine to convert the command text with timestamps to obtain a command text sequence. The multimodal data sequence set is obtained by summarizing the body posture trajectory sequence, hand operation posture sequence, action speed sequence, and instruction text sequence.

[0061] Furthermore, the semantic memory breakpoint recognition module 12 is also used to perform the following steps: Extract the first instruction text and the second instruction text from the instruction text sequence; Perform explicit and implicit semantic memory association judgment on the first instruction text and the second instruction text to obtain a first judgment result. If the first judgment result is that there is an association, add the first instruction text and the second instruction text into the initially empty memory unit to obtain the first memory unit. The first memory unit is used to perform explicit and implicit semantic memory association judgment on the third instruction text in the instruction text sequence to obtain a second judgment result. If the second judgment result is that there is an association, the third instruction text is added to the first memory unit to obtain the second memory unit. If the second judgment result is that there is no association, the second instruction text is used as a semantic memory breakpoint and added to the semantic memory breakpoint set. A memory unit reset instruction is obtained, and the third instruction text is added to the initially empty memory unit based on the memory unit reset instruction to obtain the second memory unit. Similarly, based on the second memory unit, the explicit and implicit semantic memory association judgment is performed on the fourth instruction text in the instruction text sequence, and the third memory unit is obtained according to the judgment result. The explicit and implicit semantic memory association judgment of the next instruction text is then performed until the end of the instruction text sequence is reached, and the semantic memory breakpoint set is obtained. Based on the semantic memory breakpoint set, the instruction text sequence is used to identify instruction operations and obtain an instruction operation identifier set.

[0062] Furthermore, if the first judgment result is that there is no association, the first instruction text is added to the semantic memory breakpoint set as a semantic memory breakpoint, and the second instruction text is added to the initially empty memory unit to obtain the first memory unit.

[0063] Furthermore, the semantic memory breakpoint recognition module 12 is also used to perform the following steps: The first instruction text and the second instruction text are extracted according to the preset keyword types to obtain the first keyword set and the second keyword set. The preset keyword types include action type, operation object, quantity information and timing information. Explicit semantic memory association analysis is performed on the first keyword set and the second keyword set to obtain explicit semantic memory association results; Implicit action intent association analysis is performed on the first instruction text and the second instruction text to obtain implicit semantic memory association results; Determine whether both the explicit semantic memory association results and the implicit semantic memory association results are associated; if so, the first determination result is that an association exists. If not, the first judgment result is that there is no association.

[0064] Furthermore, the semantic memory breakpoint recognition module 12 is also used to perform the following steps: Independent explicit semantic memory association filtering is performed on the first keyword set and the second keyword set to obtain the independent explicit semantic memory association coefficient; An overall explicit independent semantic memory association analysis was performed on the first keyword set and the second keyword set to obtain the overall explicit semantic memory association coefficient for screening. The explicit semantic memory association coefficient is obtained by performing a weighted analysis on the independent screening explicit semantic memory association coefficient and the overall screening explicit semantic memory association coefficient. When the explicit semantic memory association coefficient is greater than or equal to the preset coefficient threshold, the explicit semantic memory association result is an association; When the explicit semantic memory association coefficient is less than the preset coefficient threshold, the explicit semantic memory association result is no association.

[0065] Furthermore, the semantic memory breakpoint recognition module 12 is also used to perform the following steps: Calculate the timestamp difference between the first instruction text and the second instruction text, and determine whether the timestamp difference is less than or equal to a preset time window. If not, the implicit semantic memory association result is no association. If so, then from the perspective of the correlation between the first and second instruction texts in the emergency rescue process, an implicit action intent correlation analysis is performed to determine the implicit semantic memory correlation result.

[0066] Furthermore, the data fusion module 14 is also used to perform the following steps: Extract the first multimodal data subsequence set and the first instruction operation identifier from the plurality of multimodal data subsequence sets and instruction operation identifier sets; A temporal graph convolutional network is used to perform action compensation and correction on the first multimodal data subsequence set to obtain a first corrected multimodal data subsequence set. Based on the first instruction operation identifier, the first standard emergency operation video is retrieved. Using the first standard emergency operation video as a template, the first corrected multimodal data subsequence set is adaptively fused to obtain the first multimodal data fusion acquisition result, and the first multimodal data fusion acquisition result is added to the multimodal data fusion acquisition result set.

[0067] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0068] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0069] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and modifications fall within the scope of this application and its equivalents, this application intends to include such modifications and modifications.

Claims

1. A multimodal data fusion and acquisition method based on VR emergency rescue scenarios, characterized in that, The method includes: A set of multimodal scene sensing devices is deployed, and the set of multimodal scene sensing devices is used to collect multimodal data of the target user within a preset window to obtain a set of multimodal data sequences; Extract instruction text sequences from a multimodal data sequence set, perform semantic memory breakpoint identification on the instruction text sequences, and obtain a semantic memory breakpoint set and an instruction operation identifier set, wherein the semantic memory breakpoints correspond one-to-one with the instruction operation identifiers; Based on the set of semantic memory breakpoints, the multimodal data sequence set is split to obtain multiple sets of multimodal data sub-sequences; By combining the instruction operation identifier set, the multiple multimodal data subsequence sets are fused to obtain a multimodal data fusion acquisition result set.

2. The multimodal data fusion and acquisition method based on VR emergency rescue scenarios as described in claim 1, characterized in that, A multimodal scene sensing device set is deployed, and the multimodal scene sensing device set is used to collect multimodal data from a target user within a preset window to obtain a multimodal data sequence set, including: Optical acquisition devices are deployed within the target motion capture area, including an infrared camera and a VR interactive controller. An IMU (Inertial Measurement Unit) is fixed to the target user's forearm and first aid chest mold, and a head-mounted microphone is worn on the target user's head. The head-mounted microphone includes a real-time speech recognition engine for transcribing speech into timestamped instruction text. The infrared camera, VR interactive controller, IMU inertial measurement unit, and head-mounted microphone are processed with unified clock and spatial coordinates to obtain a set of multimodal scene perception devices that have been deployed.

3. The multimodal data fusion and acquisition method based on VR emergency rescue scenarios as described in claim 2, characterized in that, Also includes: Obtain a data acquisition start command, and based on the data acquisition start command, start the multimodal scene perception device set to perform multimodal data acquisition within a preset window; Among them, an infrared camera is used to collect the body posture trajectory of the target user within a preset window to obtain a body posture trajectory sequence. The speed, force, and angle of the target user's hand pressing and breathing movements within a preset window are collected using a VR interactive controller to obtain a sequence of hand operation postures. The IMU (Inertial Measurement Unit) is used to collect the acceleration and angular velocity of the target user's movements within a preset window to obtain a sequence of movement velocities. The system uses a headset microphone to collect the voice input of the target user within a preset window, and then uses a real-time speech recognition engine to convert the command text with timestamps to obtain a command text sequence. The multimodal data sequence set is obtained by summarizing the body posture trajectory sequence, hand operation posture sequence, action speed sequence, and instruction text sequence.

4. The multimodal data fusion and acquisition method based on VR emergency rescue scenarios as described in claim 1, characterized in that, Instruction text sequences are extracted from a multimodal data sequence set. Semantic memory breakpoints are identified within these sequences to obtain a semantic memory breakpoint set and an instruction operation identifier set, including: Extract the first instruction text and the second instruction text from the instruction text sequence; Perform explicit and implicit semantic memory association judgment on the first instruction text and the second instruction text to obtain a first judgment result. If the first judgment result is that there is an association, add the first instruction text and the second instruction text into the initially empty memory unit to obtain the first memory unit. The first memory unit is used to perform explicit and implicit semantic memory association judgment on the third instruction text in the instruction text sequence to obtain a second judgment result. If the second judgment result is that there is an association, the third instruction text is added to the first memory unit to obtain the second memory unit. If the second judgment result is that there is no association, the second instruction text is used as a semantic memory breakpoint and added to the semantic memory breakpoint set. A memory unit reset instruction is obtained, and the third instruction text is added to the initially empty memory unit based on the memory unit reset instruction to obtain the second memory unit. Similarly, based on the second memory unit, the explicit and implicit semantic memory association judgment is performed on the fourth instruction text in the instruction text sequence, and the third memory unit is obtained according to the judgment result. The explicit and implicit semantic memory association judgment of the next instruction text is then performed until the end of the instruction text sequence is reached, and the semantic memory breakpoint set is obtained. Based on the semantic memory breakpoint set, the instruction text sequence is used to identify instruction operations and obtain an instruction operation identifier set.

5. The multimodal data fusion and acquisition method based on VR emergency rescue scenarios as described in claim 4, characterized in that, If the first judgment result is that there is no association, then the first instruction text is added to the semantic memory breakpoint set as a semantic memory breakpoint, and the second instruction text is added to the initially empty memory unit to obtain the first memory unit.

6. The multimodal data fusion and acquisition method based on VR emergency rescue scenarios as described in claim 4, characterized in that, Perform explicit and implicit semantic memory association judgment on the first instruction text and the second instruction text to obtain a first judgment result, including: The first instruction text and the second instruction text are extracted according to the preset keyword types to obtain the first keyword set and the second keyword set. The preset keyword types include action type, operation object, quantity information and timing information. Explicit semantic memory association analysis is performed on the first keyword set and the second keyword set to obtain explicit semantic memory association results; Implicit action intent association analysis is performed on the first instruction text and the second instruction text to obtain implicit semantic memory association results; Determine whether both the explicit semantic memory association results and the implicit semantic memory association results are associated; if so, the first determination result is that an association exists. If not, the first judgment result is that there is no association.

7. The multimodal data fusion and acquisition method based on VR emergency rescue scenarios as described in claim 6, characterized in that, Explicit semantic memory association analysis was performed on the first keyword set and the second keyword set to obtain explicit semantic memory association results, including: Independent explicit semantic memory association filtering is performed on the first keyword set and the second keyword set to obtain the independent explicit semantic memory association coefficient; An overall explicit independent semantic memory association analysis was performed on the first keyword set and the second keyword set to obtain the overall explicit semantic memory association coefficient for screening. The explicit semantic memory association coefficient is obtained by performing a weighted analysis on the independent screening explicit semantic memory association coefficient and the overall screening explicit semantic memory association coefficient. When the explicit semantic memory association coefficient is greater than or equal to the preset coefficient threshold, the explicit semantic memory association result is an association; When the explicit semantic memory association coefficient is less than the preset coefficient threshold, the explicit semantic memory association result is no association.

8. The multimodal data fusion and acquisition method based on VR emergency rescue scenarios as described in claim 6, characterized in that, Implicit action intent association analysis is performed on the first and second instruction texts to obtain implicit semantic memory association results, including: Calculate the timestamp difference between the first instruction text and the second instruction text, and determine whether the timestamp difference is less than or equal to a preset time window. If not, the implicit semantic memory association result is no association. If so, then from the perspective of the correlation between the first and second instruction texts in the emergency rescue process, implicit action intent correlation analysis is performed to determine the implicit semantic memory correlation result.

9. The multimodal data fusion and acquisition method based on VR emergency rescue scenarios as described in claim 1, characterized in that, By combining the instruction operation identifier set, the multiple sets of multimodal data subsequences are fused to obtain a multimodal data fusion acquisition result set, including: Extract the first multimodal data subsequence set and the first instruction operation identifier from the plurality of multimodal data subsequence sets and instruction operation identifier sets; A temporal graph convolutional network is used to perform action compensation and correction on the first multimodal data subsequence set to obtain a first corrected multimodal data subsequence set. Based on the first instruction operation identifier, the first standard emergency operation video is retrieved. Using the first standard emergency operation video as a template, the first corrected multimodal data subsequence set is adaptively fused to obtain the first multimodal data fusion acquisition result, and the first multimodal data fusion acquisition result is added to the multimodal data fusion acquisition result set.

10. A multimodal data fusion and acquisition system based on VR emergency rescue scenarios, characterized in that, The system is used to implement the multimodal data fusion and acquisition method based on VR emergency rescue scenarios as described in any one of claims 1-9, and the system includes: A multimodal data acquisition module is used to deploy a set of multimodal scene sensing devices, and use the set of multimodal scene sensing devices to acquire multimodal data of the target user within a preset window to obtain a set of multimodal data sequences. The semantic memory breakpoint recognition module is used to extract instruction text sequences from a multimodal data sequence set, perform semantic memory breakpoint recognition on the instruction text sequences, and obtain a semantic memory breakpoint set and an instruction operation identifier set, wherein the semantic memory breakpoints correspond one-to-one with the instruction operation identifiers; The splitting module is used to split the multimodal data sequence set according to the semantic memory breakpoint set to obtain multiple multimodal data sub-sequence sets; The data fusion module is used to combine the instruction operation identifier set to fuse the multiple multimodal data subsequence sets respectively, so as to obtain a multimodal data fusion acquisition result set.

Images