A real-time monitoring method and related device for cavity environment based on multimodal perception

Abstract

This application discloses a real-time monitoring method for cavity environment based on multimodal perception and related devices. The method includes: synchronously collecting cavity visual, contact force, and electromagnetic positioning data; refining the three types of data separately; extracting YOLO features from the refined visual data, outputting dynamic tactile features from the contact force data using the Mamba model, and refining the positioning data into positioning features using an extended Kalman filter; generating classification results based on the three types of features, fusing them through weighted voting, and mapping the visual and tactile features to a unified three-dimensional coordinate for association and labeling; and finally generating an environmental status decision based on the judgment and association results. The present invention realizes multimodal real-time perception and risk warning of the cavity environment, improving surgical safety and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of robotics and computer-assisted surgery, and in particular to a real-time monitoring method for a cavity environment based on multimodal perception and related devices. Background Art

[0002] Currently, interventional surgery is rapidly becoming popular in endoscopy, minimally invasive surgery and other cavity diagnosis and treatment fields. However, clinical practice still relies on doctors' single-modal experience judgment. Visual information is easily obscured by bleeding and mucus, tactile feedback is missing or delayed, electromagnetic positioning drift and delay lead to path deviation, multi-source data are isolated and time is not synchronized, and it is difficult to identify risks such as ulcers, polyps, adhesions, and perforations in real time. The existing navigation system focuses on static reconstruction and ignores dynamic risks. It lacks a low-latency, high-precision multimodal fusion decision-making mechanism, resulting in high rates of intraoperative accidental injury, perforation, bleeding and secondary surgery, and the operation time and radiation dose are forced to be extended. There is an urgent need for a cavity environment monitoring solution that can synchronously perceive, integrate in real time and actively warn. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention relates to a real-time monitoring method of the cavity environment based on multimodal perception and related devices, which include but are not limited to a real-time monitoring device of the cavity environment based on multimodal perception, an electronic device, a computer-readable storage medium and a computer program product.

[0004] In a first aspect, a method for real-time monitoring of a cavity environment based on multimodal sensing is provided, comprising the following steps:

[0005] A. Synchronously collect visual data, contact force data, and electromagnetic positioning data in the cavity environment;

[0006] B. performing resizing, cropping, denoising, distortion correction, and illumination compensation on the visual data to obtain refined visual data; performing amplification, filtering, and voltage-to-force conversion on the contact force data to obtain refined contact force data; and performing filtering and preliminary position and posture calculation on the electromagnetic positioning data to obtain refined positioning data;

[0007] C. For the refined visual data, the YOLO model is used to extract edges, corners, texture features, color histograms, specific target detection results, and optical flow features as visual features. For the refined contact force data, the Mamba model is used to process the contact force data sequence of a preset time window length, and dynamic tactile features including at least one of normal contact, abnormal friction, excessive pressure, adhesion risk, and perforation risk are output. For the refined positioning data, a preset probe kinematic model is used as the state equation and the electromagnetic positioning data as the observation equation. The position and posture information is refined using an extended Kalman filter as the positioning feature.

[0008] D. generating a lesion type or structural abnormality type determination result based on the visual features, generating a tactile interaction state classification result based on the dynamic tactile features, generating a position deviation based on the positioning features, and obtaining a determination result using a weighted voting rule; establishing the three-dimensional spatial coordinates of the cavity based on the positioning features, mapping the pixel coordinates in the visual features to the three-dimensional spatial coordinates through perspective transformation, mapping the sensor spatial coordinates corresponding to the dynamic tactile features to the three-dimensional spatial coordinates through rigid body transformation, and associating and labeling the visual features and dynamic tactile features at the same spatial coordinates to obtain an association result;

[0009] E. Generate an environmental status decision based on the judgment result and the association result.

[0010] In combination with any embodiment of the present application, the synchronous acquisition is achieved through a hardware trigger signal or a unified timestamp protocol.

[0011] In combination with any embodiment of the present application, the preliminary position and posture solution processing is a 6-degree-of-freedom posture solution processing based on the electromagnetic positioning principle, and the 6-degree-of-freedom posture includes posture data of 3 position (X, Y, Z) degrees of freedom and 3 attitude (Roll, Pitch, Yaw) degrees of freedom.

[0012] In combination with any embodiment of the present application, the state equation of the extended Kalman filter is defined as a rigid body 6-degree-of-freedom kinematic model, and the observation equation is defined as a nonlinear mapping function of the electromagnetic positioning data; the 6-degree-of-freedom kinematic model includes velocity data of 3 position (X, Y, Z) degrees of freedom and angular velocity data of 3 attitude (Roll, Pitch, Yaw) degrees of freedom.

[0013] In conjunction with any embodiment of the present application, the lesion type includes ulcers, polyps, and bleeding spots;

[0014] The structural abnormalities include stenosis, dilatation, and diverticula;

[0015] The position deviation is calculated based on the Euclidean distance between the actual position of the probe and the planned path.

[0016] In combination with any embodiment of the present application, generating an environmental status decision based on the judgment result and the association result includes:

[0017] When the lesion type or the structural abnormality type is detected, generating a lesion position marking instruction;

[0018] When the tactile interaction state classification result is adhesion risk or perforation risk, generating a retreat operation instruction;

[0019] When the position deviation exceeds a safety threshold, a path correction instruction is generated.

[0020] In a second aspect, a real-time monitoring device for cavity environment based on multimodal sensing is provided, the device comprising:

[0021] A sensing unit, used to synchronously collect visual data, contact force data, and electromagnetic positioning data in the cavity environment;

[0022] a data processing unit for performing size change, cropping, denoising, distortion correction, and illumination compensation processing on the visual data to obtain refined visual data; performing amplification, filtering, and voltage-force value conversion processing on the contact force data to obtain refined contact force data; performing filtering and preliminary position and posture solution processing on the electromagnetic positioning data to obtain refined positioning data; and further for extracting edges, corners, texture features, color histograms, specific target detection results, and optical flow features from the refined visual data using a YOLO model as visual features; processing a contact force data sequence of a preset time window length based on a mamba model for the refined contact force data, and outputting dynamic tactile features including at least one of normal contact, abnormal friction, excessive pressure, adhesion risk, and perforation risk; and refining the position and posture information of the refined positioning data using a preset probe kinematic model as a state equation and the electromagnetic positioning data as an observation equation, and using an extended Kalman filter to refine the position and posture information as positioning features;

[0023] a registration unit, configured to generate a lesion type or structural abnormality type judgment result based on the visual feature, generate a tactile interaction state classification result based on the dynamic tactile feature, generate a position deviation based on the positioning feature, and obtain a judgment result using a weighted voting rule; and further configured to establish a three-dimensional spatial coordinate of the cavity based on the positioning feature, map the pixel coordinates in the visual feature to the three-dimensional spatial coordinates through perspective transformation, map the sensor spatial coordinates corresponding to the dynamic tactile feature to the three-dimensional spatial coordinates through rigid body transformation, and associate and mark the visual feature and the dynamic tactile feature at the same spatial coordinates to obtain an association result;

[0024] An output unit is used to generate an environmental status decision based on the judgment result and the association result.

[0025] In a third aspect, an electronic device is provided, comprising: a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is configured to store computer program code, wherein the program code comprises computer instructions. When the processor executes these instructions, the electronic device performs the method described in the second aspect and any embodiment thereof.

[0026] In a fourth aspect, another electronic device is provided, comprising: a processor, a wireless communication module, a touch screen, a speaker, and a storage unit, wherein the storage unit is configured to store computer program code, wherein the program code comprises computer instructions. When the processor executes these instructions, the electronic device performs the method described in the second aspect and any embodiment thereof.

[0027] In a fifth aspect, a computer-readable storage medium is provided, wherein a computer program is stored, wherein the program includes program instructions. When these instructions are executed by a processor, the processor will perform the method described in the second aspect and any embodiment thereof.

[0028] In a sixth aspect, a computer program product is provided, wherein the computer program product comprises a computer program or instructions. When the computer program or instructions are run on a computer, the computer will execute the method described in the second aspect and any embodiment thereof.

[0029] It should be understood that the above general description and the following detailed description are only used as examples and explanations and do not limit the present application in any way.

[0030] In this application, compared with the prior art, the present invention provides a real-time monitoring method and related device for cavity environment based on multimodal perception. Existing solutions mostly rely on single vision or offline modeling, which makes it difficult to synchronously process visual, tactile and positioning information during dynamic intervention, resulting in risk identification lag and path error accumulation. The present invention synchronously collects visual, contact force and electromagnetic positioning data in the cavity through hardware triggering or unified timestamp protocol, and extracts visual lesion features, dynamic tactile risk status and 6-degree-of-freedom posture after being refined by YOLO, mamba and extended Kalman filtering respectively; then, weighted voting is used to fuse the three types of features to generate real-time judgment, and the visual pixel coordinates and tactile sensor coordinates are mapped to a unified three-dimensional space through perspective-rigid body transformation to achieve associated marking; finally, lesion marking, retreat or path correction instructions are output in real time based on the judgment and association results. This process completes multimodal alignment, decision-making and feedback within a single frame, significantly reducing the amount of data and computing delay, making real-time perception, risk assessment and active warning possible during surgery, thereby improving the safety and efficiency of cavity intervention surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0032] The drawings herein are incorporated into the specification and constitute a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0033] Figure 1 This is a flow chart of a method for real-time monitoring of cavity environment based on multimodal sensing proposed in an embodiment of the present application.

[0034] Figure 2 This is a schematic diagram of a real-time monitoring device for cavity environment based on multimodal sensing proposed in an embodiment of the present application.

[0035] Figure 3 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In order to allow professionals in this technical field to more fully understand the technical solution of the present application, the technical solution of the present application will be explained in detail and clearly with the help of the accompanying drawings. It should be noted that the described embodiments are only some examples of the present application and do not represent all. Based on these embodiments, those skilled in the art can directly deduce all other possible implementation plans without engaging in creative thinking, and these are also included in the scope of protection of the present application.

[0037] In the specification, claims, and related drawings of this application, the terms "first," "second," and the like are used solely to distinguish between different elements and do not imply any particular order. Furthermore, the use of "including," "having," and their variations denotes non-exclusive inclusion. This means that if a process, method, system, product, or apparatus includes a series of steps or components, the process, method, system, product, or apparatus is not limited to the enumerated steps or components and may include other steps or components not listed, or other steps or units inherent to the process, method, system, product, or apparatus.

[0038] The “embodiment” mentioned in this document refers to any instance in which a particular feature, structure or characteristic is combined, and these instances may belong to at least one embodiment of the present application. The “embodiment” mentioned in this document does not necessarily refer to the same specific case, nor does it mean that they are independent or exclusive alternatives. It should be understood by those skilled in the art that the embodiments described herein can be used in conjunction with other embodiments. It should be understood that in this application, “at least one” includes one or more instances, “a plurality” means two or more instances, and “at least two” means two or more instances.

[0039] It should be understood that the method embodiment of the present application can also be implemented by a processor executing computer program code. The embodiment of the present application is described below in conjunction with the drawings in the embodiment of the present application.

[0040] See also Figure 1 , Figure 1A flow chart of a method for real-time monitoring of cavity environment based on multimodal sensing provided in an embodiment of the present application.

[0041] 101. Intraoperative multi-dimensional synchronous acquisition: Synchronous acquisition of visual data, contact force data and electromagnetic positioning data in the cavity environment.

[0042] In this embodiment, the synchronous acquisition is achieved through a hardware trigger signal or a unified timestamp protocol.

[0043] In another possible implementation, the synchronous acquisition may also be implemented based on other methods.

[0044] In another possible implementation, the synchronous acquisition may also acquire data of more modalities.

[0045] 102. Multimodal data preprocessing: Perform size change, cropping, denoising, distortion correction, and illumination compensation processing on the visual data to obtain refined visual data; perform amplification, filtering, and voltage-force value conversion processing on the contact force data to obtain refined contact force data; perform filtering and preliminary position and posture solution processing on the electromagnetic positioning data to obtain refined positioning data.

[0046] 103. Feature extraction: For the refined visual data, the YOLO model is used to extract edges, corners, texture features, color histograms, specific target detection results, and optical flow features as visual features. For the refined contact force data, the contact force data sequence of a preset time window length is processed based on the Mamba model, and dynamic tactile features including at least one state of normal contact, abnormal friction, excessive pressure, adhesion risk, and perforation risk are output. For the refined positioning data, a preset probe kinematic model is used as the state equation and the electromagnetic positioning data is used as the observation equation. The position and posture information is refined through an extended Kalman filter as the positioning feature.

[0047] In this embodiment, the YOLO model version is not lower than YOLOv8.

[0048] In this embodiment, the state equation is a rigid body 6-DOF kinematic model, and the observation equation is a nonlinear mapping of electromagnetic positioning data.

[0049] In this embodiment, the rigid body 6-DOF kinematic model includes a kinematic model of 3 position (X, Y, Z) degrees of freedom and 3 attitude (Roll, Pitch, Yaw) degrees of freedom.

[0050] In another possible implementation, the YOLO model can be replaced with other target detection methods.

[0051] 104. Perception fusion decision: Generate a lesion type or structural abnormality type judgment result based on the visual feature, generate a tactile interaction state classification result based on the dynamic tactile feature, generate a position deviation based on the positioning feature, and obtain a judgment result using a weighted voting rule; establish the three-dimensional spatial coordinates of the cavity based on the positioning feature, map the pixel coordinates in the visual feature to the three-dimensional spatial coordinates through perspective transformation, map the sensor spatial coordinates corresponding to the dynamic tactile feature to the three-dimensional spatial coordinates through rigid body transformation, associate and mark the visual feature and the dynamic tactile feature under the same spatial coordinates to obtain an association result.

[0052] In this embodiment, the judgment result is obtained by normalization after weighted voting.

[0053] 105. Decision output: Generate an environmental status decision based on the judgment result and the association result.

[0054] In this embodiment, when the lesion type or the structural abnormality type is detected, a lesion position marking instruction is generated; when the tactile interaction state classification result is adhesion risk or perforation risk, a retreat operation instruction is generated; when the position deviation exceeds the safety threshold, a path correction instruction is generated.

[0055] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0056] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0057] See also Figure 2 , Figure 2 This is a schematic diagram of a real-time monitoring device for cavity environment based on multimodal sensing proposed in an embodiment of the present application. The multimodal monitoring device 1 includes: a sensing unit 11, a data processing unit 12, a registration unit 13, and an output unit 14. Specifically:

[0058] The sensing unit 11 is used to synchronously collect visual data, contact force data and electromagnetic positioning data in the cavity environment;

[0059] The data processing unit 12 is used to perform size change, cropping, denoising, distortion correction, and illumination compensation on the visual data to obtain refined visual data; perform amplification, filtering, and voltage-force value conversion on the contact force data to obtain refined contact force data; perform filtering and preliminary position and posture solution processing on the electromagnetic positioning data to obtain refined positioning data; and is also used to use the YOLO model to extract edges, corners, texture features, color histograms, specific target detection results, and optical flow features from the refined visual data as visual features; for the refined contact force data, based on the mamba model, process a contact force data sequence of a preset time window length, and output dynamic tactile features including at least one state of normal contact, abnormal friction, excessive pressure, adhesion risk, and perforation risk; for the refined positioning data, use a preset probe kinematic model as the state equation and the electromagnetic positioning data as the observation equation, and refine the position and posture information through an extended Kalman filter as a positioning feature;

[0060] The registration unit 13 is configured to generate a lesion type or structural abnormality type judgment result based on the visual feature, generate a tactile interaction state classification result based on the dynamic tactile feature, generate a position deviation based on the positioning feature, and obtain a judgment result using a weighted voting rule; and is further configured to establish a three-dimensional spatial coordinate of the cavity based on the positioning feature, map the pixel coordinates in the visual feature to the three-dimensional spatial coordinates through perspective transformation, map the sensor spatial coordinates corresponding to the dynamic tactile feature to the three-dimensional spatial coordinates through rigid body transformation, and associate the visual feature and the dynamic tactile feature at the same spatial coordinates to obtain an association result.

[0061] The output unit 14 is configured to generate an environmental status decision based on the judgment result and the association result.

[0062] See also Figure 3 Figure 3 shows a schematic diagram of the hardware architecture of an electronic device described in an embodiment of the present application. The electronic device 2 is primarily composed of a processor 21 and a memory 22. Furthermore, the device may also include an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are interconnected via connecting components, which can be various interfaces, data cables, or communication buses, and are not specifically specified in the present embodiment.

[0063] Processor 21 may be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU may be single-core or multi-core. Optionally, processor 21 may comprise a processor group consisting of multiple GPUs, interconnected via one or more buses. Furthermore, the processor may be other types of processors, which are not specifically limited in this embodiment of the present application.

[0064] The memory 22 is designed to store computer program instructions and various program codes required to execute the solution of the present application. Optionally, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which are used to store relevant instructions and data.

[0065] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0066] It should be appreciated that in the embodiment of the present application, the memory 22 can store not only relevant instructions but also relevant data. The embodiment of the present application does not specify the specific data content stored in the memory.

[0067] It should be understood that FIG3 merely illustrates a simplified design of an electronic device. In actual use, the electronic device may also include other necessary components, such as varying numbers of input / output devices, processors, and memories. All electronic devices capable of implementing the embodiments of this application are within the scope of protection of this application.

[0068] Those skilled in the art will recognize that, according to the components and algorithm steps of each example described in the embodiments disclosed herein, these functions can be implemented by electronic hardware or by combining computer software and electronic hardware. Whether these functions are performed by hardware or software will be determined based on the specific application requirements and design limitations of the technical solution. Technicians can adopt different implementation methods according to the requirements of each specific application, but such implementation methods should not be considered to exceed the scope of protection of this application.

[0069] Professionals should understand that, for the sake of ease of description and simplification, the specific operating procedures of the above-mentioned systems, devices, and components can refer to the corresponding steps in the previous method embodiments and will not be repeated here. At the same time, professionals should also understand that each embodiment in this application has its own focus. For the sake of ease of description and simplification, the same or similar content may not be repeated in different embodiments. Therefore, if a part is not mentioned or not explained in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0070] In the several embodiments provided in this application, it should be recognized that the disclosed systems, devices and methods can also be implemented in other ways. For example, the device embodiments described are only exemplary, in which the division of the units is only a division of logical functions, and there may be different division methods in actual implementation. For example, multiple units or components may be merged or integrated into another system, or certain features may be omitted, or certain steps may not be performed. In addition, the connections between each other shown or discussed, whether direct or indirect, whether coupling or communication connection, may be implemented in electrical, mechanical or other forms through interfaces, devices or units.

[0071] Units described as independent components may or may not actually be physically separate; parts presented as units may or may not be physical entities; that is, they may be centralized in one location or distributed across multiple network nodes. Depending on actual needs, some or all of these units may be selected to achieve the objectives of this embodiment.

[0072] Furthermore, in the various embodiments of the present application, the various functional units may be integrated into a single processing unit, physically exist independently, or two or more units may be combined into a single unit. In the aforementioned embodiments, the relevant functions may be implemented in whole or in part through software, hardware, firmware, or any combination thereof. If software implementation is chosen, it may be implemented in whole or in part in the form of a computer program product. This computer program product comprises one or more computer instructions. When these instructions are loaded and executed on a computer, they will generate, in whole or in part, the processes or functions described in the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. These computer instructions may be stored in a computer-readable storage medium or transmitted via such a medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium may be any computer-accessible, usable medium, or a data storage facility such as a server or data center that integrates one or more usable media. These available media may include magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), semiconductor media (e.g., SSDs), etc. Those skilled in the art will appreciate that all or part of the process steps for implementing the above-described method embodiments can be accomplished through hardware associated with computer program instructions. These programs can be stored on computer-readable storage media. When executed, these programs will contain the processes for each of the above-described method embodiments. These storage media include, but are not limited to, various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A real-time monitoring method for cavity environment based on multimodal sensing, characterized in that: The following steps are involved: A. Synchronously collect visual data, contact force data, and electromagnetic positioning data in the cavity environment; B. performing resizing, cropping, denoising, distortion correction, and illumination compensation on the visual data to obtain refined visual data; performing amplification, filtering, and voltage-to-force conversion on the contact force data to obtain refined contact force data; and performing filtering and preliminary position and posture calculation on the electromagnetic positioning data to obtain refined positioning data; C. Using the YOLO model to extract edges, corners, texture features, color histograms, specific target detection results, and optical flow features from the refined visual data as visual features; Processing the refined contact force data in a contact force data sequence of a preset time window length based on a Mamba model, and outputting dynamic tactile features including at least one of normal contact, abnormal friction, excessive pressure, adhesion risk, and perforation risk; For the refined positioning data, a preset probe kinematic model is used as a state equation and the electromagnetic positioning data is used as an observation equation, and the position and posture information is refined by an extended Kalman filter as a positioning feature; D. generating a lesion type or structural abnormality type determination result based on the visual features, generating a tactile interaction state classification result based on the dynamic tactile features, generating a position deviation based on the positioning features, and obtaining a determination result using a weighted voting rule; establishing the three-dimensional spatial coordinates of the cavity based on the positioning features, mapping the pixel coordinates in the visual features to the three-dimensional spatial coordinates through perspective transformation, mapping the sensor spatial coordinates corresponding to the dynamic tactile features to the three-dimensional spatial coordinates through rigid body transformation, and associating and labeling the visual features and dynamic tactile features at the same spatial coordinates to obtain an association result; E. Generate an environmental status decision based on the judgment result and the association result.

2. The method according to claim 1, characterized in that The synchronous acquisition is achieved through a hardware trigger signal or a unified time stamp protocol.

3. The method according to claim 1, characterized in that The preliminary position and attitude solution processing is a 6-degree-of-freedom position and attitude solution processing based on the electromagnetic positioning principle, and the 6-degree-of-freedom position and attitude include posture data of 3 position (X, Y, Z) degrees of freedom and 3 attitude (Roll, Pitch, Yaw) degrees of freedom.

4. The method according to claim 1, wherein The state equation of the extended Kalman filter is defined as a rigid body 6-degree-of-freedom kinematic model, and the observation equation is defined as a nonlinear mapping function of the electromagnetic positioning data; the 6-degree-of-freedom kinematic model includes velocity data of 3 position (X, Y, Z) degrees of freedom and angular velocity data of 3 attitude (Roll, Pitch, Yaw) degrees of freedom.

5. The method according to claim 1, wherein The lesion types include ulcers, polyps, and bleeding spots; The structural abnormalities include stenosis, dilatation, and diverticula; The position deviation is calculated based on the Euclidean distance between the actual position of the probe and the planned path.

6. The method according to claim 1, characterized in that Generating an environmental status decision based on the judgment result and the correlation result includes: When the lesion type or the structural abnormality type is detected, generating a lesion position marking instruction; When the tactile interaction state classification result is adhesion risk or perforation risk, generating a retreat operation instruction; When the position deviation exceeds a safety threshold, a path correction instruction is generated.

7. A real-time monitoring device for cavity environment based on multimodal sensing, characterized in that: include: A sensing unit, used to synchronously collect visual data, contact force data, and electromagnetic positioning data in the cavity environment; a data processing unit configured to perform resizing, cropping, denoising, distortion correction, and illumination compensation processing on the visual data to obtain refined visual data; perform amplification, filtering, and voltage-to-force value conversion processing on the contact force data to obtain refined contact force data; perform filtering and preliminary position and posture solution processing on the electromagnetic positioning data to obtain refined positioning data; and further configured to extract edges, corners, texture features, color histograms, specific target detection results, and optical flow features from the refined visual data using a YOLO model as visual features; Processing the refined contact force data in a contact force data sequence of a preset time window length based on a Mamba model, and outputting dynamic tactile features including at least one of normal contact, abnormal friction, excessive pressure, adhesion risk, and perforation risk; For the refined positioning data, a preset probe kinematic model is used as a state equation and the electromagnetic positioning data is used as an observation equation, and the position and posture information is refined by an extended Kalman filter as a positioning feature; a registration unit, configured to generate a lesion type or structural abnormality type judgment result based on the visual feature, generate a tactile interaction state classification result based on the dynamic tactile feature, generate a position deviation based on the positioning feature, and obtain a judgment result using a weighted voting rule; and further configured to establish a three-dimensional spatial coordinate of the cavity based on the positioning feature, map the pixel coordinates in the visual feature to the three-dimensional spatial coordinates through perspective transformation, map the sensor spatial coordinates corresponding to the dynamic tactile feature to the three-dimensional spatial coordinates through rigid body transformation, and associate and mark the visual feature and the dynamic tactile feature at the same spatial coordinates to obtain an association result; An output unit is used to generate an environmental status decision based on the judgment result and the association result.

8. An electronic device, characterized in that: include: A processor and a storage unit, the storage unit is used to store computer program code, the code includes computer instructions, when the processor executes these instructions, the electronic device performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images