Accident state logic control multi-perception somatosensory interaction method based on visual identification

Abstract

The invention belongs to the field of chemical accident sensing interaction control and virtual reality drilling, and particularly discloses an accident state logic control multi-perception somatosensory interaction method based on visual identification, which is used for chemical accident control simulation training and virtual drilling. According to the accident state logic control multi-perception somatosensory interaction method based on visual identification, firstly, somatosensory equipment capable of expanding multiple perception types is adopted, and dynamic expansion, integration and control of vibration, smoke, water mist, hot air and other various types of effectors are achieved; secondly, an image-based gesture recognition and VR interaction method is adopted to replace a VR interaction handle device, and interaction operation in a VR virtual scene is realized. And finally, by adopting a polymorphic sensing dynamic logic control method based on an accident evolution path, according to the accident evolution path key node and built-in logic judgment information, in combination with interactive operation, carrying out evolution development of a VR virtual scene and control of sensing equipment.

Description

Technical Field

[0001] The present invention belongs to the field of chemical accident sensor interaction control and virtual reality drill, and in particular relates to a multi-sensory body interaction method for accident state logic control based on visual recognition. Background Art

[0002] In recent years, chemical accidents have occurred frequently both at home and abroad. Cases of accidents occurring or even continuing to expand due to ineffective initial emergency response are not uncommon. Therefore, how to effectively improve the emergency response capabilities of enterprise employees in the face of emergencies and improve the initial operational handling of grassroots employees is an issue that needs to be addressed urgently.

[0003] With the continuous development of three-dimensional simulation and virtual reality technology, new technologies and methods have been provided for emergency simulation drills and operational disposal of chemical accidents, which is conducive to improving the emergency rescue skills and emergency disposal capabilities of enterprise employees.

[0004] At present, simulation training based on virtual reality technology is mainly based on pure virtual software with interactive devices, which has the following main disadvantages:

[0005] (1) Existing VR technology builds a three-dimensional model of a real scene. Operators use virtual reality visual equipment to roam and interact in the virtual scene. This is mainly based on visual perception and lacks other multi-sensory experiences in the real world.

[0006] (2) Existing virtual simulation software mostly relies on VR handle devices to interact and provide feedback on human behavior in the form of buttons, lacking the experience of real operating behavior, and lacks the technology to integrate hand motion recognition feedback with VR operating scenario mapping.

[0007] (3) Existing VR simulation systems are mostly developed based on fixed script-based operating procedures, and the consequences of accidents are mostly displayed schematically. There is a lack of dynamic logic control of VR scenarios based on different behavioral feedbacks of the accident evolution path.

[0008] The present invention provides a multi-sensory somatosensory interaction method for accident status logic control based on visual recognition, invents a multi-sensory somatosensory interaction experience device that can expand multi-sensory and effect-generating equipment, and uses visual sensors to perform visual recognition of key operation gestures of personnel in emergency response processes, ultimately achieving VR accident scenario dynamic evolution control and multi-sensory immersive operation experience based on gesture operation feedback. Summary of the invention

[0009] The purpose of the present invention is to provide a multi-sensory somatosensory interaction method for accident state logic control based on visual recognition, which effectively solves the problems that existing VR equipment and systems lack multi-sensory functions, are limited by interactive handles and lack recognition of key operating gestures in chemical accidents, and fail to realize multi-state perception dynamic logic control based on accident evolution paths.

[0010] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0011] A multi-sensory somatosensory interaction method for accident status logic control based on visual recognition. First, somatosensory equipment that can expand multiple perception types is used to achieve dynamic expansion, integration and control of vibration, smoke, water mist, hot air and other types of effectors.

[0012] Secondly, image-based gesture recognition and VR interaction methods are used to replace VR interactive handle devices to realize interactive operations in VR virtual scenes.

[0013] Finally, a polymorphic perception dynamic logic control method based on the accident evolution path is adopted. According to the key nodes of the accident evolution path and the built-in logic judgment information, the evolution of the VR virtual scene and the control of the perception equipment are carried out in combination with interactive operations.

[0014] Furthermore, the somatosensory equipment consists of five parts: an assembleable cabin carrier, a computing and control module, a VR immersion terminal, an environmental effect module and a sensor interaction module. The assembleable cabin carrier reserves installation interfaces and spaces for sensors, effectors and control panels.

[0015] Furthermore, the image-based gesture recognition and VR interaction method specifically includes:

[0016] S1. Install three image sensors on the somatosensory equipment for photographing all spaces that can be operated by the operator's hands in the somatosensory equipment from different angles.

[0017] S2. Establish a key operation gesture library for emergency response to chemical accident scenarios.

[0018] S3. Establish an operation event mapping library to establish a mapping relationship between each operation gesture and the associated event.

[0019] S4. Identify and match the operator's gestures with the actions in the key operation gesture library, and identify and confirm the interaction between the virtual gestures and virtual scene objects based on spatial positioning based on the reconstruction of the three-dimensional hand model.

[0020] S5. Matching associated events in an operation event mapping library according to the recognition result of the interaction between the operator's gesture operation and the virtual space object.

[0021] S6. Control the display of the accident evolution results in the VR scene according to the event matching results, and realize the virtual scene interaction based on key operation gesture recognition.

[0022] Furthermore, the multi-state perception dynamic logic control method based on the accident evolution path specifically includes:

[0023] A1. Establish an evolution path for chemical scene accidents, and associate key elements related to VR operation and display at key nodes of the path.

[0024] A2. Establish the correlation between the vision-based gesture recognition results and the key nodes of the evolution path.

[0025] A3. Match the operation actions of gesture recognition with the key elements of the key nodes of the evolution path, derive the three-dimensional evolution and display results of the VR scene based on logical judgment, and display them in the form of visual feedback of the VR display device.

[0026] A4. According to the evolution of VR virtual scenes, match the control state of the environmental effect simulation equipment corresponding to the VR display effect, dynamically control the occurrence of related effect equipment, and realize multi-sensory effect feedback.

[0027] Further, in step S2, the key operation gesture library records characteristic information of key operation actions including but not limited to opening valves, closing valves, valve opening, pump start and stop, blind plate plugging, fire monitor operation, wearing of protective equipment and gas monitoring.

[0028] Furthermore, in step S1, the hand model is reconstructed and spatially positioned based on feature points and binocular vision algorithm using any two unobstructed hand posture images taken at different angles.

[0029] Furthermore, in step A1, the key elements include but are not limited to: operating role, spatial position coordinates, interactive operation object, three-dimensional model number of interactive operation object, spatial position of interactive operation object, spatial distance between the interactive operation object, key operation action, operation limit time, operation result state code, VR display effect corresponding to the operation result state code, and control state of environmental effect simulation device corresponding to the VR display effect.

[0030] Furthermore, the operation and control module is the storage, calculation, rendering and control unit of the VR system operation, multi-device control, gesture recognition and scene evolution algorithm, and is used to provide hardware and computing power support.

[0031] Furthermore, the VR immersion terminal is an integrated VR display device for providing support for the display of virtual three-dimensional scenes.

[0032] Furthermore, the environmental effect module is a combination module of multiple sensory effectors including vibration, heating, smoke, water mist, hot wind and cold wind, which is used to provide effect perception for changes in the virtual accident environment. The environmental effect module is expanded and accessed based on an assembler cabin-type carrier.

[0033] Furthermore, the sensor interaction module is a combination module of multiple perception sensors including image sensing, temperature sensing and gas sensing, which is used to provide perception collection of human body movements and environmental conditions. The sensor interaction module is expanded and accessed based on an assembleable cabin-type carrier.

[0034] The beneficial technical effects of the present invention are:

[0035] (1) The somatosensory equipment designed by the present invention can expand multiple perception types and achieve multiple real perception effects. Compared with the existing VR interactive equipment that only has vision and hearing, it provides more experience perceptions and increases the multi-perception immersive experience of operators in virtual scenes.

[0036] (2) The somatosensory equipment designed by the present invention can be expanded to multiple sensing types, and can realize the expansion of sensor devices and effect devices. New sensor effect devices can be added at any time during use, which increases the expandability and compatibility of the overall system and avoids the workload of repeated design and processing for adding new functional equipment.

[0037] (3) The somatosensory equipment designed by the present invention can be expanded to multiple types of perception, and the calculation and control modules therein can perform data collection, algorithm calculation, and signal control of multiple sensors, effectors, and various embedded algorithms, providing computing power and hardware support for equipment expansion and functional calculations.

[0038] (4) The present invention is based on image gesture recognition and VR interaction, and defines and identifies special operation gestures for various devices, equipment, emergency response, etc. in the accident emergency response process in chemical scenes. Compared with the operation gestures in normal life, the operations in accident response in chemical scenes have their own specificity, which cannot be accurately defined and identified by existing methods.

[0039] (5) The present invention uses image-based gesture recognition and VR interaction to change the existing VR interaction mode, allowing interaction in a VR virtual scene in a manner that is closer to normal human operation, thus getting rid of the limitations of handheld interactive devices and making interaction easier. At the same time, it opens up dynamic interactive control between gesture recognition and accident evolution judgment, VR feedback display, and environmental effect generation equipment.

[0040] (6) The present invention is based on the polymorphic perception dynamic logic control of the accident evolution path, which changes the current status of the existing VR system and fixed script process development of courses. According to the operator's gesture recognition results, the accident evolution path with logical judgment and various key information is associated to dynamically evolve the VR virtual scene. At the same time, the effect of the perception equipment is synchronously controlled according to the development of the accident in the VR virtual scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 It is a schematic diagram of the design of the somatosensory equipment of the present invention that can expand multiple perception types.

[0043] In the figure: 1. Removable segmented baffle; 2. Left and right hand interaction device assembly socket; 3. Vibration device assembly socket; 4. Center console assembly socket; 5. Controller assembly socket; 6. Leg sensing assembly socket; 7. Standardized peripheral interface. DETAILED DESCRIPTION

[0044] The present invention proposes a somatosensory equipment that can expand multiple perception types, improve the problem of the lack of sensory types in existing VR interactive equipment, and increase the immersion of users. An image-based gesture recognition and VR interaction method is proposed to change the existing VR interaction mode, get rid of the limitations of handheld interactive equipment, and use gesture operations consistent with real operations to complete the interaction in the VR virtual scene. A multi-state perception dynamic logic control method based on the accident evolution path is proposed. The control method combines the operation feedback of gesture recognition, and performs the evolution display of the VR scene according to the logical judgment of each key node of the accident in the method, and can realize the automatic control of the multi-sensory effect device for scene changes. Thereby, the dynamic evolution control of VR accident scenarios based on gesture operation feedback and the multi-perception immersive operation experience are realized, providing a new way and means for simulation training and virtual drills. The following is a detailed description in conjunction with the accompanying drawings and embodiments.

[0045] Embodiment 1:

[0046] A multi-sensory somatosensory interaction method for accident status logic control based on visual recognition. First, somatosensory equipment that can expand multiple perception types is used to achieve dynamic expansion, integration and control of vibration, smoke, water mist, hot air and other types of effectors.

[0047] The somatosensory equipment consists of five parts: an assembleable cabin-type carrier, a computing and control module, a VR immersive terminal, an environmental effect module and a sensor interaction module.

[0048] The assembleable cabin carrier reserves installation interfaces and spaces for sensors, effectors and control panels.

[0049] The operation and control module is the storage, calculation, rendering and control unit for VR system operation, multi-device control, gesture recognition and scene evolution algorithm, and is used to provide hardware and computing power support.

[0050] The VR immersive terminal is an integrated VR display device used to provide support for the display of virtual three-dimensional scenes.

[0051] The environmental effect module is a combination module of multiple sensory effectors including vibration, heating, smoke, water mist, hot wind and cold wind, which is used to provide effect perception for changes in virtual accident environment. The environmental effect module is expanded and accessed based on an assembler cabin-type carrier.

[0052] The sensor interaction module is a combination module of multiple perception sensors including image sensing, temperature sensing and gas sensing, which is used to provide perception and collection of human body movements and environmental conditions. The sensor interaction module is expanded and accessed based on an assembleable cabin-type carrier.

[0053] Secondly, image-based gesture recognition and VR interaction methods are used to replace VR interactive handle devices to realize interactive operations in VR virtual scenes.

[0054] Finally, a polymorphic perception dynamic logic control method based on the accident evolution path is adopted. According to the key nodes of the accident evolution path and the built-in logic judgment information, the evolution of the VR virtual scene and the control of the perception equipment are carried out in combination with interactive operations.

[0055] The somatosensory equipment designed in this embodiment can expand multiple perception types and achieve multiple real perception effects. Compared with the existing VR interactive equipment that only has vision and hearing, it provides more experience perceptions and increases the operator's multi-perception immersive experience in the virtual scene.

[0056] The somatosensory equipment designed in this embodiment can be expanded with multiple perception types, and can realize the expansion of sensor devices and effect devices. New sensor effect devices can be added at any time during use, which increases the expandability and compatibility of the overall system and avoids the workload of repeated design and processing for adding new functional equipment.

[0057] The somatosensory equipment designed in this embodiment can be expanded to various perception types, and the calculation and control modules therein can perform data collection, algorithm calculation, and signal control of multiple sensors, effectors, and various embedded algorithms, providing computing power and hardware support for equipment expansion and functional calculations.

[0058] This embodiment is based on image gesture recognition and VR interaction, and defines and identifies special operation gestures for various devices, equipment, emergency disposal, etc. in the accident emergency disposal process in chemical scenes. Compared with the operation gestures in normal life, the operations in accident disposal in chemical scenes have their own specificity, which cannot be accurately defined and identified by existing methods.

[0059] This embodiment uses image-based gesture recognition and VR interaction to change the existing VR interaction method, and interacts in a VR virtual scene in a way that is closer to normal human operation, getting rid of the limitations of handheld interactive devices, making interaction easier, and at the same time opening up dynamic interactive control between gesture recognition and accident evolution judgment, VR feedback display, and environmental effect generation equipment.

[0060] This embodiment is based on the polymorphic perception dynamic logic control of the accident evolution path, which changes the current status of the existing VR system and the fixed script process development of the course. It associates the accident evolution path with logical judgment and various key information according to the operator's gesture recognition results, and dynamically evolves the VR virtual scene. At the same time, the effect of the perception equipment is synchronously controlled according to the development of the accident in the VR virtual scene.

[0061] Embodiment 2:

[0062] A multi-sensory somatosensory interaction method for accident status logic control based on visual recognition. First, somatosensory equipment that can expand multiple perception types is used to achieve dynamic expansion, integration and control of vibration, smoke, water mist, hot air and other types of effectors.

[0063] The somatosensory equipment consists of five parts: an assembleable cabin-type carrier, a computing and control module, a VR immersive terminal, an environmental effect module and a sensor interaction module.

[0064] The assembleable cabin carrier reserves installation interfaces and spaces for sensors, effectors and control panels. Figure 1 As shown, the assembleable cabin-type carrier serves as the equipment shell, with a detachable segmented baffle 1, sockets (including sensor and effect device assembly sockets, left and right hand interaction device assembly sockets 2, vibration device assembly sockets 3, center console assembly sockets 4, controller assembly sockets 5 and leg sensing assembly sockets 6) and slides, to reserve standardized peripheral interfaces 7 and space for components such as control chips, storage devices, sensors, effectors, control panels and VR headsets, to achieve plug-in combination of multiple types of equipment, thereby realizing the expansion, integration and control of multiple sensor effectors.

[0065] The operation and control module is the storage, calculation, rendering and control unit for VR system operation, multi-device control, gesture recognition and scene evolution algorithm, and is used to provide hardware and computing power support.

[0066] In this embodiment, the operation and control module is composed of an operation unit, a storage unit, a multi-device control unit, a VR rendering unit, and a signal transmission unit, etc. It runs the accident evolution model, gesture recognition algorithm, scene and logic drive algorithm, reads data or performs corresponding operations through the lower computer control algorithm, and presents the calculation results to the operator in the form of VR immersive scene display. The operation and control module can be combined with the assembleable cabin carrier in a plug-in manner, and a rich peripheral interface is reserved to carry out peripheral communication and drive tests.

[0067] The VR immersive terminal is an integrated VR display device that provides support for the display of virtual three-dimensional scenes. The scene is rendered and the operation process is controlled by the computing and control module to achieve a VR immersive simulation experience of accident handling.

[0068] The environmental effect module is a combination module of multiple sensory effectors including vibration, heating, smoke, water mist, hot wind and cold wind, which is used to realize the effect perception of various accident environment changes. At the same time, the environmental effect module is expanded and accessed based on the assembleable cabin-type carrier. In this embodiment, the assembleable cabin-type carrier also reserves other effect device assembly sockets and communication interfaces for carrying out configuration integration of other effect devices.

[0069] The sensor interaction module is a combination module of multiple perception sensors including image sensing, temperature sensing and gas sensing, which is used to provide perception and collection of human body movements and environmental conditions. The sensor interaction module is expanded and accessed based on an assembleable cabin-type carrier.

[0070] In this embodiment, the sensor interaction module includes two types of sensors, temperature and gas, which are used to feedback the changes in environmental parameters in real time when smoke is emitted and heat is generated, and to feedback in the VR virtual scene, forming a closed loop of VR simulation and effect perception. At the same time, other sensor equipment assembly sockets and communication interfaces should be reserved on the assembleable cabin carrier for carrying out configuration tests of other sensor equipment.

[0071] Secondly, image-based gesture recognition and VR interaction methods are used to replace VR interactive handle devices to realize interactive operations in VR virtual scenes.

[0072] Finally, a polymorphic perception dynamic logic control method based on the accident evolution path is adopted. According to the key nodes of the accident evolution path and the built-in logic judgment information, the evolution of the VR virtual scene and the control of the perception equipment are carried out in combination with interactive operations.

[0073] The somatosensory equipment designed in this embodiment can expand multiple perception types and achieve multiple real perception effects. Compared with the existing VR interactive equipment that only has vision and hearing, it provides more experience perceptions and increases the operator's multi-perception immersive experience in the virtual scene.

[0074] The somatosensory equipment designed in this embodiment can be expanded with multiple perception types, and can realize the expansion of sensor devices and effect devices. New sensor effect devices can be added at any time during use, which increases the expandability and compatibility of the overall system and avoids the workload of repeated design and processing for adding new functional equipment.

[0075] The somatosensory equipment designed in this embodiment can be expanded to various perception types, and the calculation and control modules therein can perform data collection, algorithm calculation, and signal control of multiple sensors, effectors, and various embedded algorithms, providing computing power and hardware support for equipment expansion and functional calculations.

[0076] This embodiment is based on image gesture recognition and VR interaction, and defines and identifies special operation gestures for various devices, equipment, emergency disposal, etc. in the accident emergency disposal process in chemical scenes. Compared with the operation gestures in normal life, the operations in accident disposal in chemical scenes have their own specificity, which cannot be accurately defined and identified by existing methods.

[0077] This embodiment uses image-based gesture recognition and VR interaction to change the existing VR interaction method, and interacts in a VR virtual scene in a way that is closer to normal human operation, getting rid of the limitations of handheld interactive devices, making interaction easier, and at the same time opening up dynamic interactive control between gesture recognition and accident evolution judgment, VR feedback display, and environmental effect generation equipment.

[0078] This embodiment is based on the polymorphic perception dynamic logic control of the accident evolution path, which changes the current status of the existing VR system and the fixed script process development of the course. It associates the accident evolution path with logical judgment and various key information according to the operator's gesture recognition results, and dynamically evolves the VR virtual scene. At the same time, the effect of the perception equipment is synchronously controlled according to the development of the accident in the VR virtual scene.

[0079] Embodiment 3:

[0080] A multi-sensory somatosensory interaction method for accident status logic control based on visual recognition. First, somatosensory equipment that can expand multiple perception types is used to achieve dynamic expansion, integration and control of vibration, smoke, water mist, hot air and other types of effectors.

[0081] The somatosensory equipment consists of five parts: an assembleable cabin-type carrier, a computing and control module, a VR immersive terminal, an environmental effect module and a sensor interaction module.

[0082] The assembleable cabin carrier reserves installation interfaces and spaces for sensors, effectors and control panels. Figure 1As shown, the assembleable cabin-type carrier serves as the equipment shell, with a detachable segmented baffle 1, sockets (including sensor and effect device assembly sockets, left and right hand interaction device assembly sockets 2, vibration device assembly sockets 3, center console assembly sockets 4, controller assembly sockets 5 and leg sensing assembly sockets 6) and slides, to reserve standardized peripheral interfaces 7 and space for components such as control chips, storage devices, sensors, effectors, control panels and VR headsets, to achieve plug-in combination of multiple types of equipment, thereby realizing the expansion, integration and control of multiple sensor effectors.

[0083] The operation and control module is the storage, calculation, rendering and control unit for VR system operation, multi-device control, gesture recognition and scene evolution algorithm, and is used to provide hardware and computing power support.

[0084] In this embodiment, the operation and control module is composed of an operation unit, a storage unit, a multi-device control unit, a VR rendering unit, and a signal transmission unit, etc. It runs the accident evolution model, gesture recognition algorithm, scene and logic drive algorithm, reads data or performs corresponding operations through the lower computer control algorithm, and presents the calculation results to the operator in the form of VR immersive scene display. The operation and control module can be combined with the assembleable cabin carrier in a plug-in manner, and a rich peripheral interface is reserved to carry out peripheral communication and drive tests.

[0085] The VR immersive terminal is an integrated VR display device that provides support for the display of virtual three-dimensional scenes. The scene is rendered and the operation process is controlled by the computing and control module to achieve a VR immersive simulation experience of accident handling.

[0086] The environmental effect module is a combination module of multiple sensory effectors including vibration, heating, smoke, water mist, hot wind and cold wind, which is used to realize the effect perception of various accident environment changes. At the same time, the environmental effect module is expanded and accessed based on the assembleable cabin-type carrier. In this embodiment, the assembleable cabin-type carrier also reserves other effect device assembly sockets and communication interfaces for carrying out configuration integration of other effect devices.

[0087] The sensor interaction module is a combination module of multiple perception sensors including image sensing, temperature sensing and gas sensing, which is used to provide perception and collection of human body movements and environmental conditions. The sensor interaction module is expanded and accessed based on an assembleable cabin-type carrier.

[0088] In this embodiment, the sensor interaction module includes two types of sensors, temperature and gas, which are used to feedback the changes in environmental parameters in real time when smoke is emitted and heat is generated, and to feedback in the VR virtual scene, forming a closed loop of VR simulation and effect perception. At the same time, other sensor equipment assembly sockets and communication interfaces should be reserved on the assembleable cabin carrier for carrying out configuration tests of other sensor equipment.

[0089] Secondly, image-based gesture recognition and VR interaction methods are used to replace VR interactive handle devices to realize interactive operations in VR virtual scenes.

[0090] The image-based gesture recognition and VR interaction method specifically includes: S1, installing three image sensors on the somatosensory equipment, and photographing all spaces that the operator's hands can operate in the somatosensory equipment from three different angles to avoid incomplete capture of hand information due to occlusion from a single angle.

[0091] S2. The hand posture images taken without obstruction at any two different angles can be used to reconstruct the hand model and spatially locate the hand based on feature points and binocular vision algorithm.

[0092] S3. Establish a key operation gesture library for emergency response to chemical accidents. The key operation gesture library records characteristic information of key operation actions including but not limited to opening valves, closing valves, valve opening, pump start and stop, blind plate extraction and plugging, fire monitor operation, wearing protective equipment and gas monitoring.

[0093] S4. Establish an operation event mapping library to establish a mapping relationship between each operation gesture and the associated event.

[0094] S5: Hand behavior of the operator entering the operating range of the interactive object (equipment, etc.) and performing a key operation.

[0095] S6. Identify and match the operator's gestures with the actions in the key operation gesture library.

[0096] S7. Reconstruct the three-dimensional model of the hand, spatial positioning, and modeling of dynamic gesture movements based on spatial transformation based on visual sensors.

[0097] S8. Based on the recognition of the gesture action in step S6 and the modeling and spatial positioning of the dynamic gesture action in step S7, identify whether spatial interaction is performed with an object (device, etc.) in the virtual scene and what kind of interaction operation is performed.

[0098] S9. Matching associated events in an operation event mapping library according to the recognition result of the interaction between the operator's gesture operation and the virtual space object.

[0099] S10. Control the display of the accident evolution results in the VR scene according to the event matching results, and realize the virtual scene interaction based on key operation gesture recognition.

[0100] Finally, a polymorphic perception dynamic logic control method based on the accident evolution path is adopted. According to the key nodes of the accident evolution path and the built-in logic judgment information, the evolution of the VR virtual scene and the control of the perception equipment are carried out in combination with interactive operations.

[0101] The somatosensory equipment designed in this embodiment can expand multiple perception types and achieve multiple real perception effects. Compared with the existing VR interactive equipment that only has vision and hearing, it provides more experience perceptions and increases the operator's multi-perception immersive experience in the virtual scene.

[0102] The somatosensory equipment designed in this embodiment can be expanded with multiple perception types, and can realize the expansion of sensor devices and effect devices. New sensor effect devices can be added at any time during use, which increases the expandability and compatibility of the overall system and avoids the workload of repeated design and processing for adding new functional equipment.

[0103] The somatosensory equipment designed in this embodiment can be expanded to various perception types, and the calculation and control modules therein can perform data collection, algorithm calculation, and signal control of multiple sensors, effectors, and various embedded algorithms, providing computing power and hardware support for equipment expansion and functional calculations.

[0104] This embodiment is based on image gesture recognition and VR interaction, and defines and identifies special operation gestures for various devices, equipment, emergency disposal, etc. in the accident emergency disposal process in chemical scenes. Compared with the operation gestures in normal life, the operations in accident disposal in chemical scenes have their own specificity, which cannot be accurately defined and identified by existing methods.

[0105] This embodiment uses image-based gesture recognition and VR interaction to change the existing VR interaction method, and interacts in a VR virtual scene in a way that is closer to normal human operation, getting rid of the limitations of handheld interactive devices, making interaction easier, and at the same time opening up dynamic interactive control between gesture recognition and accident evolution judgment, VR feedback display, and environmental effect generation equipment.

[0106] This embodiment is based on the polymorphic perception dynamic logic control of the accident evolution path, which changes the current status of the existing VR system and the fixed script process development of the course. It associates the accident evolution path with logical judgment and various key information according to the operator's gesture recognition results, and dynamically evolves the VR virtual scene. At the same time, the effect of the perception equipment is synchronously controlled according to the development of the accident in the VR virtual scene.

[0107] Embodiment 4:

[0108] A multi-sensory somatosensory interaction method for accident status logic control based on visual recognition. First, somatosensory equipment that can expand multiple perception types is used to achieve dynamic expansion, integration and control of vibration, smoke, water mist, hot air and other types of effectors.

[0109] The somatosensory equipment consists of five parts: an assembleable cabin-type carrier, a computing and control module, a VR immersive terminal, an environmental effect module and a sensor interaction module.

[0110] The assembleable cabin carrier reserves installation interfaces and spaces for sensors, effectors and control panels. Figure 1 As shown, the assembleable cabin-type carrier serves as the equipment shell, with a detachable segmented baffle 1, sockets (including sensor and effect device assembly sockets, left and right hand interaction device assembly sockets 2, vibration device assembly sockets 3, center console assembly sockets 4, controller assembly sockets 5 and leg sensing assembly sockets 6) and slides, to reserve standardized peripheral interfaces 7 and space for components such as control chips, storage devices, sensors, effectors, control panels and VR headsets, to achieve plug-in combination of multiple types of equipment, thereby realizing the expansion, integration and control of multiple sensor effectors.

[0111] The operation and control module is the storage, calculation, rendering and control unit for VR system operation, multi-device control, gesture recognition and scene evolution algorithm, and is used to provide hardware and computing power support.

[0112] In this embodiment, the operation and control module is composed of an operation unit, a storage unit, a multi-device control unit, a VR rendering unit, and a signal transmission unit, etc. It runs the accident evolution model, gesture recognition algorithm, scene and logic drive algorithm, reads data or performs corresponding operations through the lower computer control algorithm, and presents the calculation results to the operator in the form of VR immersive scene display. The operation and control module can be combined with the assembleable cabin carrier in a plug-in manner, and a rich peripheral interface is reserved to carry out peripheral communication and drive tests.

[0113] The VR immersive terminal is an integrated VR display device that provides support for the display of virtual three-dimensional scenes. The scene is rendered and the operation process is controlled by the computing and control module to achieve a VR immersive simulation experience of accident handling.

[0114] The environmental effect module is a combination module of multiple sensory effectors including vibration, heating, smoke, water mist, hot wind and cold wind, which is used to realize the effect perception of various accident environment changes. At the same time, the environmental effect module is expanded and accessed based on the assembleable cabin-type carrier. In this embodiment, the assembleable cabin-type carrier also reserves other effect device assembly sockets and communication interfaces for carrying out configuration integration of other effect devices.

[0115] The sensor interaction module is a combination module of multiple perception sensors including image sensing, temperature sensing and gas sensing, which is used to provide perception and collection of human body movements and environmental conditions. The sensor interaction module is expanded and accessed based on an assembleable cabin-type carrier.

[0116] In this embodiment, the sensor interaction module includes two types of sensors, temperature and gas, which are used to feedback the changes in environmental parameters in real time when smoke is emitted and heat is generated, and to feedback in the VR virtual scene, forming a closed loop of VR simulation and effect perception. At the same time, other sensor equipment assembly sockets and communication interfaces should be reserved on the assembleable cabin carrier for carrying out configuration tests of other sensor equipment.

[0117] Secondly, image-based gesture recognition and VR interaction methods are used to replace VR interactive handle devices to realize interactive operations in VR virtual scenes.

[0118] The image-based gesture recognition and VR interaction method specifically includes: S1, installing three image sensors on the somatosensory equipment, and photographing all spaces that the operator's hands can operate in the somatosensory equipment from three different angles to avoid incomplete capture of hand information due to occlusion from a single angle.

[0119] S2. The hand posture images taken without obstruction at any two different angles can be used to reconstruct the hand model and spatially locate the hand based on feature points and binocular vision algorithm.

[0120] S3. Establish a key operation gesture library for emergency response to chemical accidents. The key operation gesture library records characteristic information of key operation actions including but not limited to opening valves, closing valves, valve opening, pump start and stop, blind plate extraction and plugging, fire monitor operation, wearing protective equipment and gas monitoring.

[0121] S4. Establish an operation event mapping library to establish a mapping relationship between each operation gesture and the associated event.

[0122] S5: Hand behavior of the operator entering the operating range of the interactive object (equipment, etc.) and performing a key operation.

[0123] S6. Identify and match the operator's gestures with the actions in the key operation gesture library.

[0124] S7. Reconstruct the three-dimensional model of the hand, spatial positioning, and modeling of dynamic gesture movements based on spatial transformation based on visual sensors.

[0125] S8. Based on the recognition of the gesture action in step S6 and the modeling and spatial positioning of the dynamic gesture action in step S7, identify whether spatial interaction is performed with an object (device, etc.) in the virtual scene and what kind of interaction operation is performed.

[0126] S9. Matching associated events in an operation event mapping library according to the recognition result of the interaction between the operator's gesture operation and the virtual space object.

[0127] S10. Control the display of the accident evolution results in the VR scene according to the event matching results, and realize the virtual scene interaction based on key operation gesture recognition.

[0128] Finally, a polymorphic perception dynamic logic control method based on the accident evolution path is adopted. According to the key nodes of the accident evolution path and the built-in logic judgment information, the evolution of the VR virtual scene and the control of the perception equipment are carried out in combination with interactive operations.

[0129] The multi-state perception dynamic logic control method based on the accident evolution path specifically includes: A1, establishing the chemical scene accident evolution path.

[0130] A2. Associate key elements related to VR operation and display at key nodes in the accident evolution path.

[0131] The key elements include but are not limited to: operating role, spatial position coordinates, interactive operation object, three-dimensional model number of interactive operation object, spatial position of interactive operation object, spatial distance between interactive operation object and interactive operation object, key operation action, operation limit time, operation result status code, VR display effect corresponding to the operation result status code, and control status of environmental effect simulation equipment corresponding to the VR display effect.

[0132] A3. Establish the correlation between the vision-based gesture recognition results and the key nodes of the evolution path.

[0133] A4. The operator's hand behavior when performing a key operation.

[0134] A5. Identify the operator's key operation gestures and match the operation actions of gesture recognition with the key elements of the key nodes of the evolution path.

[0135] A6. The three-dimensional evolution and display results of the VR scene are obtained based on logical judgment and displayed in the form of visual feedback from VR display devices.

[0136] A7. According to the evolution of VR virtual scenes, match the control state of the environmental effect simulation equipment corresponding to the VR display effect.

[0137] A8. According to the control status of the environmental effect simulation equipment, dynamically control the occurrence of related effect equipment to achieve feedback of multi-sensory effects.

[0138] A9. Ultimately, dynamic evolution control of VR accident scenarios and multi-sensory immersive operation experience based on gesture operation feedback are realized.

[0139] The somatosensory equipment designed in this embodiment can expand multiple perception types and achieve multiple real perception effects. Compared with the existing VR interactive equipment that only has vision and hearing, it provides more experience perceptions and increases the operator's multi-perception immersive experience in the virtual scene.

[0140] The somatosensory equipment designed in this embodiment can be expanded with multiple perception types, and can realize the expansion of sensor devices and effect devices. New sensor effect devices can be added at any time during use, which increases the expandability and compatibility of the overall system and avoids the workload of repeated design and processing for adding new functional equipment.

[0141] The somatosensory equipment designed in this embodiment can be expanded to various perception types, and the calculation and control modules therein can perform data collection, algorithm calculation, and signal control of multiple sensors, effectors, and various embedded algorithms, providing computing power and hardware support for equipment expansion and functional calculations.

[0142] This embodiment is based on image gesture recognition and VR interaction, and defines and identifies special operation gestures for various devices, equipment, emergency disposal, etc. in the accident emergency disposal process in chemical scenes. Compared with the operation gestures in normal life, the operations in accident disposal in chemical scenes have their own specificity, which cannot be accurately defined and identified by existing methods.

[0143] This embodiment uses image-based gesture recognition and VR interaction to change the existing VR interaction method, and interacts in a VR virtual scene in a way that is closer to normal human operation, getting rid of the limitations of handheld interactive devices, making interaction easier, and at the same time opening up dynamic interactive control between gesture recognition and accident evolution judgment, VR feedback display, and environmental effect generation equipment.

[0144] This embodiment is based on the polymorphic perception dynamic logic control of the accident evolution path, which changes the current status of the existing VR system and the fixed script process development of the course. It associates the accident evolution path with logical judgment and various key information according to the operator's gesture recognition results, and dynamically evolves the VR virtual scene. At the same time, the effect of the perception equipment is synchronously controlled according to the development of the accident in the VR virtual scene.

[0145] Embodiment 5:

[0146] A multi-sensory somatosensory interaction method for accident status logic control based on visual recognition. First, somatosensory equipment that can expand multiple perception types is used to achieve dynamic expansion, integration and control of vibration, smoke, water mist, hot air and other types of effectors.

[0147] The somatosensory equipment consists of five parts: an assembleable cabin-type carrier, a computing and control module, a VR immersive terminal, an environmental effect module and a sensor interaction module.

[0148] The assembleable cabin carrier reserves installation interfaces and spaces for sensors, effectors and control panels. Figure 1 As shown, the assembleable cabin-type carrier serves as the equipment shell, with a detachable segmented baffle 1, sockets (including sensor and effect device assembly sockets, left and right hand interaction device assembly sockets 2, vibration device assembly sockets 3, center console assembly sockets 4, controller assembly sockets 5 and leg sensing assembly sockets 6) and slides, to reserve standardized peripheral interfaces 7 and space for components such as control chips, storage devices, sensors, effectors, control panels and VR headsets, to achieve plug-in combination of multiple types of equipment, thereby realizing the expansion, integration and control of multiple sensor effectors.

[0149] The operation and control module is the storage, calculation, rendering and control unit for VR system operation, multi-device control, gesture recognition and scene evolution algorithm, and is used to provide hardware and computing power support.

[0150] In this embodiment, the operation and control module is composed of an operation unit, a storage unit, a multi-device control unit, a VR rendering unit, and a signal transmission unit, etc. It runs the accident evolution model, gesture recognition algorithm, scene and logic drive algorithm, reads data or performs corresponding operations through the lower computer control algorithm, and presents the calculation results to the operator in the form of VR immersive scene display. The operation and control module can be combined with the assembleable cabin carrier in a plug-in manner, and a rich peripheral interface is reserved to carry out peripheral communication and drive tests.

[0151] The VR immersive terminal is an integrated VR display device that provides support for the display of virtual three-dimensional scenes. The scene is rendered and the operation process is controlled by the computing and control module to achieve a VR immersive simulation experience of accident handling.

[0152] The environmental effect module is a combination module of multiple sensory effectors including vibration, heating, smoke, water mist, hot wind and cold wind, which is used to realize the effect perception of various accident environment changes. At the same time, the environmental effect module is expanded and accessed based on the assembleable cabin-type carrier. In this embodiment, the assembleable cabin-type carrier also reserves other effect device assembly sockets and communication interfaces for carrying out configuration integration of other effect devices.

[0153] The sensor interaction module is a combination module of multiple perception sensors including image sensing, temperature sensing and gas sensing, which is used to provide perception and collection of human body movements and environmental conditions. The sensor interaction module is expanded and accessed based on an assembleable cabin-type carrier.

[0154] In this embodiment, the sensor interaction module includes two types of sensors, temperature and gas, which are used to feedback the changes in environmental parameters in real time when smoke is emitted and heat is generated, and to feedback in the VR virtual scene, forming a closed loop of VR simulation and effect perception. At the same time, other sensor equipment assembly sockets and communication interfaces should be reserved on the assembleable cabin carrier for carrying out configuration tests of other sensor equipment.

[0155] Secondly, image-based gesture recognition and VR interaction methods are used to replace VR interactive handle devices to realize interactive operations in VR virtual scenes.

[0156] The image-based gesture recognition and VR interaction method specifically includes: S1, installing three image sensors on the somatosensory equipment, and photographing all spaces that the operator's hands can operate in the somatosensory equipment from three different angles to avoid incomplete capture of hand information due to occlusion from a single angle.

[0157] S2. The hand posture images taken without obstruction at any two different angles can be used to reconstruct the hand model and spatially locate the hand based on feature points and binocular vision algorithm.

[0158] S3. Establish a key operation gesture library for emergency response to chemical accident scenarios.

[0159] S4. Establish an operation event mapping library to establish a mapping relationship between each operation gesture and the associated event.

[0160] S5: Hand behavior of the operator entering the operating range of the interactive object (equipment, etc.) and performing a key operation.

[0161] S6. Identify and match the operator's gestures with the actions in the key operation gesture library.

[0162] S7. Reconstruct the three-dimensional model of the hand, spatial positioning, and modeling of dynamic gesture movements based on spatial transformation based on visual sensors.

[0163] S8. Based on the recognition of the gesture action in step S6 and the modeling and spatial positioning of the dynamic gesture action in step S7, identify whether spatial interaction is performed with an object (device, etc.) in the virtual scene and what kind of interaction operation is performed.

[0164] S9. Matching associated events in an operation event mapping library according to the recognition result of the interaction between the operator's gesture operation and the virtual space object.

[0165] S10. Control the display of the accident evolution results in the VR scene according to the event matching results, and realize the virtual scene interaction based on key operation gesture recognition.

[0166] Finally, a polymorphic perception dynamic logic control method based on the accident evolution path is adopted. According to the key nodes of the accident evolution path and the built-in logic judgment information, the evolution of the VR virtual scene and the control of the perception equipment are carried out in combination with interactive operations.

[0167] The multi-state perception dynamic logic control method based on the accident evolution path specifically includes: A1, establishing the chemical scene accident evolution path.

[0168] A2. Associate key elements related to VR operation and display at key nodes in the accident evolution path.

[0169] The key elements include but are not limited to: operating role, spatial position coordinates, interactive operation object, three-dimensional model number of interactive operation object, spatial position of interactive operation object, spatial distance between interactive operation object and interactive operation object, key operation action, operation limit time, operation result status code, VR display effect corresponding to the operation result status code, and control status of environmental effect simulation equipment corresponding to the VR display effect.

[0170] A3. Establish the correlation between the vision-based gesture recognition results and the key nodes of the evolution path.

[0171] A4. The operator's hand behavior when performing a key operation.

[0172] A5. Identify the operator's key operation gestures and match the operation actions of gesture recognition with the key elements of the key nodes of the evolution path.

[0173] A6. The three-dimensional evolution and display results of the VR scene are obtained based on logical judgment and displayed in the form of visual feedback from VR display devices.

[0174] A7. According to the evolution of VR virtual scenes, match the control state of the environmental effect simulation equipment corresponding to the VR display effect.

[0175] A8. According to the control status of the environmental effect simulation equipment, dynamically control the occurrence of related effect equipment to achieve feedback of multi-sensory effects.

[0176] A9. Ultimately, dynamic evolution control of VR accident scenarios and multi-sensory immersive operation experience based on gesture operation feedback are realized.

[0177] The somatosensory equipment designed in this embodiment can expand multiple perception types and achieve multiple real perception effects. Compared with the existing VR interactive equipment that only has vision and hearing, it provides more experience perceptions and increases the operator's multi-perception immersive experience in the virtual scene.

[0178] The somatosensory equipment designed in this embodiment can be expanded with multiple perception types, and can realize the expansion of sensor devices and effect devices. New sensor effect devices can be added at any time during use, which increases the expandability and compatibility of the overall system and avoids the workload of repeated design and processing for adding new functional equipment.

[0179] The somatosensory equipment designed in this embodiment can be expanded to various perception types, and the calculation and control modules therein can perform data collection, algorithm calculation, and signal control of multiple sensors, effectors, and various embedded algorithms, providing computing power and hardware support for equipment expansion and functional calculations.

[0180] This embodiment is based on image gesture recognition and VR interaction, and defines and identifies special operation gestures for various devices, equipment, emergency disposal, etc. in the accident emergency disposal process in chemical scenes. Compared with the operation gestures in normal life, the operations in accident disposal in chemical scenes have their own specificity, which cannot be accurately defined and identified by existing methods.

[0181] This embodiment uses image-based gesture recognition and VR interaction to change the existing VR interaction method, and interacts in a VR virtual scene in a way that is closer to normal human operation, getting rid of the limitations of handheld interactive devices, making interaction easier, and at the same time opening up dynamic interactive control between gesture recognition and accident evolution judgment, VR feedback display, and environmental effect generation equipment.

[0182] This embodiment is based on the polymorphic perception dynamic logic control of the accident evolution path, which changes the current status of the existing VR system and the fixed script process development of the course. It associates the accident evolution path with logical judgment and various key information according to the operator's gesture recognition results, and dynamically evolves the VR virtual scene. At the same time, the effect of the perception equipment is synchronously controlled according to the development of the accident in the VR virtual scene.

[0183] It is worth noting that, in the description of this specification, unless otherwise specified, "multiple" means two or more. In the description of this specification, the description with reference to the term "in the present embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0184] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A multi-sensory somatosensory interaction method for accident state logic control based on visual recognition, characterized in that: First, we use somatosensory equipment that can expand various types of perception to achieve dynamic expansion, integration and control of vibration, smoke, water mist, hot air and other types of effectors; Secondly, the image-based gesture recognition and VR interaction method is used to replace the VR interactive handle device to realize the interactive operation in the VR virtual scene; Finally, a polymorphic perception dynamic logic control method based on the accident evolution path is adopted. According to the key nodes of the accident evolution path and the built-in logic judgment information, the evolution of the VR virtual scene and the control of the perception equipment are carried out in combination with interactive operations.

2. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 1 is characterized in that: The somatosensory equipment consists of five parts: an assembleable cabin carrier, a computing and control module, a VR immersion terminal, an environmental effect module and a sensor interaction module. The assembleable cabin carrier reserves installation interfaces and space for sensors, effectors and control panels.

3. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 2 is characterized in that: The image-based gesture recognition and VR interaction method specifically includes: S1. Install three image sensors on the somatosensory equipment to capture all spaces that can be operated by the operator's hands in the somatosensory equipment from different angles; S2. Establish a key operation gesture library for emergency response to chemical accidents; S3, establish an operation event mapping library, and establish a mapping relationship between each operation gesture and the associated event; S4, identifying and matching the operator's gestures with the actions in the key operation gesture library, and reconstructing the three-dimensional hand model to identify and confirm the interaction between the virtual gestures and virtual scene objects based on spatial positioning; S5. Matching associated events in an operation event mapping library according to the recognition result of the interaction between the operator's gesture operation and the virtual space object; S6. Control the display of the accident evolution results in the VR scene according to the event matching results, and realize the virtual scene interaction based on key operation gesture recognition.

4. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 3 is characterized in that: The multi-state perception dynamic logic control method based on the accident evolution path specifically includes: A1. Establish the evolution path of chemical scene accidents, and associate the key elements related to VR operation and display at the key nodes of the path; A2. Establish the correlation between the visual gesture recognition results and the key nodes of the evolution path; A3. Match the operation actions of gesture recognition with the key elements of the key nodes of the evolution path, obtain the three-dimensional evolution and display results of the VR scene based on logical judgment, and display them in the form of visual feedback of the VR display device; A4. According to the evolution of VR virtual scenes, match the control state of the environmental effect simulation equipment corresponding to the VR display effect, dynamically control the occurrence of related effect equipment, and realize multi-sensory effect feedback.

5. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 3 is characterized in that: In step S2, the key operation gesture library records characteristic information of key operation actions including but not limited to opening valves, closing valves, valve opening, pump start and stop, blind plate plugging, fire monitor operation, wearing of protective equipment and gas monitoring.

6. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 5 is characterized in that: In step S1, the hand model is reconstructed and spatially positioned based on feature points and binocular vision algorithm using any two unobstructed hand gesture images taken at different angles.

7. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 4 is characterized in that: In step A1, the key elements include but are not limited to: operating role, spatial position coordinates, interactive operation object, three-dimensional model number of interactive operation object, spatial position of interactive operation object, spatial distance between interactive operation object and interactive operation object, key operation action, operation limit time, operation result state code, VR display effect corresponding to the operation result state code and control state of environmental effect simulation device corresponding to the VR display effect.

8. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 2 is characterized in that: The operation and control module is the storage, calculation, rendering and control unit for VR system operation, multi-device control, gesture recognition and scene evolution algorithm, and is used to provide hardware and computing power support.

9. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 8 is characterized in that: The VR immersive terminal is an integrated VR display device used to provide support for the display of virtual three-dimensional scenes.

10. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 9, characterized in that: The environmental effect module is a combination module of multiple sensory effectors including vibration, heating, smoke, water mist, hot wind and cold wind, which is used to provide effect perception for changes in virtual accident environment. The environmental effect module is expanded and accessed based on an assembler cabin-type carrier.

11. The multi-sensory somatosensory interaction method for accident state logic control based on visual recognition according to claim 10 is characterized in that: The sensor interaction module is a combination module of multiple perception sensors including image sensing, temperature sensing and gas sensing, which is used to provide perception and collection of human body movements and environmental conditions. The sensor interaction module is expanded and accessed based on an assembleable cabin-type carrier.

Images