Signal fusion perception system and method

Abstract

The embodiment of the application provides a signal fusion perception system and method, the system comprises: a master gateway, a slave gateway, the master gateway comprises a first optical fiber perception submodule for collecting a first backscattering light signal through a split optical fiber and extracting a first feature of the first backscattering light signal, a first wireless signal perception submodule for collecting a first wireless signal communicated between the master gateway and a terminal and extracting a second feature of the first wireless signal, and a first feature fusion algorithm submodule for performing feature fusion analysis on the first feature and the second feature to realize perception on an external environment; and the slave gateway is connected with the master gateway through a split optical fiber. Therefore, the embodiment of the application can solve the problem that the existing environment perception technology has limitations in detecting state changes of fixed targets and behavior identification of moving targets in a fiber-to-the-room (FTTR) system by using a traditional single environment perception technology.

Description

Technical Field

[0001] This application relates to the field of communication technology, and more specifically, to a signal fusion sensing system and method. Background Technology

[0002] Falls are the leading cause of injury-related death among people aged 65 and older, accounting for 42% of all injury-related deaths. Performing cardiopulmonary resuscitation (CPR) within 4-6 minutes of a fall can significantly improve survival rates. Therefore, quickly identifying abnormal situations and promptly calling for help is crucial to ensuring the safety of the elderly.

[0003] Currently, video surveillance technology is widely used in public places to monitor security. However, the quality of video images is easily affected by lighting and weather conditions, and its use in private spaces such as residences may raise privacy concerns, limiting its widespread application. While environmental sensing technology avoids these shortcomings, individual environmental sensing technologies often employ fiber optic sensing or wireless signal analysis, each with its own limitations. Fiber optic sensing technology can efficiently detect minute vibrations and displacements of fixed targets (such as walls and floors), but its ability to sense moving targets (such as people or animals) is weak. Conversely, wireless signal analysis technology can capture the behavioral characteristics of moving targets (such as people or animals), such as human posture and movement trajectory, but it has low sensitivity in detecting vibrations of fixed targets (such as walls and floors), and its signal quality is unstable in complex environments, leading to a decrease in the accuracy of environmental perception.

[0004] Therefore, no effective solution has yet been proposed in the relevant technologies. Summary of the Invention

[0005] This application provides a signal fusion sensing system and method to at least address the limitations of existing environmental sensing technologies in fiber-to-the-room (FTTR) systems, where traditional single environmental sensing techniques are used to detect state changes of fixed targets and recognize the behavior of moving targets.

[0006] According to one embodiment of this application, a signal fusion sensing system is provided, comprising: a main gateway and a slave gateway; the main gateway includes a first service module and a first sensing module, the first service module being used to implement communication services; the first sensing module includes a first optical fiber sensing submodule, a first wireless signal sensing submodule, and a first feature fusion algorithm submodule; wherein, the first optical fiber sensing submodule is used to collect a first backscattered light signal through a split optical fiber and extract a first feature of the first backscattered light signal, the first wireless signal sensing submodule is used to collect a first wireless signal between the main gateway and a terminal and extract a second feature of the first wireless signal, and the first feature fusion algorithm submodule is used to perform feature fusion analysis on the first feature and the second feature to achieve perception of the external environment; the slave gateway is connected to the main gateway through a split optical fiber.

[0007] According to another embodiment of this application, a signal fusion sensing method is provided, applied to a main gateway, comprising: acquiring backscattered light signals and wireless signals; extracting a first feature of the backscattered light signals and a second feature of the wireless signals respectively; and performing feature fusion analysis on the first feature and the second feature to achieve perception of the external environment.

[0008] According to another embodiment of this application, a signal fusion sensing method is provided, applied to a gateway, comprising: acquiring a second backscattered light signal and a second wireless signal used for communication between the gateway and a terminal; extracting a third feature of the second backscattered light signal and a fourth feature of the second wireless signal respectively; and performing feature fusion analysis on the third feature and the fourth feature to achieve perception of the external environment.

[0009] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0010] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in the above method embodiments.

[0011] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0012] The above embodiments of this application provide a signal fusion sensing system. A first optical fiber sensing submodule collects a first backscattered light signal via a split optical fiber and extracts a first feature from the first backscattered light signal. A first wireless signal sensing submodule collects a first wireless signal used in communication between a main gateway and a terminal and extracts a second feature from the first wireless signal. A first feature fusion algorithm submodule performs feature fusion analysis on the first and second features to achieve perception of the external environment. Specifically, the system collects backscattered light and wireless signals through the main gateway, extracts the first and second features from the backscattered light and wireless signals respectively, and performs feature fusion analysis on the first and second features. Through a multimodal feature fusion algorithm, it can achieve perception of the external environment, such as a person falling, an object tipping over, or a person's posture, improving the accuracy of external environment perception. Therefore, the embodiments of this application can solve the problem that existing environmental sensing technologies in fiber-to-the-room (FTTR) systems have limitations in detecting changes in the state of fixed targets and recognizing the behavior of moving targets using traditional single environmental sensing technology, thereby reducing false alarm rates and improving the accuracy and reliability of external environment perception. Attached Figure Description

[0013] Figure 1 This is a system architecture diagram of the FTTR system according to an embodiment of this application;

[0014] Figure 2 This is a system architecture diagram of a signal fusion sensing system according to an embodiment of this application;

[0015] Figure 3 This is a flowchart of a signal fusion sensing method applied to a main gateway according to an embodiment of this application;

[0016] Figure 4 This is a flowchart of a signal fusion sensing method applied to a gateway according to an embodiment of this application;

[0017] Figure 5 This is a schematic diagram of fiber optic environment sensing according to an embodiment of this application;

[0018] Figure 6 This is a schematic diagram illustrating the fusion perception decision made by the main gateway according to an embodiment of this application;

[0019] Figure 7 This is a flowchart of a centralized multimodal feature fusion algorithm according to an embodiment of this application;

[0020] Figure 8 This is a schematic diagram illustrating the signal preprocessing and reporting to the main gateway for fusion perception decision-making according to an embodiment of this application;

[0021] Figure 9This is a schematic diagram of the main gateway and the slave gateway making fusion perception decisions after deploying three-dimensional optical sensing devices according to an embodiment of this application;

[0022] Figure 10 This is a flowchart of a distributed multimodal feature fusion algorithm according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0025] Currently, falls are the leading cause of injury-related death among people over 65 years old, accounting for 42% of all injury-related deaths. The golden rescue time for cardiac arrest after a fall is 4-6 minutes. Performing cardiopulmonary resuscitation within this time window can significantly improve the survival rate. For every minute that rescue is delayed, the survival rate decreases by 10%. Therefore, it is crucial to detect abnormalities and call for help as soon as possible to save lives.

[0026] While most public places are covered by video surveillance, the quality of video images is easily affected by factors such as lighting and weather, and may raise privacy concerns, making them unsuitable for widespread application. Environmental sensing technology can avoid these shortcomings. However, single environmental sensing technologies often rely on fiber optic sensing or wireless signal analysis. The former can detect vibrations of fixed targets but has weak sensing capabilities for moving targets (such as human behavior), while the latter can detect behavioral characteristics of moving targets but has low sensitivity to vibrations of fixed targets. How to achieve effective and privacy-preserving all-weather environmental sensing, unaffected by lighting and weather conditions, especially the accurate identification and timely alerting of abnormal events such as human falls and object displacement, is a pressing problem that needs to be solved.

[0027] In view of the above problems, this application proposes an intelligent environmental perception and abnormal event alarm method based on multimodal fusion of FTTR (Fiber To The Room) system. The master-slave networking mode of the FTTR system is more conducive to dynamically sensing environmental changes and realizing abnormal event detection and alarm. Specifically, this application proposes a multimodal fusion perception method that combines fiber optic sensing and wireless signal analysis. Optical and wireless signals are collected in real time using optical fiber and Wi-Fi nodes. The optical and wireless signals are preprocessed by signal filtering and noise reduction, and features are extracted from the preprocessed optical and wireless signals. These features are then fused to obtain environmental information. Furthermore, a feature fusion algorithm is used to analyze the environmental information to complete the identification of abnormal events for moving and fixed targets. When dangerous situations such as a person falling or tripping, or objects vibrating are detected, a distress signal is sent out. Therefore, this application can monitor in real time the state changes (such as vibration and displacement) of fixed targets (such as walls and floors) and the behavioral characteristics (such as human posture and movement trajectory) of moving targets (such as people or animals) to achieve abnormal event identification and alarm.

[0028] Figure 1 This is a system architecture diagram of the FTTR system according to an embodiment of this application, such as... Figure 1 As shown, the FTTR system includes an MFU (Master Fiber Unit) and one or more SFUs (Secondary Fiber Units). Figure 1 Taking two SFUs as an example, such as SFU1 and SFU2, the MFU connects SFU1 and SFU2 through optical fiber to form an FTTR system. The user terminal or cloud server is used to receive environmental perception information and alarm information.

[0029] During the rate evolution of PON (Passive Optical Network), FTTR systems also face challenges such as signal quality, network latency, and equipment compatibility. Traditional PON networks, when increasing rates, may fail to maintain signal stability and integrity, leading to a decline in network performance. Based on this, the proposed method for PON rate evolution and multi-mode coexistence based on FTTR systems includes three parts: fiber environment awareness, wireless environment awareness, and multi-modal feature fusion, as detailed below:

[0030] (1) Fiber optic environmental sensing

[0031] When a synchronization optical pulse propagates along an optical fiber, the backscattered light signal from Rayleigh scattering returns along the fiber to the incident end, forming an echo signal. When the optical path is relatively stable and there are no external disturbances, the phase and intensity of the backscattered light signal relative to the emitted light signal are fixed. When the optical fiber is subjected to external disturbances (such as vibration or displacement), the phase and intensity of the echo signal will undergo brief changes. Once the external disturbance is eliminated, it returns to a stable state. Based on this principle, when the FTTR system operates in 2.5G symmetric / asymmetric mode, the receiving direction of the main gateway downlink optical module typically only receives 1310nm optical signals. When the FTTR system operates in 10G symmetric / asymmetric mode, the receiving direction of the main gateway downlink optical module typically only receives 1270nm optical signals. To resolve the backscattered light signal, a new channel is introduced in the receiving direction of the main gateway downlink optical module / slave gateway optical module, independent of the uplink service channel.

[0032] The new channel does not carry any specific services; it is only used to acquire the phase and intensity of backscattered light signals. The acquired phase and intensity parameters are analyzed by the main chip, which then outputs the environmental perception results.

[0033] (2) Wireless environment perception

[0034] In wireless communication, a frequency band is typically divided into multiple subcarriers, each responsible for transmitting a portion of the data. By analyzing the phase and amplitude information of each subcarrier, the propagation path of the wireless signal in space can be accurately inferred. Specific applications include:

[0035] The phase and amplitude differences on multiple subcarriers can reveal information such as the propagation path, reflection, scattering, and attenuation of wireless signals. By analyzing this information, we can determine the path the wireless signal has taken, whether it has undergone multipath propagation, and thus gain a more comprehensive understanding of how wireless signals propagate in space.

[0036] Currently, a single Customer Premises Equipment (CPE) can collect and analyze Channel State Information (CSI) signals. However, in complex scenarios such as wall obstructions, blind spots inevitably appear in the signal coverage of a single CPE. Subcarrier information within these blind spots cannot be effectively collected, making it impossible to determine the link status. Based on this, in an FTTR system, a CSI collection and analysis submodule is deployed at both the master gateway and each slave gateway. Each slave gateway independently collects CSI information (this embodiment is not limited to CSI information; it can also collect millimeter-wave signals, raw star flash signals, etc.; this embodiment uses CSI information collection as an example). The collected results are reported to the master gateway via the Wireless Management in the Control Plane Interface (WMCI) channel (this embodiment is not limited to the WMCI channel; it can also be an optical management message channel, Ethernet channel, etc.; this embodiment uses the WMCI channel as an example). The master gateway then comprehensively analyzes the CSI information from each slave gateway to make a judgment.

[0037] (3) Multimodal feature fusion algorithm

[0038] Because fiber optic environmental sensing can detect minute vibrations and displacements of fixed targets (such as walls and floors), but has a weaker ability to sense moving targets (such as people or animals); conversely, wireless environmental sensing can capture the behavioral characteristics of moving targets (such as people or animals), but has low sensitivity to vibrations and displacements of fixed targets.

[0039] Therefore, in order to simultaneously detect slow events such as vibrations of fixed targets and relatively fast events such as moving targets, this embodiment deploys fiber optic sensing modules on the main gateway and slave gateways respectively. Multiple fiber optic cables are used to collect information such as optical path vibrations along the paths of multiple sub-gateways. By optimizing the laying method of the fiber optic cables, full coverage of a specific building area can be achieved. Wireless sensing modules are also deployed on the main gateway and slave gateways respectively. These modules collect CSI information, and the main gateway and slave gateways share the collected information, or the slave gateway reports the collected CSI information to the main gateway via the WMCI channel. The main gateway or slave gateway then uses a feature fusion algorithm to analyze the collected or shared CSI information in real time, matching it with features in the event database to output abnormal events such as human falls, object tipping, and object vibrations, and uploads these to the user terminal or cloud server to issue alarm information.

[0040] Figure 2 This is a system architecture diagram of a signal fusion sensing system according to an embodiment of this application, such as... Figure 2As shown, the signal fusion sensing system includes: a main gateway and a slave gateway; the main gateway includes a first service module and a first sensing module, the first service module being used to implement communication services; the first sensing module includes a first optical fiber sensing submodule, a first wireless signal sensing submodule, and a first feature fusion algorithm submodule; wherein, the first optical fiber sensing submodule is used to collect a first backscattered light signal through a split optical fiber and extract a first feature of the first backscattered light signal, the first wireless signal sensing submodule is used to collect a first wireless signal between the main gateway and the terminal and extract a second feature of the first wireless signal, and the first feature fusion algorithm submodule is used to perform feature fusion analysis on the first feature and the second feature to achieve perception of the external environment; the slave gateway is connected to the main gateway through a split optical fiber.

[0041] In this embodiment, the signal fusion sensing system consists of two main parts: a main gateway and a slave gateway, wherein the slave gateway includes one or more.

[0042] The main gateway includes a first service module and a first sensing module. The main gateway integrates a module for routine service operation (the first service module) and a module for environmental awareness (the first sensing module). Therefore, the main gateway can perform environmental monitoring and event detection while processing communication services.

[0043] The first service module included in the main gateway ensures that the main gateway can handle routine communication services such as data transmission, voice calls, and video streaming, so that when the signal fusion sensing system is performing intelligent environmental sensing, the user's daily network use will not be interfered with.

[0044] The main gateway includes a first sensing module, which is further subdivided into a first optical fiber sensing submodule, a first wireless signal sensing submodule, and a first feature fusion algorithm submodule.

[0045] The first fiber optic sensing submodule: It collects backscattered light signals (e.g., the first backscattered light signal) through fiber optic branches and extracts environmentally relevant features (first features) from them, such as changes in light intensity, frequency, and phase, for use in detecting events such as vibration and displacement of fixed targets.

[0046] The first wireless signal sensing submodule is responsible for collecting wireless signals (e.g., the first wireless signal) between the main gateway and terminal devices within its coverage area, and extracting features related to the moving target (second features) from the wireless signal, such as CSI power and scattering changes, to capture information such as the moving target's attitude and trajectory.

[0047] The first feature fusion algorithm submodule fuses the different types of signal features extracted by the first optical fiber sensing submodule and the first wireless signal sensing submodule. Through algorithm analysis of the fused first and second features, it achieves comprehensive perception of the external environment and identifies potential abnormal events.

[0048] The gateway connects to the main gateway via a split fiber optic cable, enabling it to perform fiber optic environmental sensing, collect and process wireless signals, and then report the collected wireless signals to the main gateway via the WMCI channel. Alternatively, the gateway can perform preliminary analysis before reporting to the main gateway, thus improving the response speed to abnormal events without infringing on user privacy.

[0049] In one embodiment, the slave gateway is configured with a service module for implementing communication services; or, the slave gateway is configured with a service module and a sensing module for implementing communication services and environmental sensing.

[0050] In this embodiment, when the slave gateway is configured with a service module, the slave gateway includes a second service module for implementing communication services; when the slave gateway is configured with both a service module and a sensing module, the slave gateway includes a second service module and a second sensing module; the second service module is used to implement communication services, and the second sensing module includes a second optical fiber sensing submodule, a second wireless signal sensing submodule, and a second feature fusion algorithm submodule; wherein, the second optical fiber sensing submodule is used to collect a second backscattered light signal through a split optical fiber and extract a third feature of the second backscattered light signal; the second wireless signal sensing submodule is used to collect a second wireless signal used in communication between the slave gateway and the terminal and extract a fourth feature of the second wireless signal; the second feature fusion algorithm submodule is used to perform feature fusion analysis on the third feature and the fourth feature to achieve perception of the external environment; or, the slave gateway includes a second service module and a third sensing module; the second service module is used to implement communication services, and the third sensing module includes a third wireless signal sensing submodule used to collect a third wireless signal used in communication between the slave gateway and the terminal.

[0051] In this embodiment, the gateway includes three implementation methods. Specifically, in the first implementation method, the gateway includes a second service module for implementing the operation of communication services.

[0052] The second implementation method includes a second service module and a second sensing module from the gateway;

[0053] The second service module is used to implement the operation of communication services, ensuring the integrity and stability of the signal fusion sensing system, so that the communication service can still operate even if an interruption occurs in environmental sensing or abnormal event detection.

[0054] The second sensing module includes a second optical fiber sensing submodule, a second wireless signal sensing submodule, and a second feature fusion algorithm submodule. The second optical fiber sensing submodule collects backscattered light signals (second backscattered light signals) via optical fibers and extracts features reflecting events such as vibration and displacement of fixed targets (third features) to monitor changes in the state of fixed objects. The second wireless signal sensing submodule collects wireless signals (second wireless signals) between the gateway and terminal devices within its coverage area and extracts features reflecting moving targets (such as people or animals) (fourth features) to capture the movement trajectory and behavioral posture of moving targets. The second feature fusion algorithm submodule fuses and analyzes the third and fourth features, further improving the accuracy and comprehensiveness of environmental perception and identifying potential abnormal events through comprehensive analysis of the feature information from optical fibers and wireless signals.

[0055] The third implementation method: The gateway includes a second service module and a third sensing module;

[0056] The second service module is used to implement the operation of communication services, ensuring the integrity and stability of the signal fusion sensing system, so that the communication service can still operate even if an interruption occurs in environmental sensing or abnormal event detection.

[0057] The third sensing module includes a third wireless signal sensing submodule. The slave gateway only has wireless signal sensing capabilities and lacks a fiber optic sensing submodule. The third wireless signal sensing submodule is responsible for collecting the wireless signals (third wireless signals) between the slave gateway and the terminal device and transmitting these signals to the master gateway to monitor the behavior of moving targets. This configuration is suitable for scenarios where fiber optic environmental sensing is not required or inconvenient, such as in environments with good wireless signal coverage where fiber optic sensing is not applicable.

[0058] In one embodiment, the first wireless signal includes at least one of the following: a first channel state information (CSI), a first millimeter-wave signal, and a first satellite flash raw signal; the second wireless signal includes at least one of the following: a second channel state information (CSI), a second millimeter-wave signal, and a second satellite flash raw signal; and the third wireless signal includes at least one of the following: a third channel state information (CSI), a third millimeter-wave signal, and a third satellite flash raw signal.

[0059] In this embodiment, when the gateway includes a sensing module, the wireless signal sensing submodules of the main gateway and the slave gateway can each collect the wireless signals used for communication with the terminal. The wireless signals include, but are not limited to, CSI information, millimeter wave signals, and raw star flash signals.

[0060] In one embodiment, where the slave gateway includes a second service module and a second sensing module, the slave gateway includes a three-dimensional light sensing device for acquiring spatial images and parsing the spatial images to obtain three-dimensional light sensing results, and sending the three-dimensional light sensing results to the main gateway.

[0061] In this embodiment, a three-dimensional light sensing device (such as a lidar) is deployed in the gateway. The gateway can collect the terminal's CSI information, millimeter-wave signals, starburst raw signals, and spatial images (also known as laser images), and run a laser image recognition algorithm to output the three-dimensional light sensing results, which are then sent to the main gateway.

[0062] In one embodiment, the terminal is used to receive the parsing result after feature fusion parsing and to issue an alarm message based on the parsing result.

[0063] In this embodiment, if the main gateway or the slave gateway determines that there is an abnormal event based on the parsing result after feature fusion, the main gateway or the slave gateway will report the parsing result to the user terminal or the cloud service in real time and issue an alarm message to notify the user in real time.

[0064] The above embodiments of this application provide a signal fusion sensing system. A first optical fiber sensing submodule collects a first backscattered light signal via a split optical fiber and extracts a first feature from the first backscattered light signal. A first wireless signal sensing submodule collects a first wireless signal used in communication between a main gateway and a terminal and extracts a second feature from the first wireless signal. A first feature fusion algorithm submodule performs feature fusion analysis on the first and second features to achieve perception of the external environment. Specifically, the system collects backscattered light and wireless signals through the main gateway, extracts the first and second features from the backscattered light and wireless signals respectively, and performs feature fusion analysis on the first and second features. Through a multimodal feature fusion algorithm, it can achieve perception of the external environment, such as a person falling, an object tipping over, or a person's posture, improving the accuracy of external environment perception. Therefore, the embodiments of this application can solve the problem that existing environmental sensing technologies in fiber-to-the-room (FTTR) systems have limitations in detecting changes in the state of fixed targets and recognizing the behavior of moving targets using traditional single environmental sensing technology, thereby reducing false alarm rates and improving the accuracy and reliability of external environment perception.

[0065] This application also proposes a signal fusion sensing method operating in the aforementioned signal fusion sensing system. Figure 3 This is a flowchart of a signal fusion sensing method applied to a main gateway according to an embodiment of this application, such as... Figure 3 As shown, the method may specifically include the following steps:

[0066] Step S302: Collect backscattered light signals and wireless signals;

[0067] In this embodiment, the slave gateway is configured with a service module; or, the slave gateway is configured with a service module and a sensing module. When the slave gateway is configured with a service module, a first backscattered light signal and a first wireless signal used for communication between the main gateway and the terminal are collected. When the slave gateway is configured with both a service module and a sensing module, the method includes: when the slave gateway is configured with a second sensing module and collects the second backscattered light signal and the second wireless signal, collecting the first backscattered light signal and the first wireless signal and receiving the second backscattered light signal and the second wireless signal sent by the slave gateway; or, when the slave gateway is configured with a third sensing module and collects the third wireless signal, collecting the first backscattered light signal and the first wireless signal and receiving the third wireless signal sent by the slave gateway.

[0068] In this embodiment, the main gateway can collect the first backscattered light signal and the first wireless signal, and can also receive the second backscattered light signal and the second wireless signal or the third wireless signal reported from the gateway.

[0069] In this step, the backscattered light signal is the reflected light signal returned by the optical fiber when the external environment changes after the main gateway transmits a laser pulse to the slave gateway.

[0070] In this embodiment, the main gateway and the slave gateway are connected via fiber optic cables, and there can be one or more slave gateways.

[0071] In one embodiment, if the secondary gateway is configured with a second sensing module and acquires a second backscattered light signal and a second wireless signal, the method further includes: if the secondary gateway is deployed with a three-dimensional light sensing device, receiving the three-dimensional light sensing result after the secondary gateway has analyzed the acquired spatial image.

[0072] In this embodiment, a three-dimensional optical sensing device (such as a lidar) is deployed in the slave gateway. The slave gateway can collect the terminal's CSI information, millimeter-wave signals, starburst raw signals, and spatial images (also known as laser images), and run a laser image recognition algorithm to output the three-dimensional optical sensing results. The master gateway receives the three-dimensional optical sensing results sent by the slave gateway and runs a feature fusion sensing algorithm to determine whether an abnormal event has occurred.

[0073] In one embodiment, when the slave gateway is configured with a second sensing module, after acquiring the first backscattered light signal and the first wireless signal, the method further includes: sending the first backscattered light signal and the first wireless signal to the slave gateway.

[0074] In this embodiment, when the secondary gateway is configured with a second sensing module, the secondary gateway can run a feature fusion sensing algorithm. The primary gateway shares the first backscattered light signal and the first wireless signal it has collected with the secondary gateway through the WMCI channel, so that the secondary gateway can perform feature fusion analysis.

[0075] Step S304: Extract the first feature of the backscattered light signal and the second feature of the wireless signal respectively;

[0076] In one embodiment, before extracting the first feature of the backscattered light signal and the second feature of the wireless signal, the method further includes: preprocessing the backscattered light signal and the wireless signal, wherein the preprocessing includes at least one of the following: signal calibration, signal filtering, and signal denoising.

[0077] Step S306: Perform feature fusion analysis on the first feature and the second feature to achieve perception of the external environment.

[0078] In one embodiment, after performing feature fusion analysis on the first feature and the second feature, the method further includes: sending the first analysis result after feature fusion analysis to the terminal so that the terminal issues an alarm message based on the first analysis result, wherein the first analysis result includes at least one of the following: object tipping over, object vibration, human fall.

[0079] In this embodiment, if the main gateway determines that an abnormal event exists based on the first parsing result after feature fusion analysis, the main gateway will report the first parsing result to the user terminal or cloud server in real time and issue an alarm message to notify the user. For example, when a person falls, the main gateway will report the fall to the user terminal or cloud server in real time and issue an alarm message to notify the user.

[0080] The above embodiments of this application provide a signal fusion sensing method applied to a main gateway. By collecting backscattered light signals and wireless signals, a first feature of the backscattered light signal and a second feature of the wireless signal are extracted respectively. The first and second features are then fused and analyzed to achieve perception of the external environment. That is, through a multimodal feature fusion algorithm, abnormal events such as human falls and object tilting can be effectively identified and distinguished, effectively shortening the detection cycle of abnormal events such as human falls and improving the success rate of rescue.

[0081] Figure 4 This is a flowchart of a signal fusion sensing method applied to a gateway according to an embodiment of this application, such as... Figure 4 As shown, the method may specifically include the following steps:

[0082] Step S402: Collect the second backscattered light signal and the second wireless signal used for communication between the gateway and the terminal;

[0083] In this embodiment, the acquisition of the second backscattered light signal and the second wireless signal includes: when the second sensing module is configured on the gateway, acquiring the second backscattered light signal and the second wireless signal.

[0084] In one embodiment, if the slave gateway is configured with a second sensing module, the method further includes: if the slave gateway is deployed with a three-dimensional light sensing device, acquiring a spatial image and parsing the spatial image to obtain a three-dimensional light sensing result, and sending the three-dimensional light sensing result to the main gateway.

[0085] In this embodiment, a three-dimensional light sensing device (such as a lidar) is deployed in the gateway. The gateway can collect the terminal's CSI information, millimeter-wave signals, starburst raw signals, and spatial images (also known as laser images), and run a laser image recognition algorithm to output the three-dimensional light sensing results. The gateway then sends the three-dimensional light sensing results to the main gateway.

[0086] In one embodiment, after acquiring the second backscattered light signal and the second wireless signal, the method further includes: sending the second backscattered light signal and the second wireless signal to the main gateway.

[0087] In this embodiment, when the second sensing module is configured on the slave gateway, the second sensing module of the slave gateway can collect the second backscattered light signal and the second wireless signal, and share them with the master gateway through the WMCI channel so that the master gateway can perform feature fusion analysis.

[0088] Step S404: Extract the third feature of the second backscattered light signal and the fourth feature of the second wireless signal, respectively;

[0089] In one embodiment, before extracting the third feature of the second backscattered light signal and the fourth feature of the second wireless signal, the method further includes: preprocessing the second backscattered light signal and the second wireless signal, wherein the preprocessing includes at least one of the following: signal calibration, signal filtering, and signal denoising.

[0090] Step S406: Perform feature fusion analysis on the third feature and the fourth feature to achieve perception of the external environment.

[0091] In one embodiment, after performing feature fusion analysis on the third feature and the fourth feature, the method further includes: sending the second analysis result after feature fusion analysis to the terminal, so that the terminal issues an alarm message based on the second analysis result, wherein the second analysis result includes at least one of the following: object tipping over, object vibration, human fall.

[0092] In this embodiment, if the gateway determines that an abnormal event exists based on the second parsing result obtained through feature fusion parsing, the gateway reports the second parsing result to the user terminal or cloud server in real time and issues an alarm message to notify the user in real time. For example, when a person falls, the gateway reports the fall to the user terminal or cloud server in real time and issues an alarm message to notify the user.

[0093] In one embodiment, a third wireless signal is collected from the gateway for communication between the secondary gateway and the terminal, and the third wireless signal is sent to the primary gateway.

[0094] In this embodiment, the slave gateway only has wireless signal sensing capability and lacks a fiber optic sensing submodule. The third wireless signal sensing submodule in the third sensing module is responsible for collecting the wireless signal (third wireless signal) between the slave gateway and the terminal device and sending the third wireless signal to the master gateway for monitoring the behavior of moving targets. This configuration is suitable for scenarios where fiber optic environmental sensing is not required or inconvenient, such as in environments with good wireless signal coverage and where fiber optic sensing is not applicable.

[0095] The above embodiments of this application provide a signal fusion sensing method applied to a gateway. By collecting a second backscattered light signal and a second wireless signal used in communication between the gateway and the terminal, a third feature of the second backscattered light signal and a fourth feature of the second wireless signal are extracted respectively. The third and fourth features are then fused and analyzed to achieve perception of the external environment. That is, through a multimodal feature fusion algorithm, abnormal events such as human falls and object tilting can be effectively identified and distinguished, effectively shortening the detection cycle of abnormal events such as human falls and improving the success rate of rescue.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0097] Example 1

[0098] Figure 5 This is a schematic diagram of fiber optic environment sensing according to an embodiment of this application, such as... Figure 5 As shown, the main gateway MFU is connected to the slave gateway SFU1 and slave gateway SFU2 through two branch optical fibers respectively. The sensing module in the main gateway further includes an optical fiber sensing submodule, a wireless signal sensing submodule, and a feature fusion algorithm submodule. In this embodiment 1, the sensing module includes an optical fiber sensing submodule as an example for illustration.

[0099] (1) When the communication service is running:

[0100] Step S201: The service modules of the slave gateway SFU1 and slave gateway SFU2 send uplink messages to the master gateway MFU;

[0101] Step S202: The service module of the main gateway MFU receives the uplink message and performs service communication.

[0102] (2) When abnormal vibration occurs at a certain location of fiber optic cable 1, the following procedure shall be followed:

[0103] Step S201: Send an uplink message from the service module of gateway SFU1 to the main gateway MFU;

[0104] Step S202: The service module of the main gateway MFU receives the uplink message and performs service communication;

[0105] Step S203: The intensity and phase of the backscattered light signal from the Rayleigh scattering of fiber 1 at position ☆ change, and the signal is transmitted back to the sensing module of the main gateway MFU via the optical fiber.

[0106] Step S204: The main gateway MFU sensing module analyzes the light intensity and phase of the Rayleigh backscattered light signal to determine the possible vibration points on fiber 1.

[0107] (3) When abnormal vibration occurs at a certain location of fiber optic cable 2, the following procedure shall be followed:

[0108] Step S201: Send an uplink message from the service module of gateway SFU2 to the main gateway MFU;

[0109] Step S202: The service module of the main gateway MFU receives the uplink message and performs service communication;

[0110] Step S205: The intensity and phase of the backscattered light signal from Rayleigh scattering at the Δ position of fiber 2 change, and the signal is transmitted back to the sensing module of the main gateway MFU and the sensing module of the slave gateway SFU2 through the fiber.

[0111] Step S206: The sensing modules of the main gateway MFU and the slave gateway SFU2 analyze the light intensity and phase of the Rayleigh backscattered light signal to determine the possible vibration points on the fiber 2.

[0112] Example 2

[0113] Figure 6 This is a schematic diagram illustrating the fusion perception decision-making process performed by the main gateway according to an embodiment of this application, as shown below. Figure 6 As shown, CSI information is collected and reported from gateway SFU1 and gateway SFU2, and the decision is made by the master gateway MFU. The operating mechanism is as follows:

[0114] Steps S601 / S602 / S603: The main gateway MFU continuously collects CSI information; the slave gateways SFU1 and SFU2 continuously collect CSI information and report it to the main gateway MFU through the WMCI channel. Figure 6 The collected CSI information changes as people move through the space.

[0115] Step S604: The main gateway MFU continuously receives CSI information reported by the gateway SFU1 and the gateway SFU2 through the WMCI channel;

[0116] Step S605: The feature fusion algorithm deployed in the main gateway MFU can analyze the CSI information it collects and receives in real time;

[0117] Step S606: The feature fusion algorithm deployed on the main gateway MFU analyzes the current abnormal events, reports the analysis results to the user terminal APP (Application Software) or cloud server in real time, and issues alarm information to notify the user.

[0118] Example 3

[0119] Figure 7 This is a flowchart of a centralized multimodal feature fusion algorithm according to an embodiment of this application, such as... Figure 7 As shown, in this centralized multimodal feature fusion algorithm, the slave gateway SFU1 and slave gateway SFU2 are only responsible for collecting CSI information, while the main gateway MFU performs feature fusion perception algorithm analysis, specifically including the following steps:

[0120] Step S701: CSI information is continuously collected from SFU1 and SFU2 and reported to the MFU. The MFU collects the backscattered light signals from the optical fibers of SFU1 and SFU2.

[0121] Step S702: The main gateway MFU continuously collects CSI information;

[0122] Step S703: The feature fusion sensing algorithm deployed in the main gateway MFU analyzes the CSI information and backscattered light signal;

[0123] Step S704: The main gateway MFU determines whether an abnormal event has occurred and the specific event type;

[0124] Step S705: If an abnormal event occurs, the main gateway MFU will report the abnormal event and its specific type to the user terminal APP or cloud server and issue an alarm message to notify the user.

[0125] Example 4

[0126] Figure 8 This is a schematic diagram illustrating the process of signal preprocessing and reporting to the main gateway for fusion perception decision-making according to an embodiment of this application. Figure 8 As shown in the diagram, CSI information is collected from gateway SFU1 and gateway SFU2. After preprocessing, the CSI information is reported to the main gateway MFU, which then makes the decision. The operating mechanism is as follows:

[0127] Steps S801 / S802 / S803: The main gateway MFU continuously collects CSI information; the slave gateways SFU1 and SFU2 continuously collect CSI information, and the slave gateways SFU1 and SFU2 preprocess the CSI information and report it to the main gateway MFU through the WMCI channel.

[0128] Step S804: The master gateway MFU continuously receives preprocessed CSI information from slave gateways SFU1 and SFU2 via the WMCI channel;

[0129] Step S805: The feature fusion algorithm deployed in the main gateway MFU can analyze the pre-processed CSI information it collects and receives in real time;

[0130] Step S806: The feature fusion algorithm deployed on the main gateway MFU analyzes the current abnormal events, reports the analysis results to the user terminal APP (Application Software) or cloud server in real time, and issues alarm information to notify the user.

[0131] Example 5

[0132] Figure 9 This is a schematic diagram illustrating the fusion sensing decision-making process between the main gateway and the slave gateway after deploying the three-dimensional optical sensing device according to an embodiment of this application. Figure 9 As shown, in this embodiment, a slave gateway SFU is deployed, and a 3D optical sensing device (LiDAR) is deployed in the slave gateway SFU. The master gateway MFU and the slave gateway SFU are respectively deployed with feature fusion sensing algorithms, and the operation mechanism is as follows:

[0133] Steps S901 / S902 / S903: The main gateway MFU continuously acquires CSI information; the gateway SFU continuously acquires CSI information and laser images, and runs the laser image recognition algorithm to output 3D light perception results;

[0134] Step S904: The main gateway MFU runs a feature fusion perception algorithm to determine whether an abnormal event has occurred and the specific event type;

[0135] Step S905: The main gateway MFU shares its continuously collected CSI information with the slave gateway SFU through the WMCI channel. The slave gateway SFU can also share 3D light perception results and report CSI information with the main gateway MFU through the WMCI channel. The slave gateway runs a feature fusion perception algorithm to determine whether an abnormal event has occurred and the specific event type.

[0136] Steps S906 / S907: If an abnormal event occurs, the main gateway MFU and the slave gateway SFU shall report the abnormal event and the specific event type to the user terminal APP or cloud server respectively, and issue an alarm message to notify the user.

[0137] Example 6

[0138] Figure 10 This is a flowchart of a distributed multimodal feature fusion algorithm according to an embodiment of this application, such as... Figure 10 As shown, in this distributed multimodal feature fusion algorithm, the feature fusion perception algorithm is deployed on both the slave gateway SFU and the master gateway MFU. The algorithm can be flexibly selected to be implemented by either the slave gateway SFU or the master gateway MFU based on factors such as the CPU (Central Processing Unit) load of the terminal device and the fusion perception methods supported by the terminal device. The operating mechanism is as follows:

[0139] Step S1001: Continuously collect CSI information from the gateway SFU;

[0140] Step S1002: Continuously emit laser beams from the gateway SFU and generate and acquire corresponding laser images;

[0141] Step S1003: Continuously collect the backscattered light signal of the split optical fiber from the gateway SFU and report the backscattered light signal to the main gateway MFU;

[0142] Step S1004: Report the collected CSI information from the gateway SFU to the main gateway MFU;

[0143] Step S1005: The main gateway MFU sends the CSI information it has collected to the slave gateway SFU;

[0144] Step S1006: Analyze the CSI information, backscattered light signal, and laser image using the feature fusion algorithm from the gateway SFU;

[0145] Step S1007: Determine from the gateway SFU whether an abnormal event has occurred and the specific event type;

[0146] Step S1008: The main gateway MFU receives the backscattered optical signal and CSI information of the split optical fiber reported from the gateway SFU, as well as the CSI information collected by the main network optical itself, and runs the fusion algorithm to analyze the CSI information and backscattered optical signal.

[0147] Step S1009: The main gateway MFU determines whether an abnormal event has occurred and the specific event type;

[0148] Step S1010: If an abnormal event occurs, the main gateway MFU and the slave gateway SFU shall report the abnormal event and the specific event type to the user terminal APP or cloud server respectively, and issue an alarm message to notify the user.

[0149] It should be noted that, in the above embodiments, taking the acquisition of CSI information as an example, millimeter-wave signals, star flash raw signals, etc. can also be acquired for signal fusion sensing analysis; in the above embodiments, taking the WMCI channel as an example, information can also be shared through optical management message channels, Ethernet channels, etc.

[0150] The multimodal fusion of optical and wireless signals in this embodiment makes the system less susceptible to external factors such as climate and ambient light, and can maintain stable performance under various conditions. Furthermore, by utilizing the existing FTTR master-slave gateway networking method, no additional sensors need to be deployed, which reduces the cost of system deployment and maintenance, and enables the environmental perception and abnormal event alarm functions to be more widely used.

[0151] The embodiments of this application can be applied to home or enterprise FTTR systems to monitor and report abnormal situations such as human falls.

[0152] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0153] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0154] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0155] Embodiments of this application also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0156] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0157] According to yet another embodiment of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this disclosure.

[0158] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0159] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0160] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A signal fusion sensing system, characterized in that, include: Master gateway, slave gateway; The main gateway includes a first service module and a first sensing module. The first service module is used to implement the operation of communication services. The first sensing module includes a first optical fiber sensing submodule, a first wireless signal sensing submodule, and a first feature fusion algorithm submodule. The first optical fiber sensing submodule is used to collect a first backscattered light signal through a split optical fiber and extract a first feature of the first backscattered light signal. The first wireless signal sensing submodule is used to collect a first wireless signal between the main gateway and the terminal and extract a second feature of the first wireless signal. The first feature fusion algorithm submodule is used to perform feature fusion and analysis on the first feature and the second feature to achieve perception of the external environment. The slave gateway is connected to the master gateway via a split optical fiber.

2. The system according to claim 1, characterized in that, in, The gateway is configured with a service module to enable the operation of communication services; or, The gateway is configured with a service module and a sensing module to enable the operation of communication services and the sensing of the environment.

3. The system according to claim 2, characterized in that, When the slave gateway is configured with a service module, the slave gateway includes a second service module for implementing the operation of communication services; When the slave gateway is configured with a service module and a sensing module, the slave gateway includes a second service module and a second sensing module; the second service module is used to implement the operation of communication services, and the second sensing module includes a second optical fiber sensing submodule, a second wireless signal sensing submodule, and a second feature fusion algorithm submodule; wherein, the second optical fiber sensing submodule is used to collect a second backscattered light signal through a split optical fiber and extract a third feature of the second backscattered light signal; the second wireless signal sensing submodule is used to collect a second wireless signal used for communication between the slave gateway and the terminal and extract a fourth feature of the second wireless signal; the second feature fusion algorithm submodule is used to perform feature fusion analysis on the third feature and the fourth feature to achieve perception of the external environment; or, The slave gateway includes a second service module and a third sensing module; the second service module is used to implement the operation of communication services, and the third sensing module includes a third wireless signal sensing submodule, which is used to collect the third wireless signal of communication between the slave gateway and the terminal.

4. The system according to any one of claims 1-3, characterized in that, in, The slave gateway may include one or more.

5. The system according to claim 3, characterized in that, in, The first wireless signal includes at least one of the following: a first channel state information (CSI), a first millimeter-wave signal, and a first satellite flash raw signal; the second wireless signal includes at least one of the following: a second channel state information (CSI), a second millimeter-wave signal, and a second satellite flash raw signal; the third wireless signal includes at least one of the following: a third channel state information (CSI), a third millimeter-wave signal, and a third satellite flash raw signal.

6. The system according to claim 3, characterized in that, in, In the case where the slave gateway includes a second service module and a second sensing module, the slave gateway includes a three-dimensional light sensing device for acquiring spatial images and parsing the spatial images to obtain three-dimensional light sensing results, and sending the three-dimensional light sensing results to the main gateway.

7. The system according to claim 1, characterized in that, in, The terminal is used to receive the parsing results after feature fusion and parsing, and to issue alarm information based on the parsing results.

8. A signal fusion sensing method, characterized in that, The main gateway applied to the system of claim 1 includes: Collect backscattered light signals and wireless signals; The first feature of the backscattered light signal and the second feature of the wireless signal are extracted respectively; The first feature and the second feature are fused and analyzed to achieve perception of the external environment.

9. The method according to claim 8, characterized in that, in, The backscattered light signal is the reflected light signal received by the main gateway after it transmits a laser pulse to the slave gateway, which is the light returned when the external environment of the optical fiber changes.

10. The method according to claim 9, characterized in that, in, The main gateway and the slave gateway are connected via fiber optic cables, and there may be one or more slave gateways.

11. The method according to claim 10, characterized in that, in, The slave gateway is configured with a service module; or, the slave gateway is configured with both a service module and a sensing module.

12. The method according to claim 11, characterized in that, The acquisition of backscattered light signals and wireless signals includes: When the secondary gateway is configured with a service module, the first backscattered light signal and the first wireless signal used for communication between the main gateway and the terminal are collected. When the gateway is configured with a service module and a sensing module, it includes: When the gateway is configured with a second sensing module and acquires a second backscattered light signal and a second wireless signal, the first backscattered light signal and the first wireless signal are acquired, and the second backscattered light signal and the second wireless signal sent by the gateway are received; or, When the third sensing module is configured on the gateway and a third wireless signal is collected, the first backscattered light signal and the first wireless signal are collected, and the third wireless signal sent by the gateway is received.

13. The method according to claim 12, characterized in that, in, In the case where the gateway is configured with a second sensing module and acquires a second backscattered light signal and a second wireless signal, the method further includes: When the slave gateway is equipped with a three-dimensional light sensing device, it receives the three-dimensional light sensing result after the slave gateway has analyzed the acquired spatial image.

14. The method according to claim 12, characterized in that, When the second sensing module is configured on the gateway, after acquiring the first backscattered light signal and the first wireless signal, the method further includes: The first backscattered light signal and the first wireless signal are sent to the gateway.

15. The method according to claim 8, characterized in that, After performing feature fusion parsing on the first feature and the second feature, the process also includes: The first parsing result after feature fusion and parsing is sent to the terminal so that the terminal can issue an alarm message based on the first parsing result, wherein the first parsing result includes at least one of the following: object tipping over, object vibration, human fall.

16. A signal fusion sensing method, characterized in that, The slave gateway applied to the system of claim 1 includes: Collect the second backscattered light signal and the second wireless signal used for communication between the gateway and the terminal; The third feature of the second backscattered light signal and the fourth feature of the second wireless signal are extracted respectively; The third and fourth features are fused and analyzed to achieve perception of the external environment.

17. The method according to claim 16, characterized in that, After acquiring the second backscattered light signal and the second wireless signal used for communication between the gateway and the terminal, the method further includes: Send the second backscattered light signal and the second wireless signal to the main gateway.

18. The method according to claim 16, characterized in that, The acquisition of the second backscattered light signal and the second wireless signal used for communication between the gateway and the terminal includes: When the secondary gateway is equipped with a second sensing module, it collects a second backscattered light signal and a second wireless signal used for communication between the secondary gateway and the terminal.

19. The method according to claim 18, characterized in that, In the case where the second sensing module is configured on the gateway, the following is also included: When a three-dimensional light sensing device is deployed on the slave gateway, a spatial image is acquired and the spatial image is analyzed to obtain a three-dimensional light sensing result, which is then sent to the master gateway.

20. The method according to claim 16, characterized in that, After performing feature fusion analysis on the third feature and the fourth feature, the method further includes: The second parsing result after feature fusion parsing is sent to the terminal so that the terminal can issue an alarm message based on the second parsing result, wherein the second parsing result includes at least one of the following: object tipping over, object vibrating, or human falling.

21. The method according to claim 16, characterized in that, include: The third wireless signal used in the communication between the gateway and the terminal is collected, and the third wireless signal is sent to the main gateway.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 8-15 or 16-21.

23. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 8-15 or 16-21.

24. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 8-15 or 16-21.

Images