Device for safety monitoring

Abstract

The present invention provides a method and apparatus for obtaining a condition of an impeller wherein the apparatus comprises: a plurality of visual sensors for capturing visual images; a global navigation satellite system (GNSS) receiver; and one or more control processing units coupled with the plurality of visual sensors and the GNSS receiver, and configured to be adapted to perform a positioning process based on at least one of the visual image or the GNSS receiver.

Description

Technical Field

[0001] In general, the present invention relates to safety monitoring devices, and in particular, the present invention relates to wearable devices for safety monitoring. Background Art

[0002] Children's safety has always been a topic of concern to all parties. With the rise of technologies such as the Internet of Things (IoT) and wearable devices, new ideas and solutions have been provided to prevent various safety issues for children. For example, wearable devices such as children's smart watches and children's safety backpacks usually have voice call functions and / or positioning functions, so that children can call for help in time when they are in danger or allow parents and police to track their children's location in time.

[0003] However, there are still some common shortcomings in such safety devices. For example, although these devices are designed to look inconspicuous, they are still easy to be identified by child traffickers, who will remove them from the children as soon as possible, resulting in the loss of the children's location. In addition, the positioning function that usually relies on the Global Positioning Satellite System (GNSS) is often unstable due to problems such as the deterioration of the radio frequency environment.

[0004] Therefore, it is desirable to provide a safer and more reliable device and / or measure for monitoring or tracking the safety or location of a child. Summary of the invention

[0005] A brief overview of one or more embodiments is given below to provide a basic understanding of these embodiments. This overview is not a general summary of all contemplated embodiments, nor is it intended to identify key or important elements of all embodiments or describe the scope of any or all embodiments. Its purpose is only to provide some concepts of one or more embodiments in a simplified form as a preface to the more detailed description provided later.

[0006] In one aspect of the present disclosure, there is provided an apparatus for security monitoring, comprising: a plurality of visual sensors for capturing visual images; a global satellite navigation system (GNSS) receiver; and one or more control processing units, the one or more control processing units being coupled to the plurality of visual sensors and the GNSS receiver and configured to perform positioning processing based on at least one of the visual images or the GNSS receiver.

[0007] In another aspect of the present disclosure, a wearable device for security monitoring is provided, comprising: a plurality of visual sensors mounted on different surfaces of the wearable device and used to capture visual images; and a circuit board placed inside the wearable device and comprising a global satellite navigation system (GNSS) receiver and one or more control processing units; wherein the one or more control processing units are coupled to the plurality of visual sensors and the GNSS receiver and are configured to be suitable for performing positioning processing based on at least one of the visual images or the GNSS receiver.

[0008] In another aspect of the present disclosure, a method for security monitoring is provided, comprising: capturing visual images in multiple directions through multiple visual sensors; determining wireless channel conditions; and obtaining positioning information based on the determined wireless channel conditions by one of the following methods: obtaining the positioning information based on the visual images using a visual simultaneous localization and mapping (VSLAM) method, or obtaining the positioning information based on a global satellite navigation system.

[0009] Other aspects or various modifications of the present disclosure will become more apparent upon consideration of the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A An exemplary structure of a safety monitoring device according to one or more aspects of the present disclosure is shown.

[0011] Figure 1B Another exemplary structure having a safety monitoring device according to one or more aspects of the present disclosure is shown.

[0012] Figure 2 An exemplary system for security monitoring according to one or more aspects of the present disclosure is shown.

[0013] Figure 3A An exemplary method for safety monitoring according to one or more aspects of the present disclosure is shown.

[0014] Figure 3B Another exemplary method for safety monitoring according to one or more aspects of the present disclosure is shown.

[0015] Figure 4 The following illustrates scenarios to which the security monitoring method and device according to one or more aspects of the present disclosure are applicable.

[0016] Figure 5 is a flow chart illustrating an exemplary method for security monitoring according to one or more aspects of the present disclosure. DETAILED DESCRIPTION

[0017] A number of embodiments are now described with reference to the accompanying drawings, wherein the same reference numerals are used to indicate the same elements herein. In the following description, for ease of explanation, a large number of specific details are provided to provide a comprehensive understanding of one or more embodiments. However, it is apparent that the embodiments may also be implemented without these specific details. In other examples, known structures and devices are shown in block diagram form to facilitate description of one or more embodiments.

[0018] Currently, there is a great development of applications and devices for child tracking in consumer Internet of Things (IoT) devices. These devices are usually equipped with a Global Positioning System (GNSS) receiver for tracking geographic location, an Inertial Measurement Unit (IMU) for collecting motion and status information, and a camera for video chat or photo taking.

[0019] However, these devices still have many shortcomings when encountering child traffickers. First, child traffickers usually have a full understanding of such safety devices so that they can immediately identify and remove the safety devices on the children as they approach the children, preventing the children's location from being tracked. Although many manufacturers have tried to design safety devices to be more concealed in order to avoid the above situation, the effect may not be ideal. These devices are still relatively obvious and easy to be discovered by child traffickers. Secondly, such safety devices usually rely on GNSS systems to achieve positioning functions. However, due to the inherent characteristics of radio frequency or wireless signals, such as multipath fading, signal strength below the threshold, etc., GNSS systems often have positioning failures due to the deterioration of the wireless communication environment. For example, on rainy days, in densely built urban areas, through tunnels or in underground facilities, the positioning function of the GNSS system often has problems or instability.

[0020] In order to at least solve the above shortcomings, the present disclosure provides a device for child safety monitoring. The safety monitoring device can be installed on an item carried by the child, such as a backpack or other accessories, and the safety monitoring device may include multiple visual sensors. The multiple visual sensors can be installed on different sides of the backpack to cover a 360-degree field of view, so that the safety monitoring device can capture a complete view of the surrounding environment. With the help of visual images captured by multiple visual sensors, the positioning problem of the GNSS system under poor wireless channel conditions can be compensated. For example, relevant location information, such as landmark buildings or environments, can be obtained based on visual images. For another example, positioning information can be obtained based on visual images using a visual simultaneous localization and mapping (VSLAM) method.

[0021] In addition, the safety monitoring device may also include a GNSS receiver to obtain positioning information under excellent wireless communication conditions, such as an outdoor environment, a satellite signal reception strength exceeding a threshold, etc.

[0022] Figure 1A An exemplary structure having a safety monitoring device according to one or more aspects of the present disclosure is shown. The structure 100A has the shape of a backpack and includes a plurality of visual sensors 110-a, for example, three or more, respectively, on two straps and the front of the backpack. Examples of visual sensors may include cameras, webcams, video cameras, camcorders, or any device suitable for capturing images or videos of the external environment. The structure 100A may also include a circuit board 120-a disposed inside the backpack. The circuit board 120-a may include a GNSS receiver, a battery, one or more control processing units, a communication module, and other components.

[0023] Figure 1B Another exemplary structure having a safety monitoring device according to one or more aspects of the present disclosure is shown. The structure 100B may have a single shoulder shape and include a plurality of visual sensors 110-b on both sides of the shoulder strap, respectively. The structure 100B may also include a circuit board 120-b disposed inside the backpack.

[0024] According to one or more aspects of the present disclosure, the security monitoring device can also be installed in other objects or accessories, such as clothing, so that only the lens portion of the multiple visual sensors in the security monitoring device is exposed, and the rest is hidden inside the object, so that it is not easy to be noticed. In addition, the multiple visual sensors can be oriented in different directions to cover a 360-degree azimuth angle. For example, the fields of view of each of the multiple visual sensors can overlap with each other, or not overlap.

[0025] Figure 2 An exemplary system for safety monitoring according to one or more aspects of the present disclosure is shown. The system 200 for safety monitoring may include a plurality of visual sensors 222 (e.g., a first visual sensor 202, etc.), a first control processing unit 210, a second control processing unit 220, a microphone 203, an alarm 204, an inertial measurement unit (IMU) 205, a GNSS receiver 206, a battery 207, a cellular network communication module 208, and a personal area network communication module 209, wherein the first control processing unit 210 may be connected to the second control processing unit 220, the first control processing unit 210 may be connected to the plurality of visual sensors 222 for receiving visual input 201, and the second control processing unit 220 may be connected to other components.

[0026] According to one aspect of the present disclosure, the first control processing unit 210 may include a computing unit dedicated to visual data processing, for example, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA) designed to be suitable for processing parallel video streams. Compared with the second control processing unit 220, the first control processing unit 210 can be more efficient in processing visual data, and the processing result of the first control processing unit 210 on the visual data can be used as an input of the second control processing unit 220, so that the second control processing unit 220 further processes the input or combines the input with the processing results of other components to perform corresponding operations. The second control processing unit 220 may include a general-purpose processor, for example, a central processing unit (CUP), a main control unit (MCU), etc. The second control processing unit 220 can be used to manage and control all peripheral components, such as microphone 203, alarm 204 (e.g., buzzer), inertial measurement unit (IMU) 205, GNSS receiver 206, battery 207, cellular network communication module 208 and personal area network communication module 209, and can execute the relevant algorithms and operations described herein, such as VSLAM. Since the first control processing unit 210 processes the video and / or image data more efficiently in advance, the power of the second control processing unit 220 and even the entire system can be saved.

[0027] According to other aspects of the present disclosure, the first control processing unit 210 can perform operations such as video decoding, generating panoramic views, generating rough three-dimensional point cloud data, generating semantic information, or face recognition. For example, the first control processing unit 210 can perform image fusion to generate a panoramic view (e.g., a 360-degree panoramic view) based on multiple images captured by multiple visual sensors facing multiple directions. For another example, the first control processing unit 210 can perform object classification or recognition on the image to indicate or determine the landmarks, platforms, or faces contained in the image. The second control processing unit 220 can perform operations related to components such as the IMU 205 and the microphone 203, as well as control the alarm 204 to alarm, or control the cellular network communication module 208 and the personal area network communication module 209 to perform wireless communications, etc.

[0028] Figure 3AAn exemplary method for safety monitoring according to one or more aspects of the present disclosure is shown. Method 300A may be applicable to system 200. In step 310, the second control processing unit 220 may receive audio data from the microphone 203, and input the audio data as a feature to a trained machine learning model to predict dangerous events. For example, screaming, crying, and anxious sounds often indicate the occurrence of danger. The machine learning model can predict dangerous events, such as the probability of a dangerous event, by detecting such abnormal sounds.

[0029] In step 330, based on the probability of the occurrence of the dangerous event, it is determined whether to enable or disable the multiple visual sensors 222 and / or the positioning function. For example, when the probability is equal to or exceeds the threshold, the multiple visual sensors are enabled to capture visual images and perform positioning operations to obtain the child's safety status in time. When the probability is lower than the threshold, the multiple visual sensors and / or the positioning function are disabled to protect the child's privacy and save power.

[0030] According to other aspects of the present disclosure, in step 310, the second control processing unit 220 may receive inertial measurement data from the IMU 205 and pass the inertial measurement data as an input feature to a trained machine learning model to predict dangerous events. Similarly, movement conditions such as being picked up, grappling, and chasing may indicate the occurrence of danger. The machine learning model can predict dangerous events by detecting such abnormal movements. For example, the inertial measurement data can be passed to the machine learning model as an additional input together with the audio data, or the inertial measurement data can replace the audio data as an input feature of the machine learning model.

[0031] Figure 3B Another exemplary method for security monitoring according to one or more aspects of the present disclosure is shown. Method 300B may be applicable to system 200. In step 320, when the system 200 receives an indication signal or determines that it is away from a predetermined distance from the parent (e.g., more than 100 meters), it turns on multiple visual sensors 222 to capture images and / or start performing positioning operations. For example, the indication signal may be sent from the parent's mobile terminal (e.g., a smart phone, a smart watch, etc.) to the system 200, or the system 200 may estimate the distance to the parent's mobile terminal (e.g., receiving positioning information from the parent's mobile terminal). In step 340, multiple visual sensors 222 may continuously capture images and / or perform positioning operations, so that even if the security monitoring device is removed from the child, the image of the suspect has been captured, and the system 200 may immediately report the incident and the image of the suspect to the child's parents and / or the local police.

[0032] According to other aspects of the present disclosure, at step 320 , a plurality of visual sensors 222 may be turned on and / or a positioning operation may be started in response to reaching a predetermined time.

[0033] Figure 4 The scenario to which the safety monitoring method and device according to one or more aspects of the present disclosure are applicable is shown. The scenario 400 may include a safety monitoring device 420 wearable on a child, a network terminal device 410 (e.g., may include a server 412 and a database 414), and a remote monitoring terminal 430 (e.g., may include a parent 434 and its terminal device 432). According to one or more aspects of the present disclosure, the safety monitoring device 420 may collect images via multiple visual sensors 110-a, perform face recognition on the images, and send the results (e.g., via the cellular network communication module 208) to the network terminal device 410 via a wireless channel 402. For example, the results may be compared with the suspect information in the child trafficking suspect database 414 or with a list of safe contacts. If the comparison is successful for the suspect information or fails for the list of safe contacts, the server 412 may (e.g., via the wireless channel 402) send an alarm signal to the safety monitoring device 420 to trigger the alarm 204 to alarm and remind the child to escape in time. The server 412 may also send an alarm signal to a remote monitoring terminal 430 (eg, a parent's terminal device 432 ) or a local police (eg, via the wireless channel 402 ).

[0034] According to other aspects of the present disclosure, parents or local police can (for example, in an emergency) remotely turn on multiple visual sensors 110-a in the safety monitoring device 420 to obtain additional information about the child's surroundings to help determine the child's location. For example, the surrounding landmarks or places included in the captured image can be used to help determine the child's location. In addition, the images captured by the multiple visual sensors 110-a can also be used for VSLAM to synchronously create a three-dimensional representation of the surrounding environment and map it to a high-resolution map. The IMU in the safety monitoring device 420 can be used to generate 6-dimensional direction and rotation data for camera calibration and VSLAM. The three-dimensional point cloud data generated based on the images captured by the multiple visual sensors 110-a can also be mapped to the map.

[0035] The circuit board 120-a of the security monitoring device 420 may include at least two communication modules, for example, a cellular network communication module 208 for wireless cellular communication technology (e.g., 3G, LTE, 5G and other standards or protocols) and a personal area network communication module 209 for short-range connection (e.g., supporting Bluetooth, IrDA, HomeRF, or UWB and other protocols or technologies). The cellular network communication module 208 can be used to transmit image data, video streams and / or audio streams and positioning information to the network terminal device 410 and / or the remote monitoring terminal 430 in real time, and to receive signals from the network terminal device 410 and / or the remote monitoring terminal 430. The personal area network communication module 209 can be used to communicate with a smart device (e.g., a smart phone, a smart watch, a smart wristband or other wearable device) to configure the security monitoring device 420.

[0036] According to other aspects of the present disclosure, in order to protect the privacy of the child, the safety monitoring device 420 may not directly send the collected audio and video data to the remote monitoring terminal 430 (e.g., the parent's terminal device 432), but encrypt the audio and video data, and send and temporarily store the encrypted audio and video data (e.g., for a period of one month) on a third-party server. The stored data or real-time data stream will only be accessed at the request of an administrative department (e.g., the police).

[0037] Figure 5 5 is a flowchart showing an exemplary method for security monitoring according to one or more aspects of the present disclosure. Method 500 may be performed in combination with method 300A or method 300B. In step 510, visual images may be captured in multiple directions by multiple visual sensors, for example, the field of view of the multiple visual sensors may cover an azimuth of 360 degrees. Specifically, step 510 may also include fusing the visual images captured by the multiple visual sensors. In step 520, the wireless channel condition may be determined, for example, determining whether the signal reception strength of the satellite signal is greater than a threshold. When the signal reception strength is equal to or greater than the threshold, it indicates that the wireless channel condition is good, and method 500 may proceed to step 530, wherein positioning information may be obtained based on a global satellite navigation system. When the signal reception strength is less than the threshold, it indicates that the wireless channel condition is poor (for example, when in an indoor environment, an underground environment, or in a moving vehicle, or in a bad weather environment (for example, rainy days, heavy fog, etc.)), method 500 may proceed to step 540, wherein positioning information may be obtained based on the captured visual image using a VSLAM algorithm.

[0038] The various units, components, operations, models, and networks described in conjunction with the present disclosure may be implemented as hardware, software executed by a processor, firmware, or any combination thereof. The processor may be a general-purpose processor, or may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or other such structures. The devices, apparatus, or hardware described herein may also include a memory. The memory may store input data, output data, trained machine learning models (e.g., including optimized parameters and hyperparameters, etc.), data generated by the processor, and / or instructions executed by the processor.

[0039] In one embodiment, the safety monitoring device can perform positioning processing based on at least one of a GNSS receiver or a visual image (for example, it can obtain surrounding scene information based on image classification or recognition algorithms, obtain positioning information based on VSLAM methods, or a combination thereof. For example, the MCU can obtain positioning information by turning on the GNSS receiver and receiving its output) to adapt to different application scenarios and improve the stability and reliability of the safety monitoring device. The safety monitoring device can be used in various occasions, including but not limited to child safety monitoring, adventure activities, etc.

[0040] In one embodiment, the safety monitoring device may include visual sensors, auditory sensors, tactile sensors, motion sensors or other suitable sensors to collect sufficient information about the surrounding environment. A machine learning algorithm may be used to use the collected information as input to infer the probability of danger and provide timely warnings.

[0041] In one embodiment, the security monitoring device may include multiple control processors, for example, an acoustic processor suitable for speech processing, a graphics processor suitable for image processing, a parallel processor suitable for machine learning algorithms, etc., so that the overall computing efficiency of the security monitoring device can be improved to save power.

[0042] The above description of the present disclosure is provided to enable those skilled in the art to use or implement various embodiments. Various modifications to the above embodiments will be apparent to those skilled in the art, and the basic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the scope of the claims is not intended to be limited to the embodiments disclosed herein, but rather to be given the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A device for safety monitoring, comprising: A plurality of visual sensors for capturing visual images; Global Navigation Satellite System (GNSS) receivers; as well as One or more control processing units are coupled to the plurality of visual sensors and the GNSS receiver and are configured to perform positioning processing based on at least one of the visual images or the GNSS receiver.

2. The apparatus according to claim 1, further comprising a microphone, and the one or more control processing units are further configured to: Based on the audio data collected by the microphone, a machine learning model is used to detect dangerous events.

3. The device according to claim 2, wherein: The one or more control processing units are further configured to: In response to detecting the dangerous event, activating the plurality of visual sensors to capture visual images; In response to detecting the dangerous event, performing the positioning process; or Do both of the above.

4. The device according to claim 1, wherein: The plurality of visual sensors are used to continuously capture visual images at least during a period of time.

5. The apparatus according to claim 2, further comprising: An inertial measurement unit (IMU), a battery, a cellular network communication module, a personal area network communication module, and an alarm are coupled to the one or more control processing units.

6. The device according to claim 5, wherein: The one or more control processing units are further configured to: Face recognition is performed based on the visual image to detect suspicious persons.

7. The device according to claim 6, wherein: The one or more control processing units are further configured to: In response to detecting the suspicious person, the siren is triggered to sound an alarm.

8. The device according to claim 5, wherein: The performing positioning processing comprises: Based on the visual image, or the visual image together with the inertial data obtained by the IMU, a visual simultaneous localization and mapping (VSLAM) method is used to obtain positioning information.

9. The device according to claim 5, wherein: The machine learning model is also used to detect dangerous events based on inertial data obtained by the IMU.

10. The device according to claim 9, wherein: The one or more control processing units include: a first control processing unit, configured to perform visual data processing, wherein the visual data processing includes video decoding, generating a panoramic view, generating three-dimensional point cloud data, generating semantic information, or face recognition; and A second control processing unit is used to detect dangerous events using a machine learning model based on at least one of the inertial data and the audio data, or to map the generated three-dimensional point cloud data to a map.

11. The device according to claim 10, wherein: The second control processing unit is also used to execute the VSLAM method.

12. The device according to claim 1, wherein: The apparatus is a wearable device for safety monitoring, wherein the plurality of visual sensors are mounted on different surfaces of the wearable device; and the wearable device further comprises: A circuit board is placed inside the wearable device, and the GNSS receiver and the one or more control processing units are arranged on the circuit board.

13. The device according to claim 12, wherein: Include at least one of the following: A microphone is also provided on the circuit board, and the one or more control processing units are further configured to be suitable for: detecting dangerous events using a machine learning model based on audio data collected by the microphone; or The circuit board is also provided with: an inertial measurement unit (IMU) coupled to the one or more control processing units, a battery, a cellular network communication module, a personal area network communication module and an alarm.

14. A method for safety monitoring, comprising: Capturing visual images in multiple directions by multiple visual sensors; Based on the visual image, using a visual simultaneous localization and mapping (VSLAM) method to obtain positioning information, or / and The positioning information is obtained based on a global satellite navigation system.

15. The method according to claim 14, further comprising: Determine wireless channel conditions; The positioning information is obtained according to the determined wireless channel conditions by using a VSLAM method based on the visual image or by at least one of the methods based on the global satellite navigation system.

Images