Person falling into water monitoring method, device, electronic device and storage medium

Abstract

An embodiment of the present invention provides a method for monitoring a person falling into water. When a person falling into water is detected, an entry video of the person falling into water and hull vibration information are acquired; according to the entry video and the hull vibration information, it is determined whether the person falling into water is a person who has fallen into water; if it is determined that the person falling into water is a person who has fallen into water, a prompt is sent to relevant management departments. By combining the entry video of the person falling into water and the hull vibration information to determine whether the person falling into water is a person who has fallen into water, since the vibration conditions caused by diving and falling into water on the hull are different, therefore, on the basis of the entry video, the hull vibration information can be combined to more accurately determine whether it is a person who has fallen into water. At the same time, it can automatically prompt relevant management departments about the person who has fallen into water, improving the management efficiency of relevant management departments for scenic spots.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and particularly to a method, device, electronic device and storage medium for monitoring people falling into water. Background Art

[0002] A cruise ship is a water vehicle that provides water play for tourists. Generally, cruise ship services are offered in scenic spots where ships can be built. Tourists can board the cruise ship to sightsee on the lake. However, due to various accidental reasons, there is a possibility that tourists may fall into the water. For example, tourists may accidentally fall into the water while playing on the ship, or slip and fall into the water. However, some tourists may also jump into the water by themselves to play in the water. In the case where there is no targeted witness to the specific situation of a tourist falling into the water, it is also very difficult for other tourists to quickly distinguish whether the tourist in the water jumped into the water actively or was accidentally dropped into the water when they see someone in the water, and it takes a certain amount of time to observe, thus shortening the rescue window for the person who has fallen into the water. Summary of the Invention

[0003] An embodiment of the present invention provides a method for monitoring people falling into water, aiming to solve the problem in the prior art that when a person falling into water is found, it takes a certain amount of observation time to determine that the person falling into water is a person who has fallen into the water accidentally, thus shortening the rescue window for the person who has fallen into the water. By combining the entry video of the person falling into water and the ship body jitter information to determine whether the person falling into water is a person who has fallen into the water accidentally. Since the jitter situations caused by diving and falling into water on the ship body are different, therefore, on the basis of the entry video, it is possible to more accurately determine whether it is a person who has fallen into the water by combining the ship body jitter information. At the same time, it can automatically prompt the relevant management department about the person who has fallen into the water, improving the management efficiency of the relevant management department for the scenic spot.

[0004] In a first aspect, an embodiment of the present invention provides a method for monitoring people falling into water, the method comprising:

[0005] When a person falling into water is detected, obtain the entry video of the person falling into water and the ship body jitter information;

[0006] According to the entry video and the ship body jitter information, determine whether the person falling into water is a person who has fallen into the water accidentally;

[0007] If it is determined that the person falling into water is a person who has fallen into the water accidentally, prompt the relevant management department.

[0008] Optionally, before obtaining the entry video of the person falling into water and the ship body jitter information when the person falling into water is detected, the method further comprises:

[0009] Visually track the people on the cruise ship to obtain the tracking trajectory of the tracked people;

[0010] If the latest position of the tracking trajectory is outside the preset hull area, it is determined that the person being tracked corresponding to the tracking trajectory is a person who has fallen into the water.

[0011] Optionally, when a person falling into the water is detected, the video of the person falling into the water and the hull vibration information are obtained, including:

[0012] Determine the sampling time according to the tracking trajectory and the hull area;

[0013] Determine the video of the person falling into the water according to the sampling time, and determine the hull vibration information according to the sampling time.

[0014] Optionally, determining whether the person falling into the water is a person who has accidentally fallen into the water according to the video of the person falling into the water and the hull vibration information includes:

[0015] Perform state recognition on each frame of the video of the person falling into the water to obtain the sequence of the state of the person falling into the water;

[0016] Determine the hull vibration sequence according to the hull vibration information;

[0017] Determine whether the person falling into the water is a person who has accidentally fallen into the water according to the sequence of the state of the person falling into the water and the hull vibration sequence.

[0018] Optionally, performing state recognition on each frame of the video of the person falling into the water to obtain the sequence of the state of the person falling into the water includes:

[0019] Perform clothing recognition on each frame of the video of the person falling into the water to obtain the sequence of the clothing state of the person falling into the water;

[0020] Perform action recognition on each frame of the video of the person falling into the water to obtain the sequence of the action state of the person falling into the water;

[0021] Perform expression recognition on each frame of the video of the person falling into the water to obtain the sequence of the expression state of the person falling into the water;

[0022] Determine the sequence of the state of the person falling into the water based on the sequence of the clothing state, the sequence of the action state, and the sequence of the expression state.

[0023] Optionally, determining whether the person falling into the water is a person who has accidentally fallen into the water according to the sequence of the state of the person falling into the water and the hull vibration sequence includes:

[0024] Fuse the sequence of the state of the person falling into the water and the hull vibration sequence to obtain a fused sequence;

[0025] Determine whether the person falling into the water is a person who has accidentally fallen into the water based on the fused sequence.

[0026] Optionally, determining whether the person entering the water is a person who has fallen into the water based on the fusion sequence includes:

[0027] Transform the fusion sequence from the time domain to the frequency domain to obtain the frequency domain characteristics of the fusion sequence;

[0028] Compare the frequency domain characteristics with preset reference frequency domain characteristics;

[0029] If the comparison is successful, determine that the person entering the water is not a person who has fallen into the water;

[0030] If the comparison fails, determine that the person entering the water is a person who has fallen into the water.

[0031] In a second aspect, an embodiment of the present invention provides a device for monitoring a person falling into the water, where the device includes:

[0032] An acquisition module, configured to acquire the entry video of the person entering the water and the hull vibration information when detecting a person entering the water;

[0033] A first determination module, configured to determine whether the person entering the water is a person who has fallen into the water according to the entry video and the hull vibration information;

[0034] A second determination module, configured to, if it is determined that the person entering the water is a person who has fallen into the water, give a prompt to the relevant management department.

[0035] In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, the steps in the method for monitoring a person falling into the water provided by the embodiment of the present invention are implemented.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the method for monitoring a person falling into the water provided by the embodiment of the invention are implemented.

[0037] In the embodiment of the present invention, when detecting a person entering the water, the entry video of the person entering the water and the hull vibration information are acquired; according to the entry video and the hull vibration information, it is determined whether the person entering the water is a person who has fallen into the water; if it is determined that the person entering the water is a person who has fallen into the water, a prompt is given to the relevant management department. By combining the entry video of the person entering the water and the hull vibration information to determine whether the person entering the water is a person who has fallen into the water, since the vibration conditions of the hull caused by diving and falling into the water are different, therefore, on the basis of the entry video, the hull vibration information can be combined to more accurately determine whether it is a person who has fallen into the water. At the same time, it can automatically prompt the relevant management department about the person who has fallen into the water, improving the management efficiency of the relevant management department for the scenic area. Description of the Drawings

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0039] Figure 1 is a flowchart of a method for monitoring a person falling into water provided by an embodiment of the present invention;

[0040] Figure 2 is a schematic structural diagram of a device for monitoring a person falling into water provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed Embodiments

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0043] Please refer to Figure 1 , Figure 1 which is a flowchart of a method for monitoring a person falling into water provided by an embodiment of the present invention. As shown in Figure 1 , the method for monitoring a person falling into water is used for monitoring a person falling into water on a cruise ship. The method for monitoring a person falling into water includes the following steps:

[0044] 101. When a person entering the water is detected, obtain the video of the person entering the water and the information on the ship's body vibration.

[0045] In the embodiment of the present invention, the above-mentioned person entering the water is a person in the water. The water surface around the cruise ship can be subjected to target detection through an image monitoring device to determine whether there is a person on the water surface around the cruise ship. If there is a person on the water surface around the cruise ship, the person is determined as a person entering the water.

[0046] In a possible embodiment, the personnel on the cruise ship can be tracked and detected through an image monitoring device provided on the cruise ship, and whether there is a person entering the water can be determined according to the tracking and detection results.

[0047] It should be noted that the above-mentioned cruise ships are small or medium-sized cruise ships, which carry less than 10 tourists, making it easier to capture the vibration information of the hull.

[0048] The above-mentioned water entry video can be a video of the period before the water entry is detected, for example, it can be a video of the first 30 seconds when the water entry is detected.

[0049] The above-mentioned water entry video can be obtained through an image monitoring device installed on the cruise ship. The image monitoring device can store the collected video. When a person entering the water is detected, the video of the previous period of time is searched in the stored videos according to the time when the person entering the water is detected, so as to obtain the water entry video of the person entering the water.

[0050] The hull vibration information can be acquired by sensors installed on the hull, such as acceleration sensors, with a sampling rate equal to that of the image monitoring device. Sensor data over a period of time can be stored. When a person entering the water is detected, the stored sensor data is searched for sensor data from a previous period based on the time of detection to obtain the corresponding hull vibration information.

[0051] 102. Based on the water entry video and the hull shaking information, determine whether the person entering the water is a person who fell into the water.

[0052] In this embodiment of the present invention, the aforementioned water entry video records the visual information of the person entering the water, and the hull vibration information records the hull vibration of the person entering the water. It is understood that the aforementioned person entering the water can be a person who falls into the water or a person who dives into the water. The aforementioned person who falls into the water can be understood as passively entering the water, such as by slipping and falling into the water, while the aforementioned person who dives into the water can be understood as actively entering the water, such as by jumping into the water to swim.

[0053] The preparations, actions, and expressions of people who fall into the water and people who dive into the water are different before they enter the water. Specifically, the diver has preparatory movements before diving, has a normal expression, and the shaking of the hull is consistent with the shaking of the power-accumulation type. The person who falls into the water has no preparatory movements before falling into the water, has an abnormal expression, and the shaking of the hull does not conform to the shaking of the power-accumulation type. Therefore, the visual information in the water entry video can be used to obtain information such as the preparation, actions, and expressions of the person who enters the water. Based on the shaking information of the hull, it can be judged whether the shaking of the hull conforms to the shaking of the power-accumulation type, thereby judging whether the person who enters the water is the person who falls into the water. For example, if the preparation, actions, and expressions are consistent with those of a diver, and the shaking of the hull conforms to the shaking of the power-accumulation type, then it can be determined that the person who enters the water is the person who falls into the water. If the preparation, actions, and expressions are not consistent with those of a diver, and the shaking of the hull does not conform to the shaking of the power-accumulation type, then it can be determined that the person who enters the water is the person who falls into the water.

[0054] 103. If it is determined that the person entering or leaving the water is a person who has fallen into the water, a prompt will be sent to the relevant management department.

[0055] In the embodiment of the present invention, after it is determined that the person entering or leaving the water is a person who has fallen into the water, a prompt can be sent to the relevant management department so that the relevant management department can take rescue actions for the person who has fallen into the water. At the same time, if swimming is prohibited in the scenic area, after it is determined that the person entering or leaving the water is a diving person, a prompt can also be sent to the relevant management department so that the relevant management department can take warning actions for the diving person.

[0056] In the embodiment of the present invention, when a person entering the water is detected, the entry video of the person entering the water and the hull vibration information are obtained; according to the entry video and the hull vibration information, it is determined whether the person entering the water is a person who has fallen into the water; if it is determined that the person entering the water is a person who has fallen into the water, a prompt will be sent to the relevant management department. By combining the entry video of the person entering the water and the hull vibration information to determine whether the person entering the water is a person who has fallen into the water, since the vibration conditions caused by diving and falling into the water on the hull are different, therefore, on the basis of the entry video, the hull vibration information can be combined to more accurately determine whether it is a person who has fallen into the water. At the same time, it can automatically prompt the relevant management department about the person who has fallen into the water, improving the management efficiency of the relevant management department for the scenic area.

[0057] Optionally, before the step of obtaining the entry video of the person entering the water and the hull vibration information when a person entering the water is detected, visual tracking can also be performed on the people on the cruise ship to obtain the tracking trajectory of the tracked person; if the latest position of the tracking trajectory is outside the preset hull area, it is determined that the tracked person corresponding to the tracking trajectory is a person entering the water.

[0058] In the embodiment of the present invention, the surface of the cruise ship can be monitored by an image monitoring device. The above image monitoring device can be a panoramic monitoring device set on the ship's roof, or multiple image monitoring devices used to monitor the surface of the cruise ship in different directions.

[0059] Through the image monitoring device, the monitoring image of the surface of the cruise ship can be obtained. Whether there are people on the surface of the cruise ship is detected by a target detection algorithm. If there are people, the person will be determined as the tracked person, and visual tracking of the tracked person is performed by a video tracking algorithm to obtain the tracking trajectory of the tracked person.

[0060] Through an image segmentation algorithm, frame images in the monitoring image are segmented, and then the hull area and the water surface area of the cruise ship are segmented. After the hull area is determined, it is judged whether the latest position of the tracking trajectory corresponding to the tracked person is located in the hull area. If it is located in the hull area, visual tracking of the tracked person will continue. If it is outside the hull area (i.e., in the water surface area), it is determined that the tracked person is a person entering the water.

[0061] By determining the relationship between the tracking trajectory of the tracked person and the position of the hull area, it can be determined that the tracked person is a person who has entered the water, and it can be determined that the person who has entered the water has entered the water from the cruise ship.

[0062] Optionally, in the step of obtaining the entry video of the person entering the water and the hull vibration information when detecting the person entering the water, the sampling time can also be determined according to the tracking trajectory and the hull area; the entry video of the person entering the water is determined according to the sampling time, and the hull vibration information is determined according to the sampling time.

[0063] In the embodiment of the present invention, when the latest position of the tracking trajectory is outside the hull area, there is an intersection between the tracking trajectory and the boundary of the hull area, and the sampling start time is determined by the trajectory length extending a preset distance into the hull area from the intersection, and the sampling end time is the time when the person entering the water is detected, and the sampling time is determined according to the sampling start time and the sampling end time. For example, for a trajectory length of extending 1 meter into the hull area from the intersection, the earliest time of this extended trajectory can be used as the sampling start time.

[0064] After determining the sampling time, find the tracking video of the person entering the water in the stored surveillance video according to the sampling time, and intercept the video segment corresponding to the sampling time in the tracking video of the person entering the water as the entry video of the person entering the water. Intercept the sensor data corresponding to the sampling time in the stored sensor data according to the sampling time as the hull vibration information.

[0065] Optionally, in the step of determining whether the person entering the water is a person who has fallen into the water according to the entry video and the hull vibration information, the state of each frame of image in the entry video can be recognized to obtain the entry state sequence of the person entering the water; the hull vibration sequence is determined according to the hull vibration information; whether the person entering the water is a person who has fallen into the water is determined according to the entry state sequence and the hull vibration sequence.

[0066] In the embodiment of the present invention, the entry video includes multiple frames of images, and each frame of image corresponds to a state of the person entering the water. The water release video corresponds to the continuous state of the person entering the water from the ship to the water. It can be understood that the continuous state of the diving person is different from the continuous state of the person who has fallen into the water. The entry state sequence corresponding to the person who has fallen into the water is different from the water release state sequence corresponding to the diving person. Therefore, it can be judged whether the person entering the water is a person who has fallen into the water or a diving person according to the entry state sequence of the person entering the water.

[0067] The hull vibration information includes the vibration value of the hull. The hull vibration sequence represents consecutive hull vibration values. The vibration value is affected by the cruise ship engine, water surface waves, and human activities. Among them, the cruise ship engine has a continuous linear impact, the water surface waves have a periodic impact, and the impact of human activities is related to the number of people and their movements. Since the falling-into-water personnel and the diving personnel have different posture movements when leaving the hull, different vibration value distributions will be caused. For example, the diving personnel will have preparatory movements before diving, while the falling-into-water personnel do not have preparatory movements. The above preparatory movements may include warming up, building up strength for takeoff, etc. Therefore, it is possible to determine whether the person entering the water is a falling-into-water person or a diving person based on the hull vibration sequence.

[0068] Specifically, since the preparatory movements for diving have a certain convergence, while the movements for falling into the water have more randomness, several reference entry states sequences of diving personnel and the corresponding reference hull vibration sequences can be determined in advance according to different preparatory movements. Compare the entry state sequence of the person entering the water with the reference entry state sequence, and compare the hull vibration sequence with the reference hull vibration sequence. If the comparison is successful, it can be determined that the person entering the water is a diving person; if the comparison fails, it can be determined that the person entering the water is a falling-into-water person.

[0069] Optionally, in the step of performing state recognition on each frame image in the entry video to obtain the entry state sequence of the person entering the water, clothing recognition can be performed on each frame image in the entry video to obtain the clothing state sequence of the person entering the water; action recognition can be performed on each frame image in the entry video to obtain the action state sequence of the person entering the water; expression recognition can be performed on each frame image in the entry video to obtain the expression state sequence of the person entering the water; and based on the clothing state sequence, action state sequence, and expression state sequence, determine the entry state sequence of the person entering the water.

[0070] In the embodiment of the present invention, the above clothing recognition result includes the clothing state. The clothing state includes a dressed state and an undressed state. The above dressed state means wearing a normal upper garment, and the above undressed state means not wearing an upper garment or wearing a swimsuit. The clothing recognition algorithm can be used to perform clothing recognition on each frame image in the entry video to recognize the clothing state of the person entering the water before and after entering the water. Generally speaking, the diving personnel are in the undressed state before diving, and the falling-into-water personnel are in the dressed state before falling into the water. Arrange the clothing states corresponding to each frame image in the order of the frame images to obtain the clothing state sequence.

[0071] The above action recognition results include action states, and the action states include a preparatory action state and an abnormal action state. Among them, the preparatory action state refers to the action state that conforms to the diving preparation, and the abnormal action state refers to the action state that does not conform to the diving preparation. The action recognition algorithm can be used to perform action recognition on each frame image in the water-entry video to recognize the action states of the water-entry personnel before and after entering the water. Generally speaking, the diving personnel are in the preparatory action state before diving, and the water-entry personnel are in the abnormal action state before entering the water. Arrange the action states corresponding to each frame image in the order of the frame images to obtain an action state sequence.

[0072] The above expression recognition results include expression states, and the expression states include a normal expression state and an abnormal expression state. Among them, the normal expression state can be a neutral expression and a positive expression, such as no expression, smile, etc., and the abnormal expression state can be a negative expression, such as panic, flustered, etc. Arrange the expression states corresponding to each frame image in the order of the frame images to obtain an expression state sequence.

[0073] After obtaining the clothing state sequence, the action state sequence, and the expression state sequence, the clothing state sequence, the action state sequence, and the expression state sequence can be fused to obtain a water-entry state sequence.

[0074] Optionally, in the step of determining whether the water-entry personnel are water-drop personnel according to the water-entry state sequence and the ship body jitter sequence, the water-entry state sequence and the ship body jitter sequence can be fused to obtain a fusion sequence; based on the fusion sequence, determine whether the water-entry personnel are water-drop personnel.

[0075] In the embodiment of the present invention, the water-entry state sequence and the ship body jitter sequence can be linearly fused. Specifically, the water-entry state sequence an contains n state values and is a one-dimensional matrix of n×1. The ship body jitter sequence bn contains n jitter values and is also a one-dimensional matrix of n×1. The water-entry state sequence an and the ship body jitter sequence bn can be spliced by channels to obtain a two-dimensional matrix of n×2. A matrix multiplication calculation is performed on the two-dimensional matrix of n×2 through a 2×1 matrix. According to the matrix multiplication calculation rule of n×2*2×1 matrix, a one-dimensional matrix of n×1 can be obtained, and the calculated one-dimensional matrix of n×1 is used as the fusion sequence.

[0076] After obtaining the fusion sequence, a classification network can be used to classify the fusion sequence. The classification results include water-drop personnel and non-water-drop personnel. The above non-water-drop personnel are diving personnel. The above classification network can be a network that can process time series, such as a recurrent neural network RNN or a long short-term memory network LSTM.

[0077] Of course, since non-falling-into-water personnel are generally diving personnel, and due to the certain convergence of the preparatory actions for diving, while the actions of falling into the water have more randomness, several reference fusion state sequences of diving personnel can be determined in advance according to different preparatory actions. Compare the frequency domain features of the person entering the water with the reference fusion sequences. If the comparison is successful, it can be determined that the person entering the water is a diving personnel; if the comparison fails, it can be determined that the person entering the water is a falling-into-water personnel.

[0078] Optionally, in the step of determining whether the person entering the water is a falling-into-water personnel based on the fusion sequence, the fusion sequence can be transformed from the time domain to the frequency domain to obtain the frequency domain features of the fusion sequence; compare the frequency domain features with the preset reference frequency domain features; if the comparison is successful, it is determined that the person entering the water is a non-falling-into-water personnel; if the comparison fails, it is determined that the person entering the water is a falling-into-water personnel.

[0079] In the embodiment of the present invention, the above-mentioned fusion sequence is obtained by fusing multiple complex sequences, which contains more dimensional changes. The fusion sequence can be regarded as the waveform fusion of multiple signals, so that the fusion sequence can be transformed to the frequency domain to obtain the frequency domain features of the fusion sequence. The frequency domain features represent the frequency components of each signal in the fusion sequence. Several reference frequency domain features of diving personnel can be determined in advance, and the frequency domain features of the person entering the water are compared with the reference frequency domain features. If the comparison is successful, it can be determined that the person entering the water is a diving personnel; if the comparison fails, it can be determined that the person entering the water is a falling-into-water personnel.

[0080] It should be noted that the method for monitoring personnel falling into the water provided by the embodiment of the present invention can be applied to devices such as intelligent cameras, smartphones, computers, and servers that can perform the method for monitoring personnel falling into the water.

[0081] Optionally, please refer to Figure 2 , Figure 2 is a schematic structural diagram of a device for monitoring personnel falling into the water provided by the embodiment of the present invention. As Figure 2 shown, the device includes:

[0082] An acquisition module 201, configured to acquire the entry video of the person entering the water and the hull vibration information when detecting the person entering the water;

[0083] A first determination module 202, configured to determine whether the person entering the water is a falling-into-water personnel according to the entry video and the hull vibration information;

[0084] A second determination module 203, configured to prompt the relevant management department if it is determined that the person entering the water is a falling-into-water personnel.

[0085] Optionally, the device further includes:

[0086] A tracking module, configured to visually track the personnel on the cruise ship to obtain the tracking trajectory of the tracked personnel;

[0087] A third determination module, configured to determine that the tracked personnel corresponding to the tracking trajectory is a person who has fallen into the water if the latest position of the tracking trajectory is outside a preset hull area.

[0088] Optionally, the obtaining module 201 includes:

[0089] A first determination sub-module, configured to determine the sampling time according to the tracking trajectory and the hull area;

[0090] A second determination sub-module, configured to determine the entry video of the person who has fallen into the water according to the sampling time, and determine the hull jitter information according to the sampling time.

[0091] Optionally, the first determination module 202 includes:

[0092] An identification sub-module, configured to perform state identification on each frame of image in the entry video to obtain the entry state sequence of the person who has fallen into the water;

[0093] A third determination sub-module, configured to determine the hull jitter sequence according to the hull jitter information;

[0094] A fourth determination sub-module, configured to determine whether the person who has fallen into the water is a person who has accidentally fallen into the water according to the entry state sequence and the hull jitter sequence.

[0095] Optionally, the identification sub-module includes:

[0096] A first identification unit, configured to perform clothing identification on each frame of image in the entry video to obtain the clothing state sequence of the person who has fallen into the water;

[0097] A second identification unit, configured to perform action identification on each frame of image in the entry video to obtain the action state sequence of the person who has fallen into the water;

[0098] A third identification unit, configured to perform expression identification on each frame of image in the entry video to obtain the expression state sequence of the person who has fallen into the water;

[0099] A first determination unit, configured to determine the entry state sequence of the person who has fallen into the water based on the clothing state sequence, the action state sequence, and the expression state sequence.

[0100] Optionally, the fourth determination sub-module includes:

[0101] A fusion unit, configured to fuse the entry state sequence and the hull jitter sequence to obtain a fusion sequence;

[0102] A second determination unit, configured to determine whether the person entering the water is a person falling into the water based on the fusion sequence.

[0103] Optionally, the second determination unit includes:

[0104] A transformation subunit, configured to transform the fusion sequence from the time domain to the frequency domain to obtain the frequency domain feature of the fusion sequence;

[0105] A comparison subunit, configured to compare the frequency domain feature with a preset reference frequency domain feature;

[0106] A first determination subunit, configured to determine that the person entering the water is not a person falling into the water if the comparison is successful;

[0107] A second determination subunit, configured to determine that the person entering the water is a person falling into the water if the comparison fails.

[0108] It should be noted that the person falling into the water monitoring device provided by the embodiments of the present invention can be applied to devices such as intelligent cameras, smart phones, computers, and servers that can perform the method for monitoring a person falling into the water.

[0109] The person falling into the water monitoring device provided by the embodiments of the present invention can implement each process implemented by the person falling into the water monitoring method in the above method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be elaborated here again.

[0110] See Figure 3 , Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As Figure 3 shown, it includes: a memory 302, a processor 301, and a computer program for the person falling into the water monitoring method stored on the memory 302 and executable on the processor 301, where:

[0111] The processor 301 is configured to call the computer program stored in the memory 302 and execute the following steps:

[0112] When detecting a person entering the water, obtain the entry video of the person entering the water and the hull vibration information;

[0113] According to the entry video of the person entering the water and the hull vibration information, determine whether the person entering the water is a person falling into the water;

[0114] If it is determined that the person entering the water is a person falling into the water, prompt the relevant management department.

[0115] Optionally, before obtaining the entry video of the person entering the water and the hull vibration information when detecting a person entering the water, the method executed by the processor 301 further includes:

[0116] Visually track the personnel on the cruise ship to obtain the tracking trajectory of the tracked personnel;

[0117] If the latest position of the tracking trajectory is outside the preset hull area, determine that the tracked personnel corresponding to the tracking trajectory is a person who has fallen into the water.

[0118] Optionally, when the processor 301 executes the detection of a person who has fallen into the water, obtain the entry video of the person who has fallen into the water and the hull jitter information, including:

[0119] Determine the sampling time according to the tracking trajectory and the hull area;

[0120] Determine the entry video of the person who has fallen into the water according to the sampling time, and determine the hull jitter information according to the sampling time.

[0121] Optionally, when the processor 301 executes the determination of whether the person who has fallen into the water is a person who has accidentally fallen into the water according to the entry video and the hull jitter information, it includes:

[0122] Perform state recognition on each frame of the entry video to obtain the entry state sequence of the person who has fallen into the water;

[0123] Determine the hull jitter sequence according to the hull jitter information;

[0124] Determine whether the person who has fallen into the water is a person who has accidentally fallen into the water according to the entry state sequence and the hull jitter sequence.

[0125] Optionally, when the processor 301 executes the state recognition of each frame of the entry video to obtain the entry state sequence of the person who has fallen into the water, it includes:

[0126] Perform clothing recognition on each frame of the entry video to obtain the clothing state sequence of the person who has fallen into the water;

[0127] Perform action recognition on each frame of the entry video to obtain the action state sequence of the person who has fallen into the water;

[0128] Perform expression recognition on each frame of the entry video to obtain the expression state sequence of the person who has fallen into the water;

[0129] Based on the clothing state sequence, the action state sequence, and the expression state sequence, determine the entry state sequence of the person who has fallen into the water.

[0130] Optionally, when the processor 301 executes the determination of whether the person who has fallen into the water is a person who has accidentally fallen into the water according to the entry state sequence and the hull jitter sequence, it includes:

[0131] Fuse the water entry state sequence with the hull vibration sequence to obtain a fused sequence;

[0132] Based on the fused sequence, determine whether the person entering the water is a person who has fallen into the water.

[0133] Optionally, the step of the processor 301 determining whether the person entering the water is a person who has fallen into the water based on the fused sequence includes:

[0134] Transform the fused sequence from the time domain to the frequency domain to obtain the frequency domain characteristics of the fused sequence;

[0135] Compare the frequency domain characteristics with preset reference frequency domain characteristics;

[0136] If the comparison is successful, determine that the person entering the water is not a person who has fallen into the water;

[0137] If the comparison fails, determine that the person entering the water is a person who has fallen into the water.

[0138] The electronic device provided by the embodiments of the present invention can implement each process implemented by the method for monitoring a person falling into the water in the above method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be elaborated here.

[0139] The embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements each process of the method for monitoring a person falling into the water provided by the embodiments of the present invention, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0140] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disc, a read-only memory (ROM), or a random access memory (RAM), etc.

[0141] The above-disclosed are only the preferred embodiments of the present invention. Of course, the scope of the rights of the present invention cannot be limited by this. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope covered by the present invention.

Claims

1. A method for monitoring a person falling into water, characterized in that, For monitoring the falling of personnel on a cruise ship, it includes the following steps: When a person falling into the water is detected, obtain the falling-into-water video of the person falling into the water and the ship body vibration information; According to the falling-into-water video and the ship body vibration information, determine whether the person falling into the water is a passive falling-into-water person; specifically, perform state recognition on each frame of image in the falling-into-water video to obtain the falling-into-water state sequence of the person falling into the water; determine the ship body vibration sequence according to the ship body vibration information; according to the falling-into-water state sequence and the ship body vibration sequence, determine whether the person falling into the water is a passive falling-into-water person; If it is determined that the person falling into the water is a passive falling-into-water person, then give a prompt to the relevant management department.

2. The personnel falling into water monitoring method according to claim 1, wherein, Before obtaining the falling-into-water video of the person falling into the water and the ship body vibration information when the person falling into the water is detected, the method further includes: Perform visual tracking on the personnel on the cruise ship to obtain the tracking trajectory of the tracked person; If the latest position of the tracking trajectory is outside the preset hull area, then determine that the tracked person corresponding to the tracking trajectory is a person falling into the water.

3. The personnel falling into water monitoring method according to claim 2, characterized in that, When the person falling into the water is detected, obtaining the falling-into-water video of the person falling into the water and the ship body vibration information includes: Determine the sampling time according to the tracking trajectory and the hull area; Determine the falling-into-water video of the person falling into the water according to the sampling time, and determine the ship body vibration information according to the sampling time.

4. The personnel falling into water monitoring method according to claim 1, characterized in that, Performing state recognition on each frame of image in the falling-into-water video to obtain the falling-into-water state sequence of the person falling into the water includes: Perform clothing recognition on each frame of image in the falling-into-water video to obtain the clothing state sequence of the person falling into the water; Perform action recognition on each frame of image in the falling-into-water video to obtain the action state sequence of the person falling into the water; Perform expression recognition on each frame of image in the falling-into-water video to obtain the expression state sequence of the person falling into the water; Based on the clothing state sequence, the action state sequence and the expression state sequence, determine the falling-into-water state sequence of the person falling into the water.

5. The personnel falling into water monitoring method according to claim 1, wherein, According to the falling-into-water state sequence and the ship body vibration sequence, determining whether the person falling into the water is a passive falling-into-water person includes: Fuse the falling-into-water state sequence and the ship body vibration sequence to obtain a fusion sequence; Based on the fusion sequence, determine whether the person falling into the water is a passive falling-into-water person.

6. The personnel falling into water monitoring method according to claim 5, characterized in that, Based on the fusion sequence, determining whether the person falling into the water is a passive falling-into-water person includes: Transform the fusion sequence from the time domain to the frequency domain to obtain the frequency domain characteristics of the fusion sequence; Compare the frequency domain characteristics with the preset reference frequency domain characteristics; If the comparison is successful, then determine that the person falling into the water is a non-passive falling-into-water person; If the comparison fails, then determine that the person falling into the water is a passive falling-into-water person.

7. A personnel falling into water monitoring device, characterized in that, The device includes: An acquisition module, used to obtain the falling-into-water video of the person falling into the water and the ship body vibration information when a person falling into the water is detected; A first determination module, configured to determine whether the person falling into the water is a person who has fallen into the water passively according to the water-entry video and the hull vibration information; specifically, perform state recognition on each frame of image in the water-entry video to obtain the water-entry state sequence of the person falling into the water; determine the hull vibration sequence according to the hull vibration information; and determine whether the person falling into the water is a person who has fallen into the water passively according to the water-entry state sequence and the hull vibration sequence. A second determination module, configured to, if it is determined that the person falling into the water is a person who has fallen into the water passively, give a prompt to the relevant management department.

8. An electronic device, characterized in that, Comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the person falling-into-water monitoring method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the steps in the person falling-into-water monitoring method according to any one of claims 1 to 6 are implemented.

Images