Method and device for identifying swimming object

Abstract

The embodiment of the invention provides a method and device for recognizing a swimming object, and the method comprises the steps: carrying out the image segmentation of a target image, obtaining a segmented image, enabling the segmented image to comprise a plurality of sub-blocks, and enabling the target image to be an image obtained through the shooting of a target water area; classifying and identifying the sub-blocks in the segmented image, and determining the sub-blocks meeting a preset condition as target sub-blocks; and determining whether the swimming object exists in the target water area according to the target sub-block. Through the method and the device, the problem that swimming monitoring cannot be carried out on the open water area in the prior art is solved, and the effect of carrying out swimming monitoring on the open water area is further achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer vision, and in particular, to a method and device for identifying a swimming object. Background Art

[0002] One way to monitor swimming is through target detection. However, the posture of the human body on the water is complex and changeable, and the body parts exposed above the water are not fixed. It is often necessary to introduce additional features (such as detecting the head, shoulders, arms, etc.), which will inevitably lead to complex models and higher computing resource consumption. Another way of swimming detection requires the use of sensor data such as wearable devices for analysis, but this method is not suitable for swimming monitoring in open waters.

[0003] There is currently no effective solution to the above problems. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for identifying a swimming object, so as to at least solve the problem in the related art that swimming monitoring in open waters cannot be performed.

[0005] According to one embodiment of the present invention, a method is provided, comprising: performing image segmentation on a target image to obtain a segmented image, wherein the segmented image includes a plurality of sub-blocks, and the target image is an image taken of a target water area; classifying and identifying each of the sub-blocks through a neural network model to obtain a classification label for each of the sub-blocks, and determining a sub-block whose classification label is a target label as a target sub-block; and determining whether there is a swimming object in the target water area based on the target sub-block.

[0006] In an exemplary embodiment, determining whether there is a swimming object in the target water area according to the target sub-block includes: determining whether there is the swimming object in the target water area according to a position of the target sub-block in the segmented image.

[0007] In an exemplary embodiment, determining whether the swimming object exists in the target water area according to the position of the target sub-block in the segmented image includes: determining whether the swimming object exists in the target water area when the target sub-block is located at a non-edge position of the segmented image; and determining whether the swimming object exists in the target water area according to at least two frames of the segmented images when the target sub-block is located at an edge position of the segmented image.

[0008] In an exemplary embodiment, determining whether the swimming object exists in the target water area based on at least two frames of the segmented images includes: when a sub-block at a first edge position in a first segmented image is the target sub-block, and in at least one frame of image before or after the first segmented image, when at least one target sub-block exists among multiple sub-blocks located around the first edge position, determining that the swimming object exists in the target water area.

[0009] In an exemplary embodiment, when the sub-block at the first edge position in the first segmented image is the target sub-block, and at least one of the target sub-blocks exists in a plurality of sub-blocks located around the first edge position in at least one frame image before or after the first segmented image, determining that the swimming object exists in the target water area includes: when the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a position around the first edge position in N frames of images before the first segmented image, determining that the swimming object exists in the target water area, wherein N is an integer greater than or equal to 1; or The swimming object is determined to exist in the target water area when the sub-block on the first edge position is the target sub-block and the target sub-block exists at a peripheral position of the first edge position in the M frames of images after the first segmented image, wherein M is an integer greater than or equal to 1; or when the sub-block on the first edge position in the first segmented image is the target sub-block and the target sub-block exists at a peripheral position of the first edge position in the N frames of images before the first segmented image, and the target sub-block exists at a peripheral position of the first edge position in the M frames of images after the first segmented image, the swimming object is determined to exist in the target water area, wherein M is an integer greater than or equal to 1.

[0010] In an exemplary embodiment, when the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position at the first edge position in N frames of images before the first segmented image, determining that the swimming object exists in the target water area includes: when the sub-block at the first edge position in the first segmented image is the target sub-block, and the sub-block at the second edge position in the second segmented image is the target sub-block, determining that the swimming object exists in the target water area, wherein the second segmented image is located in any image of the N frames of images before the first segmented image, the shooting time interval between the first segmented image and the second segmented image is a first preset time length, the first edge position and the second edge position share the same edge, or the first edge position and the second edge position share the same point, or the first edge position and the second edge position are separated by a preset number of sub-blocks.

[0011] In an exemplary embodiment, when the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position at the first edge position in M ​​frames of images after the first segmented image, determining that the swimming object exists in the target water area includes: when the sub-block at the first edge position in the first segmented image is the target sub-block, and the sub-block at the third edge position in the third segmented image is the target sub-block, determining that the swimming object exists in the target water area, wherein the third segmented image is located in any image of the M frames of images after the first segmented image, the shooting time interval between the first segmented image and the third segmented image is a second preset time length, the first edge position and the third edge position share an edge, or the first edge position and the third edge position share a point, or the first edge position and the third edge position are separated by a preset number of sub-blocks.

[0012] In an exemplary embodiment, when a sub-block at a first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position of the first edge position in N frames of images before the first segmented image, and the target sub-block exists at a peripheral position of the first edge position in M ​​frames of images after the first segmented image, it is determined that the swimming object exists in the target water area, wherein M is an integer greater than or equal to 1, including: when a sub-block at a first edge position in the first segmented image is the target sub-block, and a sub-block at a second edge position in the second segmented image is the target sub-block, and a sub-block at a third edge position in the third segmented image is the target sub-block, it is determined that the swimming object exists in the target water area, wherein, The second segmented image is located in any image of the N frames of images before the first segmented image, the shooting time interval between the first segmented image and the second segmented image is a first preset time length, the first edge position and the second edge position share the same edge, or the first edge position and the second edge position share the same point, or the first edge position and the second edge position are separated by a preset number of sub-blocks, and the third segmented image is located in any image of the M frames of images after the first segmented image, the shooting time interval between the first segmented image and the third segmented image is a second preset time length, the first edge position and the third edge position share the same edge, or the first edge position and the third edge position share the same point, or the first edge position and the third edge position are separated by a preset number of sub-blocks.

[0013] In an exemplary embodiment, each of the sub-blocks is classified and identified through a neural network model to obtain a classification label for each of the sub-blocks, including: extracting features of the sub-blocks through a feature extraction module of the neural network model to obtain sub-block features; classifying and identifying the sub-block features through a classification module of the neural network model to obtain the classification label of the sub-block; and determining a sub-block whose classification label is a target label as a target sub-block, including: determining a sub-block whose classification label includes at least one of the following as the target sub-block: a swimming equipment label, a splash label, a human body part label, and a life-saving equipment label.

[0014] In an exemplary embodiment, before performing image segmentation on the target image, the method further includes: acquiring an initial image obtained by photographing the target water area; identifying the target water area in the initial image; determining a minimum enclosing rectangle of the target water area, and determining the minimum enclosing rectangle as the target image.

[0015] In an exemplary embodiment, when it is determined that the swimming object exists in the target water area, the method also includes at least one of the following: playing an alarm voice; displaying an alarm screen; controlling the distribution of life-saving equipment to the target water area; controlling the water vehicle to move to the location of the target sub-block; and sending an alarm message to a mobile terminal located near the target water area.

[0016] According to another embodiment of the present invention, there is provided a device for identifying swimming objects, comprising: a segmentation module for performing image segmentation on a target image to obtain a segmented image, wherein the segmented image comprises a plurality of sub-blocks, and the target image is an image taken of a target water area; an identification module for classifying and identifying each of the sub-blocks through a neural network model to obtain a classification label of each of the sub-blocks, and determining a sub-block whose classification label is a target label as a target sub-block; and a determination module for determining whether there is a swimming object in the target water area according to the target sub-block. According to yet another embodiment of the present invention, there is also provided a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of any of the methods described above are implemented.

[0017] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0018] According to yet another embodiment of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0019] According to yet another embodiment of the present invention, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, the steps of any of the above methods are implemented.

[0020] Through the present invention, by performing image segmentation on the image taken in the target water area, a segmented image including multiple sub-blocks is obtained, the sub-blocks are classified and identified, and the sub-blocks that meet the preset conditions are determined as target sub-blocks, and whether there is a swimming object in the target water area is determined according to the target sub-blocks. Therefore, the problem that the related art cannot monitor swimming in open waters can be solved, and the effect of monitoring swimming in open waters can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a hardware structure block diagram of a mobile terminal for a method for identifying a swimming object according to an embodiment of the present invention;

[0022] Figure 2 is a flow chart according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of a water area according to an embodiment of the present invention;

[0024] Figure 4 is a judgment schematic diagram according to an embodiment of the present invention;

[0025] Figure 5 is a flow chart of a method for identifying a swimming object according to an embodiment of the present invention;

[0026] Figure 6 is an explanatory diagram of a swimming monitoring system according to an embodiment of the present invention;

[0027] Figure 7 4 is a structural block diagram of a device for identifying a swimming object according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0030] The method for identifying a swimming object provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal for a method for identifying a swimming object according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for identifying a swimming object in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] In this embodiment, a method for operating the above Figure 1 A method for identifying swimming objects, Figure 2 is a flow chart according to an embodiment of the present invention, such as Figure 2 As shown, the process includes the following steps:

[0034] Step S202, performing image segmentation on the target image to obtain a segmented image, wherein the segmented image includes a plurality of sub-blocks, and the target image is an image captured of the target water area;

[0035] Specifically, before performing image segmentation on the target image, it includes: acquiring an initial image obtained by photographing the target water area; identifying the target water area in the initial image; determining the minimum bounding rectangle of the target water area, and determining the minimum bounding rectangle as the target image.

[0036] The target image may be the minimum circumscribed rectangle of the target water area determined after recognizing the initial image obtained by photographing the target water area, and the target sub-blocks may determine whether there is a swimming object in the target water area according to different positions of the target sub-blocks in the segmented image.

[0037] Figure 3 is a schematic diagram of a water area according to an embodiment of the present invention, such as Figure 3 As shown, the target water area may be photographed and the minimum circumscribed rectangle of the target water area may be taken. The segmented image may be a rectangular image evenly divided into sub-blocks of size w*h, and there may be a certain proportion of overlap between the sub-blocks. Using smaller w and h can achieve more precise and accurate monitoring.

[0038] Step S204, classifying and identifying each of the sub-blocks through a neural network model to obtain a classification label of each of the sub-blocks, and determining a sub-block whose classification label is a target label as a target sub-block;

[0039] Specifically, each of the sub-blocks is classified and identified through a neural network model to obtain a classification label for each of the sub-blocks, including: extracting features of the sub-blocks through a feature extraction module of the neural network model to obtain sub-block features; classifying and identifying the sub-block features through a classification module of the neural network model to obtain the classification label of the sub-block; and determining a sub-block whose classification label is a target label as a target sub-block, including: determining a sub-block whose classification label contains at least one of the following as the target sub-block: a swimming equipment label, a splash label, a human body part label, and a life-saving equipment label.

[0040] Specifically, the attributes of each of the above sub-blocks can be classified and identified through a neural network model, and the neural network model can use open source models such as ResNet, DenseNet, or use a self-designed lightweight model. Specifically, the classification model can output the following attributes: whether there is a limb, whether there is a head, whether there is a splash, whether there is life-saving equipment, whether there is a water scene, etc. The above preset conditions can be that the model outputs the results of the presence of a head, splash, life-saving equipment, etc.

[0041] Specifically, determining a sub-block that meets a preset condition as a target sub-block includes: determining a sub-block that contains at least one of the following in the result of classification and recognition of the sub-block as the target sub-block: a swimming equipment label, a water splash label, a human body part label, and a life-saving equipment label. The above labels are classification and recognition results output by the classification model.

[0042] Step S206: determining whether there is a swimming object in the target water area according to the target sub-block.

[0043] Specifically, it can be determined whether the swimming object exists in the target water area according to the position of the target sub-block in the segmented image.

[0044] As an optional implementation, when the target sub-block is located at a non-edge position of the segmented image, it is determined that the swimming object exists in the target water area;

[0045] When the target sub-block is at a non-edge position, whether there is a swimming object in the target water area can be determined by using one frame of segmented image, that is, if there is a swimming object in the target sub-block in the segmented image, it is determined that there is a swimming object in the target water area.

[0046] As an optional implementation manner, when the target sub-block is located at an edge position of the segmented image, it is determined whether the swimming object exists in the target water area based on at least two frames of the segmented images.

[0047] When the target sub-block is at an edge position, whether there is a swimming object in the target water area can be determined by at least two frames of segmented images. Only when there is a swimming object in both frames of segmented images can it be determined that there is a swimming object in the target water area, so as to improve the recognition accuracy.

[0048] Specifically, when the sub-block at the first edge position in the first segmented image is the target sub-block, and in at least one frame image before or after the first segmented image, among multiple sub-blocks located around the first edge position, there is at least one target sub-block, it is determined that the swimming object exists in the target water area.

[0049] The first segmented image is an image of the minimum circumscribed rectangle of the target water area determined after recognizing the initial image obtained by photographing the target water area. Figure 4 is a schematic diagram of judgment according to an embodiment of the present invention, such as Figure 4 As shown, the first edge position is a certain edge position in the first segmented image, such as the upper and lower edges, the left and right edges, Figure 4 The at least one frame image before or after the first segmented image may be at least one frame image before, after, before and after the first segmented image, and the plurality of sub-blocks around the first edge position may be as follows: Figure 4 The sub-blocks to the right, diagonally opposite, and below the edge sub-block shown, specifically, the multiple surrounding blocks are sub-blocks that covary, copoint, or are spaced from the edge sub-block by a preset number of sub-blocks, wherein the preset number of sub-blocks is preset in advance and may be 1, 2, 3, etc., and there is at least one target sub-block that is an edge sub-block in an image before or after the first segmented image, and there is more than one swimming object in the surrounding sub-blocks.

[0050] Specifically, when the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position of the first edge position in N frames of images before the first segmented image, it is determined that the swimming object exists in the target water area, where N is an integer greater than or equal to 1.

[0051] Among them, the N frames of images before the first segmented image are called the second segmented image, and the shooting time interval between the first segmented image and the second segmented image is a first preset time length, and the first preset time length is a preset fixed time interval for capturing different frame images, which can be 0.1s, 0.5s or 1s. The edge position of the second segmented image is the second edge position, and the first edge position and the second edge position share the same edge, or the first edge position and the second edge position share the same point, or the first edge position and the second edge position are separated by a preset number of sub-blocks.

[0052] For example, Figure 4 For example, assuming that the first segmented image is the target image captured at time T, there is a swimming object in the edge sub-block in the upper left corner of the first segmented image, t1 is the above-mentioned first preset time length, then the above-mentioned second segmented image is the target image captured at time T-t1, and there is a swimming object in the sub-block whose second edge position in the second segmented image shares the same edge, the same point, or is separated by a preset number (such as 2) with the first edge position, then it is determined that the swimming object exists in the above-mentioned target water area.

[0053] Specifically, when the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position of the first edge position in M ​​frames of images after the first segmented image, it is determined that the swimming object exists in the target water area, where M is an integer greater than or equal to 1;

[0054] Among them, the M frame images after the above-mentioned first segmented image are called third segmented images, the time interval between the above-mentioned first segmented image and the above-mentioned third segmented image is a second preset time length, the above-mentioned second preset time length is a preset fixed time interval for capturing different frame images, which can be 0.1s, 0.5s or 1s, the edge position of the above-mentioned third segmented image is the second edge position, the above-mentioned first edge position and the above-mentioned second edge position share the same edge, or, the above-mentioned first edge position and the above-mentioned third edge position share the same point, or, the above-mentioned first edge position and the above-mentioned third edge position are separated by a preset number of sub-blocks.

[0055] For example, Figure 4 For example, assuming that the first segmented image is the target image captured at time T, there is a swimming object in the edge sub-block in the upper left corner of the first segmented image, t2 is the above-mentioned second preset time length, then the above-mentioned third segmented image is the target image captured at time T+t2, and there is a swimming object in the sub-block where the second edge position in the third segmented image shares the same edge, the same point, or is separated by a preset number (such as 2) of the first edge position, then it is determined that the swimming object exists in the above-mentioned target water area.

[0056] Specifically, when the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position of the first edge position in N frame images before the first segmented image, and the target sub-block exists at a peripheral position of the first edge position in M ​​frame images after the first segmented image, it is determined that the swimming object exists in the target water area, where M is an integer greater than or equal to 1.

[0057] Among them, the N frames of images before the above-mentioned first segmented image are called second segmented images, the M frames of images after the above-mentioned first segmented image are called third segmented images, the edge position of the above-mentioned second segmented image and the third segmented image is the second edge position, there are target sub-blocks at the edge positions of the second segmented image and the third segmented image, the first edge position and the second edge position share the same edge, or the first edge position and the second edge position share the same point, or the first edge position and the second edge position are separated by a preset number of sub-blocks.

[0058] For example, Figure 4 For example, assuming that the first segmented image is the target image captured at time T, there is a swimming object in the edge sub-block in the upper left corner of the first segmented image, t1 is the above-mentioned first preset time length, then the above-mentioned second segmented image is the target image captured at time T-t1, t2 is the above-mentioned second preset time length, then the above-mentioned third segmented image is the target image captured at time T+t2, and there is a swimming object in the sub-blocks where the second edge position in the second segmented image and the third segmented image shares the same edge, the same point, or is separated by a preset number (such as 2) with the first edge position, then it is determined that the swimming object exists in the above-mentioned target water area.

[0059] As an optional implementation, when it is determined that the swimming object exists in the target water area, the method also includes at least one of the following: playing an alarm voice; displaying an alarm screen; controlling the distribution of life-saving equipment to the target water area; controlling the water vehicle to move to the location of the target sub-block; and sending an alarm message to a mobile terminal located near the target water area.

[0060] In the case that the above-mentioned swimming object exists in the above-mentioned target water area, emergency measures such as playing alarm voice, displaying alarm screen, controlling the distribution of life-saving equipment to the above-mentioned target water area, controlling the water transportation vehicle to move to the location of the above-mentioned target sub-block, and sending alarm messages to mobile terminals located near the above-mentioned target water area can be executed to rescue the above-mentioned swimming object.

[0061] As an optional implementation, the execution subject of the above steps can be a background processor, or other devices with similar processing capabilities, or a machine that integrates at least an image acquisition device and a data processing device, wherein the image acquisition device may include a graphics acquisition module such as a camera, and the data processing device may include a computer, a mobile phone and other terminals, but is not limited to this.

[0062] Through the above steps, the image captured in the target water area is segmented to obtain a segmented image including multiple sub-blocks, the sub-blocks are classified and identified, the sub-blocks that meet the preset conditions are determined as target sub-blocks, and whether there is a swimming object in the target water area is determined based on the target sub-blocks. Therefore, the problems of high computing resource consumption and inconvenient monitoring by wearable devices in the related art can be solved, and the effect of real-time swimming monitoring can be improved.

[0063] Figure 5 is a flow chart of a method for identifying a swimming object according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0064] S501: Acquire video images;

[0065] The system acquires the image data transmitted by the monitoring system in real time.

[0066] S502: Acquire water area;

[0067] There are two ways to obtain the water area in the image: 1) manually delineate the target water area; 2) use image segmentation technology to identify the water area in the image without human intervention. Specifically, images near the water area can be collected, annotated and trained. The water segmentation model can use open source image segmentation models such as HRNet and UNet. The use of image segmentation technology can cope with more special situations, such as adaptive segmentation based on high tide and low tide. This step is usually performed before the swimming monitoring algorithm is run or at a long time interval, and is not a real-time operation.

[0068] S503: Divide the water area into sub-blocks;

[0069] by Figure 3 Taking the water area diagram as an example, take the minimum circumscribed rectangle of the target water area and evenly divide it into sub-blocks of size w*h. There can also be a certain proportion of overlap between the sub-blocks. Using smaller w and h can achieve more precise and accurate monitoring. This step is usually performed before the swimming monitoring algorithm runs or at a long time interval, and is not a real-time operation.

[0070] S504: performing attribute segmentation on the sub-blocks;

[0071] Use the classification model to classify the sub-block attributes of all sub-blocks. Specifically, the classification model can output the following attributes: whether there is a limb, whether there is a head, whether there is a splash, whether there is life-saving equipment, whether it is a water scene, etc. The model can use open source models such as ResNet and DenseNet or use a self-designed lightweight model.

[0072] The training data of the model can be obtained in the following ways: 1) swimming materials in public data sets; 2) data collected by monitoring equipment near water areas; 3) model-generated data: use models such as Diffusion or GAN to generate swimming materials; 4) combined data: intercept the attribute images (human body parts, life-saving equipment, etc.) that the model needs to classify and paste them into the image of the water area. The attributes of the above-obtained materials are annotated, and the classification model can be trained after the annotation is completed. Since the attributes are not mutually exclusive, the loss function of the training model uses binary cross entropy loss.

[0073] S505: judging whether the sub-block is suspected of swimming according to the attribute result;

[0074] Analyze the classification results of the classification model to determine whether there is suspected swimming in the sub-block. If the model outputs the existence of a human head, water splashes, life-saving equipment, etc., it can be considered that the sub-block is suspected of swimming.

[0075] S506: Determine the swimming behavior by combining the results of different frame sub-blocks;

[0076] The sub-block classification results obtained in different video frames are comprehensively analyzed to confirm swimming and generate alarm information.

[0077] S507: generating an alarm;

[0078] If it is determined in step S506 that there is a sub-block suspected of swimming, an alarm is generated.

[0079] Figure 6 is an explanatory diagram of a swimming monitoring system according to an embodiment of the present invention. Figure 6 As shown, the system includes the following modules:

[0080] 1) Video image acquisition module: usually composed of a camera and its peripherals, used to acquire image data to be processed;

[0081] 2) Water area segmentation module: If the water area is manually demarcated, a drawing interface needs to be provided; if the water area is segmented by an image segmentation model, a built-in module is required;

[0082] This module contains image segmentation technology using neural networks. The implementation method of image segmentation technology using neural networks requires a series of steps. First, data collection is required, that is, collecting image data sets near the water area, annotating the images, and marking the water area. Next, a neural network model suitable for image segmentation tasks is built, such as HRNet, UNet, etc. The collected image data is preprocessed, including adjusting the image size, graying, normalizing, and other operations, so that it can be input into the neural network model for training. The preprocessed data is input into the neural network model for training. By adjusting the model parameters, the model can accurately identify the water area in the image and perform effective segmentation. The trained model is optimized, including adjusting hyperparameters, increasing the amount of data, and performing data enhancement operations to improve the accuracy and generalization ability of the model. Finally, the trained model is tested using the test data set to evaluate the performance indicators of the model.

[0083] 3) Sub-block classification module: built-in image classification model to classify sub-block attributes;

[0084] This module includes the technology of using convolutional neural network (CNN) to extract features and classify swimming materials. It includes the following steps: using pre-trained CNN models (such as VGG, ResNet, etc.) for transfer learning to accelerate the training process and improve model performance. At the same time, the data collected by the monitoring equipment is pre-processed and enhanced, including adjusting the lighting, rotation, cropping and other operations to expand the scale of the training data set and improve the robustness of the model. In addition, the generation of swimming materials is used to increase the diversity and richness of the data set, thereby improving the generalization ability of the model. Finally, the combined data is annotated with attributes, such as human body parts, life-saving equipment, etc., so as to train the model for attribute classification and further improve the accuracy and practicality of the model.

[0085] 4) Logical analysis module: 1. Determine whether the sub-block has swimming; 2. Analyze and summarize the classification results obtained at different times to further determine whether it is swimming;

[0086] 5) Alarm module: can display the alarm interface, or play alarm voice on the device.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0088] In the present embodiment, a device for identifying a swimming object is also provided, and the device is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0089] Figure 7 is a structural block diagram of a device for identifying a swimming object according to an embodiment of the present invention. Figure 7 As shown, the device includes: a segmentation module 702, which is used to segment the target image to obtain a segmented image, wherein the segmented image includes multiple sub-blocks, and the target image is an image taken of the target water area; an identification module 704, which is used to classify and identify each of the sub-blocks through a neural network model to obtain a classification label of each of the sub-blocks, and determine the sub-block whose classification label is the target label as the target sub-block; a determination module 706, which is used to determine whether there is a swimming object in the target water area according to the target sub-block. In an exemplary embodiment, the device is also used to determine whether there is the swimming object in the target water area according to the position of the target sub-block in the segmented image.

[0090] In an exemplary embodiment, the above-mentioned apparatus is further used to determine whether the swimming object exists in the target water area according to the position of the target sub-block in the segmented image.

[0091] In an exemplary embodiment, the above-mentioned device is also used to determine whether the swimming object exists in the target water area when the target sub-block is located at a non-edge position of the segmented image; when the target sub-block is located at an edge position of the segmented image, determine whether the swimming object exists in the target water area based on at least two frames of the segmented images.

[0092] In an exemplary embodiment, the above-mentioned device is also used to determine the presence of the swimming object in the target water area when the sub-block at the first edge position in the first segmented image is the target sub-block, and in at least one frame image before or after the first segmented image, there is at least one target sub-block among multiple sub-blocks located around the first edge position.

[0093] In an exemplary embodiment, the apparatus is further configured to determine that the swimming object exists in the target water area when the sub-block at the first edge position in the first segmented image is the target sub-block and the target sub-block exists at a peripheral position of the first edge position in N frames of images before the first segmented image, wherein N is an integer greater than or equal to 1; or, determine that the swimming object exists in the target water area when the sub-block at the first edge position in the first segmented image is the target sub-block and the target sub-block exists at a peripheral position of the first edge position in M ​​frames of images after the first segmented image, wherein M is an integer greater than or equal to 1; or, determine that the swimming object exists in the target water area when the sub-block at the first edge position in the first segmented image is the target sub-block and the target sub-block exists at a peripheral position of the first edge position in N frames of images before the first segmented image, and the target sub-block exists at a peripheral position of the first edge position in M ​​frames of images after the first segmented image, wherein M is an integer greater than or equal to 1.

[0094] In an exemplary embodiment, the above-mentioned device is also used to determine the presence of the swimming object in the target water area when the sub-block at the first edge position in the first segmented image is the target sub-block and the sub-block at the second edge position in the second segmented image is the target sub-block, wherein the second segmented image is located in any image of N frames before the first segmented image, the shooting time interval between the first segmented image and the second segmented image is a first preset time length, the first edge position and the second edge position share the same edge, or the first edge position and the second edge position share the same point, or the first edge position and the second edge position are separated by a preset number of sub-blocks.

[0095] In an exemplary embodiment, the above-mentioned device is also used to determine the presence of the swimming object in the target water area when the sub-block at the first edge position in the first segmented image is the target sub-block, and the sub-block at the third edge position in the third segmented image is the target sub-block, wherein the third segmented image is located in any image of the M frames of images after the first segmented image, the shooting time interval between the first segmented image and the third segmented image is a second preset time length, the first edge position and the third edge position share the same edge, or the first edge position and the third edge position share the same point, or the first edge position and the third edge position are separated by a preset number of sub-blocks.

[0096] In an exemplary embodiment, the apparatus is further configured to determine that the swimming object exists in the target water area when the sub-block at the first edge position in the first segmented image is the target sub-block, the sub-block at the second edge position in the second segmented image is the target sub-block, and the sub-block at the third edge position in the third segmented image is the target sub-block, wherein the second segmented image is located in any image of N frames of images before the first segmented image, the first segmented image and the second segmented image are shot at a first preset time interval, the first edge position and the second edge position share a common edge, or the first edge position and the second edge position share a common point, or the first edge position and the second edge position are spaced apart by a preset number of sub-blocks, and the third segmented image is located in any image of M frames of images after the first segmented image, the first segmented image and the third segmented image are shot at a second preset time interval, the first edge position and the third edge position share a common edge, or the first edge position and the third edge position share a common point, or the first edge position and the third edge position are spaced apart by a preset number of sub-blocks.

[0097] In an exemplary embodiment, the above-mentioned device is also used to extract features from the sub-block through the feature extraction module of the neural network model to obtain sub-block features; classify and identify the sub-block features through the classification module of the neural network model to obtain the classification label of the sub-block; and determine the sub-block whose classification label is the target label as the target sub-block, including: determining the sub-block whose classification label contains at least one of the following as the target sub-block: swimming equipment label, splash label, human body part label, life-saving equipment label.

[0098] In an exemplary embodiment, the above-mentioned device is also used to obtain an initial image obtained by photographing the target water area; identify the target water area in the initial image; determine the minimum enclosing rectangle of the target water area, and determine the minimum enclosing rectangle as the target image.

[0099] In an exemplary embodiment, the above-mentioned device is also used to play an alarm voice; display an alarm screen; control the distribution of life-saving equipment to the target water area; control the water vehicle to move to the location of the target sub-block; and send an alarm message to a mobile terminal located near the target water area.

[0100] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0101] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0102] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0103] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0104] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0105] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0106] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the steps of the method described in each embodiment of the present application when executed by a processor.

[0107] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for identifying a swimming object, characterized in that: include: Performing image segmentation on the target image to obtain a segmented image, wherein the segmented image includes a plurality of sub-blocks, and the target image is an image captured of the target water area; Classify and identify each of the sub-blocks through a neural network model to obtain a classification label for each of the sub-blocks, and determine a sub-block whose classification label is a target label as a target sub-block; It is determined whether there is a swimming object in the target water area according to the target sub-block.

2. The method according to claim 1, characterized in that Determining whether there is a swimming object in the target water area according to the target sub-block includes: It is determined whether the swimming object exists in the target water area according to the position of the target sub-block in the segmented image.

3. The method according to claim 2, characterized in that Determining whether the swimming object exists in the target water area according to the position of the target sub-block in the segmented image includes: In a case where the target sub-block is located at a non-edge position of the segmented image, determining that the swimming object exists in the target water area; In a case where the target sub-block is located at an edge position of the segmented image, it is determined whether the swimming object exists in the target water area according to at least two frames of the segmented images.

4. The method according to claim 3, characterized in that: Determining whether the swimming object exists in the target water area according to at least two frames of the segmented images includes: When the sub-block at the first edge position in the first segmented image is the target sub-block, and in at least one frame image before or after the first segmented image, there is at least one target sub-block among multiple sub-blocks located around the first edge position, it is determined that the swimming object exists in the target water area.

5. The method according to claim 4, characterized in that When a sub-block at a first edge position in a first segmented image is the target sub-block and at least one target sub-block exists among a plurality of sub-blocks located around the first edge position in at least one frame of image before or after the first segmented image, determining that the swimming object exists in the target water area includes: In a case where the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position at the first edge position in N frames of images before the first segmented image, it is determined that the swimming object exists in the target water area, wherein N is an integer greater than or equal to 1; or In a case where the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position at the first edge position in M ​​frames of images after the first segmented image, it is determined that the swimming object exists in the target water area, where M is an integer greater than or equal to 1; or When the sub-block at the first edge position in the first segmented image is the target sub-block, the target sub-block exists at a peripheral position of the first edge position in N frame images before the first segmented image, and the target sub-block exists at a peripheral position of the first edge position in M ​​frame images after the first segmented image, it is determined that the swimming object exists in the target water area, where M is an integer greater than or equal to 1.

6. The method according to claim 5, characterized in that In a case where the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position at the first edge position in N frames of images before the first segmented image, determining that the swimming object exists in the target water area includes: In a case where the sub-block at the first edge position in the first segmented image is the target sub-block, and the sub-block at the second edge position in the second segmented image is the target sub-block, it is determined that the swimming object exists in the target water area, wherein the second segmented image is located at any image in N frames of images before the first segmented image, the shooting time interval between the first segmented image and the second segmented image is a first preset time length, the first edge position and the second edge position share the same edge, or the first edge position and the second edge position share the same point, or the first edge position and the second edge position are separated by a preset number of sub-blocks.

7. The method according to claim 5, characterized in that In a case where the sub-block at the first edge position in the first segmented image is the target sub-block, and the target sub-block exists at a peripheral position at the first edge position in M ​​frames of images after the first segmented image, determining that the swimming object exists in the target water area includes: When the sub-block at the first edge position in the first segmented image is the target sub-block, and the sub-block at the third edge position in the third segmented image is the target sub-block, it is determined that the swimming object exists in the target water area, wherein the third segmented image is located in any image of the M frames of images after the first segmented image, the shooting time interval between the first segmented image and the third segmented image is a second preset time length, the first edge position and the third edge position share the same edge, or the first edge position and the third edge position share the same point, or the first edge position and the third edge position are separated by a preset number of sub-blocks.

8. The method according to claim 5, characterized in that In a case where a subblock at a first edge position in the first segmented image is the target subblock, and the target subblock exists at a peripheral position of the first edge position in N frames of images before the first segmented image, and the target subblock exists at a peripheral position of the first edge position in M ​​frames of images after the first segmented image, it is determined that the swimming object exists in the target water area, where M is an integer greater than or equal to 1, including: In the case where the sub-block at the first edge position in the first segmented image is the target sub-block, the sub-block at the second edge position in the second segmented image is the target sub-block, and the sub-block at the third edge position in the third segmented image is the target sub-block, it is determined that the swimming object exists in the target water area, wherein the second segmented image is located in any image of N frames of images before the first segmented image, the first segmented image and the second segmented image are shot at a first preset time interval, the first edge position and the second edge position share a common edge, or the first edge position and the second edge position share a common point, or the first edge position and the second edge position are spaced apart by a preset number of sub-blocks, and the third segmented image is located in any image of M frames of images after the first segmented image, the first segmented image and the third segmented image are shot at a second preset time interval, the first edge position and the third edge position share a common edge, or the first edge position and the third edge position share a common point, or the first edge position and the third edge position are spaced apart by a preset number of sub-blocks.

9. The method according to claim 1, characterized in that: Classifying and identifying each of the sub-blocks through a neural network model to obtain a classification label for each of the sub-blocks, including: extracting features from the sub-blocks through a feature extraction module of the neural network model to obtain sub-block features; classifying and identifying the sub-block features through a classification module of the neural network model to obtain the classification label for the sub-block; Determining the sub-block whose classification label is the target label as the target sub-block includes: determining the sub-block whose classification label includes at least one of the following as the target sub-block: swimming equipment label, water splash label, human body part label, life-saving equipment label.

10. The method according to claim 1, characterized in that Before performing image segmentation on the target image, the method further includes: Acquire an initial image obtained by photographing the target water area; Identifying the target water area in the initial image; The minimum bounding rectangle of the target water area is determined, and the minimum bounding rectangle is determined as the target image.

11. The method according to claim 1, characterized in that: In the case where it is determined that the swimming object exists in the target water area, the method further includes at least one of the following: Play alarm voice; Display alarm screen; Control the distribution of life-saving equipment to the target waters; Controlling the water vehicle to move to the location of the target sub-block; Send an alarm message to a mobile terminal located near the target water area.

12. A device for identifying a swimming object, characterized in that: include: A segmentation module, used for performing image segmentation on a target image to obtain a segmented image, wherein the segmented image includes a plurality of sub-blocks, and the target image is an image taken of a target water area; An identification module, used to classify and identify each of the sub-blocks through a neural network model, obtain a classification label of each of the sub-blocks, and determine a sub-block whose classification label is a target label as a target sub-block; A determination module is used to determine whether there is a swimming object in the target water area according to the target sub-block.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 11 when executed by a processor.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented.

Images