Water surface foreign matter early warning method, device and equipment and storage medium

Abstract

The invention provides a water surface foreign matter early warning method, device and equipment and a storage medium, and the method comprises the steps: detecting two frames of water area images of the same scene through a moving target detection model, obtaining a target detection result, and enabling the target detection result to comprise target moving information used for determining a moving target, and if the target movement information meets the water surface target screening condition, triggering water surface foreign matter alarm based on the target movement information. According to the method disclosed by the invention, when the moving target exists in the water surface area, the method is not limited by foreign matter types, and water surface foreign matter alarm is triggered, so that the problems that traditional target detection is difficult to be exhaustive and missing detection is easily caused once a new untrained type appears are solved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for early warning of foreign objects on a water surface. Background Art

[0002] Various accidents may occur in rivers, lakes, ponds, reservoirs and other waters, such as children falling, large floating objects getting stuck in the river during the flood season, etc. Therefore, video surveillance can be used to monitor these waters, and staff can be arranged to manually review the video monitoring data of the relevant waters to promptly discover unsafe accidents in the relevant waters, thereby reducing the probability of unsafe accidents.

[0003] In the related technologies, manually checking video monitoring data is inefficient and requires high manpower costs. If unsafe accidents are not discovered in time, it will cause great loss of life and property. Therefore, the target detection algorithm can be used to detect the water surface video monitoring data frame by frame, and warn of the human body, tree trunks, construction waste and other categories detected. However, due to the large range of possible types of foreign objects, traditional target detection is difficult to exhaust. Once a category that has not been trained appears, it is easy to cause missed detection. Summary of the invention

[0004] According to one aspect of the present disclosure, a method for early warning of foreign objects on a water surface is provided, comprising:

[0005] Detecting two frames of water area images of the same scene using a moving target detection model to obtain a target detection result, wherein the target detection result includes target movement information for determining a moving target;

[0006] If the target movement information meets the surface target screening condition, a surface foreign object alarm is triggered based on the target movement information.

[0007] According to another aspect of the present disclosure, there is provided a water surface foreign body warning device, comprising:

[0008] A processing module, used to detect two frames of water area images of the same scene through a moving target detection model to obtain a target detection result, wherein the target detection result includes target movement information for determining a moving target;

[0009] The control module is used to trigger a surface foreign object alarm based on the target detection result if the target movement information meets the surface target screening condition.

[0010] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0011] processor; and,

[0012] Memory for storing programs,

[0013] The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to the exemplary embodiment of the present disclosure.

[0014] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the method according to the exemplary embodiments of the present disclosure.

[0015] One or more technical solutions provided in the embodiments of the present disclosure can detect two frames of water images of the same scene through a mobile target detection model to obtain a target detection result. When the target detection result contains target movement information, it means that the existence of a mobile target can be determined based on the target movement information. If the target movement information meets the surface target screening condition, it means that the target determined based on the target movement information is located in the water surface area. It can be seen that when the target detection result includes target movement information, and the target movement information meets the surface target screening condition, it can be determined that there is a mobile target in the water surface area, and a surface foreign object alarm can be triggered based on the target detection result. Therefore, when there is a mobile target in the water surface area, the method of the exemplary embodiment of the present disclosure can trigger a surface foreign object alarm without being restricted by the type of foreign object, thereby solving the problem that traditional target detection is difficult to exhaust and is prone to missed detection once a new category that has not been trained appears. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A schematic diagram showing an example system in which various methods described herein may be implemented according to an exemplary embodiment of the present disclosure;

[0018] Figure 2 A schematic diagram showing a flow chart of a method for early warning of foreign matter on a water surface according to an exemplary embodiment of the present disclosure;

[0019] Figure 3 A schematic diagram showing the architecture of a moving target detection model according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 4 A schematic diagram of a detection process of a mobile target detection model according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 5 Another detection process schematic diagram of a mobile target detection model according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 6The figure shows a schematic block diagram of the functional modules of a water surface foreign object warning device according to an exemplary embodiment of the present disclosure;

[0023] Figure 7 The figure shows a schematic block diagram of a chip according to an exemplary embodiment of the present disclosure;

[0024] Figure 8 The figure shows a structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure. Detailed implementation manners

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0027] As used herein, the term "including" and its variations are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependent relationships.

[0028] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] Before introducing the embodiments of the present disclosure, the following interpretations are made for the relevant terms involved in the embodiments of the present disclosure:

[0030] Transition: The meaning of "trans" is "transform", and the meaning of "scene" is "scene". A transition is a scene conversion or a space-time conversion. The basic function of a transition is to separate content. That is, the content of different scenes is separated by means of a transition to avoid confusion for the audience when watching.

[0031] FFMPEG is a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams.

[0032] Video frame extraction is a method of extracting several frames at regular intervals in a video, simulating the process of taking a photo at regular intervals and stitching them together to form a video (i.e., slow-speed video recording).

[0033] The task of object detection is to find all the targets (objects) of interest in an image, determine their locations, and even their categories. It is one of the core issues in the field of computer vision.

[0034] The loss function is an operational function used to measure the difference between the model's predicted value f(x) and the true value Y. It is a non-negative real-valued function, usually represented by L(Y,f(x)). The smaller the loss function, the better the robustness of the model.

[0035] Mean Absolute Error (MAE) is a loss function used in regression models. It is the sum of the absolute differences between the target variable and the predictor variable. Therefore, it measures the average error size in a set of predictions, regardless of their direction, ranging from 0 to ∞.

[0036] The root mean square error (RMSE) is the square root of the ratio of the square of the deviation between the predicted value and the true value to the number of predictions n. It can measure the deviation between the predicted value and the true value and is more sensitive to outliers in the data.

[0037] Various accidents may occur in rivers, lakes, ponds, reservoirs, hydropower stations and other waters, such as children falling, large floating objects getting stuck in the river during the flood season, etc. Therefore, video surveillance can be used to monitor these waters, and staff can be arranged to manually check the video monitoring data of the relevant waters to promptly discover unsafe accidents in the relevant waters, thereby reducing the probability of unsafe accidents, and then assisting in river management in cities and villages, hydropower station reservoir management and abnormal prevention and control.

[0038] In the related technology, staff can be arranged to manually check the video monitoring data of the relevant waters to timely discover accidents in the relevant waters. However, since these video monitoring data are obtained through long-term video monitoring by cameras deployed in the relevant waters, for example, multiple cameras can be configured upstream and downstream of the reservoir of the hydropower station for year-round and all-weather monitoring. For another example, the background of the video detection data does not change or changes slightly over a period of time (occasionally individual birds fly over). Therefore, the video monitoring data is complex and the data volume is large, there is a large redundancy, and the manual viewing method is inefficient.

[0039] When manually viewing video monitoring data, if these crisis situations are neglected, it will lead to greater loss of life and property, resulting in more serious consequences. For example, if a large tree trunk is stuck in the door of a turbine in a reservoir during the flood season, it will cause certain potential dangers. Another example is that if someone accidentally falls into the river, it will threaten the safety of life and property. Therefore, automation and intelligence are very important for the detection and early warning of foreign objects floating on the surface of rivers, reservoirs, lakes and other waters.

[0040] The inventors found that a target detection algorithm can be used to perform frame-by-frame detection on water surface videos, and to issue early warnings for categories such as human bodies, tree trunks, and construction waste detected therein. However, due to the large range of possible types of foreign objects, traditional target detection is difficult to exhaust. Once a category that has not been trained appears, it is easy to cause missed detection problems.

[0041] In view of the above problems, the exemplary embodiments of the present disclosure provide a method for warning of foreign objects on the water surface, which can make full use of water videos of scenes such as water surface and lake surface, detect moving targets on the water surface through deep learning algorithms, and then trigger a foreign object alarm on the water surface to solve the problem that traditional target detection is difficult to exhaust and easily causes missed detection once new untrained categories appear. The method of the exemplary embodiments of the present disclosure can be applied to the warning of foreign objects on the water surface in various waters, which can include rivers, lakes, reservoirs, hydropower stations, ponds, and even oceans.

[0042] Figure 1 Schematic diagram of an example system in which various methods described herein may be implemented according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the system 100 of the exemplary embodiment of the present disclosure may include: a front-end video monitoring device 101 , a transmission network 102 and a monitoring center 103 .

[0043] like Figure 1As shown, the front-end video monitoring equipment 101 may include video cameras deployed on water banks, water surfaces, etc. to monitor the water area in real time, so as to continuously collect water area videos in an unattended state. For example, a set of video cameras may be deployed at a certain distance on the water surface and water banks in key areas of the water area to achieve all-round monitoring of the watershed without blind spots.

[0044] In order to reduce the obstruction of the water surface area by obstructions, the camera equipment can be deployed by mounting or setting up poles. For example, the camera can be mounted in an area with a high geographical location and few obstructions near the waterside. Another example is that the camera equipment can include a pole and a camera mounted on the pole, and the camera can collect high-definition video data for a long time and over a long distance. The camera equipment can select a power supply system such as a solar power supply system and a rechargeable battery system to achieve continuous operation of the camera equipment.

[0045] The above-mentioned camera can support the transmission of image data such as videos and pictures. A pan-tilt camera can be selected. The lens direction of the pan-tilt camera can be adjusted according to actual needs to capture the required monitoring object. The camera can be a digital camera. In terms of lens type, the camera can include infrared cameras, black and white cameras, color cameras, Yu Yan cameras, etc., but is not limited to this.

[0046] Exemplarily, the camera device may further include a protective structure for protecting the camera. For example, the protective structure may include a protective cover without any shielding around it, and the camera is located under the protective cover. Another example is that the protective structure may include a lightning rod to reduce the possibility of the camera being struck by lightning.

[0047] like Figure 1 As shown, the transmission network 102 can design the network bandwidth according to the number of front-end video surveillance devices, and should at least meet the access of the front-end video surveillance devices and user terminals. The transmission network 102 can be a high-traffic and high-concurrency Internet. The Internet can use optical fiber transmission or wireless transmission, or optical fiber transmission as the main transmission method and wireless transmission as the auxiliary transmission method, which can be selected according to actual needs. In order to reduce the possibility of network congestion, the front-end video surveillance equipment can use 100M bandwidth access, and the uplink convergence 1G transmission. The delay and packet loss rate of the convergence node and the core network also need to be subject to certain restrictions.

[0048] like Figure 1 As shown, the monitoring center 103 can communicate data with various front-end video monitoring devices 101 deployed in the water area through the transmission network 102. The monitoring center 103 may include a monitoring management platform 1031, a storage system 1032, and various application servers. Here, the storage system 1032 may select storage methods such as high-definition network camera storage, centralized storage, and cloud storage.

[0049] like Figure 1 As shown, the monitoring management platform 1031 can be a functional platform for users to use and meet their business needs, such as providing customers with water area video viewing, camera control, video data storage, return visits, analysis and map linkage, early warning and other functions. Taking the early warning function as an example, the monitoring management platform 1031 can communicate with the user terminal through the transmission network, and send an alarm signal to the user terminal in time when a moving foreign object is detected on the water surface. The user terminal here can be a terminal used by water area management personnel or institutions.

[0050] like Figure 1 As shown, the above-mentioned various application servers may include a streaming media server 1033, a video management server 1034, a device access server 1035, etc. The streaming media server 1033 is responsible for scheduling and forwarding the water area video collected by the front-end video monitoring device, the video management server 1034 can support the user device to obtain the index table of the water area video, and provide the user with video data on demand, return visit, etc. The device access server 1035 can be used to support the access of different signal front-end video monitoring devices. The system of the exemplary embodiment of the present disclosure may also include a spliced ​​display screen 1036 for playing the collected water area video.

[0051] The system of the exemplary embodiment of the present disclosure may further include a user terminal ( Figure 1 The terminal may be a terminal with a display function, and may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a PDA (Personal Digital Assistant), and a wearable device based on augmented reality (AR) and / or virtual reality (VR) technology.

[0052] Exemplarily, when the user terminal is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0053] The water surface foreign body warning method of the exemplary embodiment of the present disclosure can be executed by a monitoring management platform or a chip applied to the monitoring management platform. The method of the exemplary embodiment of the present disclosure is described below with reference to the accompanying drawings.

[0054] Figure 2 FIG. 1 is a flow chart showing a method for early warning of foreign matter on a water surface according to an exemplary embodiment of the present disclosure. Figure 2 As shown, the water surface foreign body warning method of the exemplary embodiment of the present disclosure may include:

[0055] Step 201: Detect two frames of water area images of the same scene through a moving target detection model to obtain a target detection result, which includes target movement information for determining a moving target. It should be understood that the two frames of water area images here can come from the same original water area video. The original water area video can be collected by a front-end video monitoring device and transmitted to a monitoring management platform through a transmission network.

[0056] In practical applications, the image contents displayed by two frames of water images of the same scene are relatively stable and similar, and the differences may include target movement, disappearance, appearance, etc. Based on this, when detecting two frames of water images of the same scene, the interference to the moving target detection model is relatively small, and the targets appearing therein can be detected more accurately. At the same time, the method of the exemplary embodiment of the present disclosure can also remove interference information such as marking information contained in each frame of water images, such as time jumps, scrolling text, etc., to avoid unnecessary interference of such information on target detection. For example, similar marking information can be removed by cropping the image, and of course, each frame of the image can also be repaired to remove similar marking information.

[0057] Step 202: Determine whether the target movement information satisfies the surface target screening condition. The surface target screening condition here can be used to determine whether the moving target determined based on the target movement information is located in the water surface area. When the moving target is not located in the water surface area, it means that the moving target is likely to be on the water bank and will not cause safety hazards to the river, lake, river surface, reservoir, etc. Therefore, step 203 can be executed. When the moving target is on the water surface, it means that there are floating objects in motion, which may cause safety hazards to the river, lake, river surface, reservoir, etc. Therefore, step 204 can be executed.

[0058] In practical applications, the above-mentioned surface target screening conditions include: the target moving area determined based on the target moving information has an intersection area with the water surface area contained in the water area image. At this time, image segmentation algorithms such as edge detection (Roberts, Sobel, Canny), Hough transform (hough), threshold segmentation, and region segmentation can be used to perform image segmentation on any one of the two water area images, thereby obtaining the position information of the water surface area and eliminating the interference caused by the water surface area (such as pedestrians and vehicles on the shore) to the detection. Then, the intersection of the position information of the water surface area and the position information of the target moving area is solved, and then the intersection area is determined based on the intersection.

[0059] When the target moving area and the water surface area have an intersection area, it means that the moving target may be located in the water surface area, and the moving target corresponding to the target moving area can be monitored as a potential floating object on the water surface. If the scene or water area is sensitive to the moving target on the water surface, the water surface target screening condition can be limited to include the intersection area between the target moving area and the water surface area contained in the water area image. When it is detected that the target moving area intersects with the water surface area contained in the water area image, it can be determined that the target moving information meets the water surface target screening condition, so as to maximize the elimination of safety hazards caused by floating objects on the water surface.

[0060] Exemplarily, the above surface target screening condition may also include that the ratio of the intersection area to the target moving area is greater than a preset ratio. The preset ratio here can be set according to actual conditions, for example: more than 50%, more than 60%, etc., and is specifically selected according to actual conditions.

[0061] When it is detected that the target moving area intersects with the water surface area contained in the water area image, it is also necessary to determine whether the proportion of the intersection area to the target moving area is greater than the preset proportion. If it is greater than the preset proportion, it can be considered that the moving target is very likely to be a floating object on the water surface, and if it is less than or equal to the preset proportion, it can be considered that the moving target is very likely to be a moving target on the water shoreline, but only part of the area is located in the water surface area. At this time, even if the target is a moving target, it will not trigger an alarm. It can be seen that the probability of early warning errors can be reduced by further limiting the proportion of the intersection area to the target moving area to be greater than the preset proportion.

[0062] Step 203: updating two water area images of the same scene based on the original water area video. The two water area images after updating may be different from the two water area images before updating, or may include one water area image before updating and one water area image obtained from the original water area video.

[0063] The method of the exemplary embodiment of the present disclosure can obtain a video segment identical to the original scene from the original water video, and then update two frames of water images based on the video segment. It can also directly obtain a video segment different from the original scene from the original water video, and then update two frames of water images based on the video segment.

[0064] Exemplarily, two frames of water images of the same scene may be obtained from the original water video according to the timing of the video stream input. If the water images need to be updated, two frames of water images with later timestamps may be obtained based on the timestamps of the two frames of water images.

[0065] Step 204: triggering a foreign object alarm on the water surface based on the target detection result. When the foreign object alarm on the water surface is triggered here, the monitoring management center may transmit an alarm message to the user terminal through a transmission network, or the alarm system may be directly connected to the alarm center. It can be seen that the method of the exemplary embodiment of the present disclosure can trigger a foreign object alarm on the water surface without being restricted by the type of foreign object when there is a moving target in the water surface area, thereby solving the problem that traditional target detection is difficult to exhaust and is prone to missed detection once a new category that has not been trained appears.

[0066] When the target detection result includes the target movement information and the frame numbers of the two frames of water images, while determining the presence of a moving target in the water surface area, it is also possible to determine the time interval in which the front-end video surveillance device collected the two frames of water images based on the timestamps, frame numbers and other timing information of the two frames of water images, thereby determining the time interval in which the moving foreign object appeared, to further assist the alarm receiving personnel in estimating and troubleshooting safety risks in advance.

[0067] When the front-end video surveillance device transmits the original water area video through the transmission network, it can also carry the geographic location information of the front-end video surveillance device. The geographic location information can indirectly reflect the position of the mobile target in the water area. Therefore, when it is determined that there is a mobile target in the water surface area, an alarm message can be generated based on the geographic location information and the time interval of the appearance of the mobile target, and the alarm message can be transmitted to the user terminal through the transmission network, so that the user terminal can not only understand the time interval when the mobile target appears, but also obtain the geographic coordinates of the mobile target, so as to further improve the accuracy of estimating security risks and the speed of troubleshooting security risks.

[0068] In an optional manner, when the exemplary embodiment of the present disclosure detects two frames of water images of the same scene through a moving target detection model and obtains a target detection result including target movement information, the moving target detection model can be used to detect two frames of water images of the same scene to obtain a target detection result, and then it is determined whether the target detection result includes target movement information for determining the moving target. The target movement information can be position change information of the moving target in two adjacent frames of water images.

[0069] If the target movement information does not include the target movement information, it means that the moving target detection model has not detected the moving target from the two water images of the same scene, and step 203 can be performed. If the target movement information includes the target movement information, it means that the moving target can be determined based on the target movement information, and therefore, step 202 can be performed.

[0070] In practical applications, the mobile target detection model of the exemplary embodiment of the present disclosure can detect the timestamp, frame number and other timing information of the two frames of water images input into the mobile target detection model while detecting the target movement information, so as to quickly determine the time node when the mobile target appears.

[0071] In an optional manner, the method of the exemplary embodiment of the present disclosure can perform frame extraction processing on the original water area video, and then perform transition detection on the formed frame extraction images to obtain multiple video clips. At this time, any two frames of water area images included in each frame of the video clip can be two frames of water area images of the same scene. Therefore, multiple groups of two frames of images of the same scene can be obtained from different video clips in a chronological order. It should be understood that after the original water area video is subjected to frame extraction processing, transition detection can be performed through FFMPEG to obtain multiple video clips, and transition judgment can also be performed on the frame extraction video through the lens parameters of the front-end video monitoring equipment.

[0072] For example, when the front-end video surveillance device captures the original water area video, the lens parameters of the front-end video surveillance device during the original water area video capture process can be recorded and associated with each frame of the water area image of the original water area video. The recorded lens parameters of the front-end video surveillance device and the original water area video are then transmitted together through the transmission network to the monitoring management platform.

[0073] When the lens parameters of the front-end video surveillance equipment corresponding to two adjacent frames of water images are constant or fluctuate within a small threshold range, the two adjacent frames of water images belong to the same scene and no transition occurs. When the lens parameters of the front-end video surveillance equipment corresponding to two adjacent frames of water images change significantly (the range of change exceeds a certain threshold range), the content of the two adjacent frames of water images becomes unstable and a large transition occurs.

[0074] For example, when the front-end video surveillance equipment such as a camera is rotating in an inspection state, the two frames of water area images collected are likely to be jittery and unstable, which is very likely to cause false alarms. Therefore, these two frames of water area images can be ignored, and two frames of water area images with stable lens parameters of the front-end video surveillance equipment can be obtained, so that the two frames of water area images are stable, thereby improving the detection accuracy of the mobile target detection model.

[0075] In the exemplary embodiment of the present disclosure, the acquisition interval of the two frames of water images in the original water video is within the preset threshold range. At this time, the acquisition interval of the two frames of water images can be ensured to be moderate. When the two frames of water images contain a target, the position change of the target in the two frames of water images is more obvious and easy to compare, while also avoiding the problem of misjudgment caused by too long an interval.

[0076] In practical applications, when two frames of water area images can be adjacent frames of a frame-drawing video, the real-time water area video sequence transmitted by the front-end video monitoring device through the transmission network can be frame-drawn at a frame-drawing frequency of every 5 seconds. At this time, the frame-drawing frequency of the frame-drawing can be proportional to the water flow speed of the water area. At this time, the acquisition interval of the two frames of water area images can be determined by the frame-drawing frequency. Assuming that the acquisition interval of the two frames of water area images is T, and the frame-drawing frequency is to extract a frame of water area image from the original water area video every 1 second to 10 seconds, then the acquisition interval of the two frames of water area images is T = 1 second to 10 seconds.

[0077] When the water flow speed in the water area is faster, for example, the river flow speed is fast during the flood season, the floating foreign objects on the water surface move faster. If the frame-by-frame video contains moving foreign objects, then the position change difference between the two adjacent frames of the frame-by-frame video obtained by the faster frame-by-frame frequency is relatively large. Therefore, the frame-by-frame frequency can be set to extract one frame of water area image from the original water area video every 1 second, so that the collection interval between two adjacent frames of water area images input to the moving target detection model is 1 second. If the target detection result obtained includes target movement information, the target movement area determined by the target movement information is relatively obvious.

[0078] When the water flow speed in the water area is faster, for example, the flow speed of the river is slower during the non-flood season, the floating foreign objects on the water surface move slower. If the frame-by-frame video contains moving foreign objects, the position change difference between the two adjacent frames of the frame-by-frame video obtained by the slower frame-by-frame frequency is relatively large. Therefore, the frame-by-frame frequency can be set to extract one frame of water area image every 10 seconds of the original water area video, so that the collection interval between two adjacent frames of water area images input into the moving target detection model is 10 seconds. If the target detection result obtained includes target movement information, the target movement area determined by the target movement information is relatively obvious.

[0079] In an optional manner, the mobile target detection model of the exemplary embodiment of the present disclosure is still essentially a target detection model. The traditional target detection model inputs an image for target detection, and the detected target detection result actually contains the target position information, or predicts the target category on this basis. The mobile target detection model of the exemplary embodiment of the present disclosure inputs two frames of water images from the same scene, and when the target detection result output includes target movement information, the mobile target and the target movement area can be determined based on the target movement information.

[0080] Figure 3 FIG. 1 is a schematic diagram showing the architecture of a mobile target detection model according to an exemplary embodiment of the present disclosure. Figure 3 As shown, the mobile target detection model 300 of the exemplary embodiment of the present disclosure may include: an image stitching layer 301 and a target detection unit 302. The image stitching layer 301 may be used to stitch two frames of water area images to obtain a stitched image, and the target detection unit 302 may be used to perform target detection on the stitched image to obtain a target detection result.

[0081] In practical applications, such as Figure 3 As shown, assuming that both frames of water area images are three-channel water area image data, then after being spliced ​​by the image splicing layer 301, the spliced ​​image obtained is actually six-channel water area image data. The splicing function of the image splicing layer can be realized by a convolutional layer (conv layer).

[0082] like Figure 3 As shown, the target detection unit 302 can be a conventional mmdet, Faster R-CNN, SSD and YOLO architecture, which is used to realize feature extraction and target detection functions. Since the image splicing layer 301 splices two frames of water area images into 6-channel water area image data, it integrates the information contained in the two frames of water area images. Therefore, by performing target detection on the spliced ​​image through the target detection unit 302, it can be detected whether there is target movement information.

[0083] Considering that the target detection results output by the mobile target detection model may be displayed in a visual manner, for example: when the target detection results include target movement information, the position change area corresponding to the position change information of the detected moving target in two frames of water images can be displayed by a bounding box.

[0084] Figure 4 FIG. 2 shows a detection process schematic diagram of a mobile target detection model of an exemplary embodiment of the present disclosure. Figure 4As shown, in terms of the acquisition order of water area images, two water area images from the same scene are respectively a front water area image 401 and a rear water area image 402. The difference between the two is that the front water area image 401 contains target a, while the rear water area image 402 does not contain a target, which can reflect the change process of target a from appearance to disappearance in the time dimension. Therefore, the front water area image 401 and the rear water area image 402 are simultaneously input into the moving target detection model 403. The target movement information included in the output target can be visualized by the border b to identify the position change of the detected target a in the front water area image and the rear water area image. Comparing the position of the border b with the target a, it can be found that the two positions are the same.

[0085] It can be seen that when one frame of water image from the same scene includes a target and another frame of water image does not include a target, and both are input into a mobile target detection model, and the target detection result outputted includes target movement information, the position change information of the mobile target determined by the target movement information can be the same as the target position information of the water image that includes the target.

[0086] Figure 5 FIG. 2 shows another detection process schematic diagram of a mobile target detection model according to an exemplary embodiment of the present disclosure. Figure 5 As shown, in terms of the acquisition order of water images, the two water images from the same scene are the front water image 501 and the rear water image 502. The difference between the two is only the position difference of the same target a in the front water image 501 and the rear water image 502. This process can reflect the position change of the target a in the time dimension. Therefore, the front water image 501 and the rear water image 502 are simultaneously input into the moving target detection model 503. The target movement information included in the output target can be visualized by the border b to identify the position change of the detected target a in the front water image and the rear water image. Comparing the position of the border b with the target a in the front water image 501 and the rear water image 502, it can be found that the border b actually reflects the change range of the target a from the position of the front water image 501 to the position of the rear water image 502.

[0087] It can be seen that when two frames of water images from the same scene include targets in different positions, the two are input into the mobile target detection model, and the target detection results output include target movement information, the position change information of the mobile target determined by the target movement information can reflect the position change area of ​​the target in the two frames of water images.

[0088] In order to achieve graded warning, when the surface foreign body alarm is triggered based on the target movement information, the moving target can be obtained from the spliced ​​image based on the target detection result, and the graded alarm can be triggered based on the category of the moving target and the target movement information. Figure 3As shown, the mobile target detection model 300 of the exemplary embodiment of the present disclosure may also include a classification unit 303, which is used to obtain the mobile target from the spliced ​​image based on the target detection result, and then classify the mobile target. The classification unit 303 may include ResNet, GoogLeNet, Faster R-CNN, VGG, etc. The classification unit 303 is connected after the image splicing layer 301 and the target detection unit 302. If the target detection result contains mobile target information, the mobile target can be obtained from the spliced ​​image based on the mobile target information, and then the category of the mobile target is predicted. It can be seen that the classification unit 303 can select the function of simultaneously realizing image segmentation and target category prediction, so the classification unit can realize its function through the target segmentation module.

[0089] In addition, a mapping relationship between the category of the mobile target and the warning level can be established. After the category of the mobile target is determined, the warning level can be determined based on the mapping relationship, and then an alarm message is generated based on the category of the mobile target and the target movement information, and then the alarm message is transmitted to the user terminal through the transmission network according to the warning level. For example, different levels of warning levels can be established for targets such as ships, human bodies, and tree trunks, so as to ensure that gradient alarms can be designed for different categories of targets.

[0090] The training set of the mobile target detection model of the exemplary embodiment of the present disclosure in the training stage may include: two frames of water sample images and moving area labels. The two frames of water sample images may be from the same scene of the water video, or may be obtained by mapping the water sample images.

[0091] In practical applications, at least one of the two water sample images of the exemplary embodiment of the present disclosure contains a target object. The moving area label includes position change information of the target object, which can be directly marked in the water sample image containing the target object or indirectly marked in the water sample image.

[0092] Exemplarily, when one of the water sample images contains the target object, the water sample image containing the target object may be a water sample image earlier in the time sequence or a water sample image later in the time sequence. At the same time, the position change information of the target object includes the position information of the target object in the water image, which may be marked in the form of a labeling box in the water image containing the target object.

[0093] When the water sample image containing the target object is the water sample image earlier in the time sequence, the water sample image after the time sequence consideration does not contain the target object. In this case, it is equivalent to simulating the change process of the target object gradually disappearing on the water surface with two frames of water sample images. The position change information of the target object can be reflected by marking the target object position area in the water sample image earlier in the time sequence.

[0094] When the water sample image containing the target object is a water sample image at a later time sequence, the water sample image before the time sequence does not contain the target object. In this case, it is equivalent to simulating the change process of the target object gradually appearing on the water surface with two frames of water sample images. The position change information of the target object can be reflected by marking the target object position area in the water sample image at a later time sequence.

[0095] Exemplarily, two frames of water sample images both contain the same target object, and the positions of the target object contained in the two frames of water sample images can be the same or different. When the positions of the target object contained in the two frames of water sample images are the same, the two frames of water sample images simulate the state of the target object being stationary on the water surface; when the positions of the target object contained in the two frames of water sample images are different, the two frames of water sample images simulate the process of the target object moving on the water surface. At the same time, when the two frames of water sample images both contain the same target object, the position change information of the target object includes the position change information of the target object in the two frames of water sample images.

[0096] In one example, the position change information of the target object can be determined by marking a labeling box at the position of the target object in two frames of water sample images, and then determining the position change information of the target object by the union of the labeling boxes of the target object position in the two frames of water sample images. In another example, the position change range of the target object in two frames of water sample images can be marked in one frame of water sample image or two frames of water sample images.

[0097] In some embodiments, in order to construct a training set, river videos with visible target objects such as ships, trees and floating objects during the flood season can be obtained respectively, and frame extraction processing can be performed in time series to form frame-extracted videos. The frame extraction frequency can be adjusted according to the scene, such as extracting 1 frame of river channel image from the river channel video at 1 second to 10 seconds. For example, during the flood season, the river flow rate is fast, and 1 frame of river channel image can be extracted from the river channel video every 1 second. During the non-flood season, the river flow rate is slow, and 1 frame of river channel image can be extracted from the river channel video every 10 seconds.

[0098] After the frame extraction is completed, all the extracted river images can be formed into a frame-extracted video in chronological order, and then the image frames in the transition process contained in the frame-extracted video can be removed by using the described transition detection or other related transition detection methods, thereby dividing the frame-extracted video into multiple video segments, each of which actually includes multiple frames of river videos of the same scene.

[0099] For example, frames are extracted from the river channel video in a manner of extracting one frame of the river channel video every 10 seconds. The acquisition time corresponding to the current frame of the river channel image of the extracted frame video is the tth second, the acquisition time corresponding to the previous frame of the river channel image of the current frame image is the t-10th second, and the acquisition time corresponding to the previous frame of the river channel image of the current frame image is the t+10th second.

[0100] Considering that water areas such as rivers and lakes are located in remote areas and have few human activities, the scene content of the water area images captured by the front-end video image monitoring equipment remains unchanged for a long time, and the data is extremely redundant. At the same time, there is little abnormal target data on the water surface, and it is difficult to obtain relevant positive samples. Based on this, the exemplary embodiment of the present disclosure can construct a data set by mapping on the basic water area image, thereby obtaining a large amount of positive sample data.

[0101] For example, two frames of water sample images may include a basic water image and a basic water image with a texture. At this time, the position change information of the target object may be reflected by a texture mark box marked on the basic water image, and the texture mark box may mark the position of the texture in the basic water image.

[0102] Another example: you can take two identical basic water images and form a map on each basic water image. The maps formed on the two basic water images are the same, but in different positions. At this time, you can mark the coordinate position change range of the map formed by the two basic water images in the form of a coordinate box on one basic water image, or you can mark the coordinate position of the corresponding map in the form of a coordinate box on each basic water image. When calculating the loss later, you can first determine the position change range of the map on the two basic water images based on the coordinate position of the map on the two basic water images, and then calculate the loss based on the position change range of the map on the two basic water images and the prediction results.

[0103] After the training set is constructed, the exemplary embodiment of the present disclosure may preprocess the water sample images included in the training set, and the preprocessing method may include resizing and data enhancement. Here, the data enhancement may include various optional data enhancement methods such as flipping and brightness adjustment.

[0104] Taking into account the motion target detection methods such as continuous frame difference method, background difference method, optical flow method, etc. used in traditional image processing to detect pixel differences and optical flow motion between continuous frames of water videos, the water surface itself has factors such as light changes, lake ripple changes, rain and fog, which have a great impact on the detection accuracy of the motion target detection algorithm and lead to many false detections.

[0105] Based on the above reasons, an exemplary embodiment of the present disclosure can pre-process the water sample images contained in the training set. For example, different environmental noises and / or camera noises can be added to two frames of water sample images to simulate the impact of environmental and camera noise on the image, thereby improving the attention of the mobile target detection model to the mobile target, reducing or avoiding attention to irrelevant external factors, improving the robustness of the model, and reducing the occurrence of false detections.

[0106] When adding noise by rotation or similar means, the positions of various annotation boxes marked on the water sample image change with the target object. When adding noise by masking and brightness, the target object on the water sample image does not change, so the positions of various annotation boxes marked thereon do not change, so that when training the mobile target detection model, the mobile target detection model is taught to pay more attention to the changes in masking and brightness, and ignore the changes in the annotation boxes caused by changes in masking and brightness, thereby improving the robustness of the model.

[0107] When environmental noise is added to the two frames of water sample images, the environmental noise can simulate the image changes caused by the water sample images being affected by the environment. Therefore, the environmental noise is related to the water environment in the water sample images. It should be understood that environmental noise can be added to the water sample images by adjusting the brightness changes of the water sample images, rotating the images, etc.

[0108] In terms of the impact of environmental noise, these environmental noises can include water surface light noise, water surface ripple noise, and image clarity noise. By adding these noises, the water sample image is enhanced. For example, water surface light noise can be image noise caused by cloudy or sunny weather, sunlight intensity, time of day, etc., while water surface ripple noise can be image noise caused by wind, rain, etc., and image clarity noise can be image noise caused by factors affecting air visibility such as rain, fog, and dust.

[0109] In terms of the way environmental noise is generated, these environmental noises can include image noise caused by weather reasons and ambient light differences caused by time factors. By adding these noises, the water sample image is enhanced. Here, the image noise caused by weather reasons can include: reflections on the water surface caused by cloudy and sunny weather and reflections caused by light differences, changes in water surface ripples caused by wind and rain, changes in image clarity caused by fog and rain, etc. The ambient light differences caused by time factors can include changes in image brightness caused by the time of day.

[0110] When camera noise is added to two frames of water sample, the camera noise can simulate the image impact caused by the camera lens posture of the water sample image. Therefore, the camera noise is related to the camera lens posture of the water sample image. The camera lens posture here may include: camera pitch angle, camera yaw angle, and camera roll angle, etc. By simulating one or more of these angles with camera simulation noise, the image impact caused by camera lens shaking, etc. can be simulated. It should be understood that the camera noise can be added to the water sample image in the form of adjusting the flip enhancement of the water sample image to enhance the picture of the water sample image.

[0111] When training the mobile target detection model of the exemplary embodiment of the present disclosure, in order to better reuse the feature extraction capability of the detection model, a target detection model pre-trained on super large detection data sets such as COCO and object365 can be selected as the target detection unit. This target detection model has a strong feature extraction capability as a target detection unit. Therefore, the image stitching layer can be trained while fixing the model parameters of the target detection unit.

[0112] On this basis, when the mobile target detection model includes not only the image stitching layer and the target detection unit, but also the classification unit, a general classification model with strong classification ability that has been trained can be used as the classification unit, and its model parameters can be fixed, and only the image stitching layer can be trained.

[0113] In the training phase of the exemplary embodiment of the present disclosure, two frames of water sample images can be input into the moving target detection model, and the two frames of water sample images can be spliced ​​using the image splicing layer therein to obtain a spliced ​​sample image, and the spliced ​​sample image can be detected by the target detection unit to obtain a target prediction result. If the target prediction result includes the predicted change information of the target position, the predicted change information and the position change information of the target object can be substituted into the loss function to calculate the loss.

[0114] If the calculated loss is less than or equal to the minimum loss, or fluctuates within a certain range, the training can be considered completed. Otherwise, the model parameters of the image stitching layer can be updated based on the predicted change information and the position change information of the target object.

[0115] When the moving target detection model of the exemplary embodiment of the present disclosure includes an image stitching layer and a target detection unit, the loss function used may include a regression loss function. When the moving target detection model also includes a classification unit, the loss function used may include not only a regression loss function but also a classification loss function.

[0116] The above-mentioned regression loss function can be mean absolute error (MAE), root mean square error RMSE (RMSE), etc., and the classification loss function can include cross entropy loss function, softmax loss function, KL-divergence loss function, etc.

[0117] In an optional manner, when the method of the exemplary embodiment of the present disclosure detects the presence of a moving foreign object on the water surface, the camera can be controlled to further track the moving target to obtain a better quality picture and achieve accurate reporting. Based on this, the method of the exemplary embodiment of the present disclosure may also include:

[0118] If the surface target screening conditions are met based on the target movement information, it means that there is a moving target on the water surface. The camera's lens attitude parameters and image quality control parameters can be controlled based on the target movement information until the camera's shooting direction matches the orientation of the moving target. Then, an updated water area video collected by the camera based on the image quality control parameters is obtained, and the original water area video is updated based on the updated water area video to obtain a high-quality water area video, thereby further improving the accuracy of mobile target detection and category prediction accuracy.

[0119] In practical applications, most cameras currently used in hydropower stations and rivers support the setting of inspection logic, rotation and zoom functions. The relative position of the camera and the moving target can be determined based on the position information of the moving target represented by the target movement information. Then, the lens attitude parameters used to control the camera lens can be determined based on the relative position. These parameters may include at least one of the pitch angle, yaw angle and roll angle of the lens, and the corresponding angle is adjusted specifically according to the relative position.

[0120] The above-mentioned image quality control parameters may include the focal length of the camera, and may also include parameters such as aperture. For example, the moving target may be enlarged or reduced by controlling the focal length. Based on this, when the shooting direction of the camera matches the orientation of the moving target, the water area update video collected by the camera with the image quality control parameters adjusted may be adapted to the needs of the monitoring management platform, and the collected water area update video may be transmitted to the monitoring management platform through the transmission network for re-moving target detection.

[0121] One or more technical solutions provided in the embodiments of the present disclosure can detect two frames of water images of the same scene through a mobile target detection model to obtain a target detection result. When the target detection result contains target movement information, it means that the existence of a mobile target can be determined based on the target movement information. When the target movement information meets the surface target screening condition, it means that the target determined based on the target movement information is located in the water surface area. It can be seen that when the target detection result includes target movement information, and the target movement information meets the surface target screening condition, it can be determined that there is a mobile target in the water surface area, and a surface foreign object alarm can be triggered based on the target detection result. Therefore, when there is a mobile target in the water surface area, the method of the exemplary embodiment of the present disclosure can trigger a surface foreign object alarm without being restricted by the type of foreign object, thereby solving the problem that traditional target detection is difficult to exhaust and is prone to missed detection once a new category that has not been trained appears.

[0122] It can be seen that the method of the exemplary embodiment of the present disclosure adopts deep learning algorithms such as an automated video processing system and a mobile target detection model, and automatically and intelligently detects and warns floating foreign objects on the water surface of rivers, lakes, reservoirs, etc., reducing the manpower cost required for on-duty personnel to pay attention at all times, and avoiding the loss of life and property caused by potential crisis situations. More importantly, the method of the exemplary embodiment of the present disclosure can alarm as long as it is determined that the target is a mobile target and is located on the water surface without paying attention to the target category, thereby avoiding the problem of missed detection caused by the inability of model training in related technologies to exhaust the types of foreign objects.

[0123] Furthermore, the moving object detection model can detect moving foreign objects for the sequential images of the previous and next frames of the same scene included in the frame-sampling video (i.e., two frames of water area images). At the same time, by adding environmental noise and camera noise to the water area sample images contained in the training set, this can not only augment the data, but also enable the moving object detection model to deeply learn the image effects caused by sunlight, ripples, rain, camera shake, etc. through the training process, thereby improving the model robustness and reducing false detections.

[0124] The above mainly introduces the scheme provided by the embodiment of the present disclosure. It is understandable that in order to achieve the above functions, the device may include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0125] The embodiments of the present disclosure may divide the functional units of the device according to the above method examples. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0126] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides a water surface foreign object warning device, which can be an electronic device or a chip applied to an electronic device. Figure 6 FIG. 2 shows a schematic block diagram of functional modules of a water surface foreign body warning device according to an exemplary embodiment of the present disclosure. Figure 6 As shown, the water surface foreign body warning device 600 includes:

[0127] Processing module 601, used to detect two frames of water area images of the same scene through a moving target detection model to obtain a target detection result, wherein the target detection result includes target movement information for determining a moving target;

[0128] The control module 602 is used to trigger a surface foreign object alarm based on the target detection result if the target movement information meets the surface target screening condition.

[0129] As a possible implementation method, the training set of the mobile target detection model in the training stage includes: two frames of water sample images and moving area labels, at least one of the two frames of the water sample images contains a target object, and the moving area label includes position change information of the target object.

[0130] As a possible implementation manner, one frame of the water area sample image contains a target object, and the position change information of the target object includes the position information of the target object in the water area sample image.

[0131] As a possible implementation manner, the two frames of the water area sample images both contain the same target object, and the position change information of the target object includes the position change information of the target object in the two frames of the water area sample images.

[0132] As a possible implementation, the moving target detection model includes:

[0133] An image stitching layer, used for stitching two frames of the water area images to obtain a stitched image;

[0134] The target detection unit is used to perform target detection on the stitched image to obtain the target detection result.

[0135] As a possible implementation method, the surface target screening condition includes: the target movement area determined based on the target movement information has an intersection area with the water surface area contained in the water area image, and the surface target screening condition also includes that the proportion of the intersection area to the target movement area is greater than a preset proportion.

[0136] As a possible implementation manner, the processing module 601 is further configured to update two frames of the water area image of the same scene based on the original water area video if the target detection result does not include target movement information.

[0137] As a possible implementation, each frame of the water area image is collected by a camera, and the control module 602 is further used to control the lens attitude parameters and image quality control parameters of the camera based on the target movement information until the shooting direction of the camera matches the orientation of the moving target;

[0138] The device further includes an acquisition module 603 for acquiring the water area update video collected by the camera, and the processing module 601 is further used to update the original water area video based on the water area update video.

[0139] Figure 7 Schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown. Figure 7 As shown, the chip 700 includes one or more (including two) processors 701 and a communication interface 702. The communication interface 702 can support the server to perform the data sending and receiving steps in the above image processing method, and the processor 701 can support the server to perform the data processing steps in the above image processing method.

[0140] Optional, such as Figure 7 As shown, the chip 700 also includes a memory 703, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0141] In some embodiments, Figure 7As shown, the processor 701 performs corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 701 controls the processing operations of any one of the terminal devices, and the processor may also be called a central processing unit (CPU). The memory 703 may include a read-only memory and a random access memory, and provides instructions and data to the processor 701. A portion of the memory 703 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Figure 7 Various buses are labeled as bus system 704 .

[0142] The method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor can be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0143] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0144] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0145] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0146] refer to Figure 8 , a block diagram of an electronic device 800 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0147] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0148] like Figure 8As shown, multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device that can input information to the electronic device 800, and the input unit 806 can receive input digital or character information, and generate key signal input related to the user settings and / or function control of the electronic device. The output unit 807 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 808 can include but is not limited to a disk, an optical disk. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth TM device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0149] like Figure 8 As shown, the computing unit 801 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the method of the exemplary embodiment of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. In some embodiments, the computing unit 801 can be configured to perform the method of the exemplary embodiment of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0150] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0151] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0153] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0154] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0155] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0156] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for early warning of foreign objects on the water surface, characterized in that: include: Detecting two frames of water area images of the same scene using a moving target detection model to obtain a target detection result, wherein the target detection result includes target movement information for determining a moving target; If the target movement information meets the surface target screening condition, a surface foreign object alarm is triggered based on the target detection result.

2. The method according to claim 1, characterized in that The training set of the mobile target detection model in the training stage includes: two frames of water sample images and moving area labels, at least one of the two frames of water sample images contains a target object, and the moving area label includes position change information of the target object.

3. The method according to claim 2, characterized in that One of the water area sample images contains a target object, and the position change information of the target object includes the position information of the target object in the water area sample image; or, The two frames of the water area sample images both contain the same target object, and the position change information of the target object includes the position change information of the target object in the two frames of the water area sample images.

4. The method according to claim 2, characterized in that: Different environmental noises and / or camera noises are added to the two water area sample images, wherein the environmental noise is related to the water area environment in the water area sample images, and the camera noise is related to the posture of the camera lens used to collect the water area sample images.

5. The method according to claim 1, characterized in that The mobile target detection model includes: An image stitching layer, used for stitching two frames of the water area images to obtain a stitched image; The target detection unit is used to perform target detection on the spliced ​​image to obtain the target detection result.

6. The method according to claim 5, characterized in that The triggering of a water surface foreign object alarm based on the target detection result includes: Acquire the moving target from the stitched image based on the target detection result; A graded alarm is triggered based on the category of the moving target and the target movement information.

7. The method according to claim 1, characterized in that The surface target screening condition includes: the target movement area determined based on the target movement information has an intersection area with the water surface area contained in the water area image, and the surface target screening condition also includes that the proportion of the intersection area to the target movement area is greater than a preset proportion.

8. The method according to any one of claims 1 to 7, characterized in that: The two frames of the water area image are from the same original water area video, and the acquisition interval duration of the two frames of the water area image is within a preset threshold range of the original water area video.

9. The method according to claim 8, characterized in that The method further comprises: If the target detection result does not include target movement information, two frames of the water area image of the same scene are updated based on the original water area video.

10. The method according to claim 8, characterized in that Each frame of the water area image is collected by a camera, and the method further includes: If the surface target screening condition is met based on the target movement information, the lens attitude parameter and the image quality control parameter of the camera are controlled based on the target movement information until the shooting direction of the camera matches the orientation of the moving target; Acquire the water area update video collected by the camera, and update the original water area video based on the water area update video.

11. A water surface foreign body warning device, characterized in that: include: A processing module, used to detect two frames of water area images of the same scene through a moving target detection model to obtain a target detection result, wherein the target detection result includes target movement information for determining a moving target; The control module is used to trigger a surface foreign object alarm based on the target detection result if the target movement information meets the surface target screening condition.

12. An electronic device, characterized in that: include: processor; as well as, Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions for causing the computer to execute the method according to any one of claims 1 to 10.

Images