Fish driving grid or fish collecting hopper operation monitoring method and system based on image generation

Abstract

The invention provides a fish driving grid or fish collecting hopper operation monitoring method and system based on image generation. The method comprises the following steps: acquiring a video stream containing a fish driving grid or a fish collecting hopper; images in the video stream are sequentially extracted according to a preset frame extraction frequency, the extracted images are input into corresponding pre-trained target detection models, corresponding target detection frames are obtained, and each target detection model is obtained through construction based on a network including YOLOv5s; sequentially accumulating the obtained preset number of target detection frame width sequences, and fitting a target motion trend according to the preset number of target detection frame width sequences to judge a current target motion state; and according to the change of the target motion state, determining the time of starting to record the video and the time of ending to record the video so as to store the target motion video. According to the embodiment of the invention, the target motion state is judged according to the target detection frame detected in the monitoring video of the fish driving grid or the fish collecting hopper, so that the target motion video is determined to be recorded and stored.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of water conservancy and hydropower engineering, and more particularly to a method, system, electronic device and computer-readable storage medium for monitoring the operation of fish-driving fences or fish-collecting buckets based on image generation. Background Art

[0002] As the number and scale of domestic water conservancy projects continue to increase, higher requirements are also put forward for the protection of fish ecology. The construction of fish lift system can remedy the impact of human activities on fish migration channels. It enables fish to smoothly cross the dam and migrate to specific waters to spawn, maintain the normal reproduction of river fish, and protect the continuity of the ecosystem. Using fish-driving fences and fish-collecting buckets to lift fish is an important step in the fish-driving process of fish lifts. In water conservancy projects, it is necessary to monitor the operating status of fish-driving fences and fish-collecting buckets.

[0003] Considering the complexity and variability of actual scenes, the appearance of fish-driving fences or fish-collecting buckets in different power stations is not exactly the same, and traditional image processing methods are not applicable. Deep learning networks are widely used in image processing, but data collection is a very important and costly issue for deep learning problems. As a rare application scenario, it is difficult to obtain image data of fish-driving fences or fish-collecting buckets in hydropower stations, and the sample data collected on site lacks diversity. Therefore, using deep learning network detection in hydropower station application scenarios to improve detection performance and control costs has become a new research trend. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide a method, system, electronic device and computer-readable storage medium for monitoring the operation of a fish-driving fence or a fish-collecting bucket based on image generation, so as to solve the aforementioned problems existing in the prior art.

[0005] In order to achieve the above objectives, the technical solutions adopted by the embodiments of the present disclosure are as follows:

[0006] In one aspect, the present disclosure provides a method for monitoring the operation of a fish-driving fence or a fish-collecting bucket based on image generation, the method comprising:

[0007] Obtain a video stream containing a fish trap or fish collecting bucket;

[0008] Images in the video stream are sequentially extracted according to a preset frame extraction frequency, and the extracted images are input into a corresponding pre-trained target detection model to obtain a corresponding target detection frame; wherein each of the target detection models is constructed based on a YOLOv5s network, and the training samples of each of the target detection models include: an initial image data set containing a target and a different style image data set generated by the initial image data set through a style transfer model, wherein the target includes a fish trap or a fish collecting bucket;

[0009] A preset number of target detection frame width sequences are accumulated in sequence, and a target motion trend is fitted according to the preset number of width sequences to determine the current target motion state;

[0010] According to the change of the target motion state, the time to start recording the video and the time to end recording the video are determined, and the recorded target motion video is saved.

[0011] Optionally, the target detection model is obtained by the following training steps:

[0012] Obtain several images containing fish-driving fences or fish-collecting buckets, and mark the fish-driving fences or fish-collecting buckets in each image to construct a corresponding initial data set;

[0013] According to the pre-selected images of different styles, the style transfer model is used to transform the styles of the corresponding initial data sets to obtain data sets corresponding to different styles;

[0014] Integrate the initial data set and corresponding data sets of different styles, and perform data transformation and amplification on the integrated data set to obtain a corresponding amplified data set;

[0015] The corresponding YOLOv5s network to be trained is trained with the corresponding amplified data set to obtain the corresponding trained target detection model.

[0016] Optionally, the loss functions used in the target detection model training process include: a positioning loss function, a confidence loss function and a classification loss function, wherein the confidence loss function and the classification loss function both use BCEloss, and the positioning loss function uses CIOU loss;

[0017] According to the corresponding weights of different loss functions, the overall loss function of the target detection model is obtained until the overall loss function converges to obtain the target detection model.

[0018] Optionally, after the style transfer model is used to perform style transformation on the corresponding initial data set, the method further includes:

[0019] Filter out blocky style images and retain style images with continuous background, prominent objects of interest and no deformation.

[0020] Optionally, before the preset number of target detection frame width sequences accumulated in sequence, the method further includes:

[0021] Determine whether the image is the first frame of the video stream. If so, calculate the aspect ratio of the target detection box in the first frame and save it;

[0022] If not, calculate the aspect ratio of the target detection frame of the current frame image to calculate the ratio of the absolute value of the difference between the aspect ratio of the target detection frame of the current frame image and the aspect ratio of the target detection in the first frame image to the aspect ratio of the target detection in the first frame image, and determine whether the ratio exceeds the first threshold. If so, the current frame target is a false detection and is skipped.

[0023] Optionally, fitting the target motion trend according to a preset number of width sequences to determine the target motion state includes:

[0024] The least square method is used to fit a preset number of target detection box width sequences to obtain the slope k of the fitting line;

[0025] If the slope k is greater than or equal to a first preset slope threshold θ, the width of the target detection frame gradually increases, and the target moves closer to the target position;

[0026] If the slope k is less than or equal to the second preset slope threshold -θ, the width of the target detection frame gradually decreases, and the target moves away from the target position;

[0027] If the slope k is less than the first preset threshold θ and greater than the second preset threshold -θ, the width of the target detection frame remains unchanged and the target is stationary.

[0028] Optionally, determining the time to start recording the video and the time to end recording the video according to the change of the target motion state includes:

[0029] When the target changes from a stationary state to moving towards the target position, the current time point is recorded as the target start movement time, and video recording begins;

[0030] When the target changes from moving towards the target position to a stationary state and the ratio of the target detection frame width in the current image frame to the target detection frame width in the first frame is greater than a second preset threshold, or there is no target detection frame in several consecutive images, the current time point is recorded as the time point when the target ends its movement, and the video recording ends.

[0031] On the other hand, the embodiment of the present disclosure provides a fish-driving fence or fish-collecting bucket operation monitoring system based on image generation, the system comprising:

[0032] An acquisition module, used for acquiring a video stream containing a fish-driving fence or a fish-collecting bucket;

[0033] A detection module is used to extract images from a video stream in sequence according to a preset frame extraction frequency, and input the extracted images into a corresponding pre-trained target detection model to obtain a corresponding target detection frame; wherein each of the target detection models is constructed based on a YOLOv5s network, and the training samples of each of the target detection models include: an initial image data set containing a target and a different style image data set generated by the initial image data set through a style transfer model, wherein the target includes a fish trap or a fish collecting bucket;

[0034] A judgment module is used to sequentially accumulate a preset number of target detection frame width sequences, and fit the target motion trend according to the preset number of width sequences to judge the current target motion state;

[0035] The recording module is used to determine the time to start and end video recording according to the change of the target motion state, and save the recorded target motion video.

[0036] Another aspect of the disclosed embodiment provides an electronic device, including: one or more processors;

[0037] The storage unit is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the method as described above.

[0038] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program can implement the method described above when executed by a processor.

[0039] The beneficial effects of the embodiments of the present disclosure are:

[0040] The disclosed embodiments propose a method and system for monitoring the operation of a fish-driving fence or a fish-collecting bucket based on image generation, which can automatically and efficiently identify the operating status of the fish-driving fence or the fish-collecting bucket based on vision, simplifying the manual recording workflow of power station staff, and saving corresponding videos also avoids the problem of false operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of a method for monitoring the operation of a fish-driving fence or a fish-collecting bucket based on image generation according to an embodiment of the present disclosure;

[0042] Figure 2 It is a flow chart of a method for monitoring the operation of a fish-driving fence or a fish-collecting bucket based on image generation, including a target detection model training process, according to an embodiment of the present disclosure;

[0043] Figure 3 is a schematic diagram of the style transfer model training process of an embodiment of the present disclosure;

[0044] Figure 4 This is a schematic diagram of the structure of a fish-driving fence or fish-collecting bucket operation monitoring system based on image generation according to an embodiment of the present disclosure;

[0045] Figure 5 3 is a schematic diagram of target detection results of the fish-driving fence in an embodiment of the present disclosure, wherein the fish-driving fence is located farthest from the camera and gradually approaches the camera when it starts to move. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiment of the present disclosure more clear, the embodiment of the present disclosure is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described here are only used to explain the embodiment of the present disclosure and are not used to limit the embodiment of the present disclosure.

[0047] In order to automatically monitor the operating status of fish driving fences or fish collecting buckets, standardize the work of power station staff, save corresponding video segments as supervision evidence or count the number of times the fish hoist system operates, the embodiments of the present disclosure propose a method and system for monitoring the operation of fish driving fences or fish collecting buckets based on image generation.

[0048] like Figure 1 As shown, the embodiment of the present disclosure provides a method for monitoring the operation of a fish-driving fence or a fish-collecting bucket based on image generation, the method comprising:

[0049] Step S100: obtaining a video stream including a fish-driving fence or a fish-collecting bucket.

[0050] The fish-driving fence and the fish-collecting bucket in the disclosed embodiment are both part of the fish transport system. First, the fish-driving fence moves horizontally to transport the fish to the target position, and then the fish-collecting bucket lifts the fish vertically. However, the fish transport system is relatively large in size, and a camera can only capture the clear movement process of one of the targets, the fish-driving fence or the fish-collecting bucket. Therefore, cameras are installed near the fish-driving fence or the fish-collecting bucket, and the shooting range of the corresponding camera covers the movement range of the corresponding fish-driving fence or the fish-collecting bucket. The steps of the subsequent monitoring method for the operation of the two targets are the same, wherein the target detection model for the fish-driving fence or the fish-collecting bucket in step S200 is also trained by the same method, and the difference is only that the parameter weights of the target detection model are different. Therefore, the method of the disclosed embodiment can realize the monitoring of the fish-driving fence or the fish-collecting bucket. The fish-driving fence and the fish-collecting bucket need to respectively set up and fix the camera at a suitable position to capture the target movement process, ensure that the fish-driving fence or the fish-collecting bucket target is facing the camera, so that the change in the size of the target detection frame can reflect the movement change of the target. The camera of the fish-driving fence is set in front of it, and the camera of the fish-collecting bucket is set above it. For the fish-driving fence, the camera is set in front of the target position, which can capture the whole process of the fish-driving fence operation, where the front refers to the direction of transporting the fish to the target position, and the back refers to the direction away from the target position; for the fish-collecting bucket, the camera is set above it, which can capture the whole process of the fish-collecting bucket operation. When the fish-driving fence or the fish-collecting bucket starts to operate, it will move closer to the camera. The camera for the fish-driving fence cannot capture the fish-collecting bucket, and the camera for the fish-collecting bucket cannot capture the fish-driving fence either. Moreover, there is only one target detection frame in each frame of the image, and multiple detections are false detections.

[0051] Step S200, extracting images in the video stream in sequence according to a preset frame extraction frequency, inputting the extracted images into a corresponding pre-trained target detection model to obtain a corresponding target detection frame, wherein each of the target detection models is constructed based on a YOLOv5s network, and each of the target detection model training samples includes: an initial image data set containing a target and a different style image data set generated by the initial image data set through a style transfer model, and the target includes a fish driving grid or a fish collecting bucket.

[0052] The disclosed embodiment processes images in a video stream in real time, detects targets using a YOLOv5s network, obtains a target detection frame and coordinates, and uses the coordinates of the target detection frame to calculate the height and width of the target detection frame, saves the aspect ratio of the target detection frame in the first frame of the video, and calculates the width w of the target detection frame at the current moment based on the target detection frame. t The video stream is processed frame by frame, including target detection, false detection screening, and recording the width of the target detection frame for each frame to determine the target motion state.

[0053] In the embodiment of the present disclosure, a target frame of a fish-driving fence or a fish-collecting bucket in each frame of a video is obtained in real time. The specific process includes inputting the real-time video into a target detection model to obtain a target frame of interest, such as Figure 5 As shown; at the same time, some misdetected targets are filtered out according to the aspect ratio of the target detection frame in the first frame image.

[0054] Exemplarily, before inputting the extracted image into the corresponding pre-trained target detection model, the method further includes: preprocessing the image to obtain an image with a long side of 640 pixels.

[0055] It should be noted that in order to ensure real-time detection and edge implementation, the disclosed embodiment adopts a one-stage target detector based on YOLOv5s for frame skipping detection. For example, detection is performed every 5 frames, and the image frame to be detected is scaled to a long side of 640 pixels and input into the backbone feature extraction network Backbone. Then, three layers of feature maps of different sizes are output through the Neck layer network, and then the 1×1 convolution prediction result is used in the Head layer.

[0056] Exemplarily, the target detection model is obtained by the following training steps:

[0057] Step S210, obtaining a plurality of images containing fish-driving fences or fish-collecting buckets, and marking the fish-driving fences or fish-collecting buckets in each image to construct a corresponding initial data set.

[0058] In step S210, the disclosed embodiment collects images containing fish-driving fences or fish-collecting buckets in actual scenes of different hydropower stations, and respectively marks the fish-driving fences or fish-collecting buckets in the images to form a corresponding initial data set. The initial data set contains images of fish-driving fences or fish-collecting buckets at multiple angles and distances. The sizes of fish-driving fences or fish-collecting buckets vary depending on the distance from the camera.

[0059] Step S220: Based on the pre-selected images of multiple different styles, a style transfer model is used to perform style transformation on the corresponding initial data sets to obtain data sets corresponding to different styles.

[0060] In step S220, the disclosed embodiment selects multiple target style images, and uses the style transfer model to convert the real images containing the fish trap or the fish collecting bucket obtained in step S210 into corresponding style images. In step S220, 12 target style images containing different textures, shapes, colors and distribution characteristics are selected, and 12 different style data sets are generated based on the initial data set with the help of the style transfer model, and then manually screened. The screening process includes: retaining style images with continuous background, prominent objects of interest and no large deformation, and filtering out style images with obvious blockiness.

[0061] In the embodiment of the present disclosure, a style transfer model is used to convert real image data into style data, and its corresponding network architecture is as follows: Figure 3 As shown. The style transfer model consists of an image generation network and a loss network. The image generation network is essentially an autoencoder network, which consists of a pair of downsampling and upsampling modules and 5 residual modules. Through the image generation network, the style of the input image can be transformed into the expected style while keeping the high-level semantic information of the input image unchanged. The loss network is essentially a feature extraction network, which is composed of a pre-trained VGG-16 network and is used to measure the differences in content and style between the generated style image and the input content image and style image. In order to change the style of the input image while ensuring that the image content remains unchanged, the style transfer model uses feature reconstruction loss and style reconstruction loss to constrain the generated style image. The original style transfer model includes three loss functions, namely feature reconstruction loss, style reconstruction loss and total variation regularization loss. Among them, the purpose of total variation regularization loss is to make the generated style image smoother. The style transfer model of the embodiment of the present disclosure only uses the first two to constrain the generated image, so that the foreground and background are more clearly distinguished, and is trained until the weighted sum of the two converges.

[0062] Feature reconstruction loss: The shallow feature maps of the loss network often contain pixel information similar to the input, including color, texture, etc., while the deep feature maps often contain abstract semantic information, such as spatial structure information. Feature reconstruction loss constrains the deep features of the loss network so that the generated image changes the color and texture information while retaining the original shape and contour information. Let x represent the input content image and y represent the input style image. represents the generated image, C j , H j , W j Respectively represent the number of channels and height and width of the feature variables, φ j (·) represents the j-th layer feature of the loss network, then the feature reconstruction loss function can be expressed as:

[0063]

[0064] Style reconstruction loss: The Gram matrix is ​​the inner product operation of matrices and can be used to measure the correlation between matrices. In the process of style transfer, the embodiment of the present disclosure uses the Gram matrix to measure the correlation between the generated image and the input style image at different feature layers. If the two are close in different feature layers, they have similar color and texture information, that is, similar styles. The Gram matrix can be expressed as:

[0065]

[0066] Among them, c represents all channels of the jth layer features of the loss network, and c' represents the transpose of c. j (x) Adjust to C j ×H j W j The form of can make the Gram matrix be calculated efficiently, then G j φ (x) = ψψ T / C j H j W j The style reconstruction loss reflects the difference between the Gram matrix of the generated image and the input style image. Minimizing the style reconstruction loss can make the style of the generated image shift to the expected style. The style reconstruction loss function can be expressed as:

[0067]

[0068] Where ψ represents C j ×H j W j A matrix of shape. F represents the Frobenius norm of the matrix, which is the square root of the sum of the squares of each element of the matrix.

[0069] Combining feature reconstruction loss and style reconstruction loss, the overall loss function of the style transfer model can be expressed as:

[0070]

[0071] Among them, λ C and λ S Represent the weights of feature reconstruction loss and style reconstruction loss respectively, J represents the style feature layer, j∈J. C Set to 1, λ S Set to 100. j is only for a single layer, and J represents a multi-layer network collection.

[0072] Step S230: Integrate the initial data set and corresponding data sets of different styles, and perform data transformation and amplification on the integrated data set to obtain a corresponding amplified data set.

[0073] In step S230, the disclosed embodiment integrates the style data set after filtering in step S220 and the corresponding initial data set obtained in step S210 to form a training set for the corresponding target detection network. Step S230 can also further perform data transformation and amplification on the integrated data set to obtain a corresponding amplified data set to train a fish-driving fence or fish-collecting bucket target detection model based on the YOLOv5s network framework. Train a fish-driving fence or fish-collecting bucket target detection network based on the YOLOv5s framework. Data transformation methods include: any one or more methods such as mosaic, scale scaling, random flipping and translation.

[0074] Step S240: train the corresponding YOLOv5s network to be trained with the corresponding amplified data set to obtain a corresponding trained target detection model.

[0075] In the embodiment of the present disclosure, the diversity of the data set is improved in steps S210 to S230, which prevents the target detection model from learning irrelevant features, so that the trained target detection model has stronger generalization ability and improves the overall performance. After comprehensively considering the detection accuracy and detection speed, the YOLOv5s network is selected as the target detection network.

[0076] Exemplarily, the loss functions used in the target detection model training process include: a positioning loss function, a confidence loss function and a classification loss function, wherein the confidence loss function and the classification loss function both use BCEloss, and the positioning loss function uses CIOU loss;

[0077] According to the corresponding weights of different loss functions, the overall loss function of the target detection model is obtained until the overall loss function converges to obtain the target detection model.

[0078] The overall loss function of the target detection model in the embodiment of the present disclosure comprehensively considers three factors: the overlapping area of ​​the rectangular frame, the center point distance and the aspect ratio, so as to improve the stability and convergence speed of the target detection model training.

[0079] Step S300: accumulating a preset number of target detection frame width sequences in sequence, and fitting the target motion trend according to the preset number of target detection frame width sequences to determine the current target motion state.

[0080] The disclosed embodiment calculates the width of the target detection frame based on the target detection frame coordinates. The disclosed embodiment can obtain a preset sequence of target detection frame widths for a period of time to fit the target motion trend and determine the target motion state. It can also be performed once every 5 frames, and 24 detection results are fitted to determine the target motion trend. For a camera with a frame rate of 30, the preset time is about 4 seconds. If the target detection frame width does not change significantly, it is caused by normal jitter of the detection results of different frames.

[0081] For every 24 frames of detection results, the slope method is used to determine the current motion state of the target. The specific process is: the least squares method is used to fit the 24 target detection frame width data, and the data trend of the sequence is determined according to the slope k of the fitted straight line. In the disclosed embodiment, the target motion state can be obtained according to the width change of the target detection frame. The width change and the motion state change are related to the camera installation position. Therefore, the disclosed embodiment installs the camera position directly in front of the fish-driving fence or directly above the fish-collecting bucket, and a clearer working state video of the fish-driving fence or the fish-collecting bucket can be obtained. The disclosed embodiment detects and determines the motion state of the target in the video stream in real time. For example, the video stream is sequentially extracted according to the preset frame extraction frequency, and the frame is detected frame by frame. The target motion state is judged once for each accumulated 24 frames of detection results. The first motion state judgment is for the 1st to 24th frames, and the second motion state judgment is for the 2nd to 25th frames, and so on. Whether to start recording the video is determined according to the target motion state change judgment result.

[0082] like Figure 2 As shown, exemplarily, before sequentially accumulating a preset number of target detection frame sequences, the method further includes:

[0083] Determine whether the image is the first frame of the video stream. If so, calculate the aspect ratio of the target detection box in the first frame and save it;

[0084] If not, calculate the aspect ratio of the target detection frame of the current frame image to calculate the ratio of the absolute value of the difference between the aspect ratio of the target detection frame of the current frame image and the aspect ratio of the target detection in the first frame image to the aspect ratio of the target detection in the first frame image, and judge whether the ratio exceeds the first threshold. If so, the target of the current frame image is a false detection and is skipped; if not, accumulate the target detection frame of the current frame image.

[0085] In general, the first frame of the video stream assumes that the target has not started to move at this time and is located at the farthest distance from the camera. Because the camera can capture the complete movement process of the target, theoretically there is a target in each frame of the image. When the absolute value of the difference between the aspect ratio r1 of the current frame target detection frame and the aspect ratio r0 of the first frame target detection and the ratio of the target detection aspect ratio in the first frame image exceeds the first preset threshold, the difference in aspect ratio between the current target and the first frame target is large, and the current frame target detection frame is determined to be a falsely detected target. In the disclosed embodiment, the first preset threshold is about 0.3, that is, when |r1-r0| / r0≥0.3, it is determined to be a falsely detected target, and the images of the falsely detected targets are filtered out until the target detection frames of a preset number of images are detected.

[0086] Exemplarily, fitting the target motion trend according to a preset number of target detection frame width sequences to determine the target motion state includes:

[0087] The least square method is used to fit a preset number of target detection box width sequences to obtain the slope k of the fitting line;

[0088] If the slope k is greater than or equal to a first preset slope threshold θ, the width of the target detection frame gradually increases, and the target moves closer to the target position;

[0089] If the slope k is less than or equal to the second preset slope threshold -θ, the width of the target detection frame gradually decreases, and the target moves away from the target position;

[0090] If the slope k is less than the first preset slope threshold θ and greater than the second preset slope threshold -θ, the width of the target detection frame remains substantially unchanged and the target is stationary.

[0091] Specifically, in the disclosed embodiment, the slope method is used to fit a preset number of m frames of target detection frame width sequence, where m can be 24 frames, and the optimal linear function matching of m data is found by the minimum sum of squares of the error, that is, the optimal slope and intercept are calculated according to the given discrete data. This embodiment mainly considers the slope calculation, and the specific formula is as follows:

[0092]

[0093] Among them, p represents the data sequence number, q represents the width of the target frame of each frame, Represents the average value of the target width of m frames, represents the mean of the data sequence number, p i Indicates the i-th data sequence number. In the disclosed embodiment, m is 24, p i The value range is 1 to 24.

[0094] When the slope k≥the first preset slope threshold θ, θ>0, it is determined that the target is moving toward the target position, that is, the target is approaching the camera; when the slope k≤the second preset slope threshold -θ, it is determined that the target is moving away from the target position, that is, the target is moving in the direction away from the camera; when the second preset slope threshold -θ<slope k<first preset slope threshold θ, it is determined that the target is in a stationary state.

[0095] Step S400: Determine the time to start saving the video and the time to end recording the video according to the change of the motion state of the adjacent target, so as to save the recorded target motion video.

[0096] That is, the closer the fish-driving fence or fish-collecting bucket is to the target position, the closer it will be to the camera. At this time, the video is recorded and saved. When the fish-driving fence or fish-collecting bucket is reset, the video is not recorded. Specifically, when the target of interest, that is, the fish-driving fence or fish-collecting bucket, changes from a static state to an approaching motion state, it is marked as the target start running time, and the video writing starts at the same time; when the target of interest, that is, the fish-driving fence or fish-collecting bucket, changes from an approaching state to a static state, it is marked as the target end motion time, and the video writing ends at this time.

[0097] The present invention discloses a method for monitoring the operation of fish driving fences or fish collecting buckets based on image generation. Based on intelligent recognition technology and information technology, it simplifies the process of manual recording of the operation time of the fish hoist system by power station operators, avoids the possibility of false alarm of operation, and saves the video of the target operation time period, effectively improving the efficiency of subsequent review and analysis; based on image generation technology, it expands the original data set, reduces the cost of data collection, enhances the generalization ability of the target detection model, and improves the overall performance.

[0098] Exemplarily, determining the time to start recording a video and the time to end recording a video according to the change of the motion state of the adjacent target includes:

[0099] When the target changes from a stationary state to moving towards the target position, the current time point is recorded as the target start movement time, and video recording begins;

[0100] When the target changes from moving towards the target position to a stationary state and the ratio of the target detection frame width in the current image frame to the target detection frame width in the first frame is greater than a second preset threshold, or when the target cannot be detected in several consecutive frames of images, that is, the target is partially or completely beyond the image field of view, the current time point is recorded as the time point when the target ends its movement, and the video recording ends.

[0101] Considering that it may be difficult to capture the complete motion trajectory of the target of interest at the actual camera installation location, the disclosed embodiment also marks the end of motion time when the target of interest is partially or completely out of the camera's visual range. In order to increase the robustness of the algorithm, the disclosed embodiment records the aspect ratio of the fish trap or fish bucket target in the first frame of the video as a constraint condition to filter out some misdetected targets; in order to avoid misjudgment of the motion state caused by the jitter of the detection frame in the video, the disclosed embodiment also sets the change in the width of the target frame as another judgment condition for the end of motion time.

[0102] It should be noted that in order to avoid the influence of detection frame jitter on the target motion state, the embodiment of the present disclosure also sets the target frame width growth ratio as a judgment condition for ending the video recording node, that is, when the target motion state changes, the ratio of the target width in the current image frame to the target width in the first frame must be greater than the set threshold before the video recording is ended.

[0103] The disclosed embodiment adopts a more generalized deep learning target detection method, which realizes the video preservation of the supervision of the start and end of the operation of the fish driving grid and the fish collecting bucket during the operation of the fish lift system, which is conducive to subsequent review and analysis.

[0104] like Figure 4 As shown, another aspect of the present disclosure provides a fish-driving fence or fish-collecting bucket operation monitoring system based on image generation, the system comprising:

[0105] An acquisition module 100 is used to acquire a video stream containing a fish-driving fence or a fish-collecting bucket;

[0106] The detection module 200 is used to extract images in the video stream in sequence according to a preset frame extraction frequency, input the extracted images into a corresponding pre-trained target detection model, and obtain a corresponding target detection frame; wherein each of the target detection models is constructed based on a YOLOv5s network, and the training samples of each of the target detection models include: an initial image data set containing a target and a different style image data set generated by the initial image data set through a style transfer model, wherein the target includes a fish trap or a fish collecting bucket;

[0107] A judgment module 300 is used to sequentially accumulate a preset number of target detection frame width sequences, and fit the target motion trend according to the preset number of target detection frame width sequences to judge the current target motion state;

[0108] The recording module 400 is used to determine the time to start recording the video and the time to end recording the video according to the change of the target motion state, so as to save the recorded target motion video.

[0109] The disclosed embodiment mainly proposes data augmentation based on a style transfer model, using a deep learning target detection network to monitor the target position information of fish-driving screens or fish-collecting buckets in real time and judge the operating status based on the results, and determine whether to record based on the target operating status. The target detection model (trained detection model weights) is exported to the rknn format, deployed to the RK3588 platform, and the embedded neural network processor NPU is called for acceleration to achieve real-time processing at the edge. The RK3588 platform is an edge computing platform, and the above-mentioned reasoning process is carried out on this platform. The edge refers to an AI industrial host installed at a suitable location in a hydropower station, equipped with an RK3588 processor, and the camera video stream is directly transmitted to the edge computing platform to complete the detection, reasoning, video storage and other processes.

[0110] The system further comprises: a false detection judgment module, used to judge whether the image is the first frame image of the video stream, and if so, calculate and save the aspect ratio of the target detection frame in the first frame image;

[0111] If not, calculate the aspect ratio of the target detection frame of the current frame image to calculate the ratio of the absolute value of the difference between the aspect ratio of the target detection frame of the current frame image and the aspect ratio of the target detection in the first frame image to the aspect ratio of the target detection in the first frame image, and determine whether the ratio exceeds the first threshold. If so, the target of the current frame image is a false detection and is skipped.

[0112] Another aspect of the disclosed embodiment provides an electronic device, including: one or more processors;

[0113] The storage unit is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the method as described above.

[0114] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program can implement the above-mentioned method when executed by a processor.

[0115] The method of the disclosed embodiment first collects image data of fish driving grids or fish collecting buckets in actual scenes of hydropower stations. Taking into account the problems of few training samples, many actual application scenarios, and differences between fish driving grids or fish collecting buckets in different hydropower stations, the present invention uses a style transfer model to generate diversified data, screens and annotates them to expand the training set; trains a target detection network YOLOv5s according to a synthetic data set to achieve fish driving grid or fish collecting bucket target detection; performs frame-by-frame detection on the video stream, uses the slope method to fit the width change trend of the target detection frame in the image frame, and sets a threshold to judge the target motion state; records the start and end time of the target according to the change of the motion state of the fish driving grid or fish collecting bucket, and saves the corresponding video segment for subsequent reference.

[0116] The above is only a preferred implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be considered as the protection scope of the embodiment of the present disclosure.

Claims

1. A method for monitoring the operation of a fish-driving fence or a fish-collecting bucket based on image generation, characterized in that: The method comprises: Get a video stream containing a fish trap or fish collecting bucket; Extracting images from the video stream in sequence according to a preset frame extraction frequency, inputting the extracted images into a corresponding pre-trained target detection model to obtain a corresponding target detection frame, wherein each of the target detection models is constructed based on a YOLOv5s network, and the training samples of each of the target detection models include: an initial image data set containing a target and a different style image data set generated by the initial image data set through a style transfer model, wherein the target includes a fish trap or a fish collecting bucket; A preset number of target detection frame width sequences are accumulated in sequence, and a target motion trend is fitted according to the preset number of width sequences to determine the current target motion state; According to the change of the target motion state, the time to start recording the video and the time to end recording the video are determined, and the recorded target motion video is saved.

2. The method according to claim 1, characterized in that: The target detection model is obtained using the following training steps: Obtain several images containing fish-driving fences or fish-collecting buckets, and mark the fish-driving fences or fish-collecting buckets in each image to construct a corresponding initial data set; According to the pre-selected images of different styles, the style transfer model is used to transform the styles of the corresponding initial data sets to obtain data sets corresponding to different styles; Integrate the initial data set and corresponding data sets of different styles, and perform data transformation and amplification on the integrated data set to obtain a corresponding amplified data set; The corresponding YOLOv5s network to be trained is trained with the corresponding amplified data set to obtain the corresponding trained target detection model.

3. The method according to claim 2, characterized in that The loss functions used in the target detection model training process include: positioning loss function, confidence loss function and classification loss function, wherein the confidence loss function and the classification loss function both use BCEloss, and the positioning loss function uses CIOUloss; According to the corresponding weights of different loss functions, the overall loss function of the target detection model is obtained until the overall loss function converges to obtain the target detection model.

4. The method according to claim 2, characterized in that: After the style transfer model is used to perform style transformation on the corresponding initial data set, the method further includes: Filter out blocky style images and retain style images with continuous background, prominent objects of interest and no deformation.

5. The method according to any one of claims 1 to 4, characterized in that: Before the preset number of target detection frame width sequences accumulated in sequence, the method further includes: Determine whether the image is the first frame of the video stream. If so, calculate the aspect ratio of the target detection frame of the first frame and save it. If not, calculate the aspect ratio of the target detection frame of the current frame image to calculate the ratio of the absolute value of the difference between the aspect ratio of the target detection frame of the current frame image and the aspect ratio of the target detection in the first frame image to the aspect ratio of the target detection in the first frame image, and determine whether the ratio exceeds the first threshold. If so, the target of the current frame image is a false detection and is skipped.

6. The method according to any one of claims 1 to 4, characterized in that: The step of fitting the target motion trend according to a preset number of width sequences to determine the target motion state includes: The least square method is used to fit the preset number of target detection box width sequences to obtain the slope k of the fitting line; If the slope k is greater than or equal to a first preset slope threshold θ, the width of the target detection frame gradually increases, and the target moves closer to the target position; If the slope k is less than or equal to the second preset slope threshold -θ, the width of the target detection frame gradually decreases, and the target moves away from the target position; If the slope k is less than the first preset slope threshold θ and greater than the second preset slope threshold -θ, the width of the target detection frame remains unchanged and the target is stationary.

7. The method according to claim 6, characterized in that The step of determining the time to start recording the video and the time to end recording the video according to the change of the target motion state includes: When the target changes from a stationary state to moving towards the target position, the current time point is recorded as the target start movement time, and video recording begins; When the target changes from moving towards the target position to a stationary state and the ratio of the target detection frame width in the current image frame to the target detection frame width in the first frame is greater than a second preset threshold, or there is no target detection frame in several consecutive images, the current time point is recorded as the time point when the target ends its movement, and the video recording ends.

8. A fish-driving fence or fish-collecting bucket operation monitoring system based on image generation, characterized in that: The system comprises: An acquisition module, used for acquiring a video stream containing a fish-driving fence or a fish-collecting bucket; A detection module is used to extract images from a video stream in sequence according to a preset frame extraction frequency, and input the extracted images into a corresponding pre-trained target detection model to obtain a corresponding target detection frame; wherein each of the target detection models is constructed based on a YOLOv5s network, and the training samples of each of the target detection models include: an initial image data set containing a target and a different style image data set generated by the initial image data set through a style transfer model, wherein the target includes a fish trap or a fish collecting bucket; A judgment module is used to sequentially accumulate a preset number of target detection frame width sequences, and fit the target motion trend according to the preset number of width sequences to judge the current target motion state; The recording module is used to determine the time to start and end video recording according to the change of the target motion state, and save the recorded target motion video.

9. An electronic device, characterized in that: include: one or more processors; A storage unit, used to store one or more programs, which, when executed by the one or more processors, can enable the one or more processors to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement the method according to any one of claims 1 to 7.

Images