A fish in-situ video processing method and device

Abstract

A method and device for in-situ video processing of fish, including: performing image quality assessment on the collected video and screening out valid video frames; identifying the category, bounding box, and mask information of each individual fish in the video through instance segmentation technology; using monocular relative depth estimation technology, combined with reference information and computational photography principles, to calculate the instantaneous swimming speed of the fish instance; and editing and correcting the extracted fish instance information through an interactive annotation function. The device includes an in-situ video acquisition and preprocessing module, a multimedia resource management module, a video frame granularity multi-sensor data structured binding module, an instance granularity biometric association module, and an interactive annotation module, which realizes the capture, screening, multimodal data binding, biometric association, and interactive annotation functions of video frames. The present invention effectively solves the problems of low effective frame density and single data dimension in traditional video analysis, and provides comprehensive and accurate technical support for fish ecology research.

Description

Technical Field

[0001] The invention relates to computer vision technology, and in particular to a method and device for processing fish in-situ videos. Background Art

[0002] As a key link in ecology, the biome has very important academic research value. With the help of information such as biological habitats and movement trajectories, biologists can establish connections with corresponding ecological resources and discover the underlying natural laws. On the one hand, the biome can be used as a measure of the effectiveness of species diversity protection, and on the other hand, it can also be used as an early warning indicator before the deterioration of the ecological environment. As one of the many species in the ocean, fish are extremely widely distributed, from the high-latitude poles to the low-latitude equator, from the coast close to the land to the depths of the blue ocean, from the surface of the sea to the abyss of about 10,000 meters. The diversity of the living environment has contributed to the diversity of marine fish. Many fish individuals also constitute large and small fish communities, which are of high research value to biologists.

[0003] There are many ways to conduct scientific research on fish communities. One of them is to use high-definition cameras mounted on underwater working machines such as ROVs, AUVs, and landers to automatically collect underwater images in situ, and then observe and analyze the collected image content after the collection is completed.

[0004] In the traditional collection and research and analysis process, there is usually a problem of extremely low density of valid frames of interest related to fish, which causes useless videos to occupy a large amount of storage space, and thus only a smaller amount of valid videos can be stored in the limited disk space. In addition, fish biologists as researchers will waste a lot of time and manpower costs when manually screening videos to eliminate invalid video frames and locate valid video frames.

[0005] With the rise of artificial intelligence-related technologies with deep learning as the mainstream core, more and more tasks in the field of computer vision, including target detection and image segmentation, have begun to use related methods for improvement, which has also brought new optimization ideas and references for underwater image processing and fish identification. As a master in the field of target detection, YOLO has a wide range of engineering practice applications. At the same time, with the advancement and evolution of ultralytics, YOLO has gradually become an engineering framework for general computer vision tasks, which has provided great convenience for various downstream applications.

[0006] Most existing technologies are limited to the recognition of visual features on the surface of fish, simple counting of the number of organisms in a school of fish, or minor improvements and adjustments to the recognition algorithms. They lack integration and linkage with corresponding data collection hardware equipment, and the data dimension is relatively single.

[0007] In general, because the biodiversity in the deep sea is relatively low compared to that in the shallow sea and on land, the frames and areas of interest in the video image data collected by the underwater platform are very sparse, that is, there are actually only a very small number of video frames in the massive video collected that need to be actually paid attention to, and researchers need to spend a lot of time and manpower to manually find these very few video frames from the massive video. In addition, generally speaking, a lander will carry more than one scientific payload and sensor in a dive mission, and the data they collect is strongly bound to time, but this part of the data is often separated from the video multimedia resources that also have strong temporal properties when conducting analysis and research.

[0008] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0009] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a method and device for processing fish in-situ video.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] A fish in-situ video processing method comprises the following steps:

[0012] S1: Perform image quality assessment on the captured video, determine the image quality value through quantitative analysis, and compare it with the preset threshold to identify and exclude invalid frames;

[0013] S2: Perform instance segmentation on the valid video frames filtered by step S1, identify and extract the category, bounding box and mask information of each fish individual in the video;

[0014] S3: using monocular relative depth estimation technology, combining the instance segmentation result obtained in step S2 and the reference object information, and according to the principles of computational photography and kinematics, calculating the instantaneous swimming speed of the fish instance in the video;

[0015] S4: Edit and correct the fish instance information extracted in steps S2 and S3 by interactively annotating on the video canvas.

[0016] A fish in-situ video processing device, comprising:

[0017] In-situ video acquisition and preprocessing module, used to capture and filter valid video frames for subsequent processing;

[0018] Multimedia resource management module, used to manage video and image resources, including import, export, preview and retrieval functions;

[0019] The video frame granularity multi-sensor data structured binding module is used to structure the video frame and multi-sensor data in the time dimension to achieve data synchronization;

[0020] Instance-granular biometric association module, used to associate dynamic features obtained from video content analysis with static biometric features of individual fish, supporting custom reference objects and feature calculation formulas;

[0021] The interactive annotation module is used to allow users to interactively annotate and edit video content through a graphical user interface to enhance the accuracy and usability of data.

[0022] The present invention has the following beneficial effects:

[0023] The present invention provides a method for in-situ video processing of fish, which significantly improves the effective utilization rate of the storage space of the deep-sea lander by screening effective video frames through image quality assessment, and avoids the problem of a large number of invalid video frames occupying storage resources. The method uses instance segmentation technology to accurately identify the category, bounding box and mask information of each individual fish in the video, and combines the monocular relative depth estimation technology to calculate the instantaneous swimming speed of the fish instance, thereby realizing the dynamic analysis of fish behavior. In addition, through the interactive annotation function, researchers can edit and correct the extracted fish instance information, further improving the accuracy and availability of the data. This method not only solves the problem of low effective frame density in traditional video analysis, but also provides more comprehensive data support for fish ecology research through the integration and analysis of multimodal data.

[0024] The present invention also provides a fish in-situ video processing device, which efficiently captures and screens effective video frames through the in-situ video acquisition and preprocessing module, significantly reducing the time and labor cost of researchers to screen video frames. The multimedia resource management module supports the convenient management of video and image resources, including import, export, preview and retrieval functions, which improves the efficiency of data operations. The video frame granularity multi-sensor data structured binding module realizes the time dimension synchronization of video frames and multi-sensor data, providing a basis for the unified analysis of multimodal data. The instance granularity biometric association module associates the dynamic features of fish instances with static biometric features, supports custom reference objects and feature calculation formulas, and enhances the flexibility of data analysis. The interactive annotation module allows users to perform visual annotation and editing through a graphical interface, further improving the accuracy of the data and research efficiency. Through the collaboration of software and hardware, the device realizes the full process optimization of fish in-situ video processing, providing efficient and accurate technical support for fish ecological research.

[0025] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic flow chart of a fish in-situ video processing method provided in one embodiment of the present invention;

[0027] Figure 2 The module diagram of a fish in-situ video processing device provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0029] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0030] See also Figure 1 The embodiment of the present invention provides a fish in-situ video processing method, comprising the following steps:

[0031] Step S1: Perform image quality assessment on the captured video, determine the image quality value through quantitative analysis, and compare it with a preset threshold to identify and exclude invalid frames.

[0032] In a preferred embodiment, step S1 specifically includes: segmenting the captured video frames frame by frame, scoring each frame using an image quality assessment method based on a lightweight backbone network, wherein the scoring range is based on a non-reference image quality assessment method, and the output value is a quantitative image quality value; simulating the subjective opinions of different annotators through a multi-view attention learning module, and combining knowledge distillation technology to reduce computational complexity and retain high-resolution image details; and screening out valid frames that meet quality requirements and excluding invalid frames according to a preset image quality threshold.

[0033] Step S2: Perform instance segmentation on the valid video frames filtered by step S1 to identify and extract the category, bounding box and mask information of each fish individual in the video.

[0034] In a preferred embodiment, step S2 specifically includes: performing instance segmentation on the valid video frames screened out in step S1, using an instance segmentation model based on a deep convolutional neural network, scaling the image to the model input size by downsampling, and sequentially performing feature extraction, multi-scale feature fusion and feature enhancement processing to output the category, bounding box and segmentation mask of each fish individual; the instance segmentation model is pre-trained and fine-tuned on a fish dataset to adapt to the fish recognition task.

[0035] Step S3: Using the monocular relative depth estimation technology, combined with the instance segmentation result and reference object information obtained in step S2, the instantaneous swimming speed of the fish instance in the video is calculated according to the principles of computational photography and kinematics.

[0036] In a preferred embodiment, step S3 specifically includes: performing monocular relative depth estimation on the video frame processed by step S2, using a pre-trained monocular depth estimation model to output pixel-level depth information, combining the distance between the reference object and the camera lens, and based on the "near big and far small" principle in computational photography and the properties of similar triangles, calculating the actual displacement of the individual fish instance in the real world; according to the kinematic formula and Euclidean distance, combined with the approximate assumption, calculating the instantaneous swimming speed of the fish instance in each frame.

[0037] Step S4: Edit and correct the fish instance information extracted in steps S2 and S3 by interactively annotating on the video canvas.

[0038] In a preferred embodiment, step S4 specifically includes: providing an interactive annotation function on the video canvas, allowing the user to annotate and correct the fish instances in the video frame by selecting a rectangular area or using a brush in the annotation mode; when the user selects a fish instance, the system automatically displays the static biological feature value of the instance, and supports the user to edit and supplement the feature value according to the actual fishing data to establish an association relationship between the fish instance and the biological feature.

[0039] See also Figure 2, a fish in-situ video processing device, including an in-situ video acquisition and preprocessing module, a multimedia resource management module, a video frame granularity multi-sensor data structured binding module, an instance granularity biometric feature association module, and an interactive annotation module. Among them, the in-situ video acquisition and preprocessing module is used to capture and filter valid video frames for subsequent processing; the multimedia resource management module is used to manage video and image resources, including import, export, preview and retrieval functions; the video frame granularity multi-sensor data structured binding module is used to structure the video frame and multi-sensor data in the time dimension to achieve data synchronization; the instance granularity biometric feature association module is used to associate the dynamic features obtained by video content analysis with the static biometric features of individual fish, and supports custom reference objects and feature calculation formulas; the interactive annotation module is used to allow users to interactively annotate and edit video content through a graphical user interface.

[0040] In a preferred embodiment, the processing of the in-situ video acquisition and preprocessing module specifically includes:

[0041] In-situ video is captured by a high-definition camera, and the video stream is temporarily stored in the file system; image quality assessment is performed on the stored video frames, and based on the comparison of the quantitative image quality value with the preset threshold, valid frames that meet the quality requirements are screened out, and the valid frames are divided into sets that logically belong to different videos according to the acquisition time period, and stored in a designated storage medium; the threshold of the image quality assessment can be flexibly adjusted according to actual needs.

[0042] In a preferred embodiment, the processing of the multimedia resource management module specifically includes:

[0043] Batch import, export, preview, search and sort image and video resources according to user operation instructions; display the collected video frames in a list, which contains the video name, duration and modification time attributes; provide a visual player function, allowing users to jump to the video frame at a specified time point for viewing by dragging the progress bar.

[0044] In a preferred embodiment, the processing of the video frame granularity multi-sensor data structured binding module specifically includes:

[0045] The multi-dimensional sensor data collected by the deep-sea lander are structured and bound to the video frames in the time dimension; the sensor data includes depth, three-dimensional posture, conductivity, temperature, pH value and light intensity information, and is imported into the system in a structured file format; the matching relationship between the sensor data and the video frames is displayed on the interface, and the comparison viewing function is supported.

[0046] In a preferred embodiment, the processing of the instance-granularity biometric association module specifically includes:

[0047] The individual fish instances identified in the video frames are associated with their static and dynamic biological features; the instantaneous swimming speed of the fish instances is estimated based on the camera intrinsic parameters, reference object information and computational photography principles; the volume, weight and age of the fish instances are parsed and calculated according to the biological prior calculation formula input by the user; the fish instances are explicitly identified on the interface through identification boxes and masks of different colors, and their biological feature values ​​are displayed in real time.

[0048] In a preferred embodiment, the processing of the interactive annotation module specifically includes:

[0049] A graphical user interface is provided to allow users to interactively annotate fish instances in video frames, including annotations of bounding boxes, masks, and biometric features; it allows users to correct the biometric values ​​intelligently identified by the system and supplement the corrected data into the system in a visual manner to complete the establishment of the association relationship between fish instances and biometric features.

[0050] The fish in-situ video processing method and device provided by the present invention effectively solve the pain points of low effective frame density of current fish in-situ videos, and related similar software can only realize simple target detection / classification single function for pictures / videos, and cannot make better use of cloud computing resources under networking conditions. The present invention allows the limited disk space of the deep-sea lander to store relatively more effective video frames of interest to researchers, and can effectively reduce the time and labor costs of researchers to screen effective frames; the embodiment of the present invention supports the estimation of instantaneous swimming speed of fish, supports the association of computer vision recognition results with biological feature numerical calculations, and binds with the measurement values ​​of various other sensors, realizing unified global perspective management of multimodal data; it can provide the possibility of private deployment when the cloud computing power is sufficient, improve the utilization rate of cloud resources, and facilitate research collaboration between teams, while supporting the full use of local computing power to achieve recognition effects based on lightweight difference models when networking is not possible.

[0051] The specific embodiments of the present invention are further described below.

[0052] In some embodiments, a method for processing in-situ videos of fish includes: performing image quality assessment on the in-situ captured video, and if the quantitative image quality value is lower than a given threshold, it is considered to be an invalid frame; performing instance segmentation on the in-situ video to obtain the category, bounding box and mask of each individual fish in the video frame; performing monocular relative depth estimation on the in-situ video, combining the reference object and the result of instance segmentation, and calculating the instantaneous swimming speed of the fish instance in the corresponding video frame based on the relevant principles of computational photography and kinematics; and supporting interactive editing of annotation content on the canvas where the video is located.

[0053] A fish in-situ video processing device comprises: an in-situ video acquisition and preprocessing module, a multimedia resource management module, a video frame granularity multi-sensor data structured binding module, an instance granularity biometric feature association module, and an interactive annotation module; the in-situ video acquisition and preprocessing module is used to interface with a video acquisition peripheral, and correctly store the in-situ video that is initially screened and identified as valid in a designated storage medium; the multimedia resource management module is used to import and export pictures and video files individually or in batches, and supports preview, viewing in list form, searching by name, filtering by type, and sorting by time. For video type resources, it also supports jumping to view frames of a designated time progress in the form of a player; the video The frame-granularity multi-sensor data structured binding module is used to realize the import and export of structured sensor data such as depth, pH value, three-dimensional posture, conductivity, light intensity, etc. that are aligned with the video content in the time dimension, and supports the corresponding comparison viewing function; the instance-granularity biometric association module is used to associate dynamic features such as instantaneous swimming speed derived from the video content with static features such as volume, weight, age, etc. with individual fish instances in each video frame, and supports customized reference objects and feature calculation formulas; the interactive annotation module is used to complete user visualization annotation behavior based on a graphical user interface, involving the annotation of individual fish instance bounding boxes, masks, and real biometric features in the video. Embodiment 1:

[0054] Figure 1 FIG. 1 is a flow chart of a method for processing fish in-situ video provided in one embodiment of the present invention. Figure 1 As shown, the embodiment of the present invention provides a fish in-situ video processing method, including steps 101 to 104, each of which is specifically as follows:

[0055] Step 101: Segment the collected video frames frame by frame, and score each frame using the MobileIQA method. The underlying kernel of this method uses a lightweight backbone network for efficient image quality assessment, while retaining image details through high-resolution (such as 4K) input. This method uses a multi-view attention learning (MAL) module to capture diverse opinions and simulate the subjective opinions provided by different annotators during the dataset annotation process. This method involves the model using a teacher model to guide the learning of the student model through knowledge distillation, which significantly reduces the computational complexity while maintaining high performance. This method belongs to the NR-IQA (Non-Reference Image Quality Assessment) method range, and its output value range is [0, 100]. The larger the score, the better the quality of the frame. The threshold here can be manually adjusted according to the actual situation.

[0056] Step 102: Use the ultralytics YOLOv8 framework to identify the images that have been processed in step 101 and meet the image quality requirements. The specific identification steps include: scaling the image to the input size of the deep convolutional neural network by downsampling, extracting image features through the Backbone part with the C2f module as the basic component unit, and then through the Neck part, multi-scale fusion of feature maps from different stages of the Backbone to enhance the feature representation capability, and finally predicting the category, bounding box, and segmentation mask of each target through the Head part using the detection head and segmentation head. The s-size seg type model is used here to complete the instance segmentation task in computer vision. UltralyticsYOLOv8 officially provides model structures of various models of different sizes, including "n", "s", "m", "l", and "x", which respectively represent "nano", "small", "medium", "large", and "extra large". These models have different depths, widths, and numbers of channels, which leads to differences in their performance. From the "n" specification to the "x" specification, the inference speed of the model gradually slows down and the inference accuracy gradually increases. Since the original pre-trained model was trained on the COCO dataset, and the COCO dataset lacks the fish category, we need to use the pre-trained model here to fine-tune the fish dataset. The fish dataset uses the DeepFish dataset, which contains about 40,000 images from 20 habitats in the tropical marine environment of Australia. It contains classification, point-level and segmentation labels. This step uses the segmentation label. The hyperparameters used in pre-training include epoch=600, early stop=200, batch=64, image size=640, workers=32, optimizer=SGD, lr 0 =0.01, lr f =0.01. The difference between the seg type model and the commonly used object detection model is that its detection head is different, which can output the category, bounding box and mask of each fish individual in the video frame. The adjustable parameters during prediction include confidence and IoU of the NMS link. The values ​​of these two parameters can be adjusted according to the actual situation to adapt to different scenarios.

[0057] Step 103: Use the Depth Anything model to perform pixel-level monocular relative depth estimation on the in-situ video. The pre-trained model of size s is used, and the output is a binary grayscale image. The pre-trained model specifications of Depth Anything are divided into "s", "b", and "l", which represent "small", "big", and "large", respectively, and their model parameters increase in sequence. According to the grayscale value of the corresponding pixel point in the binary grayscale image combined with the actual distance between the reference object and the camera lens, the distance between the fish instance individual and the camera can be obtained; according to the basic principle of "near big and far small" in computational photography, combined with the properties of similar triangles in mathematics and the value of the fixed internal parameter of the camera, the actual displacement of the fish instance individual on the XY plane in the real world can be obtained; based on the kinematic formula and Euclidean distance, combined with the premise of partial approximation, the instantaneous swimming speed of each fish instance individual in each frame of the video frame can be calculated.

[0058] Step 104: The user can switch between "annotation mode" and "preview mode" at will through the buttons on the interface. In "annotation mode", the user can correct and errata the original annotation by selecting a rectangular area on the canvas where the video is located or using a brush to apply pixel dots. The underlying QPainter drawing class based on QT is used to complete this. At the same time, when the mouse click position is within the corresponding rectangular bounding box or the connected area painted by the brush, that is, when a certain fish instance is selected, the user can observe that the static biological characteristic values ​​including volume, weight, and age are switched accordingly. If the user confirms the true value of the characteristic of the corresponding fish instance through actual fishing and other behaviors, it can be edited on the interface and the true value can be added to the software system to complete the establishment of the association relationship. Embodiment 2:

[0059] Figure 2 FIG. 1 is a module division diagram of a fish in-situ video processing provided in one embodiment of the present invention. Figure 2 As shown, the embodiment of the present invention provides a fish in-situ video processing device, including: an in-situ video acquisition and preprocessing module, a multimedia resource management module, a video frame granularity multi-sensor data structured binding module, an instance granularity biometric association module, and an interactive annotation module. The specific responsibilities of each module are as follows:

[0060] The responsibilities of the in-situ video acquisition and preprocessing module include converting external image information into structured data through hardware, and selecting high-quality content for storage. During the process of the deep-sea lander diving, landing on the bottom and floating up, the 4K images collected in situ by the high-definition camera are stored in the hard disk externally connected to the lower computer main control board. The storage here is pre-storage, and the real-time video stream will be temporarily stored in the file system, and then handed over to the image quality judgment module to judge the quality of the stored video frame. If the quantitative image quality value is lower than the given threshold, the video frame is discarded, that is, it is deleted from the hard disk file system of the lower computer main control board. Finally, what is stored on the hard disk are independent video frame pictures that meet the image quality requirements. These pictures are divided into sets that logically belong to different videos according to the acquisition time period. The threshold can be flexibly adjusted as needed.

[0061] The responsibility of the multimedia resource management module is to enable users to conveniently add, delete, modify, and check image and video resources. After the deep-sea lander floats up, the hard disk connected to the lower computer main control board is transferred to the laboratory workstation, the recognition and analysis software is started, and the collected images are imported from the hard disk into the software in batches for subsequent processing. The imported images will be displayed on the interface in the form of a list with three columns of video inherent attributes: video name, duration, and last modification time. As a user, the researcher clicks to select different records in the list, and the corresponding video will be displayed in the form of a visual player on the software interface. There is a progress bar below the video player. The user drags the video progress bar, and the video player can switch to the video frame corresponding to different scales for display.

[0062] The responsibility of the video frame granularity multi-sensor data structured binding module is to connect various sensors and video information from both the hardware and software levels. The multi-dimensional sensor data collected by the scientific payload peripherals connected to the deep-sea lander are structured and stored together with the corresponding acquisition timestamps in the hard disk of the lower computer main control board. The scientific payload peripherals include CTD, TCM, PH sensor, and lighting. Users can import the structured stored depth (pressure), three-dimensional posture (heading, pitch, roll), conductivity, temperature, PH value, and light intensity information into the software system in the format of .csv or .xlsx files, and the corresponding information will be displayed in the "Sensor Information" section on the interface according to the matching relationship with the acquisition time of the corresponding video frame.

[0063] The responsibility of the instance-granular biometric association module is to establish an association between the individual fish instances in the video frame and the static and dynamic features of the organism. The image recognition results, video frames and corresponding timestamps are used as input, combined with the inherent internal parameters of the camera, pre-selected reference objects, and the "near big far small" similar triangle principle in computational photography, to estimate the approximate instantaneous swimming speed of each fish instance in each frame. In the software interface, the biological prior calculation formulas of different fish are entered through the input box of the "Intelligent Identification of Biometrics" section, which specifies the meanings of different variables, uses Python syntax to parse the written formulas, and estimates the volume, weight, and age of the fish. It supports explicitly identifying different fish instances from the picture through identification boxes and masks of different colors. When the user clicks on each fish instance, the calculated instantaneous swimming speed, volume, weight, and age are displayed in real time in the "Intelligent Identification of Biometrics" section of the interface.

[0064] The interactive annotation module is responsible for allowing users to complete the conversion of visual information into structured fields in a visual way. Users can start the recognition process by clicking the "Recognize" button, input the picture files one by one to the image recognition module, and the image recognition module outputs the specific fish category in each video frame and the mask in the image, and stores the corresponding numerical structure in the lower computer main control board. If the user believes that the biological feature value intelligently recognized by the system is incorrect, the corresponding data item can be corrected through the "Add Actual Value" function.

[0065] In summary, the fish in-situ video processing method and device provided by the present invention can significantly improve the storage space utilization and data analysis efficiency by screening effective video frames through image quality assessment; accurately identify individual fish and calculate instantaneous swimming speed by using instance segmentation and monocular depth estimation technology, thus realizing dynamic analysis of fish behavior; and support unified management and user visual editing of multimodal data through multi-sensor data binding and interactive annotation functions, thus enhancing data accuracy and research flexibility. The present invention effectively solves the problems of low effective frame density and single data dimension in traditional video analysis, significantly improves storage space utilization and data analysis efficiency, and provides efficient and accurate technical support for fish ecology research.

[0066] The embodiment of the present invention further provides a fish in-situ video processing system, which refers to a whole of coordinated software and hardware.

[0067] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0068] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0069] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0070] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0071] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0072] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0073] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0074] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0075] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0076] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0077] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0078] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0079] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A fish in-situ video processing method, characterized in that: The steps include: S1: Perform image quality assessment on the captured video, determine the image quality value through quantitative analysis, and compare it with the preset threshold to identify and exclude invalid frames; S2: Perform instance segmentation on the valid video frames filtered by step S1, identify and extract the category, bounding box and mask information of each fish individual in the video; S3: Using the monocular relative depth estimation technology, combined with the instance segmentation results and reference information obtained in step S2, according to the principles of computational photography and kinematics, the instantaneous swimming speed of the fish instance in the video is calculated; wherein, the video frame processed in step S2 is subjected to monocular relative depth estimation, and the pre-trained monocular depth estimation model is used to output pixel-level depth information, and the actual displacement of the individual fish instance in the real world is calculated based on the "near big far small" principle in computational photography and the properties of similar triangles in combination with the distance between the reference object and the camera lens; according to the kinematic formula and Euclidean distance, combined with the approximate assumption, the instantaneous swimming speed of the fish instance in each frame is calculated; S4: Edit and correct the fish instance information extracted in steps S2 and S3 by interactively annotating on the video canvas; wherein, an interactive annotation function is provided on the video canvas, allowing the user to annotate and correct the fish instances in the video frame by selecting a rectangular area or using a brush in the annotation mode; when the user selects a fish instance, the system automatically displays the static biological feature value of the instance, and supports the user to edit and supplement the feature value according to the actual fishing data, so as to establish the association relationship between the fish instance and the biological feature.

2. The fish in-situ video processing method according to claim 1, characterized in that: Step S1 specifically includes: The captured video frames are segmented frame by frame, and each frame is scored using an image quality assessment method based on a lightweight backbone network. The scoring range is based on a non-reference image quality assessment method, and the output value is a quantitative image quality value. The subjective opinions of different annotators are simulated through a multi-view attention learning module, and the knowledge distillation technology is combined to reduce the computational complexity and retain high-resolution image details. According to the preset image quality threshold, valid frames that meet the quality requirements are screened out, and invalid frames are excluded.

3. The fish in-situ video processing method according to claim 1 or 2, characterized in that: Step S2 specifically includes: The valid video frames screened out in step S1 are segmented by instances. An instance segmentation model based on a deep convolutional neural network is used to scale the image to the model input size by downsampling. Feature extraction, multi-scale feature fusion and feature enhancement are performed in sequence to output the category, bounding box and segmentation mask of each fish individual. The instance segmentation model is pre-trained and fine-tuned on a fish dataset to adapt to the fish recognition task.

4. A fish in-situ video processing device, characterized in that: include: An in-situ video acquisition and preprocessing module, used to capture and filter valid video frames, and process the filtered valid video frames, wherein the valid video frames are instance segmented, and the category, bounding box and mask information of each fish individual in the video are identified and extracted; Multimedia resource management module, used to manage video and image resources, including import, export, preview and retrieval functions; The video frame granularity multi-sensor data structured binding module is used to structure the video frame and multi-sensor data in the time dimension to achieve data synchronization; Instance-granular biometric association module, used to associate dynamic features obtained from video content analysis with static biometric features of individual fish, supporting custom reference objects and feature calculation formulas; An interactive annotation module, used to allow users to interactively annotate and edit video content through a graphical user interface to enhance the accuracy and usability of data; Among them, the processed video frames are subjected to monocular relative depth estimation, and the pre-trained monocular depth estimation model is used to output pixel-level depth information. Combined with the distance between the reference object and the camera lens, the actual displacement of the individual fish instance in the real world is calculated based on the "near is big and far is small" principle in computational photography and the properties of similar triangles. According to the kinematic formula and Euclidean distance, combined with the approximate assumption, the instantaneous swimming speed of the fish instance in each frame is calculated; Among them, an interactive annotation function is provided on the video canvas, allowing users to annotate and correct fish instances in the video frame by selecting a rectangular area or using a brush in the annotation mode; when the user selects a fish instance, the system automatically displays the static biological feature value of the instance, and supports users to edit and supplement the feature value according to actual fishing data to establish the association relationship between the fish instance and the biological feature.

5. The fish in-situ video processing device according to claim 4, characterized in that: The processing of the in-situ video acquisition and preprocessing module specifically includes: The original video is captured by a high-definition camera, and the video stream is temporarily stored in the file system; the image quality of the stored video frames is evaluated, and based on the comparison between the quantitative image quality value and the preset threshold, the valid frames that meet the quality requirements are screened out, and the valid frames are divided into sets that logically belong to different videos according to the acquisition time period, and stored in a designated storage medium; the threshold of the image quality evaluation can be flexibly adjusted according to actual needs; The processing of the multimedia resource management module specifically includes: batch importing, exporting, previewing, retrieving and sorting operations on image and video resources according to user operation instructions; displaying the collected video frames in the form of a list, the list includes the video name, duration and modification time attributes; providing a visual player function, allowing the user to jump to the video frame at a specified time point for viewing by dragging the progress bar.

6. The fish in-situ video processing device according to claim 4 or 5, characterized in that: The processing of the video frame granularity multi-sensor data structured binding module specifically includes: The multi-dimensional sensor data collected by the deep-sea lander are structured and bound to the video frames in the time dimension; the sensor data includes depth, three-dimensional posture, conductivity, temperature, pH value and light intensity information, and is imported into the system in a structured file format; the matching relationship between the sensor data and the video frames is displayed on the interface, and the comparison viewing function is supported.

7. The fish in-situ video processing device according to claim 4 or 5, characterized in that: The processing of the instance granularity biometric association module specifically includes: The individual fish instances identified in the video frames are associated with their static and dynamic biological features; the instantaneous swimming speed of the fish instances is estimated based on the camera intrinsic parameters, reference object information and computational photography principles; the volume, weight and age of the fish instances are parsed and calculated according to the biological prior calculation formula input by the user; the fish instances are explicitly identified on the interface through identification boxes and masks of different colors, and their biological feature values ​​are displayed in real time.

8. The fish in-situ video processing device according to any one of claims 4 to 5, characterized in that: The processing of the interactive annotation module specifically includes: A graphical user interface is provided to allow users to interactively annotate fish instances in video frames, including annotations of bounding boxes, masks, and biometric features; it allows users to correct the biometric values ​​intelligently identified by the system and supplement the corrected data into the system in a visual manner to complete the establishment of the association relationship between fish instances and biometric features.

Images