Multi-dimensional fish school sensing device and method

Abstract

The present invention provides a multi-dimensional fish school sensing device, which can be applied to the field of fish school sensing technology, including: a fixed workbench, a plurality of pan-tilt cameras and an omnidirectional scanning sonar, wherein the omnidirectional scanning sonar is located at the center of the fixed workbench, and the omnidirectional scanning sonar is used to obtain point cloud information within a 360-degree range around the device; a plurality of pan-tilt cameras are evenly distributed on the fixed workbench, and the distances from the plurality of pan-tilt cameras to the center position are equal; the pan-tilt camera can rotate with a single degree of freedom in the horizontal direction; and the pan-tilt camera can actively rotate according to the distribution of optical image feature points in the environment. Based on the image data and sonar point cloud data collected by the device, the fusion of optical and sonar features and the three-dimensional target detection of fish schools are completed. The present invention also provides a multi-dimensional fish school sensing method.

Description

Technical Field

[0001] The present invention relates to the field of fish school sensing technology, and in particular to a multi-dimensional fish school sensing device and method. Background Art

[0002] Fish school perception is an important means for the development and protection of marine biological resources. It includes key technical elements such as fish school detection, positioning, counting, tracking, and behavior recognition. Traditional fish school perception technology is mostly based on underwater sonar. Underwater sonar sensors have strong penetration capabilities and long measurement distances, but low resolution, high latency, and are sensitive to environmental noise. In recent years, with the advancement of computer vision technology, optical sensors have demonstrated excellent perception capabilities. However, when used underwater, optical sensors are significantly affected by the absorption, refraction, and scattering effects of water bodies, and have a short perception distance. Therefore, there is an urgent need for an efficient fusion method of sonar and optical sensors, and based on this, multi-dimensional element perception of fish schools can be achieved. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] In order to solve the technical problems existing in the prior art of fish school perception, the embodiments of the present invention provide a multi-dimensional fish school perception device and method. The device is composed of a multi-pan-tilt camera and an omnidirectional scanning sonar. The camera can actively rotate to improve the global perception accuracy or the perception accuracy of the area of ​​interest. It can achieve the complementary advantages of sonar and optical vision, and obtain multi-dimensional information such as the spatial position, number, movement path, and high-order behavior categories of the fish school. The device and method provided by the present invention can be flexibly deployed on various underwater robots to monitor fish resources.

[0005] (II) Technical solution

[0006] In view of the above technical problems, an embodiment of the present invention proposes a multi-dimensional fish school sensing device and method.

[0007] According to a first aspect of the present invention, a multi-dimensional fish school sensing device is provided, comprising: a fixed workbench, a plurality of pan-tilt cameras and an omnidirectional scanning sonar, wherein the omnidirectional scanning sonar is located at the center of the fixed workbench, and the omnidirectional scanning sonar is used to obtain point cloud information within a 360-degree range around the device; the plurality of pan-tilt cameras are evenly distributed on the fixed workbench, and the plurality of pan-tilt cameras are equidistant from the center position; the pan-tilt cameras can rotate with a single degree of freedom in the horizontal direction; and the pan-tilt cameras can actively rotate according to the distribution of optical image feature points in the environment.

[0008] According to a second aspect of the present invention, a multi-dimensional fish school perception method is provided, comprising: using the above-mentioned multi-dimensional fish school perception device to obtain a multi-view optical image and original sonar data of the fish school; obtaining a first 3D feature map of the fish school based on the multi-view optical image; obtaining a second 3D feature map of the fish school based on the original sonar data; extracting fusion features based on the first 3D feature map and the second 3D feature map to obtain a fusion feature map; and based on the fusion feature map, using a 3D target detection method based on a self-attention mechanism to obtain a 3D detection result of the target fish school.

[0009] In some exemplary embodiments, obtaining multi-perspective optical images and raw sonar data of a school of fish includes collecting multi-perspective optical images and raw sonar data of an underwater environment using an underwater robot equipped with the multi-dimensional fish sensing device to obtain the multi-perspective optical images and raw sonar data.

[0010] In some exemplary embodiments, a first 3D feature map of a school of fish is obtained based on a multi-view optical image, including: inputting the multi-view optical image into a first convolutional neural network to obtain feature encoding of the multi-view optical image; based on the feature encoding, using a multi-scale feature fusion method of an adaptive feature pyramid to obtain fused 2D features; and based on the 2D features, multiple gimbal camera internal parameters and dynamic external parameter information, using a spatial projection method to complete depth prediction and obtain a first 3D feature map.

[0011] In some exemplary embodiments, a second 3D feature map of a school of fish is obtained based on the original sonar data, including correcting the original sonar data to obtain point cloud data with stronger time alignment; performing voxel conversion on the point cloud data to obtain voxel data; and obtaining a second 3D feature map based on the voxel data using a sparse 3D convolutional neural network.

[0012] In some exemplary embodiments, the original sonar data is corrected to obtain point cloud data with a stronger time alignment, including: filtering, denoising, correcting and normalizing the original sonar data to obtain preprocessed sonar data; using an autoregressive sliding average model to predict the delay value of the original sonar data; based on the delay value, correcting the preprocessed sonar data to obtain point cloud data with a stronger time alignment.

[0013] In some exemplary embodiments, based on the first 3D feature map and the second 3D feature map, fusion features are extracted, including: cascading data of the first 3D feature map and the second 3D feature map according to channels, and further extracting fusion features using a second convolutional neural network.

[0014] In some exemplary embodiments, a 3D target detection method using a self-attention mechanism is used based on a fused feature map to obtain a 3D detection result of a target fish school, including: based on the fused feature map at the current moment, spatial features in the fused features are obtained through multi-layer convolution, global pooling, and fully connected layers; optical flow features based on a displacement vector field are obtained through a continuous sequence of fused feature maps; based on the optical flow features, temporal features in the fused features are obtained through multi-layer convolution, global pooling, and fully connected layers; and temporal features and spatial features are merged layer by layer and input into a feature fusion layer to obtain the behavior detection result of the target fish school, wherein the spatial features include the relative position and scene information of the fish school; and the temporal features include the position changes of the fish school at different times, which are used to reflect the motion information of the fish school.

[0015] According to a third aspect of the present invention, a multi-dimensional fish school perception system is provided, comprising: an acquisition module, used to use the above-mentioned multi-dimensional fish school perception device to acquire multi-view optical images and original sonar data of the fish school; a first feature extraction module, used to obtain a first 3D feature map of the fish school based on the multi-view optical image; a second feature extraction module, used to obtain a second 3D feature map of the fish school based on the original sonar data; a feature fusion module, used to extract fusion features based on the first 3D feature map and the second 3D feature map to obtain a fusion feature map; and a result detection module, used to obtain a 3D detection result of the target fish school based on the fusion feature map using a 3D target detection method based on a self-attention mechanism.

[0016] According to a fourth aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0017] According to a fifth aspect of the present invention, a storage medium is provided, characterized in that: computer-readable instructions are stored in the storage medium, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the above method.

[0018] (III) Beneficial effects

[0019] It can be seen from the above technical solutions that the multi-dimensional fish school sensing device and method provided by the embodiments of the present invention have at least the following beneficial effects:

[0020] (1) The perception device consists of a multi-pan-tilt camera and an omnidirectional scanning sonar. The camera can actively rotate to improve the global perception accuracy or the perception accuracy of the area of ​​interest. The perception method is based on the fusion framework of optical image data and sonar point cloud data, which is divided into a visual branch, a sonar branch, and a fusion branch. It can complete the fusion of optical and sonar features and the three-dimensional target detection of fish schools.

[0021] (2) The optical image data comes from the gimbal camera, and the model explicitly considers the dynamic external parameter information involved in the camera. The sonar point cloud data is delayed corrected using the autoregressive sliding average model to improve the time alignment.

[0022] (3) A sonar data delay correction method based on an autoregressive sliding model is provided. In order to realize the behavior recognition of fish schools, a spatiotemporal dual-stream recognition network is designed based on the data fusion framework. It can realize the complementary advantages of sonar and optical vision and obtain multi-dimensional information such as the spatial position, number, movement path, and high-order behavior categories of fish schools.

[0023] (4) The device and method provided by the present invention can be flexibly deployed on various underwater robots to monitor fish resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0025] Figure 1 A schematic diagram of the structure of a multi-dimensional fish school sensing device according to an embodiment of the present invention is shown;

[0026] Figure 2 A schematic diagram of a fusion framework of optical image data and sonar point cloud data according to an embodiment of the present invention is shown schematically;

[0027] Figure 3 A schematic diagram of a flow chart of a multi-dimensional fish school sensing method according to an embodiment of the present invention is shown;

[0028] Figure 4 A schematic diagram of a process of obtaining a first 3D feature map of a school of fish based on a multi-view optical image according to an embodiment of the present invention is schematically shown;

[0029] Figure 5 A schematic diagram of a process of obtaining a second 3D feature map of a school of fish based on original sonar data according to an embodiment of the present invention is shown;

[0030] Figure 6 A schematic diagram of a process of correcting raw sonar data according to an embodiment of the present invention is shown;

[0031] Figure 7 A schematic diagram of a process flow of a sonar correction module according to an embodiment of the present invention is shown;

[0032] Figure 8 A schematic diagram of a spatiotemporal dual-stream fish school behavior network structure according to an embodiment of the present invention is shown;

[0033] Fig. 9A schematic diagram of a process of obtaining a behavior detection result of a target fish school by using a 3D target detection method based on a self-attention mechanism based on a fusion feature map according to an embodiment of the present invention is shown;

[0034] Fig.10 A schematic diagram of a structure of a multi-dimensional fish school perception system according to an embodiment of the present invention is shown; and

[0035] Fig.11 A block diagram of an electronic device for extracting a faint and elongated star image according to an embodiment of the present invention is schematically shown.

[0036] Reference numerals:

[0037] 1-Workbench; 2-Pan-tilt camera; 3-Omnidirectional scanning sonar. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Figure 1 A schematic structural diagram of a multi-dimensional fish school sensing device according to an embodiment of the present invention is shown schematically.

[0040] like Figure 1 As shown, a multi-dimensional fish school sensing device according to an embodiment of the present invention includes a fixed workbench 1, an omnidirectional scanning sonar 3 and a plurality of pan-tilt cameras 2, wherein the omnidirectional scanning sonar 3 is located at the center of the fixed workbench 1, and the omnidirectional scanning sonar 3 is used to obtain point cloud information within a 360-degree range around the device; the plurality of pan-tilt cameras 2 are evenly distributed on the fixed workbench 1, and the distances between the plurality of pan-tilt cameras 2 and the center position are equal; the pan-tilt camera 2 can rotate with a single degree of freedom in the horizontal direction; and the pan-tilt camera 2 can actively rotate according to the distribution of optical image feature points in the environment, thereby improving the overall positioning accuracy or the accuracy of the area of ​​interest. Figure 1 The number of the pan-tilt cameras 2 is 6. It can be understood that the number of the pan-tilt cameras 2 is not limited to 6, and the number can be designed according to specific needs.

[0041] The device according to the embodiment of the present invention can be fixedly installed on an underwater robot to constitute its perception system.

[0042] Figure 2 A schematic diagram of a framework for fusing optical image data and sonar point cloud data according to an embodiment of the present invention is schematically shown.

[0043] like Figure 2 As shown, a fusion framework of optical image data and sonar point cloud data according to an embodiment of the present invention is divided into three parts: a visual branch, a sonar branch, and a fusion branch. In the visual branch, the multi-view optical image collected by the pan-tilt camera 2 is first input into the basic feature extraction module composed of a convolutional neural network to complete the preliminary feature encoding, and then input into the multi-scale feature fusion module based on the adaptive feature pyramid. The fused 2D features are input into the 2D→3D space projection module together with the internal parameters and dynamic external parameter information of the multi-camera to complete the depth prediction and obtain the 3D feature map of the visual branch. In the sonar branch, the original sonar data obtained by the omnidirectional scanning sonar 3 is first input into the sonar correction module to optimize the data delay and obtain point cloud data with stronger time alignment. Then the sonar point cloud data is input into the voxel conversion module to obtain voxel data, thereby reducing the complexity of the data in the vertical direction. The voxel data is further input into the 3D feature extraction module based on sparse 3D convolution to obtain the 3D feature map of the sonar branch. The fusion branch receives 3D feature information from the visual branch and the sonar branch, cascades the data of the two branches according to the channel, and then uses the convolution-based feature fusion module to further extract the fusion features. Furthermore, the fusion features are input into the 3D target detection module based on the self-attention mechanism, and the weights of the sonar and visual features are dynamically adjusted according to the current environment, finally obtaining the 3D detection results of the fish school target.

[0044] based on Figure 1 The device shown and Figure 2 The framework shown, combined with Figure 3~Figure 9 A multi-dimensional fish school perception method of the present invention is described.

[0045] Figure 3 A schematic flowchart of a multi-dimensional fish school perception method according to an embodiment of the present invention is shown.

[0046] like Figure 3 As shown, a multi-dimensional fish school perception method according to an embodiment of the present invention includes steps S110-S150.

[0047] In step S110, a multi-view optical image and raw sonar data of a school of fish are acquired.

[0048] In some exemplary embodiments, using Figure 1 The underwater robot of the device shown collects multi-view optical images and original sonar data of the underwater environment to obtain multi-view optical images and original sonar data.

[0049] In step S120, a first 3D feature map of the school of fish is obtained based on the multi-view optical image.

[0050] In some exemplary embodiments, step S120 includes steps S121-S123, see Figure 4 .

[0051] In step S121, the multi-view optical image is input into a first convolutional neural network to obtain feature encoding of the multi-view optical image.

[0052] In step S122, based on feature coding, a multi-scale feature fusion method of an adaptive feature pyramid is used to obtain fused 2D features.

[0053] In step S123, based on the 2D features, the internal parameters of the plurality of gimbal cameras 2 and the dynamic external parameter information, a spatial projection method is used to complete the depth prediction and obtain the first 3D feature map. The first 3D feature map corresponds to Figure 2 3D feature map of the optical branches in .

[0054] In step S130, a second 3D feature map of the school of fish is obtained based on the original sonar data.

[0055] In some exemplary embodiments, step S130 includes steps S131-S133, see Figure 5 .

[0056] In step S131, the original sonar data is corrected to obtain point cloud data with stronger time alignment.

[0057] The data delay problem of sonar sensors is a key factor that weakens the sonar visual data fusion effect. The embodiment of the present invention implements real-time correction of sonar data based on a sonar data delay prediction algorithm based on historical data and current environmental information.

[0058] In some exemplary embodiments, step S131 includes steps S1311-S1313, see Figure 6 .

[0059] In step S1311, the original sonar data is filtered, denoised, corrected and normalized to obtain preprocessed sonar data.

[0060] In step S1312, the delay value of the original sonar data is predicted using an autoregressive moving average model.

[0061] In the embodiment of the present invention, the Auto-Regressive Moving Average Model (ARMA) model:

[0062] AR(p): (1)

[0063] MA(q): (2)

[0064] Among them, X t is the value of the time series at time t, c is a constant term, is the autoregressive coefficient, is the moving average coefficient, is white noise, μ is the mean of the sequence, and p and q are the orders of autoregression and moving average, respectively.

[0065] In step S1313, the pre-processed sonar data is corrected based on the delay value to obtain point cloud data with stronger time alignment.

[0066] In the embodiment of the present invention, the sonar data sequence is input into formulas (1) and (2) for real-time correction. The correction formula is:

[0067] (3)

[0068] in, is the predicted sonar data delay value, The actual observed sonar data value, is the corrected sonar data value.

[0069] Figure 7 The flowchart of the sonar correction module according to an embodiment of the present invention is schematically shown.

[0070] like Figure 7 As shown in the figure, after completing the acquisition of raw sonar data, it is necessary to make a stationary sequence judgment on the data, build the ARMA model after the data is stabilized, and then perform model parameter estimation and data prediction, and finally perform data correction.

[0071] In step S132 , voxel conversion is performed on the point cloud data to obtain voxel data.

[0072] Point cloud data is a collection of discrete points in three-dimensional space, and is widely used in many fields such as three-dimensional modeling, autonomous driving, and virtual reality. However, the discreteness and large scale of point cloud data make direct processing more complicated and inefficient. Voxels, as a representation method that discretizes three-dimensional space into regular small cubes, can simplify data processing to a certain extent and provide a unique analytical perspective. Converting point cloud data into a voxel structure can provide a convenient basic data form for a variety of subsequent applications.

[0073] In step S133, based on the voxel data, a sparse 3D convolutional neural network is used to obtain a second 3D feature map. The second 3D feature map corresponds to Figure 2 3D feature map of the sonar branch in .

[0074] Comparison Figure 3In step S140, based on the first 3D feature map and the second 3D feature map, fusion features are extracted to obtain a fusion feature map.

[0075] For example, according to the channel cascade of the data of the first 3D feature map and the second 3D feature map, the fusion feature is further extracted using the second convolutional neural network.

[0076] In step S150, based on the fused feature map, a 3D target detection method based on the self-attention mechanism is used to obtain a 3D detection result of the target fish school.

[0077] Figure 8 The schematic diagram of the spatiotemporal dual-stream fish school behavior network structure according to an embodiment of the present invention is schematically shown.

[0078] like Figure 8 As shown, Figure 2 The outputs of the feature fusion module at continuous moments are combined into feature sequences, which are input into the spatial stream and temporal stream branches respectively. The spatial stream branch is based on the feature map at the current moment, and is condensed through multi-layer convolution, global pooling, and fully connected layers to obtain the spatial features of the fusion features. The spatial features focus on the relative position and scene information of the fish school. The temporal stream branch uses the continuous feature sequence to obtain the optical flow features based on the displacement vector field, and then condenses through multi-layer convolution, global pooling, and fully connected layers to obtain the temporal features of the fusion features. The temporal features focus on the position changes of the fish school at different moments, reflecting the movement information of the fish school. The temporal features and spatial features are merged layer by layer and input into the feature fusion layer, and finally the results of fish school behavior recognition are obtained, thereby judging the satiety, activity, and health status of the fish.

[0079] based on Figure 8 The spatiotemporal dual-stream fish school behavior network structure shown in FIG. 1 , step S150 includes steps S151-S154, see Fig. 9 .

[0080] In step S151, based on the fusion feature map at the current moment, the spatial features in the fusion features are condensed through multi-layer convolution, global pooling and full connection layers, where the spatial features include the relative position of the fish school and scene information.

[0081] In step S152, a continuous sequence of fused feature maps is used to obtain an optical flow feature based on a displacement vector field.

[0082] In step S153, based on the optical flow features, multi-layer convolution, global pooling and full connection layer are used to condense the time features in the fusion features, where the time features include the position changes of the fish school at different times, which are used to reflect the movement information of the fish school.

[0083] In step S154, the temporal features and the spatial features are merged layer by layer and input into the feature fusion layer to obtain the 3D detection result of the target fish school.

[0084] Fig.10 A structural block diagram of a multi-dimensional fish school perception system according to an embodiment of the present invention is schematically shown.

[0085] like Fig.10 As shown, a multi-dimensional fish school perception system 800 according to an embodiment of the present invention includes an acquisition module 810, a first feature extraction module 820, a second feature extraction module 830, a feature fusion module 840 and a result detection module 850.

[0086] Acquisition module 810, used to utilize Figure 1 The multi-dimensional fish school sensing device shown obtains multi-perspective optical images and original sonar data of fish schools.

[0087] The first feature extraction module 820 is used to obtain a first 3D feature map of the fish school based on the multi-view optical image.

[0088] The second feature extraction module 830 is used to obtain a second 3D feature map of the fish school based on the original sonar data.

[0089] The feature fusion module 840 is used to extract fusion features based on the first 3D feature map and the second 3D feature map to obtain a fusion feature map.

[0090] The result detection module 850 is used to obtain the 3D detection result of the target fish school based on the fused feature map and the 3D target detection method based on the self-attention mechanism.

[0091] In some specific embodiments, any multiple modules among the acquisition module 810, the first feature extraction module 820, the second feature extraction module 830, the feature fusion module 840 and the result detection module 850 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0092] In some specific embodiments, at least one of the acquisition module 810, the first feature extraction module 820, the second feature extraction module 830, the feature fusion module 840 and the result detection module 850 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by an appropriate combination of any of them. Alternatively, at least one of the acquisition module 810, the first feature extraction module 820, the second feature extraction module 830, the feature fusion module 840 and the result detection module 850 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0093] Fig.11 A block diagram of an electronic device for a multi-dimensional fish school sensing method according to an embodiment of the present invention is schematically shown.

[0094] like Fig.11 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an on-board memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0095] In RAM603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM602 and RAM603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to an embodiment of the present invention by executing the program in ROM602 and / or RAM603. It should be noted that the program can also be stored in one or more memories other than ROM602 and RAM603. Processor 601 can also perform various operations of the method flow according to an embodiment of the present invention by executing the program stored in one or more memories.

[0096] In some specific embodiments, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0097] An embodiment of the present invention shows a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0098] In some specific embodiments, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0099] An embodiment of the present invention shows a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0100] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0101] In some specific embodiments, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0102] In some specific embodiments, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0103] In some specific embodiments, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-dimensional fish school perception method, characterized in that: The method comprises: Use a multi-dimensional fish school sensing device to obtain multi-view optical images and raw sonar data of fish schools; Obtaining a first 3D feature map of the school of fish based on the multi-view optical image; Obtaining a second 3D feature map of the school of fish based on the original sonar data; Extracting fusion features based on the first 3D feature map and the second 3D feature map to obtain a fusion feature map; and Based on the fused feature map, a 3D target detection method based on a self-attention mechanism is used to obtain the 3D detection result of the target fish school. Wherein, obtaining a first 3D feature map of the school of fish based on the multi-view optical image includes: Inputting the multi-view optical image into a first convolutional neural network to obtain feature encoding of the multi-view optical image; Based on the feature encoding, a multi-scale feature fusion method using an adaptive feature pyramid is used to obtain fused 2D features; and Based on the 2D features, multiple gimbal camera internal parameters and dynamic external parameter information, a spatial projection method is used to complete depth prediction and obtain a first 3D feature map; The multi-dimensional fish school sensing device includes: a fixed workbench, multiple pan-tilt cameras and an omnidirectional scanning sonar. Wherein, the omnidirectional scanning sonar is located at the center of the fixed workbench, and the omnidirectional scanning sonar is used to obtain point cloud information within a 360-degree range around the device; The plurality of pan-tilt cameras are evenly distributed on the fixed workbench, and the plurality of pan-tilt cameras are at equal distances from the center position; the pan-tilt cameras can rotate with a single degree of freedom in the horizontal direction; and the pan-tilt cameras can actively rotate according to the distribution of optical image feature points in the environment; The pan-tilt camera is used to acquire multi-viewing angle optical images.

2. The method according to claim 1, characterized in that The method of obtaining multi-perspective optical images and original sonar data of a school of fish comprises collecting multi-perspective optical images and original sonar data of an underwater environment using an underwater robot equipped with the device of claim 1 to obtain multi-perspective optical images and original sonar data.

3. The method according to claim 1, characterized in that The step of obtaining a second 3D feature map of the school of fish based on the original sonar data comprises: Correcting the original sonar data to obtain point cloud data with stronger time alignment; Performing voxel conversion on the point cloud data to obtain voxel data; Based on the voxel data, a second 3D feature map is obtained using a sparse 3D convolutional neural network.

4. The method according to claim 3, characterized in that The correcting the original sonar data to obtain point cloud data with a stronger time alignment includes: Filtering, denoising, correcting and normalizing the raw sonar data to obtain preprocessed sonar data; Use the autoregressive moving average model to predict the delay value of the original sonar data; Based on the delay value, the pre-processed sonar data is corrected to obtain point cloud data with stronger time alignment.

5. The method according to claim 1, characterized in that The extracting fusion features based on the first 3D feature map and the second 3D feature map includes: cascading data of the first 3D feature map and the second 3D feature map according to channels, and further extracting fusion features using a second convolutional neural network.

6. The method according to claim 1, characterized in that The 3D target detection method based on the fused feature map and using the self-attention mechanism to obtain the 3D detection result of the target fish school includes: Based on the fusion feature map at the current moment, the spatial features in the fusion feature are condensed through multi-layer convolution, global pooling and full connection layer; Using a continuous sequence of fused feature maps, optical flow features based on the displacement vector field are obtained; Based on the optical flow features, the temporal features in the fusion features are condensed through multi-layer convolution, global pooling and full connection layers; and The temporal features and the spatial features are merged layer by layer and input into the feature fusion layer to obtain the target fish school behavior detection result. Wherein, the spatial features include the relative position of the school of fish and scene information; The time feature includes the position change of the fish school at different times, which is used to reflect the movement information of the fish school.

7. A multi-dimensional fish school sensing system, characterized in that: The system comprises: An acquisition module, used to acquire multi-view optical images and raw sonar data of fish schools using a multi-dimensional fish school sensing device; A first feature extraction module, configured to obtain a first 3D feature map of the school of fish based on the multi-view optical image; A second feature extraction module, used for obtaining a second 3D feature map of the school of fish based on the original sonar data; a feature fusion module, configured to extract fusion features based on the first 3D feature map and the second 3D feature map to obtain a fusion feature map; and A result detection module is used to obtain a 3D detection result of the target fish school based on the fused feature map and a 3D target detection method based on a self-attention mechanism. Wherein, obtaining a first 3D feature map of the school of fish based on the multi-view optical image includes: Inputting the multi-view optical image into a first convolutional neural network to obtain feature encoding of the multi-view optical image; Based on the feature encoding, a multi-scale feature fusion method using an adaptive feature pyramid is used to obtain fused 2D features; and Based on the 2D features, multiple gimbal camera internal parameters and dynamic external parameter information, a spatial projection method is used to complete depth prediction and obtain a first 3D feature map; The multi-dimensional fish school sensing device includes: a fixed workbench, multiple pan-tilt cameras and an omnidirectional scanning sonar. Wherein, the omnidirectional scanning sonar is located at the center of the fixed workbench, and the omnidirectional scanning sonar is used to obtain point cloud information within a 360-degree range around the device; The plurality of pan-tilt cameras are evenly distributed on the fixed workbench, and the plurality of pan-tilt cameras are at equal distances from the center position; the pan-tilt cameras can rotate with a single degree of freedom in the horizontal direction; and the pan-tilt cameras can actively rotate according to the distribution of optical image feature points in the environment; The pan-tilt camera is used to acquire multi-viewing angle optical images.

8. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 6.

Images