Method and System for Safety Monitoring of Deep-Sea Operation Equipment Based on Panoramic Sonar Images

Abstract

The present invention provides a safety monitoring method and system for deep-sea operation equipment based on panoramic sonar images. The method includes: obtaining sonar signals returned by all sonar devices after seafloor detection during the operation of the deep-sea operation equipment, constructing a panoramic sonar image based on the sonar signals, performing pixel measurement on the panoramic sonar image, obtaining the coordinate values of the detected target objects in the actual space and the sizes of the target objects, determining the current safety level of the deep-sea operation equipment according to the coordinate values of the target objects in the actual space and the sizes of the target objects, and controlling the deep-sea operation equipment to perform corresponding operation actions according to the current safety level of the deep-sea operation equipment. The present invention realizes a safety warning mechanism for deep-sea operation equipment, avoids the operation risks of deep-sea operation equipment, and improves the service life of deep-sea operation equipment.

Description

Technical Field

[0001] The present invention relates to the field of deep-sea operation technology, and in particular to a method, device and system for safety monitoring of deep-sea operation equipment based on panoramic sonar images, a terminal device and a computer-readable storage medium. Background Art

[0002] A monitoring system can monitor trencher operations during deep-sea operations. However, due to complex geological conditions and turbid offshore waters, existing monitoring systems cannot fully understand the trencher's operating environment, posing safety risks and shortening the trencher's service life. Summary of the Invention

[0003] The main purpose of the present invention is to provide a deep-sea operating equipment safety monitoring method, device and system, terminal equipment and computer-readable storage medium based on panoramic sonar images, aiming to solve the technical problem that the existing monitoring system is still unable to fully understand the working environment of the trencher, there are safety hazards, and thus shorten the service life of the trencher.

[0004] In a first aspect, the present invention provides a method for safety monitoring of deep-sea operating equipment based on panoramic sonar images, wherein sonar equipment is installed in each direction of the deep-sea operating equipment, and the method comprises:

[0005] Acquiring sonar signals returned by all the sonar devices after seabed detection when the deep-sea operation equipment is in operation;

[0006] constructing a panoramic sonar image according to the sonar signal;

[0007] Performing pixel measurement on the panoramic sonar image to obtain coordinate values of the detected target in real space and the size of the target;

[0008] determining a current safety level of the deep-sea operation equipment according to the coordinate values of the target object in real space and the size of the target object;

[0009] According to the current safety level of the deep-sea operation equipment, the deep-sea operation equipment is controlled to perform corresponding operating actions.

[0010] In a specific embodiment, the pixel measurement of the panoramic sonar image to obtain the coordinate value of the detected target in the actual space and the size of the target includes:

[0011] Calibrate the pixel resolution of the panoramic sonar image to obtain a calibration ratio;

[0012] Measuring the pixel position and number of pixels occupied by the target object in the panoramic sonar image;

[0013] Calculate the coordinate values of the detected target object in the actual space and the size of the target object according to the calibration ratio, the pixel position, and the number of pixels.

[0014] In a specific embodiment, the calibration of the pixel resolution of the panoramic sonar image to obtain a calibration ratio includes:

[0015] Obtain the sonar signal of a standard object with a known distance and a known size within the detection range of the sonar device;

[0016] Construct a panoramic sonar image for calibration according to the sonar signal;

[0017] Measure the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration;

[0018] Determine the proportional relationship between the pixels of the panoramic sonar image for calibration and the distance and the known size according to the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration, so as to obtain the calibration ratio.

[0019] In a specific embodiment, the determination of the current safety level of the deep-sea operation device according to the coordinate values of the target object in the actual space and the size of the target object includes:

[0020] Predict the operation risk of the deep-sea operation device according to the coordinate values of the target object in the actual space, the size of the target object, and the operation state of the deep-sea operation device;

[0021] Determine the current safety level of the deep-sea operation device according to the operation risk of the deep-sea operation device and a preset safety threshold.

[0022] In a specific embodiment, the safety levels include a first-level operation risk, a second-level operation risk, and a third-level operation risk. The control of the deep-sea operation device to perform corresponding operation actions according to the current safety level of the deep-sea operation device includes:

[0023] When it is determined that the current safety level of the deep-sea operation device is the first-level operation risk, control the deep-sea operation device to continue to perform the current operation;

[0024] When it is determined that the current safety level of the deep-sea operation device is the second-level operation risk, control the deep-sea operation device to perform an avoidance action;

[0025] When it is determined that the current safety level of the deep-sea operation device is the third-level operation risk, control the deep-sea operation device to perform an action of immediately stopping operation.

[0026] In a specific embodiment, the method further includes:

[0027] Controlling a display device to display the panoramic sonar image.

[0028] In a second aspect, the present invention provides a safety monitoring device for deep - sea operation equipment based on a panoramic sonar image. Sonar devices are installed in each direction of the deep - sea operation equipment. The device includes:

[0029] A sonar signal acquisition module, configured to acquire sonar signals returned by all the sonar devices after detecting the seabed when the deep - sea operation equipment is running;

[0030] A panoramic sonar image construction module, configured to construct a panoramic sonar image according to the sonar signals;

[0031] A pixel measurement module, configured to perform pixel measurement on the panoramic sonar image to obtain the coordinate values of the detected target in the actual space and the size of the target;

[0032] A safety level determination module, configured to determine the current safety level of the deep - sea operation equipment according to the coordinate values of the target in the actual space and the size of the target;

[0033] An operation action execution module, configured to execute corresponding operation actions according to the current safety level of the deep - sea operation equipment.

[0034] In a third aspect, the present invention provides a safety monitoring system for deep - sea operation equipment based on a panoramic sonar image, including a plurality of sonar devices, a deep - sea operation equipment, a panoramic sonar image construction device, and a control device;

[0035] The plurality of sonar devices are respectively installed in each direction of the deep - sea operation equipment and are configured to acquire sonar signals returned after detecting the seabed when the deep - sea operation equipment is running;

[0036] The panoramic sonar image construction device is electrically connected to the plurality of sonar devices and is configured to: construct a panoramic sonar image according to the sonar signals; and perform pixel measurement on the panoramic sonar image to obtain the coordinate values of the detected target in the actual space and the size of the target;

[0037] The control device is electrically connected to the panoramic sonar image construction device and the deep - sea operation equipment respectively and is configured to: determine the current safety level of the deep - sea operation equipment according to the coordinate values of the target in the actual space and the size of the target; and control the deep - sea operation equipment to execute corresponding operation actions according to the current safety level of the deep - sea operation equipment.

[0038] Fourth aspect, the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a safety monitoring method for deep-sea operation equipment based on panoramic sonar images as described in the first aspect.

[0039] Fifth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute a safety monitoring method for deep-sea operation equipment based on panoramic sonar images as described in the first aspect.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows: by obtaining the sonar signals returned by all sonar devices after seafloor detection during the operation of deep-sea operation equipment, constructing a panoramic sonar image based on the sonar signals, performing pixel measurement on the panoramic sonar image, obtaining the coordinate values of the detected target objects in the actual space and the sizes of the target objects, determining the current safety level of the deep-sea operation equipment according to the coordinate values of the target objects in the actual space and the sizes of the target objects, and controlling the deep-sea operation equipment to perform corresponding operation actions according to the current safety level of the deep-sea operation equipment, a safety warning mechanism for the deep-sea operation equipment is realized, the operation risk of the deep-sea operation equipment is avoided, and the service life of the deep-sea operation equipment is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a schematic flowchart of a safety monitoring method for deep-sea operation equipment based on panoramic sonar images provided by an embodiment of the present invention;

[0042] Figure 2 is a schematic structural diagram of a safety monitoring system for deep-sea operation equipment based on panoramic sonar images provided by an embodiment of the present invention;

[0043] Figure 3 is a schematic structural diagram of a safety monitoring device for deep-sea operation equipment based on panoramic sonar images provided by an embodiment of the present invention;

[0044] Figure 4 is a schematic structural diagram of a terminal device provided by an embodiment of the present invention.

[0045] Wherein:

[0046] The realization, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by terms such as "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, "a plurality of" means two or more unless otherwise specifically and clearly defined.

[0049] In the description of the present invention, it should be noted that unless otherwise clearly specified and defined, the terms "mounted", "connected", and "coupled" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium, and it can be the communication inside two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0050] In the present invention, unless otherwise clearly specified and defined, the first feature being "above" or "below" the second feature may include the direct contact between the first and second features, or may include the situation where the first and second features are not in direct contact but in contact through other features therebetween. Moreover, the first feature being "above", "over", and "on top of" the second feature includes that the first feature is directly above and obliquely above the second feature, or merely indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature being "below", "beneath", and "under" the second feature includes that the first feature is directly below and obliquely below the second feature, or merely indicates that the horizontal height of the first feature is lower than that of the second feature.

[0051] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a method for safety monitoring of deep - sea operation equipment based on panoramic sonar images provided by an embodiment of the present invention.

[0052] A method for safety monitoring of deep - sea operation equipment based on panoramic sonar images according to an embodiment of the present invention, where sonar devices are installed in each direction of the deep - sea operation equipment, and the method includes the following steps:

[0053] S100. Obtain the sonar signals returned after the seabed detection by all the sonar devices when the deep-sea operation equipment is running;

[0054] S200. Construct a panoramic sonar image based on the sonar signals;

[0055] S300. Perform pixel measurement on the panoramic sonar image to obtain the coordinate values of the detected target object in the actual space and the size of the target object;

[0056] S400. Determine the current safety level of the deep-sea operation equipment according to the coordinate values of the target object in the actual space and the size of the target object;

[0057] S500. Control the deep-sea operation equipment to perform corresponding running actions according to the current safety level of the deep-sea operation equipment.

[0058] Specifically, a safety monitoring method for deep-sea operation equipment based on panoramic sonar images in an embodiment of the present invention can be applied to a safety monitoring system for deep-sea operation equipment based on panoramic sonar images. Please refer to Figure 2 , Figure 2 which is a schematic structural diagram of a safety monitoring system for deep-sea operation equipment based on panoramic sonar images provided in an embodiment of the present invention.

[0059] A safety monitoring system for deep-sea operation equipment based on panoramic sonar images provided in an embodiment of the present invention includes a plurality of sonar devices, deep-sea operation equipment, a panoramic sonar image construction device, and a control device;

[0060] The plurality of sonar devices are respectively installed in all directions of the deep-sea operation equipment and are used to obtain the sonar signals returned after the seabed detection when the deep-sea operation equipment is running;

[0061] The panoramic sonar image construction device is electrically connected to the plurality of sonar devices and is used to: construct a panoramic sonar image according to the sonar signals; and perform pixel measurement on the panoramic sonar image to obtain the coordinate values of the detected target object in the actual space and the size of the target object;

[0062] The control device is electrically connected to the panoramic sonar image construction device and the deep-sea operation equipment respectively and is used to: determine the current safety level of the deep-sea operation equipment according to the coordinate values of the target object in the actual space and the size of the target object; and control the deep-sea operation equipment to perform corresponding running actions according to the current safety level of the deep-sea operation equipment.

[0063] It can be understood that the deep-sea operation equipment safety monitoring system based on panoramic sonar images provided in an embodiment of the present invention can perform all the steps and functions of the deep-sea operation equipment safety monitoring method based on panoramic sonar images provided in any embodiment of the present invention.

[0064] Specifically, a deep-sea operation equipment safety monitoring method based on panoramic sonar images in an embodiment of the present invention provides a deep-sea operation equipment safety early warning mechanism, combined with Figure 2 As shown, a sonar device is installed in each of the four directions of the deep-sea operation equipment. After acquiring sonar signals returned by the sonar device after detecting the seabed at various angles, each sonar device transmits its sonar signal to a panoramic sonar image construction device to construct a 360° panoramic sonar image. In one embodiment, the panoramic sonar image can also be displayed in real time on a display screen. Secondly, the panoramic sonar image construction device measures the pixels of the panoramic sonar image, obtains the coordinates of the target detected by the sonar device in real space and the target's own dimensions, and feeds the detection signal back to the control device. The control device uses a safety condition prediction calculation to determine the safety level faced by the deep-sea operation equipment for continued operation. Based on the safety level, it sends corresponding control signals to cause the deep-sea operation equipment to perform corresponding operational actions, such as immediate stop, steering, etc., thus implementing a safety early warning mechanism for the deep-sea operation equipment and reducing operational risks.

[0065] In a specific embodiment, the deep-sea operation equipment includes a trencher, and the control equipment includes a controller.

[0066] During the seafloor detection process in step S100, sonar devices in various directions acquire seafloor information by emitting sound waves and receiving reflected signals. Common sonar devices, such as the Multibeam Echo Sounder System (MBES), utilize multiple sound beams to simultaneously detect the seafloor from different angles, generating high-resolution seafloor topography maps. Side-scan sonar (SSS) is primarily used for large-scale seafloor coverage and can identify obstacles and man-made structures. The sound waves emitted by these sonar devices reflect when they encounter various objects and terrain on the seafloor. The reflected signals are received by the sonar device's receiver, forming a sonar signal containing seafloor information.

[0067] In step S200, the sonar signals collected by each sonar device are transmitted to the panoramic sonar image construction device, which is configured with complex algorithms and processing models, such as geometric projection mapping model, spatial transformation model and image reconstruction model, to process the sonar signals.

[0068] Specifically, the geometric projection mapping model projects and maps sonar signals from the time-frequency domain to the spatial domain to determine the spatial position distribution of sonar signals. Next, the spatial transformation model performs multi-dimensional transformation on the projected spatial domain data to generate a multi-dimensional spatial transformation matrix to optimize the data spatial distribution and improve the image resolution and clarity. Finally, the image reconstruction model generates a single-frame sonar image based on the multi-dimensional spatial transformation matrix, and combines multiple single-frame images into a 360° panoramic sonar image through image stitching and fusion techniques.

[0069] In a specific embodiment, the panoramic sonar image construction device may refer to the single-frame reconstruction system of sonar images for seabed detection in the invention patent CN118799434A "A Method and System for Single-frame Reconstruction of Sonar Images for Seabed Detection", which can implement the step S200, and no more description will be made on this step here.

[0070] In a specific embodiment, the image stitching and fusion technique includes the following steps:

[0071] Feature extraction and matching: Extract feature information such as feature points, edges or textures from adjacent single-frame sonar images, and use feature matching algorithms such as gray-based correlation matching, SIFT, SURF algorithms based on feature points, etc. to find the corresponding relationships and overlapping regions between adjacent single-frame sonar images.

[0072] Image registration: According to the feature matching results, calculate the transformation parameters such as translation, rotation and scaling between adjacent single-frame sonar images, and accurately register the adjacent single-frame sonar images to align them spatially.

[0073] Image fusion: Adopt image fusion algorithms such as weighted average method and median filtering method to perform fusion processing on the overlapping regions of the registered adjacent single-frame sonar images, eliminate the stitching traces, achieve smooth transition of the images, and generate a complete and continuous panoramic sonar image.

[0074] In a specific embodiment, the step S300 measures the pixels of the panoramic sonar image to obtain the coordinate values of the detected target in the actual space and the size of the target, including the following steps:

[0075] S310. Calibrate the pixel resolution of the panoramic sonar image to obtain a calibration ratio;

[0076] S320. Measure the pixel positions and pixel numbers occupied by the target in the panoramic sonar image;

[0077] S330. Calculate the coordinate values of the detected target in the actual space and the size of the target according to the calibration ratio, the pixel positions and the pixel numbers.

[0078] In this embodiment, the coordinate values of the target object in the actual space and the size of the target object are calculated by using the proportional relationship between the image pixels and the actual size (i.e., the calibration ratio), the pixel position and the number of pixels occupied by the target object in the panoramic sonar image. Specifically, it is divided into the following steps:

[0079] Pixel resolution calibration: First, it is necessary to calibrate the pixel resolution of the panoramic sonar image, which can be achieved by performing sonar detection on a standard object with a known distance and size and measuring the number of pixels it occupies in the sonar image. For example, place a standard object with a known length of L meters within the sonar detection range, and measure that the corresponding pixel length in the panoramic sonar image is P pixels. Then, the proportional relationship k = L / P between the pixel and the actual length can be obtained.

[0080] Measurement of the target object size: After obtaining the calibration ratio, for the detected target object in the panoramic sonar image, measure the number of pixels it occupies in different directions in the panoramic sonar image. For example, the number of horizontal pixels is Px, and the number of vertical pixels is Py. Then, the horizontal size Dx = k × Px, and the vertical size Dy = k × Py of the target object.

[0081] Calculation of the target object coordinates: According to the number of pixels, pixel position of the target object in the panoramic sonar image, and the above proportional relationship, combined with the position and attitude information of each sonar device, etc., the coordinate values and size of the target object in the actual space are further calculated.

[0082] In a specific embodiment, step S310 calibrates the pixel resolution of the panoramic sonar image to obtain a calibration ratio, including the following steps:

[0083] S311. Obtain the sonar signal of a standard object with a known distance and known size within the detection range of the sonar device;

[0084] S312. Construct a panoramic sonar image for calibration according to the sonar signal;

[0085] S313. Measure the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration;

[0086] S314. Determine the proportional relationship between the pixels of the panoramic sonar image for calibration and the distance and the known size respectively according to the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration, so as to obtain the calibration ratio.

[0087] In this embodiment, by imaging a standard object with known distance and size within the sonar detection range, the proportional relationship between pixels and the actual length is determined to obtain a calibration ratio. The method is simple and effective. It can be understood that this calibration step can be executed before the deep-sea operation equipment operates. The calibration ratio can be stored in the panoramic sonar image construction device and the data can be retrieved when needed.

[0088] In a specific embodiment, step S400 determines the current safety level of the deep-sea operation equipment according to the coordinate values of the target object in the actual space and the size of the target object, and includes the following steps:

[0089] S410. Predict the operation risk of the deep-sea operation equipment according to the coordinate values of the target object in the actual space, the size of the target object, and the operation state of the deep-sea operation equipment;

[0090] S420. Determine the current safety level of the deep-sea operation equipment according to the operation risk of the deep-sea operation equipment and a preset safety threshold.

[0091] In this embodiment, the operation states of the deep-sea operation equipment include stop, forward, turn, etc. Predicting the operation risk of the deep-sea operation equipment according to the coordinate values of the target object in the actual space, the size of the target object, and the operation state of the deep-sea operation equipment can be converted into corresponding score values, and then combined with the preset safety threshold to determine the current safety level of the deep-sea operation equipment. For example, the safety levels include first-level operation risk, second-level operation risk, and third-level operation risk, and the operation risks of each level increase in turn. Determine which level of the preset safety threshold the operation risk of the deep-sea operation equipment reaches to determine the current safety level of the deep-sea operation equipment.

[0092] In a specific embodiment, the safety levels include first-level operation risk, second-level operation risk, and third-level operation risk. Step S500 controls the deep-sea operation equipment to perform corresponding operation actions according to the current safety level of the deep-sea operation equipment, and includes the following steps:

[0093] S510. When it is determined that the current safety level of the deep-sea operation equipment is the first-level operation risk, control the deep-sea operation equipment to continue to perform the current operation;

[0094] S520. When it is determined that the current safety level of the deep-sea operation equipment is the second-level operation risk, control the deep-sea operation equipment to perform an avoidance action;

[0095] S530. When it is determined that the current safety level of the deep-sea operation equipment is the third-level operation risk, control the deep-sea operation equipment to perform an action of immediately stopping operation.

[0096] In this embodiment, the control device triggers the deep-sea operation device to perform corresponding operation actions according to the obtained safety level. For example, when it is determined that the current safety level of the deep-sea operation device is a first-level operation risk, the control device controls the deep-sea operation device to continue to perform the current operation; when it is determined that the current safety level of the deep-sea operation device is a second-level operation risk, indicating that there may be a certain collision risk for the deep-sea operation device, the control device will control the deep-sea operation device to perform avoidance actions such as turning, so as to avoid the operation risk of the deep-sea operation device; when it is determined that the current safety level of the deep-sea operation device is a third-level operation risk, indicating that the deep-sea operation device is about to face serious dangers such as collisions, the control device will immediately issue an instruction to make the deep-sea operation device perform the action of immediately stopping operation. The safety warning mechanism of the deep-sea operation device is realized.

[0097] In a specific embodiment, the method further includes the following steps:

[0098] S600. Control the display device to display the panoramic sonar image.

[0099] In this embodiment, please refer to Figure 2 , the display device is electrically connected to the panoramic sonar image construction device. After the panoramic sonar image construction device generates the panoramic sonar image, it sends the panoramic sonar image to the display device, and the display device displays the panoramic sonar image in real time for the operator to monitor, so that the operator can intuitively understand the seabed conditions around the deep-sea operation device.

[0100] In summary, a safety monitoring method for a deep-sea operation device based on a panoramic sonar image provided by an embodiment of the present invention obtains sonar signals returned by all sonar devices after seabed detection during the operation of the deep-sea operation device, constructs a panoramic sonar image according to the sonar signals, performs pixel measurement on the panoramic sonar image, obtains the coordinate values of the detected target objects in the actual space and the sizes of the target objects, determines the current safety level of the deep-sea operation device according to the coordinate values of the target objects in the actual space and the sizes of the target objects, and controls the deep-sea operation device to perform corresponding operation actions according to the current safety level of the deep-sea operation device, realizes the safety warning mechanism of the deep-sea operation device, avoids the operation risk of the deep-sea operation device, and improves the service life of the deep-sea operation device.

[0101] Please refer to Figure 3 , Figure 3 is a schematic structural diagram of a safety monitoring device for a deep-sea operation device based on a panoramic sonar image provided by an embodiment of the present invention.

[0102] For a safety monitoring device for a deep-sea operation device based on a panoramic sonar image according to an embodiment of the present invention, sonar devices are installed in each direction of the deep-sea operation device, and the device includes:

[0103] A sonar signal acquisition module, configured to acquire sonar signals returned by all the sonar devices after seafloor detection during the operation of the deep-sea operation device;

[0104] A panoramic sonar image construction module, configured to construct a panoramic sonar image according to the sonar signals;

[0105] A pixel measurement module, configured to perform pixel measurement on the panoramic sonar image to obtain the coordinate values of the detected target in the actual space and the size of the target;

[0106] A safety level determination module, configured to determine the current safety level of the deep-sea operation device according to the coordinate values of the target in the actual space and the size of the target;

[0107] An operation action execution module, configured to execute corresponding operation actions according to the current safety level of the deep-sea operation device.

[0108] A safety monitoring device for a deep-sea operation device based on a panoramic sonar image provided by an embodiment of the present invention can execute all steps and functions of a safety monitoring method for a deep-sea operation device based on a panoramic sonar image provided by any of the above embodiments. The specific functions of this device will not be elaborated here.

[0109] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of a terminal device provided by an embodiment of the present invention.

[0110] The terminal device includes: a processor, a memory, and a computer program stored in the memory and configured to be run by the processor. When the processor executes the computer program, it implements the steps of a safety monitoring method for a deep-sea operation device based on a panoramic sonar image in each of the above embodiments, such as Figure 1 the steps S100 to S500 shown. Alternatively, when the processor executes the computer program, it implements the functions of each module in each of the above device embodiments.

[0111] Exemplarily, the computer program can be divided into one or more modules. The one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of completing specific functions, and these instruction segments are used to describe the execution process of the computer program in a terminal device. For example, the computer program can be divided into several modules, and the specific functions of each module have been described in detail in a safety monitoring method for a deep-sea operation device based on a panoramic sonar image provided by any of the above embodiments. The specific functions of this device will not be elaborated here.

[0112] The described terminal device can be a computing device such as a desktop computer, notebook, handheld computer, and cloud server, etc. The described terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the schematic diagram is only an example of a terminal device, and does not constitute a limitation on a terminal device. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the described terminal device may also include input / output devices, network access devices, buses, etc.

[0113] The so-called processor can be a central processing unit (CPU), or it can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor is the control center of the described terminal device, and connects all parts of the entire terminal device through various interfaces and lines.

[0114] The memory can be used to store the computer program and / or module. The processor realizes various functions of the described method for safety monitoring of deep-sea operation equipment based on panoramic sonar images by running or executing the computer program and / or module stored in the memory, and by calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one magnetic disk storage device, flash device, or other volatile solid-state storage devices.

[0115] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute a method for safety monitoring of deep-sea operation equipment based on a panoramic sonar image as described in any one of the above embodiments.

[0116] If a module integrated in a terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0117] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A safety monitoring method for deep-sea operation equipment based on panoramic sonar images, characterized in that, Sonar devices are installed in all directions of the deep - sea operation equipment. The method includes: Obtaining sonar signals returned by all the sonar devices after seafloor detection when the deep - sea operation equipment is running; Constructing a panoramic sonar image based on the sonar signals, including: Projecting and mapping the sonar signals from the time - frequency domain to the spatial domain through a geometric projection mapping model to determine the position distribution of the sonar signals in space; performing multi - dimensional transformation on the projected spatial - domain data using a spatial transformation model to generate a multi - dimensional spatial transformation matrix to optimize the data spatial distribution; generating a single - frame sonar image according to the multi - dimensional spatial transformation matrix through an image reconstruction model, and combining multiple single - frame images into a 360° panoramic sonar image through image stitching and fusion techniques; Performing pixel measurement on the panoramic sonar image to obtain the coordinate values of the detected target in the actual space and the size of the target, including: Calibrating the pixel resolution of the panoramic sonar image to obtain a calibration ratio, including: obtaining sonar signals of a standard object with a known distance and a known size within the detection range of the sonar device; constructing a panoramic sonar image for calibration based on the sonar signals; measuring the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration; determining the proportional relationships between the pixels of the panoramic sonar image for calibration and the distance and the known size respectively according to the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration to obtain the calibration ratio; Measuring the pixel position and the number of pixels occupied by the target in the panoramic sonar image; Calculating the coordinate values of the detected target in the actual space and the size of the target according to the calibration ratio, the pixel position, and the number of pixels; Determining the current safety level of the deep - sea operation equipment according to the coordinate values of the target in the actual space and the size of the target. The safety level includes first - level operation risk, second - level operation risk, and third - level operation risk; Controlling the deep - sea operation equipment to perform corresponding operation actions according to the current safety level of the deep - sea operation equipment, including: When it is determined that the current safety level of the deep - sea operation equipment is the first - level operation risk, controlling the deep - sea operation equipment to continue to perform the current operation; when it is determined that the current safety level of the deep - sea operation equipment is the second - level operation risk, controlling the deep - sea operation equipment to perform an avoidance action; when it is determined that the current safety level of the deep - sea operation equipment is the third - level operation risk, controlling the deep - sea operation equipment to perform an action of immediately stopping operation.

2. The method for safety monitoring of deep-sea operation equipment based on panoramic sonar images according to claim 1, wherein, The determining the current safety level of the deep - sea operation equipment according to the coordinate values of the target in the actual space and the size of the target includes: Predicting the operation risk of the deep - sea operation equipment according to the coordinate values of the target in the actual space, the size of the target, and the operation state of the deep - sea operation equipment; Determining the current safety level of the deep - sea operation equipment according to the operation risk of the deep - sea operation equipment and a preset safety threshold.

3. The safety monitoring method for deep-sea operation equipment based on panoramic sonar images according to any one of claims 1-2, characterized in that, The method further includes: Control the display device to display the panoramic sonar image.

4. A safety monitoring device for deep-sea operation equipment based on panoramic sonar images, characterized in that, Sonar devices are installed in each direction of the deep-sea operation device. The device includes: A sonar signal acquisition module, configured to acquire sonar signals returned after seafloor detection by all the sonar devices when the deep-sea operation device is operating. A panoramic sonar image construction module, configured to construct a panoramic sonar image based on the sonar signals, including: Projecting and mapping the sonar signals from the time-frequency domain to the spatial domain through a geometric projection mapping model to determine the position distribution of the sonar signals in space; performing multi-dimensional transformation on the projected spatial domain data using a spatial transformation model to generate a multi-dimensional spatial transformation matrix to optimize the data spatial distribution; generating a single-frame sonar image according to the multi-dimensional spatial transformation matrix through an image reconstruction model, and combining multiple single-frame images into a 360° panoramic sonar image through image stitching and fusion techniques. A pixel measurement module, configured to perform pixel measurement on the panoramic sonar image to obtain the coordinate values of the detected target in the actual space and the size of the target, including: Calibrating the pixel resolution of the panoramic sonar image to obtain a calibration ratio, including: acquiring sonar signals of a standard object with a known distance and a known size within the detection range of the sonar device; constructing a panoramic sonar image for calibration based on the sonar signals; measuring the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration; determining the proportional relationship between the pixels of the panoramic sonar image for calibration and the distance and the known size according to the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration to obtain the calibration ratio. Measuring the pixel position and the number of pixels occupied by the target in the panoramic sonar image. Calculating the coordinate values of the detected target in the actual space and the size of the target according to the calibration ratio, the pixel position, and the number of pixels. A safety level determination module, configured to determine the current safety level of the deep-sea operation device according to the coordinate values of the target in the actual space and the size of the target. The safety level includes a first-level operation risk, a second-level operation risk, and a third-level operation risk. An operation action execution module, configured to execute corresponding operation actions according to the current safety level of the deep-sea operation device, including: When it is determined that the current safety level of the deep-sea operation device is the first-level operation risk, controlling the deep-sea operation device to continue to execute the current operation; when it is determined that the current safety level of the deep-sea operation device is the second-level operation risk, controlling the deep-sea operation device to execute an avoidance action; when it is determined that the current safety level of the deep-sea operation device is the third-level operation risk, controlling the deep-sea operation device to execute an action of immediately stopping operation.

5. A safety monitoring system for deep-sea operation equipment based on panoramic sonar images, characterized in that, It includes multiple sonar devices, a deep-sea operation device, a panoramic sonar image construction device, and a control device. The multiple sonar devices are respectively installed in each direction of the deep-sea operation device and are used to acquire sonar signals returned after seafloor detection when the deep-sea operation device is operating. The panoramic sonar image construction device is electrically connected to the multiple sonar devices and is configured to: construct a panoramic sonar image according to the sonar signals, including: Projecting and mapping the sonar signals from the time-frequency domain to the spatial domain through a geometric projection mapping model to determine the position distribution of the sonar signals in space; performing multi-dimensional transformation on the projected spatial domain data by using a spatial transformation model to generate a multi-dimensional spatial transformation matrix so as to optimize the data spatial distribution; generating a single-frame sonar image according to the multi-dimensional spatial transformation matrix through an image reconstruction model, and combining multiple single-frame images into a 360° panoramic sonar image through image stitching and fusion techniques; And, performing pixel measurement on the panoramic sonar image to obtain the coordinate values of the detected target in the actual space and the size of the target, including: Calibrating the pixel resolution of the panoramic sonar image to obtain a calibration ratio, including: obtaining the sonar signals of a standard object with a known distance and a known size within the detection range of the sonar device; constructing a panoramic sonar image for calibration according to the sonar signals; measuring the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration; determining the proportional relationships between the pixels of the panoramic sonar image for calibration and the distance and the known size respectively according to the pixel position and the number of pixels occupied by the standard object in the panoramic sonar image for calibration to obtain the calibration ratio; Measuring the pixel position and the number of pixels occupied by the target in the panoramic sonar image; Calculating the coordinate values of the detected target in the actual space and the size of the target according to the calibration ratio, the pixel position and the number of pixels; The control device is electrically connected to the panoramic sonar image construction device and the deep-sea operation device respectively, and is configured to: determine the current safety level of the deep-sea operation device according to the coordinate values of the target in the actual space and the size of the target, where the safety level includes a first-level operation risk, a second-level operation risk and a third-level operation risk; and, Controlling the deep-sea operation device to perform corresponding operation actions according to the current safety level of the deep-sea operation device, including: When it is determined that the current safety level of the deep-sea operation device is the first-level operation risk, controlling the deep-sea operation device to continue to perform the current operation; when it is determined that the current safety level of the deep-sea operation device is the second-level operation risk, controlling the deep-sea operation device to perform an avoidance action; when it is determined that the current safety level of the deep-sea operation device is the third-level operation risk, controlling the deep-sea operation device to perform an action of immediately stopping operation.

6. A terminal device, characterized in that, Including: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, where when the processor executes the computer program, it implements a method for monitoring the safety of a deep-sea operation device based on a panoramic sonar image as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute a method for safety monitoring of a deep-sea operation device based on a panoramic sonar image as described in any one of claims 1 to 3.

Images