Underwater robot based on trichromatic camera and visual perception method thereof

Abstract

The present disclosure provides a kind of underwater robot based on three-color camera and its visual perception method.The underwater robot visual perception method includes: video stream is collected by the three-color camera of the underwater robot;The luminance feature vector of the current frame of the video stream is calculated and the calculated luminance feature vector is input into the pre-trained Gaussian mixture model;When it is determined by the Gaussian mixture model that the current illumination belongs to normal distribution, the three-color image frame of the current frame is input into the target detection network, when it is determined by the Gaussian mixture model that the current illumination does not belong to normal distribution, event frame is synthesized by video-to-event algorithm, and the synthesized event frame is input into the target detection network;The position and category of target are output by the target detection network;And the motion of the underwater robot is controlled based on the position and category of the target.Based on statistical model adaptive switching, all-weather robust perception is realized while ensuring computational efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of underwater robot environmental perception and computer vision technology, specifically to an underwater robot based on a three-color camera and its visual perception method. Background Technology

[0002] Underwater robots are used to perform tasks such as ocean exploration and ecological monitoring, and visual perception is one of their core capabilities. However, the underwater environment is complex and changeable, especially the drastic fluctuations in lighting conditions such as the dark light of the deep sea and the shimmering light on the water surface, which seriously affect the stability of traditional vision systems based on three-color images (RGB images).

[0003] Current solutions generally employ image enhancement algorithms or utilize event cameras. Image enhancement algorithms attempt to restore the color and contrast of RGB images, but they often fail under extreme low light conditions and are computationally intensive. Dynamic vision sensors (DVS, also known as event cameras) have high dynamic range and low latency characteristics, making them suitable for handling extreme lighting changes. However, DVS hardware is extremely expensive, and there is a lack of high-quality underwater labeled datasets, making it difficult to deploy on a large scale in low-cost underwater robots.

[0004] Therefore, a visual perception solution is needed that can leverage the low-cost advantage of standard three-color cameras (RGB cameras) while also possessing DVS robustness. Summary of the Invention

[0005] To address the aforementioned technical issues, this disclosure proposes an underwater robot based on a three-color camera and its visual perception method. This method utilizes a common three-color camera to achieve a simulated dynamic visual sensor (DVS) effect and can adaptively switch perception modes according to lighting conditions for underwater target detection.

[0006] In one general aspect, a visual perception method for underwater robots based on a three-color camera is provided. The method includes: acquiring a video stream using the underwater robot's three-color camera; calculating the luminance feature vector of the current frame of the video stream and inputting the calculated luminance feature vector into a pre-trained Gaussian mixture model; when the Gaussian mixture model determines that the current illumination belongs to a normal distribution, inputting the three-color image of the current frame into a target detection network; when the Gaussian mixture model determines that the current illumination does not belong to a normal distribution, synthesizing an event frame from the three-color image of the current frame using a video-to-event algorithm, and inputting the synthesized event frame into the target detection network; outputting the position and category of the target through the target detection network; and controlling the movement of the underwater robot based on the position and category of the target.

[0007] According to an embodiment, calculating the luminance feature vector of the current frame of the video stream includes: converting the three-color image of the current frame into a grayscale image; dividing the grayscale image into multiple regions; and calculating the average luminance of each of the multiple regions as the luminance feature vector.

[0008] According to the embodiment, the log-likelihood value of the brightness feature vector is calculated using the Gaussian mixture model. When the log-likelihood value is greater than or equal to a predetermined threshold, it is determined that the current illumination belongs to a normal distribution. When the log-likelihood value is less than the predetermined threshold, it is determined that the current illumination does not belong to a normal distribution.

[0009] According to an embodiment, the predetermined threshold is dynamically adjusted based on the historical ambient light intensity. The underwater robot collects a set of light intensity samples during the offline phase, and the historical ambient light intensity is obtained by fitting the light distribution through the Gaussian mixture model based on the collected light intensity sample set.

[0010] According to an embodiment, the target detection network is trained through the following steps: acquiring sample data and label data; converting the original video frames captured by the three-color camera into grayscale images, calculating the log-likelihood value of the illumination intensity change of the grayscale image, and using the original video frames as sample data; inputting the original video frames and label data into the target detection network, determining the loss value through a predetermined loss function; and adjusting the detection accuracy of the target detection network based on the loss value.

[0011] According to an embodiment, the underwater robot visual perception method further includes: when the log-likelihood value of the change in illumination intensity is greater than the predetermined threshold, synthesizing an event frame from the three-color image of the current frame using a video-to-event algorithm, using the synthesized event frame as sample data, and using the label data of the current original video frame as the label data of the generated synthesized event frame; inputting the synthesized event frame and label data into a target detection network, determining a loss value through a predetermined loss function; and adjusting the detection accuracy of the target detection network based on the loss value.

[0012] According to an embodiment, controlling the movement of the underwater robot based on the position and category of the target includes: guiding the movement of the underwater robot by means of PID control, fuzzy PID control or model predictive control based on the position and category of the target to perform target tracking or obstacle avoidance.

[0013] In another general aspect, an underwater robot based on a three-color camera is provided. The underwater robot includes: a data acquisition module, including a three-color camera, for acquiring video streams; a virtual event generation module, which calculates the luminance feature vector of the current frame of the video stream and inputs the calculated luminance feature vector into a pre-trained Gaussian mixture model; when the Gaussian mixture model determines that the current illumination belongs to a normal distribution, it inputs the three-color image of the current frame into a target detection network; when the Gaussian mixture model determines that the current illumination does not belong to a normal distribution, it synthesizes an event frame from the three-color image of the current frame using a video-to-event algorithm and inputs the synthesized event frame into the target detection network; and a control module, which outputs the position and category of a target through the target detection network and controls the movement of the underwater robot based on the position and category of the target.

[0014] In another general aspect, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the underwater robot visual perception method based on a tri-color camera as described above.

[0015] In another general aspect, a computer device is provided, the computer device comprising: a processor; and a memory storing a computer program that, when executed by the processor, implements the underwater robot visual perception method based on a three-color camera as described above.

[0016] This disclosure addresses the high cost of dynamic vision sensors by achieving the same effect using a standard three-color camera.

[0017] This disclosure addresses the scarcity of DVS data and the difficulty of manual annotation due to its unique imaging principles. It employs label inheritance, eliminating the need for additional annotation of the generated event data.

[0018] This disclosure proposes an adaptive switching mechanism based on a statistical model to address sudden changes in underwater illumination, achieving robust all-weather sensing while ensuring computational efficiency. Attached Figure Description

[0019] The above and other aspects, features and advantages of this disclosure will become clearer from the following detailed embodiments, taken in conjunction with the accompanying drawings, in which: Figure 1 A flowchart illustrating a visual perception method for underwater robots based on a three-color camera according to an embodiment of the present disclosure is shown.

[0020] Figure 2 A Gaussian mixture model of an underwater robot visual perception method based on a three-color camera, according to an embodiment of the present disclosure, is shown.

[0021] Figure 3 A block diagram of an underwater robot based on a tri-color camera according to an embodiment of the present disclosure is shown.

[0022] Figure 4 A data flow diagram of an underwater robot visual perception method based on a three-color camera, according to an embodiment of the present disclosure, is shown.

[0023] Figure 5 A flowchart illustrating the training process of a target detection network for an underwater robot visual perception method based on a three-color camera, according to an embodiment of the present disclosure, is shown.

[0024] Figure 6 The image shows a comparison of the imaging effects of RGB images and virtual DVS images under different lighting conditions. Detailed Implementation

[0025] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, after understanding the disclosure of this application, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will be readily apparent. For example, the order of operations described herein is merely illustrative and is not limited to the order set forth herein; rather, changes that will be readily understood after understanding the disclosure of this application are possible, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known after understanding the disclosure of this application may be omitted.

[0026] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus and / or systems described herein, many of which will become clear upon understanding the disclosure of this application.

[0027] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0028] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0029] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0030] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0031] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0032] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0033] To enable those skilled in the art to better understand this disclosure, specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0034] Figure 1 A flowchart illustrating a visual perception method for underwater robots based on a three-color camera according to an embodiment of the present disclosure is shown.

[0035] like Figure 1 As shown, the underwater robot visual perception method based on a three-color camera according to an embodiment of this disclosure includes: In step 101, a video stream is acquired using the underwater robot's tri-color camera.

[0036] In step 102, the luminance feature vector of the current frame of the video stream is calculated and the calculated luminance feature vector is input into the pre-trained Gaussian mixture model.

[0037] In step 103, when the Gaussian mixture model determines that the current illumination belongs to a normal distribution, the tri-color image of the current frame is input into the target detection network. When the Gaussian mixture model determines that the current illumination does not belong to a normal distribution, an event frame is synthesized from the tri-color image of the current frame using a video-to-event algorithm, and the synthesized event frame is input into the target detection network.

[0038] In step 104, the target location and category are output by the target detection network.

[0039] In step 105, the movement of the underwater robot is controlled based on the position and category of the target.

[0040] Based on the steps described above, a video stream is acquired by an underwater robot camera. The brightness feature vector of the current frame, calculated in real-time by illumination analysis, is input into the Gaussian Mixture Model (GMM). The processing path is selected based on the probability values ​​output by the GMM. In this case, path A has a higher probability of being executed and path B has a lower probability. In path A, the three-color image is directly input into the object detection network. In path B, the video-to-event (v2e) algorithm is called to synthesize a DVS event frame, which is then input into the object detection network. The object detection network can then perform object recognition based on the above inputs and output the object's location (bounding box) and category.

[0041] As an example, controlling the movement of the underwater robot based on the position and category of the target includes: guiding the movement of the underwater robot through PID control, fuzzy PID control, or model predictive control based on the position and category of the target to perform target tracking or obstacle avoidance. The controller of the underwater robot may include, but is not limited to, a PID controller, a fuzzy PID controller, or a model predictive controller (MPC).

[0042] The underwater robot visual perception method based on a three-color camera according to embodiments of this disclosure has the advantages of low cost and versatility. It utilizes software to implement sensor technology, endowing ordinary underwater cameras with "event perception" capabilities without increasing hardware costs. Furthermore, the underwater robot visual perception method based on a three-color camera according to embodiments of this disclosure has the advantage of high robustness. It uses a Gaussian mixture model (GMM) to evaluate ambient lighting in real time, autonomously switching between RGB features and virtual event features, ensuring stable detection by utilizing texture information under normal lighting conditions and motion edge information in dark or overexposed environments.

[0043] This disclosure employs an adaptive illumination switching mechanism based on a Gaussian Mixture Model (GMM) to balance performance and efficiency on computationally limited underwater platforms, avoiding the excessive resource consumption of dual-stream parallel processing. A pre-trained Gaussian Mixture Model is used to determine whether the current illumination is within a normal distribution. If the illumination is not within a normal distribution (too dark or too bright), the system automatically switches to a virtual DVS mode.

[0044] As an example, according to an embodiment of this disclosure, calculating the luminance feature vector of the current frame of the video stream includes: converting the tri-color image of the current frame into a grayscale image; dividing the grayscale image into multiple regions; and calculating the average luminance of each of the multiple regions as the luminance feature vector.

[0045] Figure 2 A Gaussian mixture model of an underwater robot visual perception method based on a three-color camera, according to an embodiment of the present disclosure, is shown.

[0046] Specifically, refer to Figure 2 The tricolor image I∈R of each current frame in the video stream. H×W×3 First, convert it to grayscale. gray .

[0047] To reduce computational load, the grayscale image I gray The grid is uniformly divided into M non-overlapping N×N grid blocks {B1, B2, B3, ..., B... M For example, M can be 4, in which case the grayscale image I can be... gray Divide into 4 blocks by the central cross line.

[0048] Calculate the average brightness of each block Bi. This allows us to construct a low-dimensional brightness feature vector x=[x1,x2,x3,...,x...] M ] T ∈R M .

[0049] As an example, the log-likelihood value of the brightness feature vector is calculated using the Gaussian mixture model. When the log-likelihood value is greater than or equal to a predetermined threshold... When it is determined that the current illumination belongs to a normal distribution, the log-likelihood value is less than a predetermined threshold. At that time, it was determined that the current illumination was not in a normal distribution.

[0050] As an example, the predetermined threshold is dynamically adjusted according to the historical ambient light intensity, wherein the underwater robot collects a sample set of light intensity during the offline phase, and the historical ambient light intensity is obtained by fitting the light distribution through the Gaussian mixture model based on the collected sample set of light intensity.

[0051] Specifically, during the robot's operation, the brightness vector x is calculated for the newly acquired tri-color image. test And calculate its log-likelihood under the pre-trained GMM, as shown in Equation 1 below: (1) Set a robust illumination threshold (or a predetermined threshold). Predetermined threshold It can be a preset fixed value. Optionally, a preset threshold. It can be an adaptive value that is dynamically adjusted based on historical ambient light intensity.

[0052] When log p (x test )≥ When the current (which may refer to the current frame) illumination is determined to be in a normal distribution, the original three-color image is input into the target detection network, which can utilize the rich texture and color information of the original three-color image for high-precision detection.

[0053] When log p (x test )< When it is determined that the current (referring to the current frame) illumination does not belong to the normal distribution (illumination abnormality), the illumination is either too bright or too dark, deviating from the normal training distribution. At this time, the system activates the virtual DVS generation path, synthesizes an event frame from the three-color image of the current frame using a video-to-event algorithm, and inputs the synthesized event frame into the object detection network. Image recognition based on the synthesized event frame can utilize the wide dynamic range characteristics of event data to combat illumination interference.

[0054] The target detection network can employ the general YOLO series or other detection architectures, trained using a mixture of RGB and software-implemented virtual DVS data. The target's position and category can be input into the robot controller to achieve vision-guided tracking or obstacle avoidance.

[0055] As an example, the target detection network is trained through the following steps: acquiring sample data and label data; converting the original video frames captured by the three-color camera into grayscale images, calculating the log-likelihood value of the light intensity change of the grayscale image, and using the original video frames as sample data when the log-likelihood value of the light intensity change is less than or equal to a predetermined threshold; inputting the original video frames and label data into the target detection network, determining the loss value through a predetermined loss function; and adjusting the detection accuracy of the target detection network based on the loss value.

[0056] As an example, the underwater robot visual perception method further includes: when the log-likelihood value of the change in illumination intensity is greater than the predetermined threshold, synthesizing an event frame from the three-color image of the current frame using a video-to-event algorithm, using the synthesized event frame as sample data, and using the label data of the current original video frame as the label data of the generated synthesized event frame; inputting the synthesized event frame and label data into a target detection network, determining the loss value through a predetermined loss function; and adjusting the detection accuracy of the target detection network based on the loss value.

[0057] Specifically, this disclosure proposes a software-based virtual event stream generation and label inheritance method, which can omit the physical DVS sensor and does not increase the sample labeling volume. The Video-to-Event (V2E) algorithm can reconstruct event data from a standard RGB video stream. The V2E process includes the following steps: First, the three-color image is converted to a grayscale image, and a logarithmic intensity transformation is applied to simulate the photoreceptor response; then, the log-likelihood value of the brightness feature vector is calculated using the Gaussian mixture model. When the log-likelihood value is less than a predetermined threshold, it is determined that the current illumination is not within a normal distribution, thus triggering a virtual event. This process simulates the physical characteristics of the DVS sensor's "response to motion and light intensity changes" at the software level. The generated virtual DVS image retains a clear target outline under low light conditions and removes color noise caused by low light.

[0058] Since virtual DVS data is a synthetic event frame generated from the tri-color image of the current frame using a video-to-event algorithm, there is a direct pixel correspondence between the synthetic event frame and the corresponding original tri-color image; they are strictly aligned in spatial pixel coordinates. Utilizing this spatiotemporal alignment, the labels from the RGB dataset can be directly transferred to the virtual event data, significantly reducing the training threshold for the model. Therefore, high-precision manual annotations (bounding boxes) from the RGB dataset can be directly reused through label inheritance (also known as label sharing). In other words, RGB frames I at the same time t... t The label L t It can be directly used as a virtual event frame E t This avoids the tedious task of separately annotating virtual DVS data, making it possible to train a light-resistant model using existing massive RGB datasets.

[0059] Figure 3 A block diagram of an underwater robot based on a tri-color camera according to an embodiment of the present disclosure is shown.

[0060] like Figure 3 As shown, the underwater robot based on a three-color camera according to an embodiment of this disclosure includes: The acquisition module 301 includes a three-color camera for acquiring video streams; a virtual event generation module 302 calculates the brightness feature vector of the current frame of the video stream and inputs the calculated brightness feature vector into a pre-trained Gaussian mixture model; when the Gaussian mixture model determines that the current illumination belongs to a normal distribution, it inputs the three-color image of the current frame into a target detection network; when the Gaussian mixture model determines that the current illumination does not belong to a normal distribution, it synthesizes an event frame from the three-color image of the current frame using a video-to-event algorithm and inputs the synthesized event frame into the target detection network; and a control module 303 outputs the position and category of the target through the target detection network and controls the movement of the underwater robot based on the position and category of the target.

[0061] The virtual event generation module 302 and control module 303 of the underwater robot based on a three-color camera according to embodiments of the present disclosure can be used to execute the underwater robot visual perception method based on a three-color camera according to embodiments of the present disclosure as described above.

[0062] Figure 4 A data flow diagram of an underwater robot visual perception method based on a three-color camera, according to an embodiment of the present disclosure, is shown.

[0063] like Figure 4 As shown, the data flow diagram of the underwater robot visual perception method based on a three-color camera according to an embodiment of this disclosure can be based on the original three-color image to synthesize event frames, i.e., DVS images, and after the illumination condition analysis of the Gaussian mixture model, the direct extraction of RGB features or the extraction of features from the virtual DVS is selected as the input of the target detection network, and the target detection network with shared weights outputs the detection results and guides the robot's movement.

[0064] Figure 5 A flowchart illustrating the training process of a target detection network for an underwater robot visual perception method based on a three-color camera, according to an embodiment of the present disclosure, is shown.

[0065] like Figure 5 As shown, the target detection network of the underwater robot visual perception method based on a three-color camera according to the embodiments of this disclosure can first be trained by synthesizing a DVS based on a normal RGB image. Through label inheritance, the virtual DVS can directly inherit the annotation information of the original RGB data without additional manual annotation.

[0066] This disclosure verifies, through comparative experiments and actual underwater control experiments, that the underwater robot visual perception method based on a three-color camera according to embodiments of this disclosure has significant advantages and effectiveness under low-cost hardware conditions.

[0067] Tests on standard underwater datasets and self-built pool datasets show that this disclosure solves the problem of traditional RGB methods failing in low light, as shown in Table 1 below: Table 1

[0068] Referring to Table 1, under normal lighting conditions, Example 1 of the underwater robot visual perception method based on a three-color camera according to embodiments of the present disclosure maintains extremely high detection accuracy (mAP@50>94%) using RGB features, which is close to that of the comparative example according to conventional methods.

[0069] In low-light environments: the detection accuracy of the comparative example based on the traditional RGB method drops significantly, with mAP@50 at only 6.8%, making it practically unusable. However, Example 1 of the underwater robot visual perception method based on a three-color camera according to embodiments of this disclosure maintains a detection accuracy of over 60% by switching to a virtual DVS and utilizing motion edge information.

[0070] In dynamic continuous tracking tests, the stability of this method was compared with that of traditional methods, as shown in Table 2 below: Table 2

[0071] Referring to Table 1, under low-light conditions, the comparative example based on the traditional RGB method exhibited severe continuous frame loss, with the maximum number of consecutive target frames lost exceeding 130, causing the underwater robot to completely lose sight of the target. However, Example 2 of the underwater robot visual perception method based on a three-color camera, according to embodiments of this disclosure, under the same low-light conditions, showed a maximum number of consecutive frame losses of only 4.3 frames, and the F1-score (comprehensive evaluation index) increased from 15.6% in the comparative example to 81.6%, ensuring continuous input to the control system.

[0072] Figure 6 The image shows a comparison of the imaging effects of RGB images and virtual DVS images under different lighting conditions.

[0073] Reference Figure 6 Under normal lighting conditions, both RGB images and virtual DVS images can be clearly imaged. However, under extremely low lighting conditions, RGB images lose texture details, leading to detection failure, while virtual DVS images can still retain the complete target edge contours, thus maintaining stable detection results.

[0074] Underwater closed-loop control experiments, using a biomimetic robotic fish as an example of an underwater robot, verified the engineering practicality of the underwater robot visual perception method based on a tri-color camera according to embodiments of this disclosure. In dynamic scenes with drastic lighting changes (e.g., light on -> light off -> light on), it can accurately identify sudden changes in lighting and automatically switch between RGB mode and virtual DVS mode. When encountering sudden darkness or overexposure of strong light, robots with traditional vision systems may collide or deviate from their trajectory due to target loss, while the underwater robot based on a tri-color camera according to embodiments of this disclosure can stably lock onto the target throughout the process, successfully completing target tracking and obstacle avoidance tasks.

[0075] The underwater robot and its visual perception method based on a three-color camera according to embodiments of this disclosure can be implemented based on a common RGB three-color camera, without the need for a DVS sensor.

[0076] By using a label inheritance mechanism, the existing labels of the RGB dataset can be directly reused, avoiding manual labeling of the DVS data, which makes the training and deployment of the algorithm extremely simple.

[0077] An exemplary embodiment of the present disclosure also provides a computer-readable storage medium storing a computer program. The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to execute the underwater robot visual perception method based on a three-color camera according to the present disclosure. The computer-readable recording medium is any data storage device capable of storing data read by a computer system. Examples of computer-readable recording media include: read-only memory, random access memory, read-only optical disc, magnetic tape, floppy disk, optical data storage device, and carrier waves (such as data transmission via the Internet through wired or wireless transmission paths).

[0078] An exemplary embodiment of the present disclosure also provides a computer device. The computer device includes a processor and a memory. The memory stores a computer program. The computer program is executed by the processor, causing the processor to execute the computer program of the underwater robot visual perception method based on a three-color camera according to the present disclosure.

[0079] Therefore, the exemplary embodiments of this disclosure can be implemented as methods in a computer or a non-transitory computer-readable medium storing computer-executable instructions. In the exemplary embodiments, when executed by a processor, the computer-readable instructions can perform a method according to at least one aspect of this disclosure.

[0080] Furthermore, the methods according to exemplary embodiments of this disclosure can be implemented in the form of program instructions that can be executed by various computer devices and recorded on a computer-readable medium.

[0081] Computer-readable media may include program instructions, data files, data structures, etc., individually or in combination. Program instructions recorded on a computer-readable medium may be specifically designed and configured for the exemplary embodiments of this disclosure, or may be known and available to those skilled in the art of computer software. Computer-readable recording media may include hardware devices configured to store and execute program instructions. For example, computer-readable recording media may be or include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as optical-magnetic disks; ROM; RAM; flash memory; etc. Program instructions may include not only machine language code generated by a compiler, but also high-level language code executable by a computer through an interpreter, etc.

[0082] While this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner and / or if components in the described system, architecture, apparatus, or circuit are replaced or supplemented by other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents shall be construed as included in this disclosure.

Claims

1. A visual perception method for underwater robots based on a three-color camera, characterized in that, The underwater robot visual perception method includes: Video streams are acquired using the underwater robot's three-color camera; Calculate the luminance feature vector of the current frame of the video stream and input the calculated luminance feature vector into a pre-trained Gaussian mixture model; When the Gaussian mixture model determines that the current illumination belongs to a normal distribution, the tri-color image of the current frame is input into the target detection network. When the Gaussian mixture model determines that the current illumination does not belong to a normal distribution, an event frame is synthesized from the tri-color image of the current frame using a video-to-event algorithm, and the synthesized event frame is input into the target detection network. The target detection network outputs the target's location and category; and The movement of the underwater robot is controlled based on the location and category of the target.

2. The underwater robot visual perception method according to claim 1, characterized in that, Calculating the luminance feature vector of the current frame of the video stream includes: Convert the tri-color image of the current frame to a grayscale image; The grayscale image is divided into multiple regions; The average brightness of each of the plurality of regions is calculated as the brightness feature vector.

3. The underwater robot visual perception method according to claim 2, characterized in that, The log-likelihood value of the brightness feature vector is calculated using the Gaussian mixture model. When the log-likelihood value is greater than or equal to a predetermined threshold, the current illumination is determined to be within a normal distribution. When the log-likelihood value is less than the predetermined threshold, the current illumination is determined to be outside a normal distribution.

4. The underwater robot visual perception method according to claim 3, characterized in that, The predetermined threshold is dynamically adjusted based on historical ambient light intensity. The underwater robot collects a set of light intensity samples during the offline phase, and obtains the historical environmental light intensity by fitting the light distribution through the Gaussian mixture model based on the collected light intensity sample set.

5. The underwater robot visual perception method according to claim 1, characterized in that, The target detection network is trained through the following steps: Obtain sample data and label data; The original video frames captured by the three-color camera are converted into grayscale images, and the log-likelihood value of the change in light intensity of the grayscale image is calculated. When the log-likelihood value of the change in light intensity is less than or equal to a predetermined threshold, the original video frames are used as sample data. The original video frames and label data are input into the object detection network, and the loss value is determined by a predetermined loss function. as well as Adjust the detection accuracy of the target detection network based on the loss value.

6. The underwater robot visual perception method according to claim 5, characterized in that, The underwater robot visual perception method also includes: When the log-likelihood of the change in light intensity is greater than the predetermined threshold, an event frame is synthesized from the three-color image of the current frame using a video-to-event algorithm. The synthesized event frame is used as sample data, and the label data of the current original video frame is used as the label data of the generated synthesized event frame. Synthetic event frames and label data are input into the object detection network, and the loss value is determined using a predetermined loss function; and Adjust the detection accuracy of the target detection network based on the loss value.

7. The underwater robot visual perception method according to claim 1, characterized in that, Controlling the movement of the underwater robot based on the location and category of the target includes: Underwater robots can be guided to track or avoid obstacles using PID control, fuzzy PID control, or model predictive control.

8. An underwater robot based on a three-color camera, characterized in that, The underwater robot includes: The acquisition module includes a three-color camera for acquiring video streams; A virtual event generation module calculates the luminance feature vector of the current frame of the video stream and inputs the calculated luminance feature vector into a pre-trained Gaussian mixture model. When the Gaussian mixture model determines that the current illumination belongs to a normal distribution, it inputs the tri-color image of the current frame into a target detection network. When the Gaussian mixture model determines that the current illumination does not belong to a normal distribution, it synthesizes an event frame from the tri-color image of the current frame using a video-to-event algorithm and inputs the synthesized event frame into the target detection network. The control module outputs the position and category of the target through a target detection network, and controls the movement of the underwater robot based on the position and category of the target.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the underwater robot visual perception method based on a three-color camera as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The computer device includes: processor; The memory stores a computer program that, when executed by a processor, implements the underwater robot visual perception method based on a three-color camera as described in any one of claims 1 to 7.

Images