Target detection method, device and equipment under vision of unmanned ship and medium

Abstract

The invention discloses a target detection method, device and equipment under vision of an unmanned ship and a medium. Correcting the original visual image acquired by the visual sensor according to the acquired attitude data acquired by the inertial measurement unit and the acquired three-dimensional distance information acquired by the laser radar to obtain a corrected to-be-recognized image; the to-be-recognized image is input to the trained target detection model, a marked image and target position information and confidence score for a marked frame included in the marked image are obtained, and the marked frame is used for frame selection of a target entity in the marked image; based on the confidence score, screening out a to-be-detected entity from the target entity; and extracting first feature information corresponding to the to-be-detected entities, and performing motion track detection on each to-be-detected entity based on the first feature information of each to-be-detected entity and the corresponding target position information. Therefore, the multi-dimensional data are fused to obtain accurate image data to identify the entity, and continuous motion track detection is performed on the entity, so that the target detection accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, equipment and medium for target detection under the vision of an unmanned surface vessel. Background Technology

[0002] Unmanned surface vessels (USVs) are playing an increasingly important role in many fields such as marine monitoring and maritime rescue. Specifically, USVs use sensors to collect images of their surrounding environment for target detection.

[0003] In related technologies, model training is performed based on collected environmental information image data. During the inference phase, environmental information images around the unmanned surface vessel are collected so that the trained model can be used to perform target detection on the real-time collected environmental information images to detect targets around the unmanned surface vessel.

[0004] However, target detection in related technologies is mainly based on model training using stable environmental information images. But there are usually undulating waves on the sea surface, which can interfere with the sensor. For example, waves can intermittently block the line of sight of the visual sensor, causing problems such as unclear or blurry targets in the acquired images. In addition, the waves on the sea surface can cause the unmanned vessel to sway. Under these circumstances, the quality of the acquired image data is poor. Using the acquired environmental information images for target detection will reduce the accuracy of target detection. Summary of the Invention

[0005] This application provides a target detection method, apparatus, device, and medium under the vision of an unmanned surface vessel (USV), which can improve the accuracy of visual target detection of USVs in wave-like environments.

[0006] Firstly, this application provides a target detection method under the vision of an unmanned surface vessel, including: Acquire a target sensing dataset, which includes at least the raw visual images acquired by the visual sensor, the attitude data acquired by the inertial measurement unit, and the three-dimensional distance information acquired by the lidar. The original visual image is corrected based on the pose data and the three-dimensional distance information to obtain the corrected image to be recognized. The image to be identified is input into the trained target detection model to obtain a labeled image and target location information and confidence score for the bounding boxes contained in the labeled image. The bounding boxes are used to select target entities in the labeled image. Based on the confidence score, entities to be detected are selected from the target entities; Extract the first feature information corresponding to the entity to be detected, and perform motion trajectory detection on each entity to be detected based on the first feature information of each entity to be detected and the corresponding target position information.

[0007] Secondly, this application provides a target detection device under unmanned surface vessel vision, comprising: The acquisition unit is used to acquire a target sensing dataset, which includes at least the original visual images acquired by the visual sensor, the attitude data acquired by the inertial measurement unit, and the three-dimensional distance information acquired by the lidar. A correction unit is used to perform correction processing on the original visual image based on the pose data and the three-dimensional distance information to obtain a corrected image to be recognized. The input unit is used to input the image to be identified into the trained target detection model to obtain a labeled image and target location information and confidence score for the labeled boxes contained in the labeled image. The labeled boxes are used to select target entities in the labeled image. A filtering unit is used to filter out entities to be detected from the target entities based on the confidence score; The detection unit is used to extract the first feature information corresponding to the entity to be detected, and to perform motion trajectory detection on each entity to be detected based on the first feature information of each entity to be detected and the corresponding target position information.

[0008] In some embodiments, the correction unit is further configured to: Calculate the attitude transformation matrix based on the attitude data; The original visual image is corrected according to the pose transformation matrix to obtain a corrected initial visual image; The initial visual image is spatially mapped based on the three-dimensional distance information to obtain the corrected image to be recognized.

[0009] In some embodiments, the acquiring unit is further configured to: Acquire the time signal transmitted by the Global Positioning System and generate a second pulse signal based on the time signal; Using the second pulse signal as a time reference, the system acquires raw visual images through the visual sensor, acquires attitude data of the local unmanned surface vessel through the inertial measurement unit, and acquires three-dimensional distance information through the lidar. A target sensing dataset is generated based on the original visual image, the pose data, and the three-dimensional distance information.

[0010] In some implementations, the target detection model includes at least a backbone network, an attention module, and a fusion module; the input unit is further used for: The image to be identified is input into the trained target detection model, and the backbone network is used to extract features from the image to be identified to obtain multiple initial image features. The attention module enhances the representation of each initial image feature in both channel and spatial dimensions to obtain each target feature map. The fusion module performs a fusion process on multiple target feature maps to obtain a fused feature map; Based on the fused feature map, a prediction is made to obtain a labeled image and target location information and confidence score for the labeled bounding boxes contained in the labeled image.

[0011] In some embodiments, the input unit is further configured to: For each initial image feature, a first initial feature map is obtained by global average pooling, and a second initial feature map is obtained by global max pooling. Channel attention weights are determined based on the first and second initial feature maps, and the initial image features are weighted by the channel dimension using the channel attention weights to obtain channel feature maps. The channel feature map is subjected to average pooling in the channel dimension to obtain a third initial feature map, and the channel feature map is subjected to max pooling in the channel dimension to obtain a fourth initial feature map. Spatial attention weights are determined based on the third initial feature map and the fourth initial feature map, and the channel feature map is subjected to spatial dimension weighting based on the spatial attention weights to obtain each target feature map.

[0012] In some embodiments, the target detection device under the vision of the unmanned surface vessel further includes a training unit, which is also used for: A sample image to be identified is obtained, and the sample image to be identified is annotated to obtain a sample labeled image and sample location information and sample confidence score for the sample labeled box in the sample labeled image. The sample labeled box is used to select the sample target entity in the sample labeled image. The sample image to be identified is input into a preset model to obtain a predicted labeled image; The target loss is determined based on the difference between the predicted labeled image and the sample labeled image; The preset model is trained based on the target loss to obtain the trained target detection model.

[0013] In some implementations, the training unit is further configured to: Multiple sample visual images were acquired, including images of multiple sea areas at different acquisition times and under different weather conditions; Each sample visual image is transformed to obtain a transformed sample visual image. The transformation process includes geometric transformation, color transformation, and noise addition. A set of sample visual images is generated based on the plurality of sample visual images and the transformed sample visual images; Select any one sample visual image from the set of sample visual images as the sample image to be identified.

[0014] In some embodiments, the detection unit is further configured to: For each entity to be detected, initial trajectory information is constructed by combining the corresponding first feature information and the corresponding target location information; The target original visual image of the next frame adjacent to the original visual image is obtained, as well as the corresponding target pose data and target three-dimensional distance information. The target original visual image of the next frame is corrected according to the target pose data and the target three-dimensional distance information to obtain the corrected target image to be identified. The target image to be identified is input into the target detection model to obtain the current labeled image and the current target location information and current confidence score for the current labeled box contained in the current labeled image. The current labeled box is used to select the current target entity in the current labeled image. For each entity to be detected, the positional distance between the corresponding target position information and each current target position information is determined, and the second feature information corresponding to each current target entity is extracted from the current marked image. The feature similarity between the first feature information and each second feature information is determined. The positional distance of each current target entity and the corresponding feature similarity are weighted and calculated to obtain the comprehensive similarity. A comprehensive similarity matrix is ​​obtained by combining each comprehensive similarity. Based on the comprehensive similarity matrix, the real target entity corresponding to each entity to be detected in the current labeled image is determined by the Hungarian algorithm. The current target state corresponding to each entity to be detected is determined according to the current target position information corresponding to each real target entity and the target position information corresponding to the entity to be detected. By combining the initial trajectory information and the current target state of each entity to be detected, motion trajectory detection is performed on each entity to be detected.

[0015] Furthermore, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described target detection method under unmanned surface vessel vision.

[0016] Furthermore, embodiments of this application also provide a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the aforementioned target detection method under unmanned surface vessel vision.

[0017] This application embodiment acquires a target sensing dataset, which includes at least raw visual images acquired by a visual sensor, attitude data acquired by an inertial measurement unit, and three-dimensional distance information acquired by a lidar. The raw visual images are corrected based on the attitude data and three-dimensional distance information to obtain a corrected image to be identified. This image is then input into a trained target detection model to obtain a labeled image, along with target location information and a confidence score for the bounding boxes contained within the labeled image. The bounding boxes are used to select target entities in the labeled image. Based on the confidence score, entities to be detected are selected from the target entities. First feature information corresponding to each entity to be detected is extracted, and motion trajectory detection is performed on each entity based on its first feature information and corresponding target location information.

[0018] As can be seen from the above, this application obtains multi-dimensional data such as the original visual image, posture data, and three-dimensional distance information, and corrects the original visual image by combining the posture data and three-dimensional distance information to obtain the corrected image to be identified. This avoids the impact of wave occlusion, unmanned surface vessel swaying, etc., on the quality of the visual image, thus improving the quality of the image to be identified. Then, a target detection model is used to detect targets based on the image to be identified, and the target entities in the image to be identified are marked by bounding boxes to obtain the target location information and confidence score of the target entities. Further, the entities to be detected are filtered according to the size of the confidence score. Finally, the first feature information corresponding to the entity to be detected is extracted, and the motion trajectory of each entity to be detected is tracked and detected based on the first feature information and the corresponding target location information. Therefore, compared with related technologies where poor image data quality leads to reduced target detection accuracy, this application improves the quality of image data by combining the attitude data and three-dimensional distance information of the unmanned surface vessel to correct the original visual image. The corrected image is then input into the target detection model to identify potential target entities. Based on the confidence score, the target entities with higher recognition accuracy are selected. By combining the feature information of the target entities with the initial target position information, continuous motion trajectory detection is performed, thereby improving the target detection accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A scene diagram illustrating the target detection system under unmanned surface vessel vision provided in an embodiment of this application; Figure 2 A schematic flowchart illustrating the steps of the target detection method under unmanned surface vessel vision provided in an embodiment of this application; Figure 3 A schematic diagram of the target detection device under the vision of an unmanned surface vessel provided in an embodiment of this application; Figure 4 This is a schematic diagram of the terminal structure provided in the embodiments of this application; Figure 5 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It is understood that in the specific implementation of this application, the original visual images, posture data, and related data such as three-dimensional distance information, images to be identified, target location information, confidence scores, and motion trajectories are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the subject is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0023] Furthermore, when this application embodiment needs to obtain relevant data, it will obtain separate permission or separate consent for the original visual image, posture data, and related data such as three-dimensional distance information, image to be identified, target location information, confidence score, and motion trajectory through pop-up windows or redirection to a confirmation page. Only after clearly obtaining separate permission or separate consent for the original visual image, posture data, and related data such as three-dimensional distance information, image to be identified, target location information, confidence score, and motion trajectory will the necessary data for enabling the application embodiment to operate normally be obtained.

[0024] It should be noted that while some processes described in the specification, claims, and accompanying drawings contain multiple steps that appear in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not represent any particular order of execution. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] This application provides a target detection method, apparatus, device, and medium based on the vision of an unmanned surface vessel (USV). Specifically, the target detection method based on the vision of an USV can be implemented in a computer device, which can be a server or a user terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The user terminal device can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart home appliance, vehicle terminal, smart voice interaction device, aircraft, drone, etc., but is not limited to these.

[0027] For ease of understanding, this application will describe the implementation process of the target detection method under unmanned surface vessel vision through several embodiments, as follows: This application provides a target detection method based on the vision of an unmanned surface vessel (USV). It primarily involves fusing multi-dimensional sensor data to calibrate the visual image, and then performing target recognition based on the calibrated image to detect target entities and their motion trajectories. Specifically, it acquires multi-dimensional data including the original visual image, attitude data, and 3D distance information. The original visual image is then calibrated using the attitude data and 3D distance information to obtain a calibrated image to be identified. This avoids issues such as wave occlusion and USV body swaying that could affect the quality of the visual image, thus improving the quality of the image to be identified. Then, a target detection model is used to detect targets based on this image, marking target entities in the image with bounding boxes to obtain target location information and confidence scores. Further, entities to be detected are selected according to their confidence scores. Finally, the first feature information corresponding to each entity to be detected is extracted, and the motion trajectory of each entity is tracked and detected based on its first feature information and corresponding target location information. Please refer to the following specific embodiments for details.

[0028] It should be noted that the target detection method under the vision of this unmanned surface vessel can be executed by the terminal alone, or by the terminal and the server together.

[0029] For example, see Figure 1 This is a schematic diagram of an information push system provided in an embodiment of this application. The system includes a terminal 110 and / or a server 120.

[0030] Taking the standalone execution by a terminal as an example, each terminal 110 can have a target application installed, through which corresponding application services can be run. This target application can be called a client. Taking terminal 110 as an unmanned surface vessel (USV) as an example, the USV can collect target sensing datasets. The target sensing datasets include at least the original visual images collected by the visual sensor, the attitude data collected by the inertial measurement unit, and the three-dimensional distance information collected by the lidar. The original visual images are corrected based on the attitude data and the three-dimensional distance information to obtain the corrected image to be identified. The image to be identified is input into the trained target detection model to obtain a labeled image and the target position information and confidence score of the bounding boxes contained in the labeled image. The bounding boxes are used to select target entities in the labeled image. Based on the confidence score, the entities to be detected are selected from the target entities. The first feature information corresponding to the entities to be detected is extracted, and the motion trajectory of each entity to be detected is detected based on the first feature information and the corresponding target position information.

[0031] Furthermore, taking the target detection method under the vision of an unmanned surface vessel (USV) jointly executed by a terminal and a server as an example, the target sensing dataset can be uploaded from the terminal 110 to the server 120. For example, if the terminal 110 is used as an USV, the USV can collect a target sensing dataset, which includes at least the raw visual images collected by the visual sensor, the attitude data collected by the inertial measurement unit, and the three-dimensional distance information collected by the lidar. The client on the terminal 110 can upload the target sensing dataset to the server 120, and the server can then execute the target detection method under the vision of the USV.

[0032] The server 120 can be a single service node, a distributed system composed of multiple service nodes, or a service node within a distributed system. For example, the server 120 can be a cloud service, an edge node, etc., without limitation. The server 120 executes the steps of the target detection method under the vision of the unmanned surface vessel. Specifically, it can acquire a target sensing dataset, which includes at least the original visual image acquired by the visual sensor, the attitude data acquired by the inertial measurement unit, and the three-dimensional distance information acquired by the lidar; it corrects the original visual image based on the attitude data and the three-dimensional distance information to obtain a corrected image to be identified; it inputs the image to be identified into the trained target detection model to obtain a labeled image and target location information and confidence score for the labeled bounding boxes contained in the labeled image. The bounding boxes are used to select target entities in the labeled image; based on the confidence score, it filters out the entities to be detected from the target entities; it extracts the first feature information corresponding to the entity to be detected, and performs motion trajectory detection on each entity to be detected based on the first feature information and the corresponding target location information.

[0033] Therefore, by acquiring multi-dimensional data such as the original visual image, posture data, and 3D distance information, and combining the posture data and 3D distance information to correct the original visual image, a corrected image to be identified is obtained. This avoids the impact of wave occlusion, unmanned surface swaying, and other factors on the quality of the visual image, thus improving the quality of the image to be identified. Then, a target detection model is used to detect targets based on the image to be identified, marking target entities in the image with bounding boxes to obtain the target location information and confidence score of the target entities. Further, the entities to be detected are filtered according to the confidence score. Finally, the first feature information corresponding to the entity to be detected is extracted, and the motion trajectory of each entity to be detected is tracked and detected based on the first feature information and the corresponding target location information. Therefore, compared with related technologies where poor image data quality leads to reduced target detection accuracy, this application improves the quality of image data by combining the attitude data and three-dimensional distance information of the unmanned surface vessel to correct the original visual image. The corrected image is then input into the target detection model to identify potential target entities. Based on the confidence score, the target entities with higher recognition accuracy are selected. By combining the feature information of the target entities with the initial target position information, continuous motion trajectory detection is performed, thereby improving the target detection accuracy.

[0034] For ease of understanding, the steps of the target detection method under unmanned surface vessel vision will be described in detail below. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0035] See Figure 2 , Figure 2 This is a flowchart illustrating the steps of the unmanned surface vessel (USV) vision-based target detection method provided in this embodiment. In this embodiment, the USV vision-based target detection method can be executed by a computer device, such as a server. The specific process is as follows: 101. Obtain the target sensor dataset.

[0036] Unmanned surface vessels (USVs) are playing an increasingly important role in numerous fields such as marine monitoring and maritime rescue. For example, USVs use sensors to collect images of their surrounding environment for target detection. However, the sea surface is often characterized by undulating waves, which can interfere with the sensors. For instance, waves can intermittently obstruct the vision of the visual sensors, resulting in images that are either clear or blurry. Furthermore, waves can cause the USV to rock, leading to poor image quality. When these environmental images are used for target detection, the accuracy of target detection is reduced.

[0037] To address the above issues, this application embodiment fuses multi-dimensional sensor data to calibrate the visual image, and performs target recognition based on the calibrated visual image to detect target entities and detect their motion trajectories. Specifically, by acquiring multi-dimensional data such as the original visual image, attitude data, and three-dimensional distance information, and combining the attitude data and three-dimensional distance information to calibrate the original visual image, a calibrated image to be identified is obtained. This avoids the impact of wave occlusion, unmanned surface vessel swaying, etc., on the quality of the visual image, thus improving the quality of the image to be identified. Then, a target detection model is used to detect targets based on the image to be identified, marking target entities in the image with bounding boxes to obtain the target location information and confidence score of the target entities. Further, the entities to be detected are filtered according to the confidence score. Finally, the first feature information corresponding to the entity to be detected is extracted, and the motion trajectory of each entity to be detected is tracked and detected based on the first feature information and the corresponding target location information.

[0038] The unmanned surface vessel (USV) serves as the terminal for collecting target sensing datasets. The USV is equipped with at least a visual sensor, an inertial measurement unit (IMU), and a lidar. Thus, the target sensing dataset is obtained by acquiring raw visual images through the visual sensor, attitude data through the IMU, and three-dimensional distance information through the lidar.

[0039] The target sensing dataset includes at least the raw visual images acquired by the visual sensor, the attitude data acquired by the inertial measurement unit, and the three-dimensional distance information acquired by the lidar.

[0040] The visual sensor utilizes a high-resolution, low-light-performance industrial-grade camera, enabling it to acquire clear images even in dimly lit maritime environments. It is also equipped with autofocus and image stabilization to handle the swaying motion of the unmanned surface vessel (USV). The lens angle of the visual sensor is selected based on actual needs, such as a common 120° wide-angle lens, ensuring coverage of a large monitoring area. The visual sensor continuously acquires maritime image data at a set frame rate, optimized to 30 frames per second, capturing raw visual images. These raw visual images can be of the surroundings of the USV and can encompass various types of targets, including entities such as fishing boats, cargo ships, speedboats, buoys, and offshore platforms.

[0041] The Inertial Measurement Unit (IMU) is a device that measures an object's three-axis attitude angles (or angular velocities) and acceleration. Specifically, it employs a high-precision microelectromechanical system (MEMS) capable of measuring the unmanned surface vessel's (USV) three-axis acceleration, three-axis angular velocity, and three-axis magnetic field strength at sufficiently high frequencies. Based on these data, the USV's attitude data can be calculated. For example, when measuring the USV's attitude data, a frequency of 100Hz can be used to acquire the attitude data, which may include roll angle, pitch angle, and yaw angle.

[0042] The lidar is used to scan the surrounding environment to measure three-dimensional distance data, which refers to the distance data obtained by scanning the environment around the unmanned surface vessel (USV). Specifically, the lidar on the USV emits laser beams at a specific scanning frequency (e.g., 10Hz) to scan the surrounding environment, measuring and outputting the "relative position information of the target or environmental object in three-dimensional space relative to the USV." The core is to use "X, Y, and Z coordinates" to describe where objects are on the sea surface, how far away they are from the USV, and their altitude. It should be noted that the lidar can specifically be a mechanical rotating lidar, which can quickly acquire three-dimensional point cloud data of the surrounding environment, providing depth information to supplement visual images and helping to accurately identify the position and distance of targets in complex wave environments.

[0043] In some implementations, a specific time signal can be used as a time reference, enabling multiple sensors to simultaneously acquire raw visual images, attitude data, and 3D distance information. For example, step 101 may include: acquiring a time signal sent by a global positioning system and generating a second pulse signal based on the time signal; using the second pulse signal as a time reference, acquiring raw visual images through a visual sensor, acquiring attitude data of the local unmanned surface vessel through an inertial measurement unit, and acquiring 3D distance information through a lidar; and generating a target sensing dataset based on the raw visual images, attitude data, and 3D distance information.

[0044] The time signal can be a high-precision time signal based on the Global Positioning System (GPS). Specifically, the GPS module on the unmanned surface vessel receives high-precision time signals from multiple satellites (the satellites are equipped with atomic clocks, with time accuracy down to the nanosecond level) and synchronizes itself to UTC (Coordinated Universal Time). Every second, the GPS module automatically outputs a second pulse signal (i.e., second pulse PPS). The rising or falling edge of the pulse strictly corresponds to the whole second of UTC time (such as 00:00:00, 00:00:01, etc.), and its time accuracy is guaranteed by the satellite signal, requiring no additional calculation.

[0045] The pulse signal serves as a time reference signal for controlling the time synchronization between the vision sensor, inertial measurement unit, and lidar. Through hardware circuits and software algorithms, the time synchronization between the vision sensor, inertial measurement unit, and lidar is achieved, ensuring that the time error of the data collected by each sensor is controlled within the microsecond level, thus guaranteeing the accuracy of the fused data (such as the corrected image to be identified).

[0046] Specifically, the unmanned surface vessel receives a high-precision time signal from the Global Positioning System (GPS) and then generates a pulse-per-second (PPS) signal based on this time signal. This PPS signal acts like a "uniform alarm clock," ensuring that the time is aligned when all subsequent sensors collect data. Next, using this second pulse signal as a unified time reference, three key sensors are activated simultaneously: First, a visual sensor (such as an industrial-grade camera) captures images of the marine environment at a set frequency (30 frames / second in the document), acquiring raw visual images containing marine targets (such as ships and buoys). Second, an inertial measurement unit (IMU) captures the unmanned surface vessel's attitude changes in real time at a higher frequency (100Hz in the document), such as the hull's roll, pitch, and yaw angles. This data reflects the unmanned surface vessel's swaying caused by wave undulations. Third, a lidar (a mechanical rotating lidar in the document) scans the surrounding environment at a certain frequency (10Hz in the document), calculating the three-dimensional spatial position information of surrounding objects (including targets) through laser reflection, i.e., three-dimensional distance data (e.g., how many meters in front of the unmanned surface vessel, how many meters to its right, and how many meters above it). Finally, the three types of data collected under the same time reference—"raw visual images," "unmanned surface vessel attitude data," and "3D distance information"—are integrated together to form a complete, time-synchronized target sensing dataset. This dataset contains not only the visual features of the target, but also the attitude background of the unmanned surface vessel and the actual spatial distance information of the target, providing reliable basic data for subsequent multi-sensor data fusion, target detection, and tracking.

[0047] Using the above methods, attitude data of the local unmanned surface vessel can be collected by an inertial measurement unit, and three-dimensional distance information can be collected by a lidar to form a target sensing dataset. This allows for the subsequent fusion of multi-dimensional data from the target sensing dataset to obtain a corrected visual image, thereby enabling the identification and detection of entities on the sea surface around the unmanned surface vessel by combining multi-dimensional data, thus improving the accuracy of target detection.

[0048] 102. Based on the pose data and three-dimensional distance information, the original visual image is corrected to obtain the corrected image to be recognized.

[0049] In this embodiment, after obtaining the target sensing dataset, the attitude data, 3D distance information, and original visual image in the target sensing dataset can be fused. Specifically, the original visual image is corrected based on the attitude data and 3D distance information to obtain the corrected image to be identified. In this way, the fusion of multi-dimensional sensor data such as attitude data, 3D distance data, and original visual image is achieved, removing the interference of factors such as waves, spray, and swaying on the sea surface on the visual image, improving the quality of the fused image used for target detection, so that the target entity in the corrected image to be identified can be identified subsequently. In this way, the accurate identification of the environment around the unmanned surface vessel (such as any ship, sea buoy, offshore operation platform, etc.) can be achieved, thereby improving the accuracy of target detection.

[0050] The image to be identified is a corrected image that may contain information about the environment around the unmanned surface vessel, such as entities like ships, buoys, and offshore platforms.

[0051] It should be noted that a multi-sensor data fusion model based on Bayesian estimation theory can be established. This model treats data from different sensors as interrelated random variables, describing their relationships through a joint probability distribution. In this model, visual sensor data provides visual feature information of the target, IMU data provides attitude information of the unmanned surface vessel, and LiDAR data provides distance information of the target entity. These three complement each other, jointly constructing a more accurate description of the target and its environment. The joint probability distribution is the foundation of Bayesian estimation, and Bayesian estimation is the foundation of Kalman filtering.

[0052] In some implementations, road topology information and road incident information can be combined for encoding to generate a path distance threshold, and a dynamic obstacle probability distribution can be calculated by combining vehicle driving status information and reference driving status information of vehicles surrounding the target vehicle. For example, step 102, "converting road topology information, vehicle driving status information, and road incident information into a path distance threshold and a dynamic obstacle probability distribution," may include: (102.1) Calculate the attitude transformation matrix based on the attitude data; (102.2) Correct the original visual image according to the pose transformation matrix to obtain the corrected initial visual image; (102.a.3) Based on the three-dimensional distance information, the initial visual image is spatially mapped to obtain the corrected image to be identified.

[0053] Specifically, firstly, based on the attitude data of the inertial measurement unit and the motion model of the target entity (the entity to be detected), the Kalman filter algorithm is used to predict the current target state of the next frame of the original visual image of the target in the next moment, that is, to predict the target position information and attitude (velocity) of a certain target entity in the next frame of the "marked image". The "current target state" includes the predicted target position information and attitude (velocity).

[0054] For example, assuming the target entity moves on a two-dimensional plane, its historical state vector in the previous frame can be represented as: ,in and These are the historical location coordinates of the target entity. and It represents the historical velocity components of the target entity. According to kinematic principles, the state transition matrix... It can be represented as:

[0055] in, This is the time interval, determined by the image acquisition frame rate. The state transition matrix is ​​a transformation matrix in Kalman filtering that maps the relationship between the current state and the predicted state. Historical state estimate at the previous time step It can predict the first target state at the current moment. .

[0056] The measurement data, including the raw visual images acquired by the visual sensor and the 3D distance data obtained by the lidar, are fused with the predicted first target state. First, the actual position of the visual sensor in space and the attitude transformation matrix R are calculated based on the attitude data measured by the IMU. The attitude transformation matrix R refers to the rotation matrix corresponding to the attitude changes of the unmanned surface vessel (USV). Then, this transformation matrix is ​​used to correct the visual image, eliminating image distortion caused by the vessel's swaying. Simultaneously, a measurement matrix H is constructed by combining the distance data from the lidar. The measurement matrix H maps the target's state vector to the measurement space; for example, it constructs a measurement matrix based on the target position information measured by the visual sensor. The measurement matrix is ​​as follows:

[0057] Through measurement matrix and actual detection location (measured value) ), utilizing Kalman gain Update the predicted state to obtain a more accurate state estimate. The Kalman gain is expressed as follows:

[0058] in, It is the covariance matrix of the predicted state. This is the covariance matrix of the measurement noise. It should be noted that after each state update of the target entity, the covariance matrix of the state estimate also needs to be updated. To reflect the uncertainty of the estimate, the covariance matrix is ​​expressed as:

[0059] The update formula for the covariance matrix is ​​as follows: ,in It is an identity matrix.

[0060] Thus, by continuously performing state prediction, measurement updates, and covariance updates, the Kalman filter algorithm can effectively fuse data from different sensors, eliminate the interference of wave fluctuations on visual sensor data, and improve the accuracy and stability of the data.

[0061] To facilitate understanding, the generation process of the corrected image to be identified is further described. The corrected image is obtained by eliminating the distortion caused by the hull movement of the original visual image through "attitude compensation based on Kalman filtering". The specific process is as follows: First, the original visual image and attitude prediction values ​​are acquired. The visual sensor acquires the original visual image (containing the target but may be tilted or offset). At the same time, the Kalman filter predicts the "attitude offset of the visual sensor at the current moment" (such as roll angle deviation α and pitch angle deviation β) based on the attitude data of the IMU. Then, the image correction state transition matrix is ​​calculated. Based on the predicted attitude offset (α, β) and combined with the installation parameters of the visual sensor (such as the relative position of the camera and the hull), the "attitude transformation matrix" is calculated. This attitude transformation matrix describes "how to rotate and translate the tilted original image to a horizontal state" (for example, if the offset is α, rotate α in the opposite direction to cancel the tilt). Next, geometric correction is performed on the original visual image. A state transformation matrix is ​​used to recalculate the coordinates of each pixel in the original visual image (e.g., mapping the original pixel (x,y) to the corrected new coordinates (x',y')). This geometric adjustment of the original visual image horizontalizes the tilted image, adjusts the offset target entity to its true position, and eliminates visual distortion caused by the hull's movement, resulting in a corrected initial visual image. Finally, details are optimized using LiDAR data. Specifically, the 3D distance data from the LiDAR is used to supplement the initial visual image with depth information of the target entity, such as marking "a target entity has coordinates (x,y) in the image, actual distance 30 meters." However, the core "attitude correction" (eliminating tilt and offset) mainly relies on the attitude estimate output by the Kalman filter. Thus, the final corrected image retains the visual features of the target while eliminating distortion caused by waves, and can be directly used as input to the target detection model, improving the model's target recognition accuracy.

[0062] The above method allows for the fusion of attitude data, 3D distance information, and original visual images from the target sensing dataset. Specifically, the original visual image is corrected based on the attitude data and 3D distance information to obtain a corrected image to be identified. This removes the interference from factors such as undulating waves, spray, and swaying on the sea surface, improving the quality of the fused image used for target detection. This allows for subsequent identification of target entities in the corrected image, thus enabling accurate identification of entities in the environment surrounding the unmanned surface vessel (such as any vessel, buoy, or offshore platform) and improving target detection accuracy.

[0063] 103. Input the image to be identified into the trained target detection model to obtain the labeled image and the target location information and confidence score of the bounding boxes contained in the labeled image.

[0064] In this embodiment, after obtaining the corrected image to be identified, the image can be input into the trained target detection model so that the target detection model can perform target entity recognition based on the image to be identified, detect the target entities in the image to be identified, and mark the target entities in the image to be identified using bounding boxes, thus obtaining a marked image containing the bounding boxes. In this way, the bounding boxes are used to select the target entities in the marked image, and the coordinate information of each bounding box represents the position information of the corresponding target entity. In addition, a confidence score is output for the bounding boxes, indicating the probability or likelihood of the presence of a target entity at the location of the bounding box. In this way, the target entities in the image to be identified are detected, so as to prepare for the identification and description of the environmental information around the unmanned surface vessel, which has reliability.

[0065] The target detection model can be an image recognition model. In some implementations, YOLOv5 can be used as the model base, as it has advantages such as fast detection speed and high accuracy, making it suitable for applications in scenarios like unmanned surface vessels (USVs) where real-time performance is critical. Specifically, the overall architecture of YOLOv5 includes an input layer, a backbone network, a neck, and a prediction layer. The backbone network (CSPDarknet53) is responsible for extracting basic image features. The neck uses a Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) structure to fuse feature maps of different scales to adapt to the detection of targets of different sizes. The prediction layer then predicts the target category and location based on the fused feature maps.

[0066] Furthermore, in this embodiment, the model is modified to embed an attention mechanism module. For example, within the backbone network (CSPDarknet53), a Convolutional Block Attention Module (CBAM) is embedded at a suitable location, such as after the backbone network. The CBAM module mainly consists of a Channel Attention Module and a Spatial Attention Module. Feature maps are weighted sequentially by the Channel Attention Module and the Spatial Attention Module to highlight the features of the target entity. Then, these features pass through the Neck and the prediction layer to output a labeled image. Therefore, the modified object detection model can architecturally include a backbone network, an attention module, a fusion module (i.e., the neck), and an output layer.

[0067] In some implementations, an initial image feature can be proposed based on the image to be identified by an object detection model, and the initial image feature can be enhanced using an attention mechanism. A prediction can then be made based on the enhanced image feature to obtain the labeled image. For example, the object detection model includes at least a backbone network, an attention module, and a fusion module. Step 103 may include: (103.1) Input the image to be identified into the trained target detection model, and extract features from the image to be identified through the backbone network to obtain multiple initial image features; (103.2) Enhance the representation of each initial image feature by channel dimension and spatial dimension through the attention module to obtain the feature map of each target; (103.3) Multiple target feature maps are fused using a fusion module to obtain a fused feature map; (103.4) Based on the fused feature map, prediction is performed to obtain the labeled image and the target location information and confidence score of the labeled bounding box contained in the labeled image.

[0068] The labeled image can contain environmental information about the unmanned surface vessel (USV), such as various target entities including ships, buoys, and offshore platforms. These target entities are marked with bounding boxes to select the location region of each target entity in the image to be identified. Each bounding box can correspond to one or more coordinate information, that is, the bounding box can be represented by one or more coordinate information. For example, the target location information of the bounding box can be represented by the coordinates of the upper left corner and the lower right corner.

[0069] First, after obtaining the corrected image to be identified, it can be input into the trained object detection model. The backbone network of the object detection model extracts features from the image to obtain multiple initial image features. Specifically, the backbone network is the "bottom layer" of the model, and its core function is to process the input image layer by layer, extracting visual features at different levels and with different levels of abstraction, providing "feature materials that can be used to distinguish the target from the background and identify the target category" for subsequent modules. For example, the backbone network abstracts features layer by layer. Specifically, the backbone network can include shallow and deep layers. Shallow layers (the first few convolutional layers and residual blocks) extract "low-level features," such as image edges, textures, and color distribution (e.g., the outline edges of maritime targets, the texture of waves, and the color of ships). These features have high resolution (close to the original image) but low abstraction, and are mainly used to locate fine details of target entities. Secondly, deep layers extract "high-level features," such as the global shape of the target and semantic relationships (e.g., the overall outline proportion of a "ship," the head-to-body ratio of a "person," or the outline proportion of other target entities). These features have low resolution (e.g., about 1 / 10 of the original image to be identified). While the scale is 6 or even 1 / 32, it has a high degree of abstraction and can directly reflect "what the target entity is," possessing strong classification capabilities. Thus, multi-scale feature maps are obtained. For example, to adapt to target entities of different sizes (such as large cargo ships, small buoys, or people in the water), the backbone network will output multiple feature maps of different scales, resulting in multiple initial image features. For example, assuming the input image is 416×416, the output will be 52×52, 26×26, and 13×13. The smaller scale feature map (13×13) has a large receptive field, suitable for extracting global features of large targets; while the larger scale feature map (52×52) has a small receptive field, suitable for extracting local detail features of small targets. The above is just an example. It should be noted that each initial image feature can represent a feature of a potentially existing target entity.

[0070] Then, after obtaining multiple initial image features, for each initial image feature, an attention module can be used to enhance its representation in both the channel and spatial dimensions to obtain each target feature map. Specifically, the attention module can include a channel attention submodule and a spatial attention submodule. For each initial image feature, the channel attention submodule first enhances the initial image feature in the channel dimension to obtain a channel feature map; further, the spatial attention submodule enhances the channel feature map in the spatial dimension to obtain the corresponding target feature map. In this way, the representation of each initial image feature is enhanced in both the channel and spatial dimensions to obtain each target feature map, and the feature regions of each target entity are extracted. It should be noted that the processing of this attention mechanism can also be understood as a feature extraction process. Embedding an attention module in the backbone network allows the model to pay more attention to the features of the region where the target entity is located when extracting and processing features, especially when facing complex backgrounds caused by undulating waves, thereby improving the feature extraction capability of the target entity.

[0071] Next, the neck layer performs "fusion and enhancement" on the multi-scale target feature maps. By fusing cross-scale features, the limitations of single-scale features are addressed, making the features more adaptable to the detection needs of targets of different sizes. This neck layer consists of FPN (Feature Pyramid Network) + PAN (Path Aggregation Network). The feature maps output by the backbone network have a "scale fragmentation" problem. After processing by the attention mechanism, there is still a problem of inconsistent scales. Since the deep small-scale feature map (13×13) can identify large targets, it loses the details of small targets, while the shallow large-scale feature map (52×52) can capture the details of small targets, it lacks global semantics. Therefore, the neck layer uses the "top-down fusion" of FPN (transmitting deep semantic features to the shallow layer to supplement semantic information for small target features) and the "bottom-up fusion" of PAN (transmitting shallow detailed features to the deep layer to supplement positional details for large target features) to finally obtain a fused feature map. This fused feature map can include three fused multi-scale sub-feature maps. These multi-scale sub-feature maps contain both deep semantic information (which can distinguish between "ships" and "buoys") and shallow detailed information (which can accurately locate the bounding boxes of small buoys).

[0072] Finally, the prediction layer performs predictions based on the fused feature map to obtain a labeled image, along with target location information and confidence scores for the bounding boxes contained in the labeled image. Additionally, it may include category labels representing the target entities. Specifically, the prediction layer analyzes the fused feature map and ultimately outputs a labeled image, along with three core detection results for the target entities in the labeled image: "target category," "target location (bounding box)," and "detection confidence." This describes "what target entities are in the image to be identified, where the target entities are located, and how reliable the detection results are."

[0073] For example, firstly, the fused feature map is mapped to a "class probability vector" through convolutional layers. The vector length is equal to the number of target entity classes the model needs to identify (e.g., if "ship", "buoy", and "person" need to be identified, the vector length is 3). The vector is converted into probability values ​​through the Softmax function, and the class with the highest probability is the "predicted class of the target entity" (e.g., "ship" has a probability of 0.92, meaning the target is identified as a ship). Secondly, target location prediction (regression branch) also outputs a "boundary box parameter vector" through convolutional layers, containing parameters such as the "center coordinates (x, y)" and "width and height (w, h)" of the target bounding box. These parameters are transformed according to the scale of the feature map (e.g., converting the relative coordinates on the feature map to the pixel coordinates of the original image), ultimately obtaining a "rectangular bounding box that can enclose the target entity" (e.g., the coordinates of the top left corner (x1, y1) and bottom right corner (x2, y2) of the target entity in the original image to be identified). Thirdly, the confidence prediction (confidence branch) measures whether the target entity actually exists within the predicted bounding box and the accuracy of the bounding box location. A higher confidence score (e.g., 0.85) indicates a more reliable detection result. Conversely, if the confidence score is below a set threshold (e.g., 0.5 mentioned in the document), it indicates that the detection is not feasible and can be classified as "no target entity," thus filtering out false positives.

[0074] In some implementations, for each initial image feature, channel feature maps are obtained by weighting the channel dimension using channel attention weights, and corresponding target feature maps are obtained by weighting the channel feature maps using spatial attention weights. For example, step (103.2) may include: for each initial image feature, obtaining a first initial feature map by global average pooling and a second initial feature map by global max pooling; determining channel attention weights based on the first and second initial feature maps; weighting the initial image features by the channel dimension using the channel attention weights to obtain channel feature maps; further, performing average pooling on the channel feature maps along the channel dimension to obtain a third initial feature map, and performing max pooling on the channel feature maps along the channel dimension to obtain a fourth initial feature map; determining spatial attention weights based on the third and fourth initial feature maps; and weighting the channel feature maps by the spatial dimension using the spatial attention weights to obtain each target feature map.

[0075] Specifically, the attention module can include a channel attention submodule and a spatial attention submodule. After obtaining multiple initial image features, for each initial image feature, the channel attention mechanism provided by the channel attention submodule can be used. First, the initial image features are compressed in the spatial dimension by global average pooling to obtain a first initial feature image. Then, the initial image features are compressed in the spatial dimension by global max pooling to obtain a second initial feature image. Both the first and second initial feature images are 1x1xC vectors, where C represents the number of channels in the first and second initial feature images. Then, these two vectors, the first and second initial feature images, are input into a shared multilayer perceptron (MLP) layer. After passing through an activation function (such as ReLU) and a fully connected layer, the two outputs are used to obtain the target result. Finally, the channel attention weights are obtained through the Sigmoid activation function. In this way, the channel attention weights are multiplied with the original initial image features along the channel dimension to achieve adaptive weighting of different channel features, so as to highlight the feature information of important channels and obtain the channel feature map corresponding to the current initial image features.

[0076] For example, the channel attention submodule compresses the feature map spatially using global average pooling and global max pooling operations, resulting in two 1×1×C vectors (where C is the number of channels in the feature map). These two vectors are then input into a shared multilayer perceptron (MLP), processed by activation functions (such as ReLU) and fully connected layers, and the outputs are summed. Finally, a sigmoid activation function is used to obtain the channel attention weights. These weights are multiplied by the original feature map along the channel dimension, achieving adaptive weighting of features from different channels and highlighting the feature information of important channels. The mathematical expression is as follows:

[0077] in, This represents the initial image features of the input. It is the Sigmoid activation function. and These represent global average pooling and global max pooling operations, respectively.

[0078] Furthermore, the spatial attention submodule performs average pooling on the channel feature maps processed by the channel attention submodule to compress them along the channel dimension, resulting in a third initial feature map. It then performs max pooling on the channel feature maps to compress them along the channel dimension, resulting in a fourth initial feature map. This yields two H×W×1 feature maps (H and W represent the height and width of the feature map, respectively, with 1 indicating one channel). These third and fourth initial feature maps are then concatenated to obtain a concatenated H×W×2 feature map. Next, a convolutional layer (typically with a 7×7 kernel) is used to fuse features and reduce dimensionality in the concatenated feature map. Finally, a sigmoid activation function is applied to obtain spatial attention weights. These spatial attention weights are multiplied spatially by the channel feature maps, achieving adaptive weighting for features at different spatial locations, focusing on the spatial region where the target entity is located, thus obtaining the final target feature map corresponding to each initial image feature. The mathematical expression is as follows:

[0079] in, It is the channel feature map after processing by the channel attention module. This represents a convolution operation. Therefore, by embedding an attention module into the backbone network, the object detection model can pay more attention to the features of the target region when faced with complex backgrounds caused by undulating waves, thus improving its ability to extract target features.

[0080] In this embodiment, sample data and sample labels can be acquired and used to train a preset model to obtain a trained object detection model. The structure of the preset model is the same as that of the object detection model. For example, the preset model includes an input layer, a backbone network, a convolutional block attention module (CBAM), a neck, and a prediction layer.

[0081] In some implementations, an initialized preset model is obtained. Using pre-defined sample images to be identified and sample labeled images, with the sample images to be identified as input and the sample labeled images as output labels, the preset model is trained. This training is performed based on a target loss to obtain a trained target detection model. For example, the training process of the target detection model includes: (A.1) Obtain the sample image to be identified, annotate the sample image to be identified, and obtain the sample labeled image and the sample location information and sample confidence score for the sample labeled box in the sample labeled image. The sample labeled box is used to select the sample target entity in the sample labeled image. (A.2) Input the sample image to be identified into the preset model to obtain the predicted labeled image; (A.3) Determine the target loss based on the difference between the predicted labeled image and the sample labeled image; (A.4) Train the preset model based on the target loss to obtain the trained target detection model.

[0082] The image to be identified can be a real image captured on the sea surface, which may include any entity such as a ship, a buoy, and / or an offshore platform.

[0083] First, obtain the initialized preset model. For example, use the parameters of a YOLOv5 model pre-trained on a large-scale public image dataset (such as the COCO dataset) to initialize the improved preset model. It should be noted that the COCO dataset contains rich image categories and scenes; through transfer learning, the preset model can quickly acquire general image feature extraction capabilities, resulting in the initialized preset model. Therefore, fine-tuning the training on image data under undulating environments can accelerate the model's convergence speed and reduce training time and computational resource consumption.

[0084] Next, sample images to be identified are acquired and labeled. Specifically, sample target entities in the image are labeled using bounding boxes. Each bounding box can represent its position using one or more coordinates. A corresponding confidence score is assigned to each bounding box to indicate its credibility. Multiple bounding boxes can be labeled, with different confidence scores assigned based on the actual presence of target entities. This information is used for subsequent model training, enabling the trained model to distinguish between real and fake entities and improving target entity recognition performance. Thus, labeled images and the sample position information and confidence scores for the bounding boxes within these images are obtained. For example, a professional image annotation tool is used to label the acquired image data, identifying the category (e.g., ships, buoys) and location information (using bounding box coordinates) of each target. The annotation format adopts the VOC dataset annotation format, where each image corresponds to an XML file containing detailed information such as the target category and the coordinates of the top-left and bottom-right corners of the bounding box.

[0085] Next, the sample image to be identified is input into the preset model so that the preset model can perform recognition processing based on the sample image to be identified. The recognition processing process can refer to the description process of the model application inference stage above, which will not be repeated here. In this way, the prediction layer of the preset model will output a predicted label image, as well as the predicted position information and prediction confidence score of the predicted label box in the predicted label image. The predicted label box is used to select the sample target entity that is predicted to exist in the image.

[0086] Furthermore, the target loss is determined based on the differences between the predicted labeled image and the sample labeled image. Specifically, the target loss is determined based on the positional difference between the predicted position information of each predicted bounding box in the predicted labeled image and the sample position information of the sample bounding box in the sample labeled image, and the score difference between the predicted confidence score of each predicted bounding box in the predicted labeled image and the sample confidence score of the sample bounding box in the sample labeled image.

[0087] Finally, the preset model is trained based on the target loss until the preset training termination conditions are met, such as reaching the specified number of training iterations, the model parameters no longer changing, and the model prediction performance being the same as or infinitely close to the expected effect. At this point, iterative training is stopped, and the trained target detection model is obtained.

[0088] For example, the Adam optimizer is used, which combines the advantages of adaptive learning rate gradient algorithms (Adagrad and RMSProp), enabling it to adaptively adjust the learning rate and exhibiting good convergence performance during training. The parameters of the Adam optimizer... Set to 0.9, The learning rate is set to 0.999, and ε is set to 1e-8. Furthermore, a cosine annealing learning rate adjustment strategy can be used. The initial learning rate is set to 0.001, and during training, the learning rate gradually decreases according to a cosine function as the number of training epochs increases. This strategy enables the model to converge quickly in the early stages of training and optimizes more stably in the later stages, avoiding the problem of the model failing to converge due to an excessively high learning rate or the problem of the model taking too long to train due to an excessively low learning rate. Additionally, the training termination condition can be the total number of training epochs. Specifically, the total number of training epochs can be set to 300, with validation performed on the validation set every 10 epochs. During validation, the model's detection accuracy metrics (such as mean accuracy) and loss values ​​(such as classification loss, regression loss, etc.) on the validation set are calculated. Based on the validation results, the model parameters are adjusted promptly. For example, if overfitting occurs, the learning rate can be appropriately reduced or a regularization term can be added to prevent overfitting and improve the model's generalization ability.

[0089] In some implementations, real sample visual images can be acquired and transformed to expand the dataset, resulting in a set of sample visual images. Then, any sample visual image from this set can be selected as the image to be identified. For example, step (A.1), "obtaining the image to be identified," can include: acquiring multiple sample visual images, including images from multiple sea areas at different acquisition times and under different weather conditions; transforming each sample visual image to obtain a transformed sample visual image, where the transformation includes geometric transformation, color transformation, and noise addition; generating a set of sample visual images based on the multiple sample visual images and the transformed sample visual images; and selecting any sample visual image from the set as the image to be identified.

[0090] The sample visual images are images of targets of different sizes and shapes, collected under target sea areas, target times, and target weather conditions. Specifically, multiple sea area locations, different collection times, and different weather conditions are determined, and multiple sample visual images are collected at each sea area location based on the different collection times and weather conditions. For example, to enable the improved model to adapt to target detection tasks under various wave conditions, a large amount of marine image data is collected in different sea areas, at different times, and under different weather conditions. The collected images cover various types of targets, such as vessels of different sizes and shapes (including fishing boats, cargo ships, speedboats, etc.), marine buoys, and offshore operating platforms; at the same time, environmental information such as wave height, wave frequency, and light intensity at the time of image collection is recorded so that the data can be processed more effectively during subsequent training.

[0091] Specifically, multiple sample visual images corresponding to different sea areas, times, and weather conditions are collected. Further, these sample visual images undergo transformation processing. For example, geometric transformations are used to randomly rotate the original sample visual images, with rotation angles ranging from -15° to 15°, to simulate sample visual images collected by the unmanned surface vessel under different attitudes. Alternatively, random scaling is performed, with scaling ratios ranging from 0.8 to 1.2, enabling the model to adapt to targets of different distances and sizes. Furthermore, random flipping (horizontal and vertical flipping) can be performed to increase data diversity. Another example is color transformation, where color parameters such as brightness, contrast, and saturation of the sample visual images are randomly adjusted. For instance, brightness is adjusted by a factor of 0.8 to 1.2, contrast by a factor of 0.8 to 1.2, and saturation by a factor of 0.8 to 1.2, to enhance the model's adaptability to different lighting conditions and color changes, resulting in transformed sample visual images. Furthermore, Gaussian noise is added to the sample visual images by adding noise, with a noise standard deviation set to 0.05, to simulate noise interference that may exist in the real environment, making the model more robust and obtaining the transformed sample visual images. Thus, a sample visual image set is formed based on the real-world sample visual images and the transformed sample visual images; any sample visual image is then selected from this set as the sample image to be identified.

[0092] Using the above method, a target detection model can be used to identify target entities based on the image to be identified, thereby detecting the target entities in the image and marking them with bounding boxes to obtain a marked image containing the bounding boxes. In this way, the bounding boxes are used to select target entities in the marked image, and the coordinate information of each bounding box represents the location information of the corresponding target entity. In addition, a confidence score is output for the bounding boxes, indicating the probability or likelihood of the presence of a target entity at the location of the bounding box. In this way, the target entities in the image to be identified can be detected, in order to identify and describe the environmental information around the unmanned surface vessel, which has reliability.

[0093] 104. Based on confidence scores, select entities to be detected from the target entities.

[0094] In this embodiment, after obtaining the marked image and the target location information and confidence score of the marked boxes in the marked image, the confidence score indicates the credibility of the target detection model in determining that the corresponding image region contains a target entity. A higher confidence score indicates a greater likelihood that the corresponding image region contains a target entity, and vice versa. Based on this, target entities with higher recognition accuracy can be selected based on the confidence score; these are the truly existing entities to be detected. This allows for continuous detection of only the motion trajectories of these truly existing entities, reducing error judgments and focusing on more valuable target entities, thus improving the accuracy of target entity detection.

[0095] In some implementations, based on the confidence score corresponding to each bounding box in the labeled image, target bounding boxes with higher confidence scores are selected, and the target entities corresponding to these target bounding boxes are identified as entities to be detected. Specifically, the confidence score corresponding to each bounding box in the labeled image is compared with a preset confidence threshold to obtain a comparison result; based on the comparison result, target bounding boxes in the labeled image with confidence scores greater than or equal to the preset confidence threshold are selected, and the target entities corresponding to each target bounding box are identified as entities to be detected.

[0096] The preset confidence threshold is used to determine whether a bounding box in the image contains a real target entity. If the confidence score is greater than or equal to the preset confidence threshold, it indicates the presence of a real target entity; conversely, if the confidence score is less than the preset confidence threshold, it indicates the absence of a real target entity. This preset confidence threshold can be set to 0.5, and the specific value can be adjusted according to the detection requirements. A higher preset confidence threshold indicates stricter requirements, while a lower threshold can retain more potential target entities; no specific limitation is made here.

[0097] For example, suppose there are three bounding boxes in the labeled image, corresponding to "ship" (confidence score 0.9), "wave" (misclassified as a target entity, confidence score 0.4), and "buoy" (confidence score 0.6). Assuming the preset confidence threshold is set to 0.5, after comparison, it is found that "ship" 0.9 ≥ 0.5, "wave" 0.4 < 0.5, and "buoy" 0.6 ≥ 0.5. Therefore, the target bounding boxes corresponding to "ship" and "buoy" are selected, and "ship" and "buoy" are finally determined to be the entities to be detected, while "wave" is excluded because of its low confidence score. This can reduce misclassification and focus on the truly valuable target entities.

[0098] By using the above methods, target entities with relatively high recognition accuracy can be screened out based on the confidence score, i.e., real entities to be detected. This allows for continuous detection of the motion trajectories of real entities to be detected, which not only reduces error judgments but also focuses on more valuable target entities, thus improving the detection accuracy of target entities.

[0099] 105. Extract the first feature information corresponding to the entity to be detected, and perform motion trajectory detection on each entity to be detected based on the first feature information and the corresponding target position information.

[0100] In this embodiment, after filtering out the entities to be detected, the corresponding first feature information can be extracted from the image to be identified or the marked image, and the target location information corresponding to the entity to be detected can be determined. Based on the first feature information and target location information corresponding to each entity to be detected, the motion trajectory of each entity to be detected can be continuously detected. In this way, the environmental information (such as fishing boats, cargo ships, speedboats, marine buoys, marine operation platforms, etc.) on the sea surface around the unmanned surface vessel can be identified and the motion trajectory can be continuously detected to meet the target detection requirements on the sea surface.

[0101] The first feature information can be the image feature information of the corresponding entity to be detected. Specifically, after identifying an entity to be detected, a specialized feature extraction algorithm can be used to extract features from the entity in the image. In addition to extracting the bounding box coordinates of the target, the algorithm will also extract the color features, texture features, etc. of the entity to be detected.

[0102] In some implementations, the step "extracting the first feature information corresponding to the entity to be detected" may include: extracting the target bounding box feature vector for the entity to be detected from the marked image, extracting the color histogram feature vector for the entity to be detected in the image color space, extracting the texture feature vector corresponding to the entity to be detected, and finally merging the target bounding box feature vector, the color histogram feature vector and the texture feature vector to obtain the first feature information.

[0103] For example, a specialized feature extraction algorithm is used to extract features from the entity to be detected in the image. In addition to extracting the bounding box coordinates of the target, the color histogram features of the entity are extracted. These color histogram features describe the color characteristics of the entity by statistically analyzing the distribution of different colored pixels within the target region. Specifically, the image color space is converted from red, green, and blue (RGB) to hue, saturation, and lightness (HSV) space. Then, in the HSV space, the hue range is divided into 18 intervals, and saturation and lightness are each divided into 3 intervals. The percentage of pixels in each interval is calculated, resulting in a 162-dimensional color histogram feature vector. Furthermore, the texture features of the target are extracted using the Gray-Level Co-occurrence Matrix (GLCM) method to calculate the gray-level co-occurrence relationships of pixels at different directions and distances within the target region, obtaining a texture feature description of the target to enhance the uniqueness and recognizability of the target features. Thus, based on the features extracted above, the first feature information corresponding to the entity to be detected is obtained.

[0104] In some implementations, initial trajectory information can be constructed based on the first feature information and target location information of each entity to be detected. Then, the corrected target image to be identified in the next frame adjacent to the original visual image is obtained. This target image is then input into the target detection model to obtain the current labeled image, the current target location information and current confidence score for the current labeled box contained in the current labeled image, and the comprehensive similarity between each target entity in the current labeled image and the previous frame labeled image is identified, resulting in a comprehensive similarity matrix. Based on the comprehensive similarity matrix, the real target entity is determined from the current labeled image, and the motion trajectory of the corresponding entity to be detected is detected by combining the second feature information of the real target entity, the current target location information, and the initial trajectory information. For example, step 105, "detecting the motion trajectory of each entity to be detected based on the first feature information and corresponding target location information of each entity to be detected," may include: For each entity to be detected, initial trajectory information is constructed by combining the corresponding first feature information and the corresponding target position information; the next frame of the target's original visual image adjacent to the original visual image is obtained, along with the corresponding target pose data and target 3D distance information. The next frame of the target's original visual image is then corrected based on the target pose data and target 3D distance information to obtain the corrected target image to be identified; the target image to be identified is input into the target detection model to obtain the current labeled image, the current target position information of the current labeled bounding box contained in the current labeled image, and the current confidence score. The current labeled bounding box is used to select the current target entity in the current labeled image; for each entity to be detected, the position between the corresponding target position information and each current target position information is determined. The algorithm calculates the distance between each target entity and the target entity, extracts the second feature information corresponding to each current target entity from the current labeled image, and determines the feature similarity between the first feature information and each second feature information. It then calculates a weighted average similarity between the positional distance of each current target entity and its corresponding feature similarity. A comprehensive similarity matrix is ​​obtained by combining each comprehensive similarity matrix. Based on this matrix, the algorithm uses a Hungarian algorithm to determine the real target entity corresponding to each target entity in the current labeled image. The algorithm also determines the current target state of each target entity based on the current target position information of each real target entity and the target position information of the corresponding target entity. Finally, it performs motion trajectory detection on each target entity by combining its initial trajectory information and its current target state.

[0105] The initial trajectory information can represent the initial motion trajectory of the corresponding entity to be detected, such as motion trajectory points. Specifically, when the target position information (such as the coordinates of the bounding box) and the first feature information (such as the color histogram feature vector, texture features, etc.) of each entity to be detected are obtained, they are input into the target tracking algorithm to initialize and generate the initial trajectory information of the entity to be detected. When initializing the tracking trajectory, a unique identifier ID is created for the target, and the target's position, velocity (initial velocity is set to 0), and other state information in the current frame are recorded. Optimally, a data structure is established to store the historical trajectory information of the target, using a linked list structure, where each node stores the target's state information in one frame, facilitating subsequent querying and analysis of the target's motion trajectory.

[0106] Specifically, for the previously selected "entities to be detected" (such as a small boat or a buoy at sea), an "initial trajectory information" needs to be established for each target. This initial trajectory information not only records the target entity's "position information" (such as its coordinates in the current image, like "left side of the image (x=100, y=80)"), but also its "primary feature information" (such as the target's color and texture—"red hull with white stripes" and "circular buoy with a blue surface"). Combining "position" and "features" forms the "initial trajectory information" for each target, laying the foundation for subsequent continuous motion trajectory detection.

[0107] Furthermore, after initial trajectory information is generated for each entity to be detected, the next frame of the target original visual image adjacent to the previous "original visual image" is obtained, along with the target pose data and target three-dimensional distance information corresponding to the target original visual image. At this point, the target pose data and target three-dimensional distance information can be combined to perform correction processing on the next frame of the target original visual image to obtain the corrected target image to be identified. This correction process can refer to the steps in step 102 above, and will not be elaborated here.

[0108] Then, the target image to be identified is input into the target detection model to obtain the current labeled image, as well as the current target location information and current confidence score for the current labeled bounding box contained in the current labeled image. The current labeled bounding box is used to select the current target entity in the current labeled image. The recognition process of this target detection model can be referred to the description in step 103 above, and will not be repeated here.

[0109] Next, for each entity to be detected, the positional distance between its corresponding target location information and the current target location information of the corresponding current target entity is determined. For example, Euclidean distance is used to calculate the distance between the center coordinates of the two bounding boxes to obtain the positional distance. Simultaneously, the second feature information corresponding to each current target entity is extracted from the current labeled image. By comparing the feature differences between the corresponding first and second feature information, the feature similarity is determined. For example, cosine similarity is used to calculate the color feature similarity between the color histogram feature vectors of the first and second feature information, and cosine similarity is used to calculate the texture similarity between the texture feature vectors of the first and second feature information. The feature similarity is then obtained by combining the color feature similarity and texture similarity, such as by multiplication or addition, without limitation. Finally, the positional distance and feature similarity are weighted and summed. For example, the weight of positional distance is set to 0.4, and the weight of feature similarity is set to 0.6. The weighted sum is then calculated to obtain the comprehensive similarity between the entity to be detected and the corresponding current target entity.

[0110] For example, assuming image frame A is the "marked image" mentioned earlier, and image frame B is the next frame of the marked image adjacent to image A, for each "entity to be detected" in image A, the positional distance between its "target position information" in image A and the "current target position information" of the current target entity in image frame B is calculated (e.g., using coordinates to calculate the straight-line distance; the closer they are, the higher the positional similarity). Simultaneously, from the "current target entities" in image B, their "secondary feature information" (e.g., "the target in a certain box is red and has white stripes") is extracted and compared with the "first feature information" of the same entity in the previous frame A (whether the colors are similar, whether the textures are similar), to obtain the feature similarity. Further, combining the "positional distance" and "feature similarity," a weighted average is calculated (e.g., position accounts for 40%, feature accounts for 60%) to obtain a "comprehensive similarity." Finally, the comprehensive similarities of all "entities to be detected in the previous frame" and "current target entities in this frame" are compiled into a "comprehensive similarity matrix."

[0111] Furthermore, this comprehensive similarity matrix is ​​used as input to the Hungarian algorithm. The Hungarian algorithm finds the optimal matching relationship, associating the target in the current frame with the tracked target in the previous frame to determine which detected targets are continuations of previously tracked targets. This determines the true target entity corresponding to each entity to be detected in the current labeled image. Then, using the Kalman filter algorithm, the current target state of each entity to be detected is determined based on the current target position information of each true target entity and the target position information of the corresponding entity to be detected. For example, using the idea of ​​predicting the next state with Kalman filtering, based on the Kalman filter using the "state of the previous frame labeled image, i.e., target position information (x1, y1)" and the "current target position information (x2, y2)" of the current labeled image, combined with the time interval Δt (e.g., 1 / 30 second, corresponding to the visual sensor frame rate), the target velocity (Vx=(x2-x1) / Δt, Vy=(y2-y1) / Δt) can be derived. Thus, the current target state of the corresponding entity to be detected in the next frame (i.e., the current labeled image) is obtained, which includes both position and velocity.

[0112] Finally, by combining the initial trajectory information and the current target state of each entity to be detected, motion trajectory detection is performed on each entity. For example, the initial trajectory information (position and features in the previous frame) and the current target state (position and velocity in this frame) of each entity to be detected are combined to update its motion trajectory: for instance, if the trajectory in the previous frame was "(x1,y1)", it is updated to "(x1,y1)→(x2,y2)" in this frame, effectively extending the initial trajectory information of the entity to be detected. This completes the tracking of the motion trajectory of the entity to be detected "from the current frame to the next frame". Repeating this process allows for the continuous recording of the motion path of the entities to be detected, thus enabling motion trajectory detection for the corresponding entities.

[0113] In some implementations, the current target state of each entity to be detected is determined based on the current target location information corresponding to each real target entity and the target location information of the corresponding entity to be detected. The specific process is as follows: Assuming the entity to be detected moves on a two-dimensional plane, its first state vector in the previous frame (labeled image) can be represented as: ,in and These are the position coordinates of the entity to be detected. and This represents the velocity component of the entity to be detected. According to kinematic principles, the state transition matrix... It can be represented as:

[0114] in, It is a time interval, determined based on the image acquisition frame rate. This is achieved through the state transition matrix. State estimate at the previous time step It can predict the state at the current moment (i.e., for the current labeled image). Simultaneously, a measurement matrix is ​​constructed. The measurement matrix is ​​as follows:

[0115] Mapping the target's state vector to the measurement space, for example, for target position information measured by a vision sensor, the measurement matrix can be represented as:

[0116] When the actual detection position (measured value) of the target in the current frame is obtained After that, Kalman gain is used. Update the predicted state to obtain a more accurate state estimate. Kalman gain , It is the covariance matrix of the predicted state. This is the covariance matrix of the measurement noise. After each state update, the covariance matrix of the state estimate also needs to be updated. To reflect the uncertainty of the estimate, the covariance matrix The update formula is: ,in It is an identity matrix.

[0117] In some implementations, before the step "determine the current target state corresponding to each entity to be detected based on the current target position information corresponding to each real target entity and the target position information corresponding to the entity to be detected", the following steps may be included: when it is detected that the current marked image does not contain any current target entity corresponding to any target entity to be detected, determine the corresponding initial target state from the initial trajectory information corresponding to the target entity to be detected, the initial target state including initial target position information and initial velocity; predict the current target position range corresponding to the target entity to be detected based on the initial target position information and initial velocity using a Kalman filter algorithm. Then, the step "determine the current target state corresponding to each entity to be detected based on the current target position information corresponding to each real target entity and the target position information corresponding to the entity to be detected" includes: for each target entity to be detected, determine the current target state corresponding to each entity to be detected based on the corresponding current target position range and the corresponding target position information using the motion model of the target entity to be detected; when it is detected that the current marked image contains an entity to be detected corresponding to the current target entity, determine the current target state corresponding to each entity to be detected based on the current target position information corresponding to each real target entity and the target position information corresponding to the entity to be detected using the motion model of the entity to be detected.

[0118] For example, when a target entity is occluded by waves in a labeled image frame and is not recognized by the target detection model, the Kalman filter algorithm is used to predict the current target location range of the target entity in the current frame (current labeled image) based on the historical target state (when the image was labeled). Based on the motion model of the target entity and the previous state estimates, the possible current target location range in the current frame is predicted. For example, assuming the target entity's velocity before being occluded is... , direction is During the time interval of occlusion Inside, the predicted target is The displacement in the direction is ,exist The displacement in the direction is This allows us to obtain the current target location range where the entity to be detected may appear. This predicted current target location range serves as the region where the target may appear in the next frame, providing a basis for matching and tracking when the entity to be detected reappears.

[0119] In some implementations, the step "combining the initial trajectory information and the corresponding current target state of each entity to be detected, performing motion trajectory detection on each entity to be detected" may include: updating the initial trajectory information of each entity to be detected based on the current target state of each entity using the motion model of the entity to be detected, obtaining the first trajectory information of each entity to be detected, and then performing motion trajectory detection on each entity to be detected based on the first trajectory information of each entity to be detected. Specifically, updating the initial trajectory information may involve overlaying the current target state at the end of the initial trajectory information, or it may involve recording it in a target table, with the motion trajectory represented by the target state at each time point in the target table; this is not limited here.

[0120] Furthermore, in some embodiments, when the target entity to be detected is re-acquired by the visual sensor, the predicted target position information is selected within the current target position range corresponding to the target entity, based on the reproduced target position information of the target entity re-appearing. The predicted target state is determined based on the predicted target position information and historical target position information of the target entity, and the reproduced target state is determined based on the reproduced target position information and the predicted target position information. The initial trajectory information of the target entity is updated by combining the predicted target state and the reproduced target state to obtain the second trajectory information of the target entity. The motion trajectory of the target entity is detected based on the second trajectory information.

[0121] For example, when a target entity reappears in the field of view of the visual sensor, target entity matching and association are performed within the predicted location area. By comparing the feature information of the target entity (color histogram features, texture features, etc.), the reappearing target entity is associated with the previously tracked trajectory. Specifically, the feature similarity between the newly appearing target entity and the last known target in the previously tracked trajectory is calculated. If the similarity exceeds a preset feature similarity threshold (e.g., 0.8), the newly appearing target entity is considered a continuation of the previously tracked target entity, and the tracking and detection of the target entity's motion trajectory is resumed. Simultaneously, based on the reproduced target location information of the newly appearing target entity, the current target state corresponding to the previous "current target location range" is corrected, and the parameters of the Kalman filter are readjusted to better and continuously detect the subsequent motion trajectory of the target entity.

[0122] In some implementations, based on the established target motion model and combined with current wave state information (such as wave height and frequency), the trajectory of the entity to be detected is more accurately predicted and corrected. Through analysis of a large amount of maritime target motion data, and incorporating principles of fluid dynamics, a motion model of the entity under wave action is established. For example, when the wave height is large, the vertical motion amplitude of the entity increases, while its horizontal motion is also affected by wave thrust and friction. In this case, appropriately adjusting the parameters in the target motion model, such as increasing the vertical motion noise covariance and adjusting the horizontal velocity variation coefficient, reflects the greater influence of waves on the motion of the entity. Thus, optimizing the target motion model based on wave state information enables more accurate prediction of the target's trajectory in a wave-filled environment, improving the accuracy of target tracking and prediction.

[0123] Using the above methods, the motion trajectory of each entity to be detected can be continuously detected based on the first feature information and target location information corresponding to each entity. In this way, the environmental information (such as fishing boats, cargo ships, speedboats, marine buoys, marine operation platforms, etc.) around the unmanned surface vessel can be identified and the motion trajectory can be continuously detected to meet the target detection requirements on the sea surface.

[0124] As can be seen from the above embodiments, this application obtains multi-dimensional data such as the original visual image, posture data, and three-dimensional distance information, and corrects the original visual image by combining the posture data and three-dimensional distance information to obtain a corrected image to be identified. This avoids the impact of wave occlusion, unmanned surface vessel swaying, etc., on the quality of the visual image, thus improving the quality of the image to be identified. Then, a target detection model is used to detect targets based on the image to be identified, and target entities in the image to be identified are marked by bounding boxes to obtain the target location information and confidence score of the target entities. Further, the entities to be detected are selected according to the size of the confidence score. Finally, the first feature information corresponding to the entity to be detected is extracted, and the motion trajectory of each entity to be detected is tracked and detected based on the first feature information and the corresponding target location information. Therefore, compared with related technologies where poor image data quality leads to reduced target detection accuracy, this application improves the quality of image data by combining the attitude data and three-dimensional distance information of the unmanned surface vessel to correct the original visual image. The corrected image is then input into the target detection model to identify potential target entities. Based on the confidence score, the target entities with higher recognition accuracy are selected. By combining the feature information of the target entities with the initial target position information, continuous motion trajectory detection is performed, thereby improving the target detection accuracy.

[0125] To facilitate better implementation of the target detection method under unmanned surface vessel (USV) vision provided in this application, this application also provides a target detection device based on the aforementioned USV vision. The meanings of the terms used are the same as in the aforementioned USV vision-based target detection method, and specific implementation details can be found in the descriptions within the method embodiments.

[0126] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of the unmanned surface vessel (USV) vision-based target detection device provided in an embodiment of this application. The USV vision-based target detection device is integrated into the computer equipment of this application, such as a server. The USV vision-based target detection device may include an acquisition unit 401, a correction unit 402, an input unit 403, a filtering unit 404, and a sending unit 405.

[0127] The acquisition unit 401 is used to acquire the target sensing dataset, which includes at least the original visual images acquired by the visual sensor, the attitude data acquired by the inertial measurement unit, and the three-dimensional distance information acquired by the lidar. The correction unit 402 is used to perform correction processing on the original visual image based on the pose data and three-dimensional distance information to obtain the corrected image to be recognized. The input unit 403 is used to input the image to be recognized into the trained target detection model to obtain the labeled image and the target location information and confidence score of the bounding boxes contained in the labeled image. The bounding boxes are used to select target entities in the labeled image. The filtering unit 404 is used to filter out entities to be detected from the target entities based on confidence scores; The detection unit 405 is used to extract the first feature information corresponding to the entity to be detected, and to perform motion trajectory detection on each entity to be detected based on the first feature information and the corresponding target position information.

[0128] In some embodiments, the correction unit 402 is further configured to: Calculate the attitude transformation matrix based on the attitude data; The original visual image is corrected based on the pose transformation matrix to obtain the corrected initial visual image; The initial visual image is spatially mapped based on the three-dimensional distance information to obtain the corrected image to be recognized.

[0129] In some embodiments, the acquisition unit 401 is further configured to: It acquires the time signal sent by the Global Positioning System and generates a second pulse signal based on the time signal; Using the second pulse signal as the time reference, the system acquires raw visual images through a visual sensor, attitude data of the local unmanned surface vessel is acquired through an inertial measurement unit, and three-dimensional distance information is acquired through a lidar. A target sensing dataset is generated based on the original visual images, pose data, and 3D distance information.

[0130] In some implementations, the object detection model includes at least a backbone network, an attention module, and a fusion module. The input unit 403 is also used for: The image to be identified is input into the trained target detection model, and the backbone network extracts features from the image to be identified to obtain multiple initial image features. Each initial image feature is enhanced with channel and spatial dimensions using an attention module to obtain a feature map for each target. The fusion module performs a fusion process on multiple target feature maps to obtain a fused feature map; Prediction is performed based on the fused feature map to obtain the labeled image, as well as the target location information and confidence score for the labeled bounding boxes contained in the labeled image.

[0131] In some embodiments, the input unit 403 is further configured to: For each initial image feature, a first initial feature map is obtained by global average pooling, and a second initial feature map is obtained by global max pooling. Channel attention weights are determined based on the first and second initial feature maps, and the initial image features are weighted according to the channel dimension using the channel attention weights to obtain channel feature maps. The channel feature maps are subjected to average pooling along the channel dimension to obtain the third initial feature map, and the channel feature maps are subjected to max pooling along the channel dimension to obtain the fourth initial feature map. The spatial attention weights are determined based on the third initial feature map and the fourth initial feature map, and the channel feature maps are subjected to spatial dimension weighting based on the spatial attention weights to obtain each target feature map.

[0132] In some implementations, the target detection device under the vision of the unmanned surface vessel also includes a training unit, which is further used for: The sample image to be identified is obtained, and the sample image to be identified is annotated to obtain the sample labeled image, as well as the sample location information and sample confidence score of the sample labeled box in the sample labeled image. The sample labeled box is used to select the sample target entity in the sample labeled image. The sample image to be identified is input into the preset model to obtain the predicted labeled image; The target loss is determined based on the difference between the predicted labeled image and the sample labeled image; The preset model is trained based on the target loss to obtain the trained target detection model.

[0133] In some implementations, the training unit is also used for: Multiple sample visual images were acquired, including images from multiple sea areas at different acquisition times and under different weather conditions; Each sample visual image is transformed to obtain the transformed sample visual image. The transformation process includes geometric transformation, color transformation, and noise addition. A set of sample visual images is generated based on multiple sample visual images and transformed sample visual images. Select any one sample visual image from the sample visual image set as the sample image to be identified.

[0134] In some embodiments, the detection unit 405 is further configured to: For each entity to be detected, initial trajectory information is constructed by combining the corresponding first feature information and the corresponding target location information; The target original visual image of the next frame adjacent to the original visual image is obtained, as well as the corresponding target pose data and target 3D distance information. The target original visual image of the next frame is then corrected based on the target pose data and target 3D distance information to obtain the corrected target image to be identified. The target image to be identified is input into the target detection model to obtain the current labeled image, the current target location information and the current confidence score for the current labeled bounding box contained in the current labeled image. The current labeled bounding box is used to select the current target entity in the current labeled image. For each entity to be detected, the positional distance between the corresponding target position information and each current target position information is determined, and the second feature information corresponding to each current target entity is extracted from the current labeled image. The feature similarity between the first feature information and each second feature information is determined. The positional distance of each current target entity and the corresponding feature similarity are weighted and calculated to obtain the comprehensive similarity. By combining each comprehensive similarity score, a comprehensive similarity matrix is ​​obtained. Based on the comprehensive similarity matrix, the Hungarian algorithm is used to determine the real target entity corresponding to each entity to be detected in the current labeled image. The current target state corresponding to each entity to be detected is determined according to the current target position information corresponding to each real target entity and the target position information of the corresponding entity to be detected. By combining the initial trajectory information and the current target state of each entity to be detected, motion trajectory detection is performed on each entity to be detected.

[0135] As described above, this application acquires multi-dimensional data, including the original visual image, posture data, and three-dimensional distance information. It then corrects the original visual image using the posture data and three-dimensional distance information to obtain a corrected image to be identified. This avoids issues such as wave occlusion and unmanned surface vessel (USV) swaying that could affect the quality of the visual image, thus improving the quality of the image to be identified. Next, a target detection model is used to detect targets based on this image, marking target entities in the image with bounding boxes to obtain target location information and confidence scores. Further, entities to be detected are selected according to their confidence scores. Finally, the first feature information corresponding to each entity to be detected is extracted, and the motion trajectory of each entity is tracked and detected based on its first feature information and corresponding target location information. Therefore, compared with related technologies where poor image data quality leads to reduced target detection accuracy, this application improves the quality of image data by combining the attitude data and three-dimensional distance information of the unmanned surface vessel to correct the original visual image. The corrected image is then input into the target detection model to identify potential target entities. Based on the confidence score, the target entities with higher recognition accuracy are selected. By combining the feature information of the target entities with the initial target position information, continuous motion trajectory detection is performed, thereby improving the target detection accuracy.

[0136] The specific implementation of each of the above units can be found in the previous embodiments, and will not be repeated here.

[0137] Figure 4 To implement the partial structural block diagram of the terminal 110 in this application embodiment, the terminal 110 includes: a radio frequency (RF) circuit 510, a memory 515, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, and a power supply 590, etc. Those skilled in the art will understand that the terminal 110 structure shown in the figures does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0138] The RF circuit 510 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 580; in addition, it transmits uplink data to the base station.

[0139] The memory 515 can be used to store software programs and modules. The processor 580 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the memory 515.

[0140] The input unit 530 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532.

[0141] The display unit 540 can be used to display input or provided information, as well as various menus of the terminal. The display unit 540 may include a display panel 541.

[0142] Audio circuit 560, speaker 561, and microphone 562 provide an audio interface.

[0143] In this embodiment, the processor 580 included in the terminal 110 can execute the target detection method under unmanned surface vessel vision in the previous embodiment.

[0144] The terminal 110 in this application embodiment includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. This invention embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0145] Figure 5 This is a partial structural block diagram of a server 120 implementing an embodiment of this application. The server 120 can vary significantly due to different configurations or performance characteristics, and may include one or more central processing units (CPUs) 622 (e.g., one or more processors) and memory 632, and one or more storage media 620 (e.g., one or more mass storage devices) for storing application programs 642 or data 644. The memory 632 and storage media 620 may be temporary or persistent storage. The program stored in the storage media 620 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 120. Furthermore, the CPU 622 may be configured to communicate with the storage media 620 and execute the series of instruction operations in the storage media 620 on the server 120.

[0146] Server 120 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input / output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0147] The central processing unit 622 in server 120 can be used to execute the target detection method under the vision of the unmanned surface vessel according to the embodiments of this application.

[0148] This application also provides a computer-readable storage medium for storing program code for executing the unmanned surface vessel vision-based target detection methods of the foregoing embodiments.

[0149] This application also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, causing the computer device to perform the aforementioned target detection method under unmanned surface vessel vision.

[0150] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, apparatus, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may include other steps or units that are not expressly listed or that are inherent to such process, method, product or device.

[0151] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0152] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0158] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0159] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A target detection method under vision of an unmanned surface vehicle, characterized in that, The method comprises the following steps: acquiring a target sensing data set, the target sensing data set at least comprising an original visual image collected by a visual sensor, attitude data collected by an inertial measurement unit, and three-dimensional distance information collected by a laser radar; correcting the original visual image according to the attitude data and the three-dimensional distance information to obtain a corrected to-be-recognized image; inputting the to-be-recognized image into a trained target detection model to obtain a marked image and target position information and a confidence score of a marked box contained in the marked image, the marked box being used for framing a target entity in the marked image; based on the confidence score, screening a to-be-detected entity from the target entity; extracting first feature information corresponding to the to-be-detected entity, and detecting a motion trajectory of each to-be-detected entity based on the first feature information of each to-be-detected entity and the corresponding target position information.

2. The method of claim 1, wherein, The method of correcting the original visual image according to the attitude data and the three-dimensional distance information to obtain a corrected to-be-recognized image comprises the following steps: calculating a pose transformation matrix according to the attitude data; correcting the original visual image according to the pose transformation matrix to obtain a corrected initial visual image; performing spatial mapping processing on the initial visual image according to the three-dimensional distance information to obtain a corrected to-be-recognized image.

3. The method of claim 1, wherein, The method of acquiring a target sensing data set comprises the following steps: acquiring a time signal sent by a global positioning system, and generating a second pulse signal according to the time signal; collecting an original visual image through the visual sensor and collecting attitude data of a local unmanned ship through the inertial measurement unit based on the second pulse signal as a time reference, and collecting three-dimensional distance information through the laser radar; generating a target sensing data set based on the original visual image, the attitude data, and the three-dimensional distance information.

4. The method of claim 1, wherein, The target detection model at least comprises a backbone network, an attention module, and a fusion module, and the method of inputting the to-be-recognized image into the trained target detection model to obtain a marked image and a confidence score of a marked box contained in the marked image comprises the following steps: inputting the to-be-recognized image into the trained target detection model, extracting features of the to-be-recognized image through the backbone network to obtain a plurality of initial image features; enhancing the representation of each initial image feature in the channel dimension and the spatial dimension through the attention module to obtain each target feature map; performing fusion processing on a plurality of the target feature maps through the fusion module to obtain a fusion feature map; based on the fusion feature map, performing prediction to obtain a marked image and target position information and a confidence score of a marked box contained in the marked image.

5. The method of claim 4, wherein, The method of enhancing the representation of each initial image feature in the channel dimension and the spatial dimension through the attention module to obtain each target feature map comprises the following steps: For each initial image feature, a first initial feature map is obtained through global average pooling processing, and a second initial feature map is obtained through global maximum pooling processing, a channel attention weight is determined according to the first initial feature map and the second initial feature map, and the initial image feature is weighted in the channel dimension through the channel attention weight to obtain a channel feature map; The channel feature map is subjected to average pooling processing in the channel dimension to obtain a third initial feature map, and is subjected to maximum pooling processing in the channel dimension to obtain a fourth initial feature map, a spatial attention weight is determined according to the third initial feature map and the fourth initial feature map, and the channel feature map is weighted in the spatial dimension according to the spatial attention weight to obtain each target feature map.

6. The method according to claim 1 or 4, characterized in that, The training process of the target detection model comprises: obtaining a sample to-be-recognized image, labeling the sample to-be-recognized image to obtain a sample labeled image and sample position information and a sample confidence score of a sample labeled box in the sample labeled image, the sample labeled box being used to frame a sample target entity in the sample labeled image; inputting the sample to-be-recognized image into a preset model to obtain a predicted labeled image; determining a target loss according to the difference between the predicted labeled image and the sample labeled image; training the preset model according to the target loss to obtain a trained target detection model.

7. The method of claim 6, wherein, The obtaining of the sample to-be-recognized image comprises: collecting a plurality of sample visual images, the plurality of sample visual images comprising images of a plurality of sea locations under different collection times and different weather conditions; performing transformation processing on each sample visual image to obtain a sample visual image after transformation processing, the transformation processing comprising geometric transformation processing, color transformation processing and noise adding processing; generating a sample visual image set according to the plurality of sample visual images and the sample visual image after transformation processing; obtaining any sample visual image from the sample visual image set as a sample to-be-recognized image.

8. The method of claim 1, wherein, The motion trajectory detection of each to-be-detected entity based on the first feature information and the corresponding target position information of each to-be-detected entity comprises: for each to-be-detected entity, constructing initial trajectory information in combination with the corresponding first feature information and the corresponding target position information; obtaining a next frame target original visual image adjacent to the original visual image, and obtaining corresponding target attitude data and target three-dimensional distance information, and performing correction processing on the next frame target original visual image according to the target attitude data and the target three-dimensional distance information to obtain a corrected target to-be-recognized image; inputting the target to-be-recognized image into the target detection model to obtain a current labeled image and current target position information and a current confidence score of a current labeled box contained in the current labeled image, the current labeled box being used to frame a current target entity in the current labeled image; For each entity to be detected, the position distance between the corresponding target position information and each current target position information is determined, and the second feature information corresponding to each current target entity is extracted from the current marked image, and the feature similarity between the first feature information and each second feature information is determined, and the position distance and the corresponding feature similarity of each current target entity are weighted to obtain a comprehensive similarity; A comprehensive similarity matrix is obtained in combination with each comprehensive similarity, and based on the comprehensive similarity matrix, the real target entity corresponding to each entity to be detected in the current marked image is determined by using the Hungarian algorithm, and the current target state corresponding to each entity to be detected is determined according to the current target position information corresponding to each real target entity and the target position information corresponding to the entity to be detected; The motion trajectory of each entity to be detected is detected in combination with the initial trajectory information corresponding to each entity to be detected and the current target state corresponding to each entity to be detected.

9. An unmanned surface vehicle target detection device under vision, characterized in that, It comprises: An acquisition unit is configured to acquire a target sensing data set, which at least includes an original visual image collected by a visual sensor, attitude data collected by an inertial measurement unit, and three-dimensional distance information collected by a laser radar; A correction unit is configured to correct the original visual image based on the attitude data and the three-dimensional distance information to obtain a corrected image to be recognized; An input unit is configured to input the image to be recognized into a trained target detection model to obtain a marked image, target position information of a marked box included in the marked image, and a confidence score, the marked box being used to frame a target entity in the marked image; A screening unit is configured to screen out an entity to be detected from the target entity based on the confidence score; A detection unit is configured to extract first feature information corresponding to the entity to be detected, and detect the motion trajectory of each entity to be detected based on the first feature information of each entity to be detected and the target position information corresponding to each entity to be detected.

10. A computer device, comprising: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the target detection method under the vision of the unmanned surface vehicle according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, which are suitable for being loaded by the processor to execute the target detection method under the vision of the unmanned surface vehicle according to any one of claims 1 to 8.

Images