A ship offshore target perception method and system based on machine vision

Abstract

The application relates to the technical field of marine orientation sensing of ships, and discloses a ship marine target sensing method and system based on machine vision, which comprises the following steps: using an AD-YOLOv9 algorithm to process incoming image information, obtaining position information and category information of a target image; establishing a coordinate system according to the position information of the target image, marking a target on the target image by using a target prediction box, determining the center point coordinates of the target prediction box based on a deep learning network to determine the center point coordinates of the target, and associating the bow of the ship to determine the absolute orientation and the relative orientation of the target; determining the position of the target relative to the reference point of the ship to determine the direction angle of the target; determining a straight line where the center point of the target and the reference point of the ship are located according to the direction angle of the target, and recording the straight line as a target direction line; calculating and formatting the angle label of the target according to the target direction line, and selecting a position on the target direction line to draw angle label information; and displaying the processed target image.

Description

Technical Field

[0001] This invention relates to the field of ship maritime positioning perception technology, specifically to a method and system for ship maritime target perception based on machine vision. Background Technology

[0002] In traditional maritime practice, crew members primarily rely on conventional navigation tools such as compasses and radar to determine the ship's heading, and use visual observation or radar image analysis to identify the relative positions of surrounding vessels, buoys, and other targets. While this method has proven effective in practice, in port waters, due to the high density of vessels, close spacing, and high maneuverability, radar faces significant challenges in stably tracking targets, thus affecting the accuracy of target location data.

[0003] According to the radar equipment performance standard MSC.192(79) issued by the International Maritime Organization (IMO), in calm seas with a range not exceeding 1.5 nautical miles, radar should have a range resolution of at least 40 meters between two targets and a azimuth resolution of at least 2.5 degrees within 50% to 100% of its range. However, when the distance between targets is less than 40 meters or the azimuth difference is less than 2.5 degrees, radar may be unable to effectively distinguish these targets or maintain stable tracking. This situation is particularly prominent in densely populated port waters. Furthermore, radar performance is also limited by adverse weather conditions such as weather and visibility, placing higher demands on the experience and skills of the crew.

[0004] Given the limitations of radar in detecting close-range targets, target orientation perception technology based on computer vision is gradually demonstrating its unique advantages.

[0005] In the task of sensing and identifying maritime targets, computer vision-based target orientation perception technology uses cameras installed on ships to monitor environmental image information around the ship in real time. By using advanced image processing algorithms and machine learning technology, the system can automatically identify and track other key targets such as ships and buoys in the vicinity.

[0006] Methods for determining heading, bearing, and angle of attack based on computer vision, such as... Figure 1 As shown, A and B represent two different target objects. T This represents true north, which is the geographical North Pole. The target's bearing is measured using true north as a reference and is called the true azimuth angle, such as TB. A TB B, which is independent of the actual sailing direction of the ship, i.e. true course TC. On the other hand, the target bearing Q is measured based on the course line (CL). A positive or negative value of the bearing Q indicates that the target object (e.g. buoy, island, other ship, etc.) is located on the starboard or port side of the ship, respectively.

[0007] The mathematical relationship between the heading, the azimuth and the bearing can be accurately described by the following formula:

[0008] ;

[0009] where TB is the true bearing of the target, TC is the true course, and Q is the relative bearing of the target with respect to the bow.

[0010] After obtaining the real-time bow data of the ship (i.e. true course TC), the system can accurately associate the target bearing captured in the image with the bow, thereby automatically calculating the true bearing (TB) and the relative bearing (Q) of the targets and presenting them to the ship operator in an intuitive and clear manner. This method improves the intelligence level of target bearing judgment and has high practical value and application potential.

[0011] However, the ship offshore target perception and recognition technology using the YOLOv9 benchmark model still has the defects of more missed detection of ships, low recognition accuracy, large amount of calculation, and poor real-time performance, and its detection performance in complex marine environment needs to be improved.

[0012] Based on this, the applicant proposes the present application. SUMMARY

[0013] In order to solve the problems of more missed detection of ships and low recognition accuracy in the prior art, or the problems of large amount of calculation and poor real-time performance in the prior art, the present application provides a ship offshore target perception method based on machine vision, which adopts an AD-YOLOv9 offshore target recognition algorithm (referred to as AD-YOLOv9 algorithm), can complete the positioning and classification of the target in one forward propagation, has a lighter model and higher target classification recognition accuracy.

[0014] The technical scheme of the present application is as follows:

[0015] A ship offshore target perception method based on machine vision, comprising the following steps:

[0016] Step S1: using the AD-YOLOv9 algorithm to process the input image information to obtain the position information and category information of the target image;

[0017] Step S2: Establish a coordinate system according to the position information of the target image, and mark the target on the target image with a target prediction box. Determine the center point coordinates of the target prediction box based on a deep learning network, thereby determining the center point coordinates of the target, and associate the center point coordinates of the target with the heading of the ship to determine the absolute position and relative position of the target;

[0018] Step S3: Determine the position of the target relative to the reference point of the ship according to the absolute position and relative position of the target, to determine the direction angle of the target;

[0019] Step S4: Determine the straight line on which the center point of the target and the reference point of the ship are located according to the direction angle of the target, denoted as the target direction line;

[0020] Step S5: Calculate and format the angle label of the target according to the target direction line, and draw the angle label information on the target direction line;

[0021] Step S6: Display the processed target image.

[0022] Further, step S1 includes using AD-YOLOv9 algorithm to recognize the ship, buoy color, bridge and island obstacle.

[0023] Optionally, it further includes step S0: Real-time monitoring of the environmental information around the ship through the camera on the ship, and the video or image information obtained is introduced into the AD-YOLOv9 algorithm model.

[0024] Further, in step S2, the center point coordinates of the target prediction box are determined according to the following formula:

[0025] ;

[0026] wherein, represents the horizontal coordinate value of the upper left corner of the target prediction box, represents the horizontal coordinate value of the lower right corner of the target prediction box;

[0027] ;

[0028] wherein, represents the vertical coordinate value of the upper left corner of the target prediction box, represents the vertical coordinate value of the lower right corner of the target prediction box.

[0029] Further, based on the heading of the ship, the position of other targets at sea is determined according to the center point coordinates of the target prediction box:

[0030] ;

[0031] ;

[0032] wherein, represents the horizontal coordinate value of the center point of the target prediction frame, represents the horizontal coordinate value of the center point of the target prediction frame, h is the length of the picture, ω is the width of the picture, Q represents the direction of other targets.

[0033] Further, in step S6, the target identified in the target image and its corresponding direction information are dynamically displayed in graphical form on the terminal display screen, so that the operator can directly observe the position of the target and its corresponding direction information.

[0034] The application also provides a ship offshore target perception system based on machine vision, comprising:

[0035] An information extraction module is configured to process incoming image information using an AD-YOLOv9 algorithm to obtain position information and category information of the target image.

[0036] An image processing module is configured to establish a coordinate system according to the position information of the target image, mark the target on the target image using a target prediction frame, determine the center point coordinates of the target prediction frame based on a deep learning network, thereby determining the center point coordinates of the target, and associate the center point coordinates of the target with the ship heading to determine the absolute direction and relative direction of the target.

[0037] A target direction angle determination module is configured to determine the position of the target relative to the reference point of the ship based on the absolute direction and relative direction of the target, to determine the direction angle of the target.

[0038] A direction line marking module is configured to determine the straight line where the center point of the target and the reference point of the ship are located, denoted as the target direction line, according to the direction angle of the target.

[0039] An angle drawing module is configured to calculate and format the angle label of the target according to the target direction line, and select a position on the target direction line to draw the angle label information.

[0040] An image display module is configured to display the processed target image.

[0041] The above technical solution has the following beneficial effects:

[0042] The method and / or system realize automatic identification of other targets on the sea and visual judgment of the approximate relative positions thereof, and through fusion of the real-time image recognition capability of the AD-YOLOv9 deep learning algorithm and the heading data, the system can dynamically display the identified targets and the corresponding position information in the form of graphics on the terminal display screen. The method and / or system enable the operator to more intuitively observe the positions of the surrounding targets and their relative positions, thereby supporting the operator to make more rapid and accurate decisions. In the process of visual presentation, the system also uses diversified colors and icons to distinguish different target types and their relative positions, so as to further enhance the visual effect of the information. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative effort.

[0044] Figure 1 A schematic diagram of a method for judging heading, position and bearing based on computer vision;

[0045] Figure 2 A flowchart of a method for ship target perception at sea based on machine vision according to the present application;

[0046] Figure 3 A structural diagram of a system for ship target perception at sea based on machine vision according to the present application;

[0047] Figure 4 An AD-YOLOv9 network framework used in the present application;

[0048] Figure 5 A schematic diagram of sea target position prediction used in the present application;

[0049] Figure 6 A schematic diagram of ship target recognition effect using a YOLOv9 benchmark model in an actual navigation scenario;

[0050] Figure 7 A schematic diagram of ship target perception effect using the AD-YOLOv9 model of the present application in an actual navigation scenario. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments.

[0052] Examples of the embodiments are illustrated in the accompanying drawings, throughout which like or similar characters of reference denote like or similar elements or components having the same or similar functions. The embodiments described below, as well as the examples illustrated in the drawings, are exemplary and intended to explain the present application, and are not to be understood as limiting the present application.

[0053] As shown in the figure, it is a flowchart of a ship offshore target perception method based on machine vision provided by the present application, which includes the following steps: Figure 2

[0054] Step S1: using AD-YOLOv9 algorithm to process the incoming image information to obtain the position information and category information of the target image;

[0055] Step S2: establishing a coordinate system according to the position information of the target image, and marking the target on the target image with a target prediction box, determining the center point coordinates of the target prediction box based on a deep learning network, thereby determining the center point coordinates of the target, and associating the center point coordinates of the target with the heading of the ship to determine the absolute position and relative position of the target;

[0056] Step S3: determining the position of the target relative to the reference point of the ship according to the absolute position and relative position of the target to determine the direction angle of the target;

[0057] Step S4: determining the straight line on which the center point of the target and the reference point of the ship are located according to the direction angle of the target, denoted as the target direction line;

[0058] Step S5: calculating and formatting the angle label of the target according to the target direction line, and selecting a position on the target direction line to draw the angle label information;

[0059] Step S6: displaying the processed target image.

[0060] Specifically, in step S1, AD-YOLOv9 algorithm is used to identify ships, buoy colors, bridges and island obstacles.

[0061] In order to obtain image information, the present application further includes step S0: monitoring the environmental information around the ship in real time through the camera on the ship, and the video or image information obtained is introduced into the AD-YOLOv9 algorithm model.

[0062] As shown in the figure, the present application provides a ship offshore target position intelligent perception system based on the heading of the ship, which includes: Figure 3

[0063] ​​The information extraction module 10 is configured to process the target image using the AD-YOLOv9 algorithm to obtain position information and category information of the target image.

[0064] The image processing module 20 is configured to establish a coordinate system according to the position information of the target image, mark the target on the target image by using a target prediction box, determine the center point coordinates of the target prediction box based on a deep learning network, thereby determining the center point coordinates of the target, and associate the center point coordinates of the target with the heading of the ship to determine the absolute position and relative position of the target.

[0065] The target direction angle determination module 30 is configured to determine the position of the target relative to the reference point of the ship according to the absolute position and relative position of the target, so as to determine the direction angle of the target.

[0066] The direction line marking module 40 is configured to determine a straight line where the center point of the target and the reference point of the ship are located, which is referred to as a target direction line, according to the direction angle of the target.

[0067] The angle drawing module 50 is configured to calculate and format the angle label of the target according to the target direction line, and draw the angle label information at a selected position on the target direction line.

[0068] The image display module 60 is configured to display the processed target image.

[0069] The AD-YOLOv9 algorithm is proposed by improving the YOLOv9 network, including introducing a triplet attention mechanism and replacing Conv with a lightweight DualConv convolution structure. Figure 4 As shown in the formula, first, a triplet attention mechanism (Triplet Attention) is introduced in the head network of YOLOv9. This mechanism includes channel attention, spatial attention and cross attention, which respectively focus on the importance of features in different dimensions, effectively enhancing the model's ability to capture key information. Especially in ship detection at sea, due to the complex and variable background and different target sizes, the triplet attention mechanism can help the model focus more accurately on the learning of details such as color, texture and shape, thereby enhancing the accuracy of target recognition and reducing false positives and false negatives.

[0070] Secondly, in the standard convolution layer in YOLOv9, a strategy of replacing it with a lightweight DualConv convolution is proposed. DualConv is an efficient convolution structure that decomposes the standard convolution into two smaller convolution operations (usually a combination of 1x1 convolution and 3x3 convolution), significantly reducing the computational complexity and parameter quantity while maintaining similar feature extraction capability. This optimization strategy not only significantly reduces the computational burden of the model, but also improves the real-time performance, making it more practical in limited computing resource scenarios on the sea.

[0071] The AD-YOLOv9 algorithm constructed by the present application can simultaneously complete the positioning and classification of the target in one forward propagation, has a lighter model and higher target classification recognition accuracy. Therefore, the algorithm is more suitable for real-time identification of ship targets on the sea than the ordinary YOLOv9 algorithm. In the specific task of sea target identification, AD-YOLOv9 can accurately identify key targets such as ships, buoy colors, bridges, and island obstacles, laying a solid foundation for subsequent target direction judgment.

[0072] In the implementation of the present application, the following settings are made for the method of target direction judgment:

[0073] (1) The ship navigation monitoring device is set to have sufficient stability, without picture shaking and the like;

[0074] (2) The bottom midpoint position coordinate (w / 2, h) of the image captured by the motion camera is set as the starting point of the ship's heading line, and the error is ignored;

[0075] (3) Since the ship's heading data at each moment of the collected image data is difficult to obtain, it is set that the heading is kept unchanged within a short period of time.

[0076] In one specific embodiment of the present application, the input image is processed in the AD-YOLOv9 algorithm network to obtain the position information and class label of the target.

[0077] As shown in Figure 5 , a coordinate system with the upper left corner of the input image as the origin is established, where each target is surrounded by a prediction box (box). This surrounding box provides the coordinate information of the target in the image. The deep learning network identifies the coordinate information of the left upper corner and the right lower corner of the target prediction box, and the coordinate origin is based on the coordinate axis in the deep learning network. Where (x1, y1) is the left upper corner coordinate of the boundary box, and (x2, y2) is the right lower corner coordinate of the boundary box. These coordinates are used to locate the target on the image.

[0078] Through these coordinate information, the diagonal midpoint or simple average boundary box coordinates of the boundary box can be calculated to obtain the center point coordinates of the target.

[0079] The position of the target relative to the ship navigation monitoring recorder (i.e. the reference point) is calculated using the arctangent function and the angle conversion formula.

[0080] According to the calculated direction angle, a straight line is drawn on the image from the reference point to the center point of the target, which represents the direction of the other target in the marine encounter scenario.

[0081] The angle label is calculated and formatted to display the direction angle and category information of the target.

[0082] Then, the angle label text is drawn at a suitable position on the direction line so that the ship crew can clearly read the ship direction angle information.

[0083] Finally, the processed image can be displayed or saved. This setting allows the user to view the detection results of the deep learning network and the direction information of the target.

[0084] In the present application, the position coordinates of the center point of the marine other target prediction box are The calculation method is as follows:

[0085] ;

[0086] wherein, represents the horizontal coordinate value of the upper left corner of the target prediction box, represents the horizontal coordinate value of the lower right corner of the target prediction box;

[0087] ;

[0088] wherein, represents the vertical coordinate value of the upper left corner of the target prediction box, represents the vertical coordinate value of the lower right corner of the target prediction box.

[0089] The bearing of the marine other target is calculated based on the heading of the ship, and the calculation method is as follows:

[0090] ;

[0091] ;

[0092] wherein, represents the horizontal coordinate value of the center point of the target prediction box, represents the horizontal coordinate value of the center point of the target prediction box, ω is the width of the picture, h is the length of the picture.

[0093] ​The arc system is converted into the angle system to represent the bearing angle of the target, and the bearing angle is calculated according to the following formula:

[0094] ;

[0095] In order to show the superiority of the ship offshore target perception method and / or system based on machine vision of the present application, the applicant conducted a comparative experiment in the actual offshore navigation of the ship, recorded the target recognition effect of the YOLOv9 benchmark model, and also recorded the target perception effect of the AD-YOLOv9 model of the present application.

[0096] In the actual offshore navigation of the ship, the data collected by the ship sailing to the sea area near Guangzhou Port is selected as the experimental scene. The scene is complex in terms of waterway and ship type, and there are bridges, islands, buoys and other common offshore targets, which can effectively verify the generalization of the proposed algorithm.

[0097] The recognition results in the actual navigation scene of the ship with poor weather conditions and complex targets are shown in Figure 6 and Figure 7 .

[0098] Figure 6 Fig. 5 is a schematic diagram of the target recognition effect of the YOLOv9 benchmark model. The model recognizes 5 targets, including 2 buoys (buoy red 0.66 and buoy green 0.48) and 3 ships (other ship 0.71, other ship 0.75 and other ship 0.62), among which there are many missed ships. The recognition accuracy of the red buoy is 66%, and the recognition accuracy of the green buoy is 48%. The recognition accuracy of the other ships is 71%, 75% and 62% respectively.

[0099] Figure 7 Fig. 6 is a target recognition effect using the AD-YOLOv9 model of the present application. The model recognizes 6 targets, including 2 buoys (buoy red 0.68 and buoy green 0.50) and 4 ships (other ship 0.43, other ship 0.73, other ship 0.70 and other ship 0.66), and Figure 7It can be seen that the accuracy of small targets in the current scene is greatly improved. Among them, the identification effect of red and green buoys is improved by 2%, reaching an identification accuracy of 68% and 50%, and one target (other ship 0.43, Q:11, TB:331) is identified, thereby reducing the missed detection rate. In addition, the scene also realizes the bearing perception of the target, according to the coordinate information obtained by deep learning, combined with the angle information of the bow, Q is the side angle, left side is negative, right side is positive, TB is the absolute bearing, the relative bearing of the target on the ship can be judged, for example, for the red buoy, a pink line is used, and the Q value and TB value are marked on a part of the line; for the green buoy, a yellow line is used, and the Q value and TB value are marked on a part of the line; for the other three target ships, green lines are used respectively, and the Q value and TB value are marked on a part of the line.

[0100] The AD-YOLOv9 algorithm proposed in the application shows that in the test results of the actual navigation scene, the method uses the deep learning YOLOv9 framework for improvement, realizes the differentiation of different buoy colors, target bearing judgment method ability, effectively enhances the recognition accuracy of different targets on the sea, and can improve the intelligent perception effect of the target in the process of ship navigation on the sea.

[0101] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art will understand that the technical solutions described in the foregoing examples can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for ship offshore target perception based on machine vision, characterized in that, The method comprises the following steps: Step S1: processing the incoming image information using an AD-YOLOv9 algorithm to obtain the position information and category information of the target image; wherein the AD-YOLOv9 algorithm is an improvement of the YOLOv9 algorithm, a triple attention mechanism is introduced in the head network of YOLOv9, the triple attention is channel attention, spatial attention and cross attention, and a strategy of replacing the standard convolution layer of YOLOv9 with lightweight DualConv convolution is proposed; Step S2: establishing a coordinate system according to the position information of the target image, taking the top left corner of the target image as the origin, and marking the target on the target image with a target prediction box, determining the center point coordinates of the target prediction box based on the deep learning network, thereby determining the center point coordinates of the target; setting the position coordinates of the midpoint of the bottom end of the image taken by the ship motion camera as the starting point of the heading line of the ship, and associating the center point coordinates of the target with the bow direction to determine the absolute bearing and relative bearing of the target, the calculation method being as follows: ； ； wherein, the horizontal coordinate value of the center point of the target prediction frame, the horizontal coordinate value of the center point of the target prediction frame, h is the length of the picture, ω is the width of the picture, and Q represents the orientation of other targets; Step S3: determining the position of the target relative to the reference point of the ship according to the absolute bearing and relative bearing of the target to determine the direction angle of the target, the reference point being the ship motion camera; Step S4: determining the straight line on which the center point of the target and the reference point of the ship are located according to the direction angle of the target, denoted as the target direction line; Step S5: calculating and formatting the angle label of the target according to the target direction line, and drawing the angle label information at a selected position on the target direction line; Step S6: displaying the processed target image.

2. The method for machine vision based ship awareness of maritime targets according to claim 1, characterized in that, Step S1 includes using the AD-YOLOv9 algorithm to recognize ships, buoys, colors, bridges and island obstacles.

3. The method for machine vision based ship awareness of maritime targets according to claim 1, characterized in that, Further comprising: Step S0: real-time monitoring of the environmental information around the ship through the camera on the ship, and the video or image information obtained is introduced into the AD-YOLOv9 model.

4. The method for machine vision based ship awareness of maritime targets according to claim 1, characterized in that, In step S2, the center point coordinates of the target prediction frame are determined according to the following formula: According to the following formula: ； wherein, represents a left upper corner horizontal coordinate value of the target prediction frame, represents a right lower corner horizontal coordinate value of the target prediction frame; ； wherein, represents a numerical value of a vertical coordinate of the upper left corner of the target prediction frame, represents a numerical value of a vertical coordinate of the lower right corner of the target prediction frame.

5. The method for machine vision based ship awareness of maritime targets according to claim 1, characterized in that, Step S6 includes: dynamically displaying the target and its corresponding bearing information identified in the target image in a graphical form on the terminal display screen, so that the operator can directly observe the position of the target and its corresponding bearing information.

6. A machine vision based ship offshore target perception system, characterized by, Comprising: An information extraction module: for processing the incoming image information using an AD-YOLOv9 algorithm to obtain the position information and category information of the target image; wherein the AD-YOLOv9 algorithm is an improvement of the YOLOv9 algorithm, a triple attention mechanism is introduced in the head network of YOLOv9, the triple attention is channel attention, spatial attention and cross attention, and a strategy of replacing the standard convolution layer of YOLOv9 with lightweight DualConv convolution is proposed; An image processing module is configured to establish a coordinate system according to position information of the target image, the coordinate system taking a top-left corner of the target image as an origin, and marking a target on the target image by using a target prediction frame, determining a center point coordinate of the target prediction frame based on a deep learning network, thereby determining a center point coordinate of the target; setting a position coordinate of a bottom end midpoint of an image captured by the ship motion camera as a starting point of a heading line of the ship, and associating the center point coordinate of the target with a bow direction, to determine an absolute position and a relative position of the target, and the calculation mode is as follows: ； ； wherein, the horizontal coordinate value of the center point of the target prediction frame, the horizontal coordinate value of the center point of the target prediction frame, h is the length of the picture, ω is the width of the picture, and Q represents the orientation of other targets; A target direction angle determination module is configured to determine a position of the target relative to a reference point of the ship according to the absolute position and the relative position of the target, to determine a direction angle of the target, the reference point being the ship motion camera; A direction line marking module is configured to determine a straight line where the center point of the target and the reference point of the ship are located according to the direction angle of the target, and mark the straight line as a target direction line; An angle drawing module is configured to calculate and format an angle label of the target according to the target direction line, and draw the angle label information at a selected position on the target direction line; An image display module is configured to display the processed target image.

Images