A Visual Recognition Method for Illegal Operations of Crew Ropes

Abstract

The present invention provides a visual recognition method for illegal operations of crew ropes, specifically relating to the field of image recognition technology, including obtaining an operation video; performing target recognition on the operation video to obtain rope feature points; screening all the rope feature points to obtain a swing center point and a head point; calculating the application force based on the position change data of the head point and the swing center point before the rope is thrown; confirming the landing point, the throwing point and the target point based on the position change data of the head point after the rope is thrown; and obtaining the deviation degree after the rope is thrown based on the position relationship among the landing point, the throwing point and the target point. By analyzing the trajectory changes of the swing center point and the head point of the rope during the heaving line operation, the present invention determines whether the rope is evenly stressed, and evaluates the accuracy of wind direction judgment in combination with the position relationship between the landing point, the throwing point and the target point. Compared with the prior art, the present invention provides a more comprehensive and accurate reference, which helps to improve the heaving line technology of the crew.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and particularly to a visual recognition method for illegal operations of crew's cable throwing. Background Art

[0002] The cable throwing operation of the crew needs to comprehensively consider the dynamic parameters of the ship and environmental factors. That is, the crew should feel the changes of environmental variables such as the wind force and direction on the deck in real time, adjust the core force angle of the waist and abdomen through ergonomic postures when throwing the cable, and control the centrifugal trajectory of the cable by combining the rotation force of the wrist, so as to achieve a cable throwing operation that complies with safety specifications. The prior art only makes a binary judgment based on the final fitting degree between the landing point of the cable and the mooring bitt, ignoring the operation process of the crew. Therefore, there is an urgent need for a method that can effectively judge whether the cable throwing operation of the crew is compliant. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a visual recognition method for illegal operations of crew's cable throwing.

[0004] In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] The present application provides a visual recognition method for illegal operations of crew's cable throwing. The method includes: obtaining an operation video of the cable throwing process, where the cable throwing process includes the process before the cable is thrown and the process after the cable is thrown; performing target recognition on the operation video to obtain first information, where the first information includes the position change data of multiple cable feature points during the cable throwing process; screening all the cable feature points to obtain a swing center point and a head point during the cable throwing process; calculating the application force after the cable is thrown based on the position change data of the head point and the swing center point before the cable is thrown, where the application force is used to characterize the distance change between the head point and the swing center point; confirming a landing frame in the operation video based on the position change data of the head point after the cable is thrown, and performing recognition on the landing frame to obtain the landing point, the starting point of throwing, and the target point; analyzing the deviation degree after the cable is thrown based on the position relationship between the landing point, the starting point of throwing, and the target point.

[0006] In a possible implementation manner, performing target recognition on the operation video to obtain first information includes: dividing the operation video into multiple video frames; performing cable area segmentation on each video frame in chronological order by using the optical flow method to obtain at least one suspected area and the positions of at least one feature point in each suspected area; judging the cable area according to the length-width relationship of each suspected area in each video frame, and taking all the feature points of the cable area as cable feature points.

[0007] In a possible implementation, the optical flow method is an optical flow method based on pyramid layering.

[0008] In a possible implementation, determining the cable region according to the length-width relationship of each suspected region in each video frame includes: performing linear fitting on each pixel point in the suspected region to obtain a fitting line; taking the extension direction of the fitting line, extracting the length of the longest pixel point connection line in the suspected region as the length of the suspected region; taking the direction perpendicular to the fitting line, extracting the length of the longest pixel point connection line in the suspected region as the width of the suspected region; dividing the length of the suspected region by the width to obtain an extension ratio, and determining whether the suspected region is a cable region based on the extension ratio.

[0009] In a possible implementation, screening all the cable feature points to obtain a swing center point and a head point during the cable-throwing process includes: based on the position change data of each cable feature point, comparing the data changes of each cable feature point one by one to obtain the swing center point; according to the timing relationship, calculating the distance between each cable feature point and the swing center point at each moment based on the first information respectively to obtain multiple image distances corresponding to each cable feature point; respectively comparing the multiple image distances at each moment and counting to obtain the number of times when the image distance of each cable feature point is the maximum value; determining the head point based on the number of times when the image distance of each cable feature point is the maximum value, and the number of times when the image distance corresponding to the head point is the maximum is the most.

[0010] In a possible implementation, based on the position change data of each cable feature point, comparing the data changes of each cable feature point one by one to obtain the swing center point includes: constructing a corresponding scatter plot according to the position change data of each cable feature point; respectively calculating the Euclidean distance between every two points in the scatter plot to obtain a set of Euclidean distances corresponding to each scatter plot; respectively extracting the maximum Euclidean distance corresponding to each scatter plot from each set of Euclidean distances; respectively extracting the minimum Euclidean distance corresponding to each point in each scatter plot from each set of Euclidean distances; respectively calculating the swing eigenvalue corresponding to each scatter plot based on the distribution of the maximum Euclidean distance and multiple minimum Euclidean distances corresponding to each scatter plot; determining whether the cable feature point is the swing center point based on the magnitude relationship between the swing eigenvalues corresponding to each cable feature point.

[0011] In a possible implementation manner, calculating the application force after the cable is thrown based on the position change data of the head point and the swing center point before the cable is thrown includes: calculating the cable image distance at each moment based on the position change data of the head point and the swing center point; extracting the maximum cable image distance based on the cable image distance at each moment; calculating the rotation angle at each moment based on the position relationship between the swing center point and the head points at two adjacent moments; correcting the rotation angle according to the maximum cable image distance and the cable image distance at each moment to obtain the actual rotation angle corresponding to each moment; and calculating the application force based on the change of the actual rotation angle at each moment.

[0012] In a possible implementation manner, analyzing the deviation degree after the cable is thrown based on the position relationship between the landing point, the throwing starting point and the target point includes: calculating the deviation angle based on the position relationship between the landing point, the throwing starting point and the target point, where the deviation angle is the included angle between the cable throwing line segment and the target line segment, the endpoints of the cable throwing line segment are the landing point and the throwing starting point respectively, and the endpoints of the target line segment are the target point and the throwing starting point respectively; calculating the landing distance according to the position relationship between the landing point and the target point; and calculating the deviation degree based on the landing distance, the deviation angle and a preset distance threshold.

[0013] In a possible implementation manner, before analyzing the deviation degree of the cable throwing process based on the position relationship between the landing point, the throwing starting point and the target point, it further includes: analyzing the position change data of the head point after the cable is thrown to obtain whether there is a moment of sudden wind force change after the cable is thrown. If there is a moment of sudden wind force change, the landing point is updated.

[0014] In a possible implementation manner, analyzing the position change data of the head point after the cable is thrown to obtain whether there is a moment of sudden wind force change after the cable is thrown. If there is a moment of sudden wind force change, updating the landing point includes: obtaining the environmental parameters corresponding to each moment after the cable is thrown; respectively predicting the predicted position corresponding to each moment according to the environmental parameters corresponding to each moment and the position of the head point, where the predicted position is the predicted landing position corresponding to when the cable lands; constructing a head displacement trajectory diagram according to the position change data of the head point; smoothing the curve of the head displacement trajectory diagram and updating the head displacement trajectory diagram with the result of the smoothing process; calculating the second derivative corresponding to each moment based on the head displacement trajectory diagram to obtain the interference degree; judging the interference degree of each moment in ascending order of time sequence according to a preset judgment threshold, taking the moment that is first greater than the preset judgment threshold as the moment of sudden wind force change, and updating the landing point with the predicted position corresponding to the moment of sudden wind force change.

[0015] The present invention has the following beneficial effects:

[0016] According to the variation law of the swing center point and the head point of the cable during the heaving line operation, this invention determines whether the cable is evenly stressed before being thrown by the change in the distance between the swing center point and the head point. At the same time, in combination with the positional relationship between the landing point of the cable after being thrown and the starting point and the target point, it measures the judgment result of the wind force and wind direction when the cable is thrown. Compared with the prior art which only judges based on the landing point and the target point, this invention evaluates from two dimensions, making it relatively more comprehensive and accurate, and providing a scientific reference basis for improving the crew's heaving line technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following-described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic flowchart of a visual recognition method for crew cable illegal operations provided in Embodiment 1 of the present invention;

[0019] Figure 2 It is a schematic flowchart of step S2 provided in Embodiment 1 of the present invention;

[0020] Figure 3 It is a schematic flowchart of step S3 provided in Embodiment 1 of the present invention;

[0021] Figure 4 It is a schematic flowchart of step S4 provided in Embodiment 1 of the present invention;

[0022] Figure 5 It is a schematic flowchart of step S6 provided in Embodiment 1 of the present invention;

[0023] Figure 6 It is a schematic flowchart of a visual recognition method for crew cable illegal operations provided in Embodiment 2 of the present invention;

[0024] Figure 7 It is a schematic flowchart of step Step6 provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] To further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following combines the accompanying drawings and preferred embodiments to detail the specific implementation manner, structure, features and effects of a visual recognition method for illegal operations of crew ropes proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0027] First, introduce the scenario targeted by the following embodiments: First, before performing the heaving line, carefully check the rope to ensure no damage or broken strands, and prepare the corresponding heaving line head to make it firmly connected. During the operation, stand at a suitable position, separate the feet, keep stable, lean the body slightly backward, and use the strength of the waist and arms to smoothly and accurately throw the rope towards the target position along a parabolic trajectory. Control the strength and angle properly to ensure a successful heaving line at one time.

[0028] Embodiment 1:

[0029] The following specifically describes the specific solution of a visual recognition method for illegal operations of crew ropes provided by the present invention in conjunction with the accompanying drawings.

[0030] Please refer to Figure 1 , which shows a schematic flowchart of a visual recognition method for illegal operations of crew ropes provided by an embodiment of the present invention. As shown in the figure, this embodiment includes steps S1 - step S6 in total.

[0031] S1. Obtain the operation video of the heaving line process, where the heaving line process includes the process before the rope is thrown and the process after the rope is thrown.

[0032] In this step, the operation video is taken by a high-speed camera for the entire process of the crew heaving the line. Set the shooting resolution to 3840×2160 (4K) and the frame rate to 1000 frames / s. Among them, the installation scheme of the high-speed camera can be fixed on the deck observation point through a carbon fiber tripod and cooperate with a hydraulic pan-tilt head to achieve ±15° pitch adjustment; or it can be integrated into a six-axis drone flight platform and adopt a three-axis mechanical stabilization structure to hover in the air to keep the picture stable. Those skilled in the art can choose according to actual needs. The operation video in this embodiment can be written using a solid-state drive to ensure continuous writing of the 4K / 1000fps video stream without frame loss. No specific limitation is made in this embodiment.

[0033] S2. Perform object recognition on the operation video to obtain the first information, where the first information includes the position change data of multiple cable feature points during the heaving line process.

[0034] Among them, the position change data of the cable feature points mentioned in this step is the position corresponding to each moment of the cable feature points in the entire operation video. Among them, the above-mentioned position is the pixel coordinate of the cable feature point. In this embodiment, since the position of the high-speed camera remains unchanged during the acquisition process of the operation video. Therefore, the pixel coordinates of the cable feature points have practical position reference significance.

[0035] In addition, regarding the object recognition method mentioned in this step, first, the operation video is processed by equidistant slicing in time sequence to obtain multiple video frames, and then object recognition is performed on the multiple video frames to obtain the first information. Regarding the specific implementation method of the object recognition method, either the object detection method based on the YOLOv5s model can be adopted, that is, lightweight and high-precision detection is achieved through an improved Focus structure and CSP module; or it can be realized by the OpenCV contour tracking algorithm, that is, dynamic contour features are constructed by combining Canny edge detection and HSV color space analysis; in addition, a hybrid object detection method combining the optical flow method and the background difference method can also be adopted, using the Horn-Schunck optical flow equation to calculate the motion vector field, and simultaneously establishing a background difference template through the Gaussian mixture model to achieve the collaborative recognition of moving objects and static objects. At the same time, in order to facilitate the understanding of those skilled in the art, the specific steps for obtaining the first information are also provided in this step. For details, see Figure 2 , which shows the process of object recognition in the figure. Specifically, it includes steps S21 - step S23.

[0036] S21. Divide the operation video into multiple video frames.

[0037] As previously described, in this step, in order to reduce the computing resources occupied by the entire method, this embodiment takes a time interval of 1 millisecond to divide and process the operation video, thereby obtaining several video frames. Regarding the division interval of the video frames, those skilled in the art can also choose other different time intervals, and no specific restrictions are made in this embodiment.

[0038] At the same time, it should also be noted that the moments mentioned in the subsequent steps are the time points corresponding to each video frame.

[0039] S22. Respectively perform cable area segmentation on each of the video frames in chronological order by the optical flow method to obtain at least one suspected area and the positions corresponding to at least one feature point in each of the suspected areas.

[0040] Specifically, the optical flow method in this embodiment is preferably an optical flow method based on pyramid layering. Regarding the optical flow method based on pyramid layering, first, a Gaussian pyramid is constructed for each video frame, so as to obtain images with different resolutions. Then, starting from the top layer (lowest resolution) of the pyramid, feature point matching is performed using the horn_schunk algorithm or Shi-Tomasi detection to obtain a preliminary optical flow estimate. Next, the preliminary optical flow estimate result is used as the initial estimate for the next layer of the pyramid, and feature point matching continues until the bottom layer of the pyramid is reached. In this way, in this embodiment, the video frames can be segmented based on the change of the optical flow. Since there is more than one moving object in a video frame, multiple suspected regions and the positions of the feature points within the suspected regions are obtained in this step. Regarding the implementation process of segmenting the suspected regions by the hierarchical pyramid L-K optical flow method, since it is a prior art, it will not be elaborated in this embodiment.

[0041] S23. Determine the cable region according to the length-width relationship of each suspected region in each video frame, and use all the feature points of the cable region as cable feature points.

[0042] Since multiple suspected regions will be obtained in the above step, but the analysis object in this embodiment is the cable. Therefore, in this step, the multiple suspected regions will be distinguished based on the slender morphological characteristics of the cable. See steps S231 - S234 for details, which show the process of judging how to distinguish a suspected region.

[0043] S231. Perform linear fitting on each pixel point in the suspected region to obtain a fitting line.

[0044] S232. In the direction of the extension of the fitting line, extract the length of the longest pixel point connection line in the suspected region as the length of the suspected region.

[0045] S233. In the direction perpendicular to the fitting line, extract the length of the longest pixel point connection line in the suspected region as the width of the suspected region.

[0046] S234. Divide the length of the suspected region by the width to obtain an extension ratio, and judge whether the suspected region is a cable region based on the extension ratio.

[0047] Among them, it should be noted that the process of judging and realizing the suspected area is as follows: by setting a cable judgment threshold (0.7), and then comparing the extension ratio with the cable judgment threshold. If the extension ratio is greater than the cable judgment threshold, it can be considered that the suspected area is the cable area. At the same time, regarding the size of the cable judgment threshold, those skilled in the art can modify it according to the actual situation, such as 0.8 or 0.9, etc., and no specific limitation is made in this embodiment.

[0048] In this embodiment, considering that before the cable is thrown, the cable presents a state of rotating around the midpoint. And before the cable is thrown, the force exerted by the crew will directly affect the rotation speed of the cable.

[0049] In this embodiment, considering that the cable is in a rotating state before being thrown, that is, the cable presents a dynamic process of rotating around its center point. This rotating state is an inherent physical property of the cable before throwing. At the same time, the force applied by the crew before throwing the cable will directly affect the rotation rate of the cable. This is because the magnitude and direction of the force applied by the crew will change the momentum of the cable, thereby affecting its rotation rate. Therefore, before throwing the cable, the force exerted by the crew can be confirmed by analyzing the rotating state of the cable. See steps S3 and step S4 for details.

[0050] S3. Screen all the cable feature points to obtain a swing center point and a head point during the heaving line process.

[0051] At the same time, in order to accurately analyze the rotating state of the cable, in this application, the combination of the swing center of the cable and the head point of the cable is analyzed. Therefore, first analyze and obtain the swing center and the head point of the cable. See Figure 3 , the figure shows that step S3 includes steps S31 - step S34.

[0052] S31. Based on the position change data of each cable feature point, compare the data changes of each cable feature point one by one to obtain the swing center point.

[0053] Among them, in this embodiment, considering that the position change of the swing center point is relatively small before the cable is thrown. Therefore, in this step, it is confirmed by comparing the position changes of each cable feature point. See steps S311 - step S316 for details.

[0054] Step S311. Construct a corresponding scatter plot based on the position change data of each cable feature point.

[0055] In this step, taking the lower left corner of the video frame as the starting point, the horizontal direction to the right is defined as the horizontal axis and its positive direction, and the vertical direction upward is defined as the vertical axis and its positive direction, thus establishing an image rectangular coordinate system. Subsequently, according to the positions of the cable rope feature points at each moment, these positions are marked in the image rectangular coordinate system, and then a scatter plot corresponding to the cable rope feature points is formed.

[0056] Step S312: Calculate the Euclidean distance between every two points in the scatter plot respectively, and obtain the set of Euclidean distances corresponding to each scatter plot.

[0057] Step S313: Extract each set of Euclidean distances respectively to obtain the maximum Euclidean distance corresponding to each scatter plot.

[0058] Step S314: Extract each set of Euclidean distances respectively to obtain the minimum Euclidean distance corresponding to each point in each scatter plot.

[0059] Step S315: Calculate the swing characteristic value corresponding to each scatter plot respectively based on the distribution of the maximum Euclidean distance and multiple minimum Euclidean distances corresponding to each scatter plot.

[0060] Specifically, in this step, the calculation functional formula of the swing characteristic value is:

[0061] ;

[0062] where represents the swing characteristic value of the th cable rope feature point; represents the maximum Euclidean distance corresponding to the th cable rope feature point; represents the number of video frames before the cable rope is thrown, which can also be called the total number of moments before the cable rope is thrown; represents the minimum Euclidean distance at the th moment corresponding to the th cable rope feature point, and the th moment is before the cable rope is thrown.

[0063] In the above calculation functional formula, represents the maximum Euclidean distance corresponding to the th cable rope feature point. When is smaller, it proves that the change of the th cable rope feature point in multiple video frames is smaller, and it is more likely to be the swing center point; similarly, is the reciprocal of . When the value of is larger, it proves that the change between the th cable rope feature points in multiple video frames is smaller, and it is more in line with the performance of the swing center point.

[0064] Step S316: Determine whether the cable feature point is the swing center point based on the magnitude relationship between each cable feature point and the corresponding swing eigenvalue.

[0065] Therefore, in this step, the cable feature point corresponding to the maximum value among multiple swing eigenvalues can be directly screened. This cable feature point should be the swing center point.

[0066] S32: According to the timing relationship, calculate the distance between each cable feature point and the swing center point at each moment based on the first information, and obtain multiple image distances corresponding to each cable feature point.

[0067] S33: Compare and count the multiple image distances at each moment respectively, and obtain the number of times that the image distance of each cable feature point is the maximum value.

[0068] S34: Determine the head point based on the number of times that the image distance of each cable feature point is the maximum value. The head point corresponds to the maximum number of times that the image distance is the maximum value.

[0069] By comparing one by one in the above steps S32 - S34, the cable feature point that is the farthest from the swing center point at each moment is obtained, and the cable feature point with the most times of being the farthest from the swing center point is selected as the head point.

[0070] S4: Calculate the application force after the cable is thrown based on the position change data of the head point and the swing center point before the cable is thrown. The application force is used to characterize the distance change situation between the head point and the swing center point.

[0071] As described above, the rotation speed of the cable is directly affected by the force applied by the crew. At the same time, there is an angle between the circle formed by the crew waving the cable in the air and the horizontal plane. Therefore, in this embodiment, the existing position change data cannot be directly used for calculation. For details, please refer to Figure 4 , which shows the calculation process of the application force in the figure.

[0072] S41: Calculate the cable image distance at each moment based on the position change data of the head point and the swing center point.

[0073] S42: Extract the maximum cable image distance based on the cable image distance at each moment.

[0074] S43: Calculate the rotation angle at each moment based on the position relationship between the swing center point and the head points at two adjacent moments.

[0075] It should be noted that the position of the center point of swing may be different at different times. The position of the center point of swing before the cable is thrown can be obtained by taking the center points of multiple positions. This is common knowledge for those skilled in the art and will not be elaborated in this embodiment. Therefore, on the premise that the positions of the center point of swing and the head point at adjacent times are known, the rotation angle can be calculated through trigonometric functions.

[0076] S44. Correct the rotation angle according to the maximum cable image distance and the cable image distance at each moment to obtain the actual rotation angle corresponding to each moment.

[0077] Specifically, the calculation functional formula of the actual rotation angle in this step is:

[0078] ;

[0079] where, represents the actual rotation angle at the th moment; represents the rotation angle at the th moment, which is calculated from the position relationship between the head point and the center point of swing corresponding to the th moment and the th moment; represents the sigmoid function; represents the maximum cable image distance; represents the th moment of the cable image distance.

[0080] In this embodiment, it is considered that the distance from the center point of swing to the head point does not change with time in reality. If the cable image distance at the th moment changes, it may be that the circle formed by the crew waving the cable in the air at the th moment has an angle with the horizontal plane. That is, when the distance from the head point of the cable to the center point of swing is shorter at the th moment, the angle between the cable head and the center point of swing in the video frame at the th moment and the next frame of the video frame is smaller than the actual angle. Therefore, the correction coefficient can correct the rotation angle calculated in the image.

[0081] S45. Calculate the application force based on the change in the actual rotation angle at each moment.

[0082] That is, in this step, the application force of the crew is measured according to the amplitude of the change in the actual rotation angle. Specifically, it includes steps S451 - S456.

[0083] S451. Construct a first rectangular coordinate system, where the abscissa of the first quadrant of the first rectangular coordinate system is time, and the ordinate of the first quadrant of the first rectangular coordinate system is angle.

[0084] S452. Mark the actual rotation angle at each moment in the first quadrant of the first rectangular coordinate system, and perform linear fitting to obtain an angle fitting line.

[0085] S453. Calculate the deviation distance from the actual rotation angle at each moment to the angle fitting line respectively.

[0086] S454. Calculate the difference between adjacent moments based on the actual rotation angle at each moment, and obtain the angle difference corresponding to each moment.

[0087] S455. Calculate the average value based on the angle differences corresponding to each moment to obtain an average difference angle;

[0088] S456. Calculate the application force based on the average difference angle, the actual rotation angle at each moment, and the deviation distance.

[0089] Specifically, in this step, the calculation functional formula of the application force is:

[0090] ;

[0091] Where, represents the application force; represents the number of video frames before the cable is thrown, which can also be called the total number of moments before the cable is thrown; represents the th deviation distance; represents the th moment and the th moment angle difference; represents the average difference angle.

[0092] In the above calculation functional formula, and The smaller they are, the more uniform the application force change of the operator's swing is, and the more it meets the operation requirements.

[0093] Since the requirement for the cable is that the final arrival position is the area around the mooring post or the mooring hole on the dock edge, which is a target point. Therefore, in this embodiment, to judge the accuracy of the operator's cable throwing, it is evaluated by combining the final arrival position of the head point, the included angle between the line connecting the head point and the throwing point, and the line connecting the target point and the throwing point. See steps S5 - step S6 for details.

[0094] S5. Based on the position change data of the head point after the cable is thrown, confirm the landing frame in the operation video, and identify the landing point, throwing point, and target point from the landing frame.

[0095] Regarding the confirmation of the landing frame mentioned in this step, the image frame corresponding to the moment when the position of the head point no longer changes can be used as the landing frame. The landing point, throwing point, and target point mentioned in this step all include the corresponding positions, that is, the pixel coordinates of the landing point, throwing point, and target point. In this embodiment, since the position of the high-speed camera remains unchanged during the acquisition process of the operation video. Therefore, the pixel coordinates of the landing point, throwing point, and target point have practical position reference significance. Regarding the throwing point and the target point, the YOLOv8 model can be used to frame the positions of the crew and the bollard, and the center points of the framed positions are used as the throwing point and the target point respectively; while the landing point is the head point, and its specific size is the position corresponding to the moment when it no longer changes.

[0096] S6. Analyze the deviation degree after the cable is thrown based on the positional relationship between the landing point, the throwing point, and the target point.

[0097] Specifically, referring to Figure 5 , the figure shows that this step includes steps S61 - S63 in total.

[0098] S61. Calculate the deviation angle based on the positional relationship between the landing point, the throwing point, and the target point. The deviation angle is the angle between the cable - throwing line segment and the target line segment. The endpoints of the cable - throwing line segment are the landing point and the throwing point, and the endpoints of the target line segment are the target point and the throwing point.

[0099] S62. Calculate the landing distance based on the positional relationship between the landing point and the target point.

[0100] S63. Calculate the deviation degree based on the landing distance, the deviation angle, and a preset distance threshold.

[0101] In this step, the calculation function formula for the deviation degree is:

[0102] ;

[0103] Where, represents the deviation degree; represents the landing point and the target point the landing distance between them; represents the preset radius length; represents the landing point , the throwing point and the target point the deviation angle between them.

[0104] In the above calculation function, when the landing point falls within the range near the target point, that is, the landing distance is not greater than the preset radius length , not greater than 1, the greater the landing distance , the greater the deviation; when the landing point is out of the range near the target point, greater than 1, the greater the landing distance , the greater the deviation; through the above calculation function, it can better describe that the final result of the heaving line deviates more from the actual requirements.

[0105] In this embodiment, the operation of the crew during heaving line is comprehensively measured by the change law of the rotation of the heaving line trajectory before and after the heaving line is thrown and the change of the predicted landing point during the heaving line operation. It can more accurately judge the operation state of the crew and facilitate the improvement of the crew's heaving line technology in the later stage.

[0106] Embodiment 2:

[0107] The difference between this embodiment and Embodiment 1 is that the process of judging the external environment is added. See Figure 6 , the figure shows that this embodiment includes Step 1 - Step 7.

[0108] Step 1, obtain the operation video of the heaving line process, and the heaving line process includes the process before the heaving line is thrown and the process after the heaving line is thrown.

[0109] Step 2, perform target recognition on the operation video to obtain the first information, and the first information includes the position change data of multiple heaving line feature points during the heaving line process.

[0110] Step 3, screen all the heaving line feature points to obtain a swing center point and a head point during the heaving line process.

[0111] Step 4, calculate based on the position change data of the head point and the swing center point before the heaving line is thrown to obtain the application force after the heaving line is thrown, and the application force is used to characterize the distance change between the head point and the swing center point.

[0112] Step 5, confirm the landing frame in the operation video based on the position change data of the head point after the heaving line is thrown, and identify the landing point, the starting point of throwing, and the target point for the landing frame.

[0113] Step 6, analyze the position change data of the head point after the heaving line is thrown to obtain whether there is a moment of sudden wind change after the heaving line is thrown. If there is a moment of sudden wind change, update the landing point.

[0114] In this embodiment, considering that changes in wind direction and wind intensity are not uncommon during actual heaving line operations, in order to ensure good results of heaving the line, the crew should pre-judge the wind conditions and thus change the throwing angle and the applied force. However, the wind conditions at sea often change in real time, which may cause the wind direction or wind intensity to suddenly change after the line has been thrown for a certain distance, even though the crew has adjusted the force direction correctly, affecting the accuracy of the heaving line result. It is also possible that there are problems with the crew's own heaving line operation, but due to the sudden change in wind force, a good heaving line result is achieved. The existence of such situations will interfere with the judgment of the operator's compliance with the operation.

[0115] Therefore, as shown in Figure 7 , Step 6 includes steps Step61 - Step66.

[0116] Step61. Obtain the environmental parameters corresponding to each moment after the line is thrown.

[0117] Specifically, the environmental parameters mentioned in this embodiment include wind speed and wind direction.

[0118] Step62. Respectively perform predictions based on the environmental parameters corresponding to each moment and the position of the head point to obtain the predicted position corresponding to each moment, and the predicted position is the predicted landing position corresponding to when the line lands.

[0119] It should be noted that the position of the head point at each moment involved in this step is derived from the position change data of the head point. At the same time, the prediction method of the predicted position mentioned in this step can be implemented through a Kalman filter. That is, parameters such as the position of the head point, wind speed, and wind direction at the current moment are written into the current system state vector and input into the Kalman filter to obtain the predicted system state vector at the next moment . Extract the predicted value of the line head position at the next moment from as the input value for analyzing the displacement position of the line head at the next moment, and repeat the operation until the moment corresponding to the landing frame.

[0120] Step63. Construct a head displacement trajectory diagram based on the position change data of the head point.

[0121] It should be noted that in this step, the head displacement diagram also takes the lower left corner of the video frame as the starting point, defines the horizontal direction to the right as the horizontal axis and its positive direction, and the vertical direction upward as the vertical axis and its positive direction. Each data point in the head displacement diagram represents the position of the head point at a certain moment.

[0122] Step64. Smooth the curve of the head displacement trajectory diagram and update the head displacement trajectory diagram with the result of the smoothing process.

[0123] Step65. Calculate the second derivative corresponding to each moment based on the head displacement trajectory diagram to obtain the interference degree.

[0124] It should be noted that during the heaving line process, the cable will have a certain rotational speed, which causes the cable to rotate around a central axis in the air and fly forward at the same time. From a top-down perspective, it appears to jitter around the central axis. However, when the wind force suddenly changes, it will cause a large change in the moving direction. At the same time, the turning situation of the head point at each moment is reflected by the absolute value of the slope in the head displacement trajectory diagram. And the turning degree of the head point at each moment can be reflected by the second derivative corresponding to that moment. Therefore, in this embodiment, the second derivative corresponding to each moment can be used as the interference degree.

[0125] At the same time, on the other hand, when the predicted landing positions corresponding to adjacent moments differ greatly, it also indicates that there is a sudden change in wind direction or wind force between adjacent moments, which will cause changes in the prediction results. Therefore, in this embodiment, the calculation function formula of the interference degree can still be the following calculation method:

[0126] ;

[0127] Where, represents the interference degree at the th moment after the cable is thrown; represents the absolute value calculation, and Norm represents the normalization function; represents the deviation degree calculated from the predicted landing position corresponding to the th moment after the cable is thrown; represents the deviation degree calculated from the predicted landing position corresponding to the th moment after the cable is thrown; represents the slope corresponding to the th moment after the cable is thrown; represents the time interval corresponding to adjacent moments.

[0128] In the above calculation function formula, represents the degree of change in the turning of the head point at each moment, which can change greatly when affected by sudden changes in wind force or wind direction; the degree of difference between the predicted positions corresponding to adjacent moments. If there is a large difference in the predicted landing positions between adjacent moments, the value of is larger. Therefore, through the above calculation formula, it can better represent the degree of wind force or wind direction at the th moment after the cable is thrown.

[0129] Step66. According to a preset judgment threshold, judge the interference degree at each moment in ascending order of time sequence, take the moment that is first greater than the preset judgment threshold as the wind force mutation moment, and update the predicted landing position corresponding to the wind force mutation moment to the landing point.

[0130] It should be noted that the preset judgment threshold in this step is 0.7. Those skilled in the art can also modify it according to the actual situation, and no specific limitation is made in this embodiment.

[0131] Step7. Analyze the deviation degree after the cable is thrown based on the position relationship among the landing point, the throwing point and the target point.

[0132] Among them, in the specific implementation processes of Steps Step1-Step5 and Step7 in this embodiment, reference can be made to the description content of Embodiment 1, and details will not be repeated in this embodiment.

[0133] In this embodiment, the influence of the changes in wind direction and wind force intensity encountered in actual operation on the crew's operation is considered. By analyzing the position of the head point corresponding to each moment and the corresponding environmental parameters, the landing position of the head point is predicted. Subsequently, based on the change in the predicted landing position between adjacent moments and the change in the displacement trajectory of the head point, the influence degree of the wind direction and wind force on the movement of the head point at adjacent moments is comprehensively evaluated. The method adopted in this embodiment can effectively eliminate the influence of the wind direction and wind force on the cable throwing result, and further improve the accuracy of the deviation degree calculation.

[0134] It should be noted that: the above-mentioned sequence of the embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0135] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the key points of each embodiment are the differences from other embodiments.

Claims

1. A visual recognition method for illegal rope operation by crew members, characterized in that: The method comprises: Obtaining an operational video of the cable-skimming process, wherein the cable-skimming process includes a process before and after the cable is thrown out; Performing target recognition on the operation video to obtain first information, wherein the first information includes position change data of multiple cable feature points during the cable-laying process; Screening all the cable feature points to obtain a swing center point and a head point during the cable-skimming process; Based on the position change data of the head point and the swing center point before the cable is thrown, the applied force after the cable is thrown is calculated, and the applied force is used to represent the change in the distance between the head point and the swing center point; Based on the position change data of the head point after the cable is thrown, a landing frame is confirmed in the operation video, and the landing frame is identified to obtain the landing point, the starting point and the target point; Obtaining a deviation of the cable after being thrown based on a positional relationship analysis among the landing point, the throwing starting point, and the target point; All the cable feature points are screened to obtain a swing center point and a head point during the cable-skimming process, including: Based on the position change data of each cable feature point, the data change of each cable feature point is compared one by one to obtain the swing center point; According to the time sequence relationship, the distance between each of the cable feature points and the swing center point at each moment is calculated based on the first information to obtain a plurality of image distances corresponding to each of the cable feature points; Comparing and counting the multiple image distances at each moment, respectively, to obtain the number of times the image distance of each cable feature point is the maximum value; A head point is determined based on the number of times the image distance of each of the cable feature points reaches a maximum value, and the head point corresponds to the point with the largest number of times the image distance reaches a maximum value; The force applied after the cable is thrown is obtained by calculating the position change data of the head point and the swing center point before the cable is thrown, including: Calculating based on the position change data of the head point and the swing center point, obtaining the cable image distance at each moment; The maximum cable image distance is obtained based on the cable image distance extraction at each moment; Calculating the rotation angle at each moment based on the positional relationship between the swing center point and the head point at two adjacent moments; Correcting the rotation angle according to the maximum cable image distance and the cable image distance at each moment to obtain the actual rotation angle corresponding to each moment; The applied force is calculated based on the actual rotation angle change at each moment.

2. The visual recognition method for illegal rope operation by crew members according to claim 1, characterized in that: Performing target recognition on the operation video to obtain first information includes: dividing the operation video into a plurality of video frames; Segmenting the cable region of each video frame in time sequence using an optical flow method to obtain at least one suspected region and a position corresponding to at least one feature point in each suspected region; The cable area is determined according to the length-width relationship of each suspected area in each video frame, and all feature points of the cable area are used as cable feature points.

3. The visual recognition method for illegal rope operation by crew members according to claim 2, characterized in that: The optical flow method is a pyramid-layered optical flow method.

4. The visual recognition method for illegal rope operation by crew members according to claim 2, characterized in that: Determining the cable area according to the length-width relationship of each suspected area in each video frame includes: Performing straight line fitting on each pixel point in the suspected area to obtain a fitting straight line; In the extension direction of the fitting straight line, the longest pixel connection length in the suspected area is extracted as the length of the suspected area; Extracting the longest line length of pixels in the suspected area in a direction perpendicular to the fitted straight line as the width of the suspected area; An extension ratio is obtained by dividing the length of the suspected area by the width, and based on the extension ratio, it is determined whether the suspected area is a cable area.

5. The visual recognition method for illegal rope operation by crew members according to claim 1, characterized in that: Based on the position change data of each cable feature point, the swing center point is obtained by comparing the data changes of each cable feature point one by one, including: Constructing a corresponding scatter plot based on the position change data of each cable feature point; Calculating the Euclidean distance between each two points in the scatter plot to obtain a Euclidean distance set corresponding to each scatter plot; Extract each Euclidean distance set separately to obtain the maximum Euclidean distance corresponding to each scatter plot; Extract each Euclidean distance set separately to obtain the minimum Euclidean distance corresponding to each point in each scatter plot; The swing eigenvalue corresponding to each scatter plot is obtained by respectively calculating the distribution of the maximum Euclidean distance and a plurality of minimum Euclidean distances corresponding to each scatter plot; Based on the magnitude relationship between each of the cable feature points and the corresponding swing feature values, it is determined whether the cable feature point is the swing center point.

6. The visual recognition method for crew rope violations according to claim 1, characterized in that: The deviation of the cable after being thrown is obtained based on the positional relationship between the landing point, the starting point and the target point, including: The deviation angle is calculated based on the positional relationship between the landing point, the starting point, and the target point. The deviation angle is the angle between the throwing line segment and the target segment. The endpoints of the throwing line segment are the landing point and the starting point, respectively, and the endpoints of the target segment are the target point and the starting point, respectively. The landing distance is calculated based on the positional relationship between the landing point and the target point; The degree of deviation is calculated based on the landing distance, the deviation angle, and a preset distance threshold.

7. The visual recognition method for illegal rope operation by crew members according to claim 1, characterized in that: The deviation of the cable-casting process is obtained based on the positional relationship analysis between the landing point, the starting point and the target point. Previously, it also included: The position change data of the head point after the cable is thrown is analyzed to determine whether there is a sudden change in wind force after the cable is thrown. If there is a sudden change in wind force, the landing point is updated.

8. The visual recognition method for illegal rope operation by crew members according to claim 7, characterized in that: The position change data of the head point after the cable is thrown is analyzed to determine whether there is a sudden change in wind force after the cable is thrown. If there is a sudden change in wind force, the landing point is updated, including: Get the environmental parameters corresponding to each moment after the cable is thrown; The predicted position corresponding to each moment is obtained by respectively predicting the environmental parameters and the position of the head point at each moment, wherein the predicted position is the predicted landing position corresponding to when the cable falls to the ground; Construct a head displacement trajectory diagram based on the position change data of the head points; Smoothing the curve of the head displacement trajectory diagram, and updating the head displacement trajectory diagram with the result of the smoothing; Calculating the second-order derivative corresponding to each moment according to the head displacement trajectory diagram to obtain the interference degree; According to the preset judgment threshold, the interference degree at each moment is judged in ascending order of time sequence, and the moment that is first greater than the preset judgment threshold is taken as the moment of wind force mutation, and the predicted position corresponding to the moment of wind force mutation is updated to the landing point.

Images