Ship assembly operation violation intrusion identification system and method based on machine vision

Abstract

The invention provides a ship assembly operation illegal intrusion identification system and method based on machine vision, and relates to the technical field of ship manufacturing safety management. The recognition system comprises an image acquisition module, a data preprocessing module, a recognition model and a violation intrusion judgment module, the ship assembly operation violation intrusion identification method comprises the steps of image acquisition, data preprocessing, operation equipment identification, personnel target identification, violation intrusion identification and the like. The method can effectively recognize the illegal behaviors in the ship component assembling and lifting operation, and is high in accuracy. According to the method, the probability of illegal intrusion behaviors in ship assembly operation can be effectively reduced, so that the overall safety of the assembly process is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of shipbuilding safety management, and particularly relates to a system and method for identifying illegal intrusion in ship assembly operations based on machine vision. Background Art

[0002] The process of ship component assembly involves hull structure welding and manufacturing, outfitting, painting, etc. It includes a large number of high-altitude operations, large-scale lifting and turning operations, and various types of steel plate raw material welding operations. There are many risk types that may pose threats to personnel, products, and equipment. Currently, there are many problems both in terms of enterprise supervision and operation execution. Lifting operations, as one of the key links in the ship component assembly process, involve the handling and positioning of large structural components, and these operations usually require the use of heavy lifting machinery. Due to the high risk of lifting operations, any illegal intrusion or improper operation may lead to serious personal injuries or equipment damage. Although the shipbuilding industry has achieved initial achievements in informatization and intelligent manufacturing, the level of automation and intelligence in safety supervision is still limited.

[0003] Currently, shipbuilding enterprises mainly rely on on-site inspections and staff in the monitoring room for safety supervision. They identify potential safety risks through manual observation and take corresponding measures. However, this traditional supervision method has many problems: First, due to the vast area of the lifting operation area, it is difficult for monitoring personnel to observe every corner in real time, resulting in supervision blind spots. Second, manual supervision is easily affected by subjective factors and fatigue, and the supervision effect is unstable. In addition, traditional methods often cannot respond in a timely manner when illegal behaviors occur, which increases the risk of accidents to a certain extent. Summary of the Invention

[0004] The present invention provides a system and method for identifying illegal intrusion in ship assembly operations based on machine vision to solve the problem that the existing ship assembly supervision method in the prior art cannot respond to abnormal situations in a timely manner.

[0005] To solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0006] A system for identifying illegal intrusion in ship assembly operations based on machine vision includes: an image acquisition module, a data preprocessing module, an identification model, and an illegal intrusion determination module;

[0007] The image acquisition module is used to collect the operation video image data during the ship assembly process captured by the monitoring device;

[0008] The data preprocessing module is used to perform sample enhancement and sample annotation on the data collected by the image acquisition module; the sample annotation includes operation equipment frame annotation and personnel target annotation;

[0009] The recognition model includes a working equipment recognition model and a personnel target recognition model, and the training of the working equipment recognition model and the personnel target recognition model is completed according to historical data;

[0010] The illegal intrusion determination module uses the recognition model to perform working equipment recognition and personnel target recognition based on real-time data, adaptively calculates the prohibited area according to the working equipment, and determines whether there are personnel targets in the prohibited area to complete the recognition of illegal intrusion in ship assembly operations.

[0011] Preferably, the sample enhancement processing of the data preprocessing module includes invalid sample removal, occlusion addition, and photometric adjustment. The specific method of adding occlusion is as follows: add a left half occlusion area to the working equipment image and a lower half occlusion area to the personnel target as additional samples; the photometric adjustment is to perform photometric adjustment on images taken during the day, at dusk, or with insufficient lighting.

[0012] Preferably, the sample annotation uses the annotation tool LabelImg to annotate the targets in the sample images. For the annotation of working equipment, the pixel positions of the working equipment in the image are selected interactively by bounding boxes. When the working equipment is a crane, the hook, the lifting rope, and the lifted object are annotated as a whole; for the annotation of personnel targets, the pixel positions where the personnel targets are located in the image are selected interactively by bounding boxes.

[0013] Preferably, the sample annotation is saved in txt format, and each annotation file contains the annotation information of all targets in the corresponding image. The specific format is as follows: <object-class>, <x_center>, <y_center>, <width> 、 <height>, wherein, <object-class>is the target type, <x_center> and <y_center> are the coordinates of the center of the bounding box, <width>And <height>are the width and height of the border.

[0014] Preferably, the recognition model adopts an improved yolov7 model.

[0015] Preferably, when the illegal intrusion determination module performs adaptive calculation of the prohibited area, the midpoint of the lower boundary of the border of the operation device is used as the center point, and the length obtained by multiplying the length of the lower boundary by the value of the scaling factor R is used as the length.

[0016] To solve the above technical problems, the present invention also provides a method for identifying illegal intrusion in ship assembly operations based on machine vision. Applying the system for identifying illegal intrusion in ship assembly operations based on machine vision, it specifically includes the following steps:

[0017] Step S1: Image acquisition; using the image acquisition module to acquire the operation video image data during the ship assembly process captured by the monitoring device;

[0018] Step S2: Data preprocessing; performing sample enhancement and sample annotation on the data acquired by the image acquisition module;

[0019] Step S3: Identification of operation device; using the operation device recognition model to identify the operation device. If there is an operation device, proceed to the next step; otherwise, return to Step S1;

[0020] Step S4: Identification of personnel target; using the personnel target recognition model to identify the personnel target and performing adaptive calculation of the prohibited area according to the operation device;

[0021] Step S5: Identification of illegal intrusion; determine whether there is a personnel target in the prohibited area. If there is, issue an early warning for illegal intrusion in ship assembly operations. If not, return to Step S1 until the ship assembly operation is completed.

[0022] Further, in Step S1, the monitoring device is set diagonally above the operation area and can capture the entire process of the operation.

[0023] Further, in Step S2, the sample enhancement includes removal of invalid samples, addition of occlusion, and photometric adjustment. The addition of occlusion specifically means: adding an occlusion area in the left half of the operation device image and adding an occlusion area in the lower half of the personnel target as additional samples; the photometric adjustment is for images in daytime, evening, or insufficient light.

[0024] Further, in Step S4, when performing the adaptive calculation of the prohibited area, the midpoint of the lower boundary of the border of the operation device is used as the center point, and the length obtained by multiplying the length of the lower boundary by the value of the scaling factor R is used as the length.

[0025] Advantages of the present invention: The present invention provides a system and method for identifying illegal intrusion in ship assembly operations based on machine vision. The identification system includes an image acquisition module, a data preprocessing module, an identification model, and an illegal intrusion determination module. The method for identifying illegal intrusion in ship assembly operations includes steps such as image acquisition, data preprocessing, identification of operating equipment, identification of personnel targets, and identification of illegal intrusion, and can effectively identify illegal behaviors in the lifting operation of ship component assembly with high accuracy. The present invention has the following advantages: (1) Improved safety: By performing real-time monitoring and identification of machine vision on the ship component assembly process, the probability of illegal intrusion in the lifting operation is effectively reduced, thereby improving the overall safety of the assembly process; (2) Real-time supervision and warning: This method realizes real-time supervision of the ship component assembly process, can immediately identify and warn of illegal intrusion in the lifting operation, enabling the management personnel in the assembly workshop to make a quick response and reducing the possibility of accidents; (3) Increased efficiency: Through the application of machine vision, the cumbersome process of manual inspection in the traditional supervision mode is avoided, improving the management efficiency. The system can monitor the lifting process of assembly all-weather and without dead angles, discover problems in a timely manner, and reduce the workload of management personnel; (4) Reduced human misjudgment: The machine vision system identifies illegal behaviors based on a trained model. Compared with manual inspection, it can judge illegal behaviors more accurately and objectively, reducing the error of subjective human judgment and improving the accuracy of discrimination; (5) Cost savings: Through machine vision technology, the input of human resources can be reduced to a certain extent. While improving production efficiency, certain cost savings are also achieved. Description of the Drawings

[0026] Figure 1 Flowchart of a method for identifying illegal intrusion in ship assembly operations based on machine vision provided by the present invention;

[0027] Figure 2 Schematic diagram of the gray-scale processing result of the daytime image by the data preprocessing module;

[0028] Figure 3 Schematic diagram of the gray-scale processing result of the evening image by the data preprocessing module;

[0029] Figure 4 Schematic diagram of the gray-scale processing result of the image with insufficient light by the data preprocessing module;

[0030] Figure 5 Structural diagram of the identification model;

[0031] Figure 6 Flowchart of a method for identifying illegal intrusion in the lifting operation during the ship component assembly process based on machine vision provided by the present invention;

[0032] Figure 7 It is a recognition result diagram for illegal intrusion during the lifting operation in the assembly process of ship components. Detailed implementation manners

[0033] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.

[0034] Refer to Figure 1 , the present invention provides an illegal intrusion recognition system for ship assembly operations based on machine vision, including: an image acquisition module, a data preprocessing module, a recognition model, and an illegal intrusion determination module; the image acquisition module is used to acquire the operation video image data during the ship assembly process captured by the monitoring device; the data preprocessing module is used to perform sample enhancement and sample annotation on the data acquired by the image acquisition module; the sample annotation includes operation equipment frame annotation and personnel target annotation; the recognition model includes an operation equipment recognition model and a personnel target recognition model, and the operation equipment recognition model and the personnel target recognition model are trained according to historical data; the illegal intrusion determination module performs operation equipment recognition and personnel target recognition using the recognition model according to real-time data, adaptively calculates the prohibited area according to the operation equipment, and determines whether there is a personnel target in the prohibited area to complete the recognition of illegal intrusion during the ship assembly operation.

[0035] In the embodiments of the present invention, the image acquisition module uses video cameras such as mobile ball cameras and digital cameras to collect video data of the actual operation situation during the lifting operation in the ship assembly process. The overall data volume meets the minimum requirements for pre-algorithm modeling training. Each video material is recorded for 2 - 3 minutes, and the shooting angle is obliquely above the hook to ensure that the whole picture of the actual lifting operation is captured, and the materials with people and without people below the hook are collected.

[0036] In the embodiments of the present invention, the data preprocessing module mainly enhances the samples through ① removing invalid samples, ② occlusion, and ③ photometric adjustment to ensure that there is enough material for model building. The detailed operations are as follows:

[0037] ① Removing invalid samples: Filter out video clips with blurred images, overexposure or irrelevant scenes to ensure data quality.

[0038] ② Occlusion: Add occlusion areas to the image, occlude the left half of the hook and the lower half of the personnel body in the sample as additional samples to enhance the sample diversity and the generalization of the recognition model.

[0039] ③ Photometric adjustment: Adjust the photometric value, and adjust the photometric value under the conditions of daytime, evening, and insufficient light respectively for sample expansion.

[0040] First, obtain the luminance I from the RGB image. The picture has three RGB channels, and the picture pixels are composed of the values of three color channels: red (R), green (G), and blue (B). Preferably, in the present invention, the luminance uses the weighted average of the RGB values as the calculation method for luminance, and the picture is converted from a three-channel image to a grayscale image. The formula is as follows:

[0041] I = 0.2989×R + 0.5870×G + 0.1140×B

[0042] Among them, the weighting factors (0.2989, 0.5870, 0.1140) are determined according to the average value of the human eye's sensitivity to different colors.

[0043] Apply the following linear conversion formula to simulate the lighting conditions in different situations (such as daytime, evening, insufficient lighting):

[0044] I adjusted = I original ×α

[0045] Among them, I adjusted is the original image, I original is the image after adjusting the brightness, and α is the multiplication factor used to adjust the brightness level. For different situations, different α values are used, specifically:

[0046] Daytime: Usually, there is no need to adjust the brightness, or only slightly increase it, and set α = 1.0.

[0047] Evening: The light is dim, and the brightness needs to be reduced, and set α = 0.7.

[0048] Insufficient lighting: The brightness needs to be greatly reduced, and set α = 0.3.

[0049] For the results of luminance adjustment, refer to Figure 2 , Figure 3 , Figure 4 .

[0050] In order to identify the operation equipment and personnel targets, it is necessary to label the sample pictures. The present invention uses the labeling tool LabelImg to accurately label the targets in the collected pictures. Taking the lifting operation process as an example, the specific target labeling method is as follows:

[0051] ① Hook and load frame labeling: Use an interactive method to frame out the pixel positions of the hook and the load in the image. Subsequently, the recognition of the hook and the load is based on the content in this labeled frame for model training. In particular, during the lifting process, the hook, the lifting rope, and the load are labeled as a whole.

[0052] ② Personnel target labeling: Use an interactive method to frame out the pixel positions where the personnel target is located in the image.

[0053] In the present invention, the models adopted for lifting object recognition and personnel target recognition are improved YOLOv7 models. The input format of these two models is in txt format, where each file corresponds to an image, and the file name is the same as the image file name, but with a different extension.

[0054] Therefore, the annotation files are saved in txt format. Each annotation file contains the annotation information of all the targets in the corresponding image, and the format is as follows:

[0055] A single image sample file can contain multiple targets, and each line represents one target. Each line contains five values, and the format is: <object-class>, <x_center>, <y_center>, <width> 、 <height> 。

[0056] <object-class>is an integer representing the class index of the target. <x_center>, <y_center>, <width> 、 <height>Attributes of the target bounding box, expressed as proportional values relative to the entire image. Where <x_center> and <y_center> are the coordinates of the center of the bounding box, <width>And <height>is the width and height of the bounding box.

[0057] In the embodiments of the present invention, see Figure 5 , the recognition model is based on the pytorch framework, constructs the yolov7 architecture, adopts the SEnet attention mechanism, and improves the yolov7 model for model training. The network structure is shown in the following table:

[0058] Table 1 Network Structure

[0059]

[0060]

[0061] Among them, the input and output sizes of each layer of the network are shown in the table. CBS represents a combination of a convolutional layer, BatchNormalization, and the Swish activation function; E-ELAN represents an efficient inter-layer aggregation network for feature fusion; SENet is Squeeze-and-Excitation Networks, which adds channel attention; MP (MaxPooling) is the max pooling layer for reducing the spatial dimension of the feature map.

[0062] The loss function of the network model used in the method of the present invention includes three categories:

[0063] Coordinate loss L coord :

[0064]

[0065] Among them, (x i , y i , w i , h i ) respectively represent the attributes of the predicted bounding box, is the ground truth, indicates whether the j-th bounding box in the i-th cell contains the target.

[0066] Confidence loss L conf :

[0067]

[0068] Among them, C i is the predicted confidence, is the ground truth confidence, 1 for the target existing and 0 for non-existing, and λ noobj is the weight when the target is not included.

[0069] Class loss L class :

[0070]

[0071] p i (c) is the probability that the i-th cell predicts class c, which is the actual probability.

[0072] The total loss function is:

[0073] L = λ coord L coord + λ conf L conf + λ class L class

[0074] where λ is the weight coefficient of different loss functions. In the model used in the present invention, the coefficient λ coord = 0.15, λ conf = 0.15, λ class = 0.7.

[0075] (3) Adjust the network parameters. In the present invention, the network parameters used for training the model are shown in the following table:

[0076] Table 2 Network Parameters

[0077] Input Size 416,416 Data Augmentation resize, pad, normalize Training Epochs 300 Batch Size 100 Learning Rate Initial value 0.001, weight decay (1, 0.0, 0.001) Optimizer SGD

[0078] In the embodiment of the present invention, the illegal intrusion determination module includes job equipment identification, personnel target identification, prohibited area adaptive calculation, and illegal intrusion determination. Taking the lifting operation as an example, the job equipment identification includes crane load identification. The input image is detected and identified by the trained crane lifting heavy object model to determine whether a lifting operation is currently in progress. If a lifting operation is in progress, personnel target detection is performed. The personnel target identification detects and identifies the input image through the trained personnel target detection model to determine whether there are personnel in the current image. If a lifting operation is in progress and the image area contains a personnel target, the center point of the coordinate frame of each person is used to represent the position of the person. The prohibited area adaptive calculation generates an adaptive area coordinate based on the target frame of the hook lifting the heavy object. Specifically, the midpoint of the lower boundary of the target frame of the crane hook lifting the heavy object is used as the center point, and the length of the lower boundary multiplied by the value of the scaling factor R is used as the length.

[0079] In the embodiment of the present invention, the scaling factor R is set to 1.2.

[0080] In the embodiment of the present invention, the illegal intrusion determination performs a matching judgment based on the coordinates of the personnel target center point and the prohibited area to determine whether the center point of the personnel is within the prohibited area. If the match is successful, it is determined that there is a person under the lifted heavy object, triggering an early warning for illegal intrusion of the person under the heavy object.

[0081] See Figure 6 To solve the above technical problems, the present invention also provides a method for identifying illegal intrusion in ship assembly operations based on machine vision, which applies a system for identifying illegal intrusion in ship assembly operations based on machine vision, and specifically includes the following steps:

[0082] Step S1: Image acquisition; using an image acquisition module to acquire the video image data of the ship assembly process captured by a monitoring device;

[0083] Step S2: Data preprocessing; performing sample enhancement and sample annotation on the data acquired by the image acquisition module;

[0084] Step S3: Identification of operation equipment; using an operation equipment identification model to identify the operation equipment. If there is operation equipment, proceed to the next step; otherwise, return to Step S1;

[0085] Step S4: Identification of personnel targets; using a personnel target identification model to identify personnel targets and performing adaptive calculation of prohibited areas according to the operation equipment;

[0086] Step S5: Identification of illegal intrusion; determining whether there are personnel targets in the prohibited area. If there are, issue an early warning of illegal intrusion in ship assembly operations. If not, return to Step S1 until the ship assembly operation is completed.

[0087] Further, in Step S1, the monitoring device is set diagonally above the operation area and can capture the entire process of the operation.

[0088] Further, in Step S2, sample enhancement includes removing invalid samples, adding occlusions, and photometric adjustment. Adding occlusions specifically means: adding a left half occlusion area to the operation equipment image and adding a lower half occlusion area to the personnel target as additional samples; photometric adjustment is performed on images during the day, evening, or with insufficient light.

[0089] Further, in Step S4, when performing adaptive calculation of the prohibited area, the midpoint of the lower boundary of the operation equipment border is used as the center point, and the length of the lower boundary multiplied by the value of the scaling factor R is used as the length.

[0090] See Figure 7 The system and method for identifying illegal intrusion in ship assembly operations based on machine vision provided by the present invention can effectively identify illegal behaviors in ship component assembly operations with high accuracy, and specifically have the following beneficial effects:

[0091] (1) Improvement in safety: By performing real-time monitoring and identification of machine vision on the ship component assembly process, the probability of illegal intrusion in lifting operations is effectively reduced, thereby improving the overall safety of the assembly process.

[0092] (2) Real-time supervision and warning: This method realizes the real-time supervision of the ship component assembly process, can immediately identify and warn against any violations in the lifting operation, enabling the management personnel in the assembly workshop to respond quickly and reducing the likelihood of accidents.

[0093] (3) Efficiency improvement: Through the application of machine vision, the cumbersome process of manual inspection in the traditional supervision mode is avoided, and the management efficiency is improved. The system can monitor the assembly and lifting process all-weather and without dead angles, detect problems in a timely manner, and reduce the workload of management personnel.

[0094] (4) Reduction of human misjudgment: The machine vision system identifies violations based on a trained model. Compared with manual inspection, it can judge violations more accurately and objectively, reducing the error of subjective human judgment and improving the accuracy of discrimination.

[0095] (5) Cost savings: Through machine vision technology, the input of human resources can be reduced to a certain extent. While improving production efficiency, certain cost savings are also achieved.

[0096] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.< / height> < / width> < / height> < / width> < / height> < / width> < / height> < / width> < / height> < / width>

Claims

1. A system for identifying illegal intrusion in ship assembly operations based on machine vision, characterized in that, it includes: an image acquisition module, a data preprocessing module, an identification model, and an illegal intrusion determination module; the image acquisition module is used to acquire the operation video image data during the ship assembly process captured by the monitoring device; the data preprocessing module is used to perform sample enhancement and sample annotation on the data collected by the image acquisition module; the sample annotation includes operation equipment box annotation and personnel target annotation; the identification model includes an operation equipment identification model and a personnel target identification model, and the operation equipment identification model and the personnel target identification model are trained according to historical data; the illegal intrusion determination module performs operation equipment identification and personnel target identification using the identification model based on real-time data, adaptively calculates the prohibited area according to the operation equipment, and determines whether there are personnel targets in the prohibited area to complete the identification of illegal intrusion in ship assembly operations.

2. The system for identifying illegal intrusion in ship assembly operations based on machine vision according to claim 1, characterized in that, the sample enhancement process of the data preprocessing module includes removing invalid samples, adding occlusions, and photometric adjustment. The specific method of adding occlusions is: adding a left half occlusion area to the operation equipment image and adding a lower half occlusion area to the personnel target as additional samples; the photometric adjustment is to perform photometric adjustment on images during the day, evening, or with insufficient light.

3. The system for identifying illegal intrusion in ship assembly operations based on machine vision according to claim 2, characterized in that, the sample annotation uses the annotation tool LabelImg to annotate the targets in the sample images. For the annotation of operation equipment, the pixel points of the operation equipment in the image are selected by interactive means. When the operation equipment is a crane, the hook, the lifting rope, and the lifted object are annotated as a whole; for the annotation of personnel targets, the pixel points where the personnel targets are located in the image are selected by interactive means.

4. The system for identifying illegal intrusion in ship assembly operations based on machine vision according to claim 3, characterized in that, The sample annotations are saved in txt format, and each annotation file contains the annotation information of all the targets in the corresponding image. The specific format is as follows: <object-class>, <x_center>, <y_center>, <width> 、 <height>, wherein, <object-class>is the target type, <x_center> and <y_center> are the center coordinates of the bounding box, <width>And <height>is the width and height of the border. < / height> < / width> < / height> < / width> 5. The system for identifying illegal intrusion in ship assembly operations based on machine vision according to any one of claims 1-4, characterized in that, the identification model uses an improved yolov7 model.

6. The system for identifying illegal intrusion in ship assembly operations based on machine vision according to claim 5, characterized in that, when the illegal intrusion determination module performs adaptive calculation of the prohibited area, the midpoint of the lower boundary of the operation equipment border is used as the center point, and the length of the lower boundary multiplied by the value of the scaling factor R is used as the length.

7. A method for identifying illegal intrusion in ship assembly operations based on machine vision, characterized in that, applying the system for identifying illegal intrusion in ship assembly operations based on machine vision according to claim 6, specifically including the following steps: Step S1: Image acquisition; using the image acquisition module to acquire the operation video image data during the ship assembly process captured by the monitoring device; Step S2: Data preprocessing; performing sample enhancement and sample annotation on the data collected by the image acquisition module; Step S3: Identification of operating equipment; Use the operating equipment identification model to identify the operating equipment. If there is operating equipment, proceed to the next step; otherwise, return to Step S1. Step S4: Identification of personnel targets; Use the personnel target identification model to identify personnel targets and perform adaptive calculation of prohibited areas based on the operating equipment. Step S5: Identification of unauthorized entry; Determine whether there are personnel targets in the prohibited area. If there are, issue a warning for unauthorized entry into ship assembly operations. If not, return to Step S1 until the ship assembly operation is completed.

8. The system for identifying unauthorized entry into ship assembly operations based on machine vision according to claim 7, characterized in that In Step S1, the monitoring device is set diagonally above the operation area and can collect the entire process of the operation.

9. The method for identifying unauthorized entry into ship assembly operations based on machine vision according to claim 8, characterized in that In Step S2, the sample enhancement includes removal of invalid samples, addition of occlusions, and photometric adjustment. The addition of occlusions specifically means: adding an occlusion area in the left half of the operating equipment image and adding an occlusion area in the lower half of the personnel target as additional samples; the photometric adjustment is for images in daytime, evening, or with insufficient lighting.

10. The method for identifying unauthorized entry into ship assembly operations based on machine vision according to claim 9, characterized in that In Step S4, when performing the adaptive calculation of the prohibited area, the midpoint of the lower boundary of the operating equipment border is used as the center point, and the value obtained by multiplying the length of the lower boundary by the scaling factor R is used as the length.

Images