An intelligent cabin key equipment identification method based on foreground and background fusion

Abstract

The application discloses a kind of key equipment identification method based on foreground and background fusion, comprising the following steps: the detection and identification of foreground and background target are carried out;Feature extraction and image matching are carried out;The spatial topological relation of the foreground and background of the i-th equipment is constructed;Realize the accurate identification of target.The application establishes the contact of foreground target and background target by using the spatial topological relation of foreground target and background target, and the foreground equipment target is evidenced using background information, so that the accurate identification of foreground target can be realized.The application is not only suitable for large space scale system such as ship engine room, but also suitable for small space scale target identification such as electronic components.Compared with the CSP module in YOLOV5, the Ghostbottlenet module of the application not only reduces the model volume, but also effectively eliminates redundant features to obtain a lighter model, making up for the incomplete shortcomings of YOLOV5 detail features.

Description

Technical Field

[0001] This invention belongs to the field of intelligent operation and maintenance of surface ship engine room equipment, and particularly relates to a method for identifying key equipment in intelligent engine room based on foreground and background fusion. Background Technology

[0002] Intelligent engine room is one of the eight core functions of intelligent ships, and intelligent operation and maintenance (O&M) is key to the intelligence of surface ship engine rooms. Existing intelligent O&M systems rely on numerous sensors installed at various locations inside and outside the equipment to measure operational status data such as pressure, temperature, and flow. This data enables functions like condition monitoring, health assessment, condition-based maintenance, fault diagnosis, and decision support for critical equipment, thereby improving the stability and reliability of the ship's propulsion system and achieving efficiency gains through crew reduction. However, these sensors have limited sensing range, and the data is multi-source, redundant, and difficult to process. Visual sensors, on the other hand, have become widely used in the naval field in recent years. They can sense the status of multiple devices, offering advantages such as wide sensing range, large information capacity, and high transmission efficiency.

[0003] The primary challenge in using visual sensors for ship status monitoring is equipment identification. However, ship engine rooms contain numerous similar devices, such as valves, pipelines, switches, and pumps. Current equipment identification methods can identify one or more types of targets, but cannot distinguish between those with similar or identical shapes and colors. Furthermore, variations in background and foreground information due to different viewing angles can affect the accuracy of target identification during visual recognition. Therefore, sophisticated identification methods for key equipment in intelligent engine rooms have significant application value in the field of intelligent operation and maintenance. Summary of the Invention

[0004] To address the problems existing in the identification of ship engine room equipment using current target recognition technologies, this invention provides a key equipment identification method based on foreground and background fusion, which can also incorporate the background into the identification process, thereby providing more refined identification results.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: a key device identification method based on foreground and background fusion, comprising the following steps:

[0006] Step A: Detect and identify foreground and background targets.

[0007] The steps for detecting and identifying foreground and background targets in real-time surveillance images captured from the cabin are as follows:

[0008] Step A1: Select the cabin equipment monitoring images and convert the data acquired by the visual sensor into a time-varying image sequence {I1,I2,…,I…} t,...}, where t is the sequential index of the image sequence, is a positive integer, is related to the frame rate of the visual sensor, and the number of images is N, forming sample one;

[0009] Step A2: Label the locations and categories of key equipment in Sample 1 to form training samples;

[0010] Step A3: Feed the training samples obtained in step A2 into the deep learning model for training.

[0011] Step A4: Input the real-time captured intelligent cabin monitoring images into the deep learning model trained in Step A3 to obtain the mapping relationship between the location and type of all devices in the current monitoring image and the input image:

[0012] [x i ,y i ,w i ,h i ,label i ]=f(I) (1)

[0013] In the formula, x i ,y i ,w i ,h i These are the x-coordinate, y-coordinate, width, and height of the top-left corner of the bounding box of the i-th device, respectively; label i Let I be the category of the i-th device, i = 1, ..., K, where K is the total number of detected devices, and I is the input image.

[0014] Step A5: Extract the device images detected and classified in step A4.

[0015] Step B: Perform feature extraction and image matching

[0016] The steps for feature extraction and image matching of similar devices in different images are as follows:

[0017] Step B1: Select the i-th device from step A5 as the foreground target;

[0018] Step B2: Extract features from the foreground target and other images of devices belonging to the same category as the foreground target;

[0019] Step B3: Match the extracted image features and output the matching results.

[0020] Step B4: Based on the matching results, determine the set of images that exist on the same device, and calculate the corresponding spatial transformation matrix M;

[0021] Step C: Construct the spatial topological relationship between the foreground and background of the i-th device.

[0022] Construct a spatial topological relationship description matrix R between the foreground and background of the i-th device in the same set of images;

[0023] Step D: Achieve accurate target identification

[0024] Based on the spatial topological relationship description, topological relationship matching is performed to achieve accurate target identification. The steps are as follows:

[0025] Step D1: Using the spatial transformation matrix M calculated in step B4, invert the background target and transform it to the same coordinate system as the reference image;

[0026] Step D2: Take any two devices from the device image set. If they are the same device, the resulting background target position will be in a position similar to the background target in the reference image after transformation. Otherwise, it is not necessarily the case. Therefore, if the background target position after inversion has similar membership, the foreground target is considered to be the same target as the foreground target in the reference image. Otherwise, they are considered not to be the same target.

[0027] Furthermore, the deep learning model is an improved YOLOv5 model. This improved YOLOv5 model is derived by improving the accuracy of cabin equipment recognition based on the original YOLOv5 model. Ghostbottlenet is selected as the main network for the feature extraction part of the YOLOv5 network, replacing the original CSP module. Ghostbottlenet consists of two stacked GhostNet layers. The first GhostNet is used as an extension layer to increase the number of channels, while the second GhostNet reduces the number of channels to match the feature dimension. The second GhostNet applies batch normalization and the ReLU activation function after each layer.

[0028] Furthermore, the feature extraction and image matching method is the Scale Invariant Feature Transform (SIFT) algorithm. The SIFT algorithm includes extracting key points, locating key points and determining feature directions, comparing the feature vectors of each key point pairwise to find several pairs of matching feature points, and establishing the correspondence between targets.

[0029] Furthermore, the spatial topological relationship description between the foreground and background is based on fuzzy mathematics theory. Based on human experience, the spatial orientation and distance around the foreground target are classified to form spatial fuzzy sets, namely {left, right}, {up, down}, and {far, near}. Combining these three fuzzy sets through reasoning yields a fuzzy topological structure covering the entire image space. Simultaneously, the membership degree of the identified background target is defined according to its position within the entire image. That is, multiple combinations of identified targets exist, but each combination has a different membership degree. This establishes a connection between the spatial topological structure of the background target and the foreground target in the reference image, enabling accurate target identification by fusing foreground and background information during image recognition. Based on this, the constructed spatial topological relationship description matrix R is represented as:

[0030] R=[a,b,c,α] (2)

[0031] Where a represents left and right, -1 represents left, 0 represents middle, and 1 represents right; b represents up and down, 1 represents up, 0 represents middle, and -1 represents down; c represents near and far, 1 represents far, 0 represents near, and -1 represents near; α represents the membership degree of the current spatial topology.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. This invention establishes a connection between foreground and background targets by utilizing their spatial topological relationship and using background information to corroborate the foreground targets, thereby enabling accurate identification of the foreground targets.

[0034] 2. This invention is suitable not only for large-scale systems such as ship engine rooms, but also for small-scale target recognition such as electronic components.

[0035] 3. The Ghostbottlenet module of this invention is a reusable module. Compared with the cumbersome CSP module in YOLOv5, it not only reduces the model size, but also the new feature map can effectively eliminate redundant features to obtain a lighter model, making up for the incomplete detailed features of YOLOv5. Attached Figure Description

[0036] Figure 1 This is a flowchart of the device fine identification method of the present invention;

[0037] Figure 2 This is the improved YOLOv5 network model in this invention.

[0038] Figure 3 This is a schematic diagram illustrating the principle of the SIFT-based image matching algorithm in this invention.

[0039] Figure 4This is an example of SIFT-based device matching in this invention.

[0040] Figure 5 This is the spatially fuzzy topology centered on the foreground in this invention.

[0041] Figure 6 This is an example of the high-temperature freshwater pump in the engine room of the "Yukun" vessel of Dalian Maritime University in this invention. Detailed Implementation

[0042] The detailed content and technology of this invention are described below in conjunction with the accompanying drawings. Obviously, the described embodiments are merely some, not all, embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention are within the scope of protection of this invention.

[0043] The flowchart of this invention is as follows Figure 1 As shown, the specific steps include:

[0044] Step A: Detect and identify foreground and background targets.

[0045] The steps for detecting and identifying foreground and background targets in real-time surveillance images captured from the cabin are as follows:

[0046] Step A1: Select the cabin equipment monitoring images and convert the data acquired by the visual sensor into a time-varying image sequence {I1,I2,…,I…} t ,...}, where t is the sequential index of the image sequence, is a positive integer, is related to the frame rate of the visual sensor, and the number of images is N, forming sample one;

[0047] Step A2: Label the locations and categories of key equipment in Sample 1 to form training samples;

[0048] Step A3: Feed the training samples obtained in step A2 into the YOLOv5 model for training.

[0049] In this step, the improved YOLOv5 model is derived by enhancing the recognition accuracy of cabin equipment based on the original YOLOv5 model. This primarily addresses the uneven distribution density of image features caused by varying equipment sizes and the presence of multiple devices within the same image. The Ghostbottlenet module was selected as the main network for the feature extraction part of the YOLO V5 network, replacing the original CSP module. Figure 2As shown, the Ghostbottlenet module consists of two stacked GhostNet layers. The first GhostNet is used as an expansion layer to increase the number of channels, while the second GhostNet reduces the number of channels to match the feature dimensions. The second GhostNet applies batch normalization and ReLU activation after each layer. The Ghostbottlenet module is a reusable module that, compared to the cumbersome CSP module in YOLO V5, not only reduces the model size but also allows the new feature maps to effectively eliminate redundant features, resulting in a lighter model and compensating for the incomplete detailed features of YOLO V5.

[0050] Step A4: Input the real-time captured intelligent cabin monitoring images into the YOLOV5 model that has been trained in step A3 to obtain the category and location information of all devices in the current frame.

[0051] Step A5: Extract the device images detected and classified in step A4 to form a foreground device set, which will be used for subsequent matching.

[0052] Step B: Perform feature extraction and image matching

[0053] The steps for feature extraction and image matching of similar devices in different images are as follows:

[0054] Step B1: Select the i-th device from step A5 as the foreground target;

[0055] Step B2: Extract SIFT features from the foreground target and other images of devices belonging to the same category as the foreground target;

[0056] In this step, SIFT feature extraction mainly involves the following three processes: extracting key points, locating key points and determining feature directions, and comparing the feature vectors of each key point pairwise to find several pairs of matching feature points and establish the correspondence between targets.

[0057] Step B3: Perform foreground matching between the extracted SIFT image features and the prior device image, and output the matching results.

[0058] In this step, SIFT algorithm feature matching is widely used in real-world scenarios. Figure 3 This is a typical illustration of SIFT-based image matching. Figure 4 This is an example of SIFT-based device matching. Through further feature extraction and image matching, it is possible to identify similar devices with certain morphological differences, building upon object detection.

[0059] Step B4: Based on the matching results, determine the set of images that exist in the same device, obtain the set of matching prior device images, and calculate the corresponding spatial transformation matrix M;

[0060] Step C: Construct the spatial topological relationship between the foreground and background of the i-th device.

[0061] For the same set of images of a device, construct a spatial topological relationship description matrix R between the foreground and background of the i-th device;

[0062] In this step, by using fuzzy mathematics to describe the space surrounding the foreground target, we can obtain the following: Figure 5 The diagram illustrates a fuzzy spatial topology. This topology categorizes the spatial orientation and distance of foreground targets based on human experience, forming spatial fuzzy sets: {left, right}, {top, bottom}, and {far, near}. Reasoning using these three fuzzy sets yields a fuzzy topology covering the entire image space. Simultaneously, the membership degree of the identified background target can be defined based on its position within the entire image. Figure 6 Taking the high-temperature freshwater pump in the engine room of the "Yukun" vessel at Dalian Maritime University as an example, the high-temperature freshwater pump is considered a foreground target in the standard image. The solenoid valve in the red box on the left and the valve in the red box on the lower right can both be considered background targets. Therefore, the membership degree of the solenoid valve in the red box on the left can be marked as 0.5 for "far left" and "near left"; the membership degree of the valve in the red box on the lower right can be marked as 0.5 for "near lower right", "near lower down", and "far lower right".

[0063] Step D: Achieve accurate target identification

[0064] Based on the spatial topological relationship description, topological relationship matching is performed to achieve accurate target identification. The steps are as follows:

[0065] Step D1: Using the spatial transformation matrix M of the current image calculated in step B4, invert the solenoid valve and other equipment in the captured image and transform them to the same coordinate system as the prior reference image;

[0066] Step D2: Take any two devices from the device image set. If they are the same device, the resulting background target position will be in a position similar to the background target in the reference image after transformation. Otherwise, it is not necessarily the case. Therefore, if the background target position after inversion has similar membership, the foreground target is considered to be the same target as the foreground target in the reference image. Otherwise, they are considered not to be the same target.

[0067] This establishes a connection between the spatial topology of background and foreground targets in a reference image, enabling accurate target identification by fusing foreground and background information during image recognition. In summary, this invention utilizes the spatial topological relationship between foreground and background targets to establish a connection, and uses background information to corroborate foreground targets, thus achieving accurate foreground target identification. This type of application is suitable not only for large-scale systems such as ship engine rooms but also for small-scale targets such as electronic components, showing broad application prospects.

[0068] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A key device identification method based on foreground and background fusion, characterized in that: Includes the following steps: Step A: Detect and identify foreground and background targets. The steps for detecting and identifying foreground and background targets in real-time surveillance images captured from the cabin are as follows: Step A1: Select the cabin equipment monitoring images and convert the data acquired by the visual sensor into an image sequence that changes over time. ,in t The image sequence index is a positive integer, related to the frame rate of the visual sensor, and the number of images is N, forming sample one; Step A2: Label the locations and categories of key equipment in Sample 1 to form training samples; Step A3: Feed the training samples obtained in step A2 into the deep learning model for training; Step A4: Input the real-time captured intelligent cabin monitoring images into the deep learning model trained in Step A3 to obtain the mapping relationship between the location and type of all devices in the current monitoring image and the input image: In the formula, The first i The top-left x-coordinate, y-coordinate, width, and height of the bounding box of each device; For the first i Each category of equipment i =1,…, K , K The total number of devices detected. I Input image; Step A5: Extract the device images detected and classified in Step A4; Step B: Perform feature extraction and image matching The steps for feature extraction and image matching of similar devices in different images are as follows: Step B1: Take the first number from step A5 in sequence. i One device is designated as a prospective target; Step B2: Extract features from the foreground target and other images of devices belonging to the same category as the foreground target; Step B3: Match the extracted image features and output the matching results; Step B4: Based on the matching results, determine the set of images that share the same device, and calculate the corresponding spatial transformation matrix M; Step C: Construct the first i Spatial topological relationship between the foreground and background of each device Construct the first image set of the same device i Spatial topological relationship description matrix of the foreground and background of each device R ; Step D: Achieve accurate target identification Based on spatial topological relationships, topological relationship matching is performed to achieve accurate target identification. The steps are as follows: Step D1: Using the spatial transformation matrix M calculated in step B4, invert the background target and transform it to the same coordinate system as the reference image; Step D2: Take any two devices from the device image set. If they are the same device, the resulting background target position will be in a position similar to the background target in the reference image after transformation. Otherwise, it is not necessarily the case. Therefore, if the background target position after inversion has similar membership, the foreground target is considered to be the same target as the foreground target in the reference image. Otherwise, they are considered not to be the same target.

2. The key equipment identification method based on foreground and background fusion according to claim 1, characterized in that: The deep learning model is an improved YOLOv5 model. The improved YOLOv5 model is obtained by improving the accuracy of cabin equipment recognition based on the original YOLOv5 model. Ghostbottlenet is selected as the main network of the YOLOv5 network feature extraction part to replace the original CSP module. Ghostbottlenet consists of two stacked GhostNets. The first GhostNet is used as an extension layer to increase the number of channels, and the second GhostNet reduces the number of channels to match the feature dimension. The second GhostNet applies batch normalization and the ReLU activation function after each layer.

3. The key equipment identification method based on foreground and background fusion according to claim 1, characterized in that: The feature extraction and image matching method is the Scale Invariant Feature Transform (SIFT) algorithm. The SIFT algorithm includes extracting key points, locating key points and determining feature directions, comparing the feature vectors of each key point pairwise to find several pairs of matching feature points, and establishing the correspondence between targets.

4. The key equipment identification method based on foreground and background fusion according to claim 1, characterized in that: The description of the spatial topological relationship between the foreground and background is based on fuzzy mathematics theory. According to human experience, the spatial orientation and distance around the foreground target are classified to form spatial fuzzy sets, namely {left, right}, {up, down}, and {far, near}. The three fuzzy sets are combined by reasoning to obtain a fuzzy topological structure covering the entire image space. At the same time, the membership degree of the identified background target is defined according to its position in the whole image. That is, the identified targets exist in multiple combinations, but the membership degrees of each combination are different; in this way, a relationship is established between the spatial topological structure of background targets and foreground targets in the reference image, enabling accurate target identification by fusing foreground and background information during image recognition; based on this, a spatial topological relationship description matrix is ​​constructed. R Represented as: (2) in, a Represents left and right, -1 for left, 0 for middle, and 1 for right; b The numbers represent top and bottom, with 1 representing top, 0 representing middle, and -1 representing bottom. c Representing distance, 1 represents far, 0 represents near, and -1 represents near; Represents the membership degree of the current spatial topology.

Images