Bulk carrier ship cleaning operation multi-target identification method based on visual intelligent analysis

Abstract

The application provides a bulk carrier cleaning operation multi-target identification method based on visual intelligent analysis, and relates to the technical fields of computer vision and port operation monitoring. It comprises: performing target detection on the operation image sequence to obtain candidate targets of cleaning machinery and cleaning personnel, and extracting the operation state and spatial position of the cleaning machinery; generating a dynamic degradation distribution map based on the operation state of the cleaning machinery through a visual degradation field prediction model to evaluate the visual attenuation characteristics of each spatial position; taking the spatial position and visual attenuation characteristics of the cleaning machinery as conditions, extracting spatio-temporal correlation features and environmental degradation features for semantic reasoning, and calculating the cooperative relationship confidence through dual-source evidence fusion; using the cooperative relationship confidence to correct the detection confidence of the cleaning personnel, and outputting the cooperative operation unit and its confidence after verification. The application can realize reverse reasoning of the man-machine cooperative relationship in the cleaning visual degradation field, thereby improving the robustness of multi-target identification in harsh working conditions.

Description

Technical Field

[0001] This application relates to the fields of computer vision and port operation monitoring technology, and more specifically, to a multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis. Background Technology

[0002] Computer vision technology has been introduced into the cleaning operations of bulk carriers to detect and identify targets such as cleaning machinery, personnel, and bulk cargo piles in the bilge area. During the cleaning operation, loaders, bulldozers, and other machinery and cleaning personnel form a mixed human-machine working environment in the confined bilge, which is narrow and has a high workload. By using explosion-proof industrial cameras deployed at the hatch opening to collect bilge image sequences, and using deep convolutional neural networks to detect cleaning machinery and personnel in the images, the localization and classification of multiple targets can be achieved, providing visual information support for the safety monitoring and progress management of the cleaning operation.

[0003] However, the harsh working conditions unique to bulk carrier cleaning operations pose a severe challenge to existing visual multi-target recognition methods. During shoveling and unloading operations, the auxiliary components of the cleaning machinery come into intense contact with the hold floor, generating large amounts of dust that spreads and creates localized blurring and reduced contrast areas in the image. Simultaneously, the extensive posture changes of these components constantly alter the distribution of incident light from the hatch onto the hold floor, causing localized areas of underexposure or overexposure. The coupling effect of dust and lighting changes creates a dynamic visual degradation field on the image plane. Cleaning personnel targets in severely degraded areas show significantly reduced response in detectors, leading to a substantial decrease in detection confidence and frequent missed detections. Existing methods treat all such visual degradation as interference factors, attempting to compensate through image enhancement or dust removal preprocessing, but completely ignore the cleaning operation status information inherent in the spatial distribution of the visual degradation field itself. In fact, the more severe the visual degradation in the area where the cleaning personnel are located, the more it indicates that the personnel are within the high-intensity operating radius of the cleaning machinery, and the greater the likelihood of a collaborative working relationship between the two. Existing methods have failed to transform this inverse correspondence into auxiliary clues for inferring collaborative relationships, resulting in severely insufficient robustness in identifying human-machine collaborative relationships under harsh working conditions. Therefore, how to achieve inverse reasoning of human-machine collaborative relationships from the visual degradation field of cleaning, thereby improving the robustness of multi-target identification under harsh working conditions, has become a challenge for the industry. Summary of the Invention

[0004] This application provides a multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligence analysis, which can realize reverse reasoning of human-machine collaboration relationship in the visual degradation field of tank cleaning, thereby improving the robustness of multi-target recognition under harsh working conditions.

[0005] In a first aspect, this application provides a multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis, comprising the following steps: Acquire operational image sequences of the bilge area of ​​a bulk carrier; Target detection is performed on the operation image sequence to obtain candidate targets for cleaning machinery and personnel, and the operation posture, action state and spatial position of the candidate targets for cleaning machinery are extracted. Based on the working posture and action state of the candidate cleaning machinery, a pre-constructed visual degradation field prediction model is used to predict the dynamic degradation distribution map of dust and illumination changes caused by the cleaning operation on the image plane. Then, the visual attenuation characteristics of different spatial locations in the operation image sequence affected by dust and illumination changes are evaluated based on the dynamic degradation distribution map. Using the spatial location of the candidate targets for the cleaning machinery as a reference and the visual attenuation feature as a spatial condition, calculate the confidence level of the cooperative relationship between each candidate target for cleaning personnel and the candidate target for cleaning machinery. The detection confidence of candidate targets for clearing personnel is corrected based on the confidence of the cooperative relationship, and after verification of the cooperative relationship, the multi-target recognition result containing the cooperative operation unit and its confidence is output.

[0006] In this embodiment, an explosion-proof industrial camera is used to capture continuous image frames of the bilge cleaning operation at a preset frame rate, thereby obtaining an image sequence of the operation in the bilge area of ​​the bulk carrier.

[0007] In this embodiment, target detection is performed on the operation image sequence to obtain candidate targets for cleaning machinery and personnel, specifically including: The current frame in the operation image sequence is input into a pre-trained deep convolutional neural network detector to extract candidate regions of the cleaning machinery and personnel in the image. Each candidate region is classified and bounding box regressed, and the bounding boxes, category labels, and detection confidence scores of the candidate targets for cleaning machinery and personnel are output.

[0008] In this embodiment, the operational posture, motion state, and spatial location of the candidate clearing machinery are extracted, specifically including: Joint point detection is performed on the image region corresponding to the candidate target of the cleaning machinery to obtain the joint point position of the auxiliary working parts of the cleaning machinery; The attitude angle of the auxiliary working components is calculated based on the position of the joint points, and the current action state of the cleaning machinery is determined according to the displacement changes of the joint points between consecutive frames and the preset action state discrimination rules. By combining camera calibration parameters, the pixel coordinates of the candidate targets for cleaning machinery in the image are converted into three-dimensional spatial positions within the ship's working space.

[0009] In this embodiment, based on the working posture and action state of the candidate cleaning machinery, a pre-constructed visual degradation field prediction model is used to predict the dynamic degradation distribution map of dust and illumination changes caused by the cleaning operation on the image plane, specifically including: Extract the motion amplitude and direction of the auxiliary working parts of the cleaning machinery in the current operating state, as well as the contact state between the auxiliary working parts of the cleaning machinery and the bottom plate of the ship, as driving parameters of the visual degradation field; The driving parameters are input into a pre-constructed visual degradation field prediction model, which includes a dust diffusion sub-model and an illumination change sub-model. The dust diffusion sub-model takes the dust origin location and dust diffusion parameters as input and outputs a dust degradation intensity value that characterizes the degree of visual degradation caused by dust to each pixel of the image. The illumination change sub-model takes the mechanical component pose and bulkhead structure parameters as input and outputs an illumination degradation intensity value that characterizes the degree of visual degradation caused by illumination change to each pixel of the image. By combining the dust degradation intensity value and the illumination degradation intensity value, a visual degradation intensity value that characterizes the degree of visual degradation caused by dust or illumination changes to image pixels is determined, and the dynamic degradation distribution map is generated.

[0010] In this embodiment, the visual attenuation characteristics of different spatial locations in the operational image sequence affected by dust and illumination changes are evaluated based on the dynamic degradation distribution map, specifically including: The image plane of the job image sequence is divided into several grid units; Based on the dynamic degradation distribution map, the mean value of the visual degradation intensity of pixels in each grid cell is calculated, and the mean value is used as the visual attenuation feature of the corresponding spatial location.

[0011] In this embodiment, using the spatial location of the candidate targets for the cleaning machinery as a reference and the visual attenuation feature as a spatial condition, the confidence level of the cooperative relationship between each candidate target for cleaning personnel and the candidate target for cleaning machinery is calculated, specifically including: Calculate the spatial distance between the cleaning machinery and the cleaning personnel based on their spatial locations and the spatial locations of the candidate cleaning machinery and personnel. Based on the motion trajectories of the cleaning machinery and personnel between consecutive frames, determine the degree of consistency in the relative motion directions between the cleaning machinery and personnel. The consistency between the spatial distance and the relative motion direction is used as an explicit spatiotemporal correlation feature. Based on the explicit spatiotemporal correlation feature, explicit semantic reasoning is performed on the collaborative operation relationship between the cleaning machinery and the cleaning personnel to obtain an explicit collaboration index that characterizes the degree of spatiotemporal collaboration that can be directly observed. The visual attenuation features corresponding to the spatial location of each candidate target for clearing the cabin are obtained as implicit environmental features. Based on the implicit environmental features, implicit semantic reasoning is performed on the visual perception attenuation caused by dust and changes in light to obtain implicit environmental indicators that characterize the degree of impact of environmental degradation on personnel detection. By performing semantic reasoning on the explicit collaborative indicators and the implicit environmental indicators, and combining them with the preset safety rules for collaborative cleaning operations, the confidence level of the collaborative relationship between each candidate target for cleaning personnel and candidate target for cleaning machinery is determined.

[0012] In this embodiment, the explicit collaboration indicators and the implicit environmental indicators are correlated through semantic reasoning. Combined with preset safety rules for collaborative cleaning operations, the confidence level of the collaboration relationship between candidate targets for cleaning personnel and candidate targets for cleaning machinery is determined. Specifically, this includes: By using the explicit collaborative indicators and the implicit environmental indicators as two independent sources of evidence, an evidence fusion framework for determining collaborative relationships is constructed. When the explicit coordination indicator suggests a spatiotemporal coordination relationship between the cleaning machinery and the cleaning personnel, and the original detection confidence level of the candidate personnel is lower than a preset confidence threshold, the implicit environmental indicator is introduced for evidence conflict analysis: If the collaborative behavior pattern indicated by the explicit collaborative indicator conforms to the human-machine collaborative operation behavior characteristics defined in the preset cabin clearing collaborative operation safety rules, and the visual attenuation intensity indicated by the implicit environmental indicator is higher than the preset attenuation threshold, then the evidence conflict between the explicit collaborative indicator and the original detection confidence is determined to be caused by environmental degradation. A confidence redistribution strategy is adopted to suppress the weight of the original detection confidence in the collaborative relationship determination, and the collaborative confidence calculated based on the explicit collaborative indicator is used as the collaborative relationship confidence.

[0013] In this embodiment, the trust redistribution strategy includes: Based on the visual attenuation intensity indicated by implicit environmental indicators, a confidence discount factor is determined for the original detection confidence level. The visual attenuation intensity and the confidence discount factor have a preset positive correlation mapping relationship. The trust discount factor is used to discount the basic probability allocation value of the original detection confidence in the determination of collaborative relationship, thereby reducing the trust contribution of the original detection confidence in evidence fusion. The trust shares released by the discount process are redistributed to the propositions supporting the validity of cooperative relationships supported by explicit cooperative indicators, resulting in a modified basic probability allocation. Based on the corrected basic probability allocation, the confidence level of the cooperative relationship between each candidate target of the cleaning personnel and the candidate target of the cleaning machinery is calculated, so that the confidence level of the cooperative relationship is dominated by explicit cooperative indicators under the interference of environmental degradation.

[0014] In this embodiment, the detection confidence of candidate targets for clearing personnel is corrected based on the confidence of the cooperative relationship, and after verification of the cooperative relationship, a multi-target recognition result containing the cooperative work unit and its confidence is output, specifically including: The original detection confidence of the candidate targets for clearing personnel is replaced with the confidence of the cooperative relationship to obtain the corrected personnel detection results; The candidate targets for cleaning personnel whose confidence level is higher than the preset confidence threshold after correction are associated with the candidate targets for cleaning machinery whose spatial location is closest to the candidate targets for cleaning personnel, and candidate collaborative operation units are constructed. The candidate collaborative operation units are verified to check the collaborative relationship. The relative motion patterns of the candidate targets for clearing personnel and the candidate targets for clearing machinery between consecutive frames are verified to meet the preset logic constraints for collaborative operation. The candidate collaborative operation units that pass the verification are determined as formal collaborative operation units. The output includes the multi-target recognition result containing the collaborative work unit and the confidence level of the collaborative work unit.

[0015] Secondly, this application provides a computer device, the computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the above-described multi-target recognition method for bulk carrier cleaning operations based on visual intelligent analysis.

[0016] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-mentioned multi-target recognition method for bulk carrier cleaning operations based on visual intelligent analysis.

[0017] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: This application constructs a visual degradation field prediction model to transform the behavior of cleaning machinery operations into a spatial distribution of environmental degradation. Based on this model, it performs dual-source evidence fusion reasoning of explicit spatiotemporal coordination and implicit environmental degradation, ultimately achieving reverse correction of the detection results of cleaning personnel. First, based on the working posture and action state of the cleaning machinery, the visual degradation field prediction model quantitatively infers the dynamic degradation distribution of dust and illumination changes on the image plane, thereby assessing the visual attenuation characteristics at different spatial locations. Dust and illumination changes, traditionally considered as interference, are transformed into spatial signals carrying operational status information. Second, the consistency of spatial distance and relative motion direction is extracted and semantically reasoned to obtain explicit coordination indicators, while visual attenuation features are extracted and semantically reasoned to obtain implicit environmental indicators. This integrates directly observable spatiotemporal coordination evidence with environmental degradation evidence. The system is decoupled into two independent information dimensions. Then, when explicit collaborative indicators indicate the existence of spatiotemporal collaborative relationships but the original detection confidence is erroneously suppressed due to environmental degradation, implicit environmental indicators are introduced for evidence conflict analysis. After determining that the conflict is caused by environmental degradation, a confidence redistribution strategy is adopted to suppress the weight of the original detection confidence and use the collaborative confidence as the collaborative relationship confidence, thereby realizing the reverse reasoning of the human-machine collaborative relationship from the visual degradation field. Finally, the original detection confidence is replaced with the collaborative relationship confidence, and the collaborative operation unit is output after verification by the relative motion mode between consecutive frames. In summary, this application realizes the reverse reasoning of the human-machine collaborative relationship from the visual degradation field of the cabin clearing through the synergistic effect of causal modeling of the visual degradation field, dual-source evidence fusion reasoning, and reverse correction of detection confidence, which significantly improves the robustness of multi-target recognition under harsh working conditions. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the application scenario architecture of a multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis, as shown in some embodiments of this application. Figure 2 This is an exemplary flowchart of a multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis, according to some embodiments of this application; Figure 3 This is a schematic flowchart illustrating the process of determining visual attenuation characteristics according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of a computer device that implements a multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis, according to some embodiments of this application.

[0019] The diagram shows: 401, processor; 402, communication bus; 403, memory; 404, communication interface. Detailed Implementation

[0020] To better understand the technical solution of this application, the technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] refer to Figure 1 This figure is a schematic diagram of the application scenario architecture of a multi-target recognition method for bulk carrier cleaning operations based on visual intelligence analysis, according to some embodiments of this application. It mainly includes an industrial camera, an edge computing device, and a monitoring display terminal, which are interconnected via a terminal industrial Ethernet or an internal high-speed data bus. The industrial camera is deployed at the hatch coaming of the bulk carrier to continuously collect operation image sequences of the hold area and transmits the collected image frames to the edge computing device in real time. The edge computing device runs the execution code of the multi-target recognition method for bulk carrier cleaning operations based on visual intelligence analysis, performs target detection on the received operation image sequences, obtains candidate targets for cleaning machinery and cleaning personnel, and extracts the operating posture, action state, and spatial position of the cleaning machinery. Based on the operating posture and action state of the cleaning machinery, a dynamic degradation distribution map is generated through a visual degradation field prediction model, and the visual attenuation characteristics of each spatial position are evaluated. Then, using the spatial position of the cleaning machinery as a reference and the visual attenuation characteristics as spatial conditions, the confidence of the cooperative relationship is calculated. Finally, the cooperative relationship is utilized... The system corrects the detection confidence of candidate targets for cleaning personnel and generates a multi-target recognition result containing collaborative work units and their confidence levels after verifying the collaborative relationship. The monitoring and display terminal receives the multi-target recognition result output by the edge computing device and overlays the bounding box and operation status of the cleaning machinery, the bounding box and corrected confidence level of the cleaning personnel, and the association identifier of the collaborative work units on the bottom monitoring screen. It also provides a visualized collaborative operation status monitoring and abnormal warning interface to on-site safety management personnel. Through this architecture, the human-machine collaborative relationship can be inferred by using the dust and light changes caused by the cleaning operation itself. Robust recognition of the collaborative operation relationship between cleaning personnel and cleaning machinery can be achieved under harsh working conditions, thereby effectively reducing the risk of personnel being missed due to environmental degradation.

[0022] refer to Figure 2 The figure is an exemplary flowchart of a multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis, according to some embodiments of this application. This method mainly includes the following steps: In step 101, a sequence of operational images of the bilge area of ​​the bulk carrier is acquired.

[0023] In this embodiment, an explosion-proof industrial camera can be used to capture continuous image frames of the bilge cleaning operation at a preset frame rate, obtaining an image sequence of the operation in the bilge area of ​​a bulk carrier. In specific applications, the explosion-proof industrial camera is positioned at an appropriate location on the bulk carrier's hatch coaming, with the lens facing the bilge cleaning operation area, ensuring that the field of view covers the core operation area containing the cleaning machinery, personnel, and bulk cargo piles. The preset frame rate is set according to the typical operating speed of the cleaning machinery to ensure that adjacent image frames can capture the dynamic changes in dust diffusion and the machinery's posture. The explosion-proof industrial camera continuously captures images at this preset frame rate, generating a continuous image sequence arranged in chronological order. During continuous shooting, fixed exposure parameters are used for exposure control to avoid non-physical abrupt changes in image brightness within the same operation cycle due to automatic camera exposure adjustment, ensuring that illumination degradation characteristics are stably recorded in the image sequence. This resulting image sequence provides a temporally continuous and consistently exposed original image data foundation for inferring the visual degradation field from mechanical movements and analyzing human-machine collaboration.

[0024] In step 102, target detection is performed on the operation image sequence to obtain candidate targets for cleaning machinery and personnel, and the operation posture, action state and spatial position of the candidate targets for cleaning machinery are extracted.

[0025] In this embodiment, target detection is performed on the operation image sequence to obtain candidate targets for cleaning machinery and personnel. This can be achieved through the following steps: The current frame in the operation image sequence is input into a pre-trained deep convolutional neural network detector to extract candidate regions of the cleaning machinery and personnel in the image. Each candidate region is classified and bounding box regressed, and the bounding boxes, category labels, and detection confidence scores of the candidate targets for cleaning machinery and personnel are output.

[0026] In practical application, firstly, the current image frame to be processed is extracted from the sequence of images to be processed. The current frame is input into a pre-trained deep convolutional neural network detector. The backbone network of the deep convolutional neural network detector performs multi-layer convolution and pooling operations on the input image to extract image feature maps with different scales and abstraction levels. On the image feature maps, the region proposal network scans each feature position in a sliding window manner and generates multiple anchor boxes with different aspect ratios and scales at each feature position. For each anchor box, a target existence score and an initial bounding box offset are predicted simultaneously. Anchor boxes with a high probability of target presence are selected based on the target existence score. The position and size of the selected anchor boxes are initially corrected based on the initial bounding box offset. The corrected anchor boxes are mapped back to the original current frame image to obtain a set of candidate regions for cleaning machinery and personnel. Then, for each candidate region, the deep convolutional neural network detector... The image features corresponding to the candidate region are input into a classification branch network and a regression branch network, with the two branches operating in parallel. The classification branch network performs a fully connected operation on the features of the candidate region, outputting a predicted score indicating whether the candidate region belongs to the cleaning machinery or cleaning personnel category. The category with the higher predicted score is determined as the category label of the candidate region, and a normalized exponential function is used to normalize the predicted score, mapping it to a probability value between 0 and 1. This probability value is the detection confidence of the candidate region. The regression branch network performs a fully connected operation on the features of the candidate region, outputting a bounding box offset correction. The bounding box offset correction includes the horizontal offset of the bounding box center coordinates, the vertical offset of the bounding box center coordinates, the scaling factor of the bounding box width, and the scaling factor of the bounding box height. The original bounding box of the candidate region is finely adjusted using the bounding box offset correction to obtain the final bounding box of the candidate region. The category label, detection confidence, and final bounding box of each processed candidate region are combined to form a complete detection result for a cleaning machinery candidate target or a cleaning personnel candidate target. The set of detection results for all candidate targets is used as the target detection output for the current frame.

[0027] It should be noted that the auxiliary operating components of the cleaning machinery in this application refer to the bucket or pusher installed at the end of the main body of the cleaning machinery and directly performing bulk cargo shoveling or scraping operations.

[0028] In this embodiment, the extraction of the operational posture, motion state, and spatial location of the candidate clearing machinery can be achieved through the following steps: Joint point detection is performed on the image region corresponding to the candidate target of the cleaning machinery to obtain the joint point position of the auxiliary working parts of the cleaning machinery; The attitude angle of the auxiliary working components is calculated based on the position of the joint points, and the current action state of the cleaning machinery is determined according to the displacement changes of the joint points between consecutive frames and the preset action state discrimination rules. By combining camera calibration parameters, the pixel coordinates of the candidate targets for cleaning machinery in the image are converted into three-dimensional spatial positions within the ship's working space.

[0029] In practical application, firstly, joint point detection is performed on the image region within the bounding box corresponding to each candidate target of the cleaning machinery. This joint point detection is achieved through a pre-trained joint point detection network. This network uses cleaning machinery image samples with joint point annotations on auxiliary working components of the cleaning machinery as training data to learn the mapping relationship from the machinery image region to the joint point positions. The network extracts features from the input cleaning machinery image region and predicts heatmaps corresponding to multiple predefined joint points on the auxiliary working components of the cleaning machinery on the feature map. The response value of each pixel in the heatmap represents the probability that the pixel is the corresponding joint point position, and the position with the strongest response in the heatmap is the image coordinate of the corresponding joint point. The predefined joint points on the auxiliary working components of the cleaning machinery include the connection joint points between the auxiliary working components and the main body of the cleaning machinery, and the joint points at the ends of the auxiliary working components. The set of coordinates of all detected joint points in the image is determined as the joint point positions of the auxiliary working components of the cleaning machinery.

[0030] Secondly, obtain the joint locations of the auxiliary working components of the cleaning machinery, and select at least two joints, including the first joint located at the connection between the auxiliary working component and the main body of the cleaning machinery, and the second joint located at the end of the auxiliary working component. The direction of the line connecting the first and second joints is calculated within the image plane. The angle between this line and a preset horizontal or vertical reference line is determined as the attitude angle of the auxiliary working component. Simultaneously, the position records of the joints on the same cleaning machinery auxiliary working component in several consecutive frames preceding the current frame are obtained. The displacement vector of the same joint between adjacent frames is calculated, and the displacement vector includes displacement distance and displacement direction. The displacement distance is compared with a preset static displacement threshold, and the displacement direction is matched with a preset typical motion direction range. The preset static displacement threshold is an empirical value set based on the maximum pixel displacement that the cleaning machinery auxiliary working component may produce due to image noise in a static state. The preset typical motion direction range includes the digging motion direction range and the lifting motion direction range. The digging motion direction range corresponds to the direction interval in which the end joint of the auxiliary working component moves obliquely downward toward the bottom plate, and the lifting motion direction range corresponds to the direction interval in which the end joint of the auxiliary working component moves obliquely upward away from the bottom plate. The above direction intervals are defined by the angle range between the joint displacement vectors of consecutive frames and the vertical or horizontal direction of the image. When the displacement distance of key points between consecutive frames is greater than the static displacement threshold, and the displacement direction falls within the range of the digging action direction, the current action state of the cleaning machinery is determined to be digging; when the displacement direction falls within the range of the lifting action direction, the current action state of the cleaning machinery is determined to be lifting; when the displacement distance of key points between consecutive frames is less than or equal to the static displacement threshold, the current action state of the cleaning machinery is determined to be static. The above attitude angle calculation results and action state determination results are respectively used as the attitude angle and current action state output of the candidate target of the cleaning machinery.

[0031] Then, the bounding box of the candidate cleaning machinery is obtained, and the pixel coordinates of the midpoint of the bottom edge of the bounding box in the image are selected as the reference pixel coordinates of the candidate cleaning machinery in the image. The camera calibration parameters obtained in advance through camera calibration are obtained. The camera calibration parameters are obtained by offline calibration of the explosion-proof industrial camera deployed at the hatch. During the calibration process, a calibration plate of known size is placed at multiple positions on the bottom plane of the hatch and photographed. The correspondence between the pixel coordinates of the corner points of the calibration plate in the image and the known three-dimensional coordinates in the world coordinate system of the hatch operation space is extracted, and the camera intrinsic parameter matrix and camera extrinsic parameter matrix are obtained by solving. The camera intrinsic parameter matrix describes the transformation relationship between the image pixel coordinate system and the camera coordinate system, and the camera extrinsic parameter matrix describes the rotation and translation relationship between the camera coordinate system and the world coordinate system of the hatch operation space. The reference pixel coordinates of the candidate cleaning machinery target are transformed from the image pixel coordinate system to the camera coordinate system using the camera intrinsic parameter matrix, obtaining the 3D coordinates in the camera coordinate system. Then, the 3D coordinates in the camera coordinate system are transformed to the world coordinate system of the ship's working space using the camera extrinsic parameter matrix, obtaining the 3D spatial position of the candidate cleaning machinery target within the ship's working space. By converting the reference pixel coordinates of the candidate cleaning machinery target in the image to its 3D spatial position within the ship's working space, the true spatial location of the candidate cleaning machinery target is determined.

[0032] In step 103, based on the working posture and action state of the candidate cleaning machinery, a dynamic degradation distribution map of dust and illumination changes caused by the cleaning operation on the image plane is predicted by a pre-constructed visual degradation field prediction model. Then, the visual attenuation characteristics of different spatial locations in the operation image sequence affected by dust and illumination changes are evaluated based on the dynamic degradation distribution map.

[0033] In this embodiment, based on the working posture and action state of the candidate cleaning machinery, the dynamic degradation distribution map of dust and illumination changes caused by the cleaning operation on the image plane can be predicted using a pre-built visual degradation field prediction model. This can be achieved through the following steps: Extract the motion amplitude and direction of the auxiliary working parts of the cleaning machinery in the current operating state, as well as the contact state between the auxiliary working parts of the cleaning machinery and the bottom plate of the ship, as driving parameters of the visual degradation field; The driving parameters are input into a pre-constructed visual degradation field prediction model, which includes a dust diffusion sub-model and an illumination change sub-model. The dust diffusion sub-model takes the dust origin location and dust diffusion parameters as input and outputs a dust degradation intensity value that characterizes the degree of visual degradation caused by dust to each pixel of the image. The illumination change sub-model takes the mechanical component pose and bulkhead structure parameters as input and outputs an illumination degradation intensity value that characterizes the degree of visual degradation caused by illumination change to each pixel of the image. By combining the dust degradation intensity value and the illumination degradation intensity value, a visual degradation intensity value that characterizes the degree of visual degradation caused by dust or illumination changes to image pixels is determined, and the dynamic degradation distribution map is generated.

[0034] It should be noted that the visual degradation intensity value in this application is a characteristic index that measures the degree of visual degradation of a single pixel in an image caused by dust and lighting changes caused by the cleaning operation. This index comprehensively reflects the degree of pixel blurring caused by dust obstruction and the degree of pixel underexposure or overexposure caused by sudden changes in lighting. The dynamic degradation distribution map is an image used to reflect the spatial distribution of the degree of visual degradation at each spatial location on the current frame image plane under the combined influence of dust and lighting changes caused by the cleaning operation.

[0035] It should also be noted that the visual degradation field prediction model in this application is a predictive model used to quantitatively predict the spatial distribution of visual degradation caused by dust and illumination changes resulting from the cleaning operation on the image plane, based on the working posture and action state of the cleaning machinery. The technical principle of this model lies in establishing a causal mapping relationship between the cleaning machinery's working behavior and environmental visual degradation. When the auxiliary working parts of the cleaning machinery perform shoveling, unloading, or scraping actions, their movement amplitude, direction, and contact state with the bottom plate of the hull constitute the driving source of environmental disturbance. When the auxiliary working parts contact the bottom plate and generate large displacement movements, material particles are excited to form dust. The dust spreads from the dust-generating point to the surrounding area and gradually attenuates, which is manifested in the image as blurring of local pixels and a decrease in contrast. At the same time, the posture changes and spatial displacement of the auxiliary working parts of the cleaning machinery will change the occlusion distribution and reflection path of the incident light from the hatch on the bottom plate, causing local areas of the image to appear too dark or overexposed. The visual degradation field prediction model, through parametric modeling of the aforementioned physical processes, transforms mechanical motion features into visual degradation intensity values ​​for each pixel on the image plane, ultimately generating a dynamic degradation distribution map. The pre-construction process of the visual degradation field prediction model includes: collecting sample image sequences of cleaning machinery operations under different motion states at the cleaning operation site; manually labeling the true value of the visual degradation degree of each pixel in each frame of the image; using the motion amplitude, motion direction, contact state, mechanical component pose, and bulkhead structural parameters of the cleaning machinery's auxiliary operating components as input features; and using pixel-level visual degradation intensity values ​​as output, supervised training of the initial prediction model is performed. By minimizing the error between the predicted visual degradation intensity values ​​and the manually labeled true values, the model parameters are iteratively adjusted to obtain the trained visual degradation field prediction model.

[0036] In practical application, firstly, the motion state determination result of the candidate target of the cleaning machinery in the current frame and the joint position of the auxiliary working parts are obtained. From the joint position, the end joint of the auxiliary working parts is selected, and the displacement distance of the end joint between the current frame and the previous frame is recorded. The displacement distance is used as the motion amplitude of the auxiliary working parts of the cleaning machinery in the current motion state. At the same time, the displacement direction of the end joint between adjacent frames is recorded, and the displacement direction is used as the motion direction of the auxiliary working parts of the cleaning machinery in the current motion state. Based on the spatial relationship between the end joint position of the auxiliary working parts of the cleaning machinery and the reference position of the bottom plate in the image, the contact state between the auxiliary working parts of the cleaning machinery and the bottom plate is determined. The reference position of the bottom plate in the image is obtained in the following way: Before the cleaning operation begins, a bottom image of the hull with no dust and good lighting conditions is acquired. The boundary of the bottom plate area is manually marked or automatically extracted by an image segmentation algorithm along the boundary line between the bottom plate and the bulkhead in the bottom image. The set of pixel coordinates of each point on the boundary of the bottom plate area is used as the reference position of the bottom plate in the image. When the distance between the end joint position and the reference position of the ship's bottom plate in the image is less than the preset contact determination threshold, the contact state is determined to be contact; when the distance is greater than or equal to the preset contact determination threshold, the contact state is determined to be non-contact. The motion amplitude, motion direction and contact state are combined into the driving parameters of the visual degradation field.

[0037] Secondly, the driving parameters of the visual degradation field are input into a pre-constructed visual degradation field prediction model. The visual degradation field prediction model includes a dust diffusion sub-model and an illumination change sub-model. The two sub-models process different components in the driving parameters in parallel. The dust diffusion sub-model receives the dust origination point location and dust diffusion parameters as input. The dust origination point location is determined by the coordinates of the end joint of the auxiliary working component of the cleaning machinery in the image. The dust diffusion parameters are determined by the motion amplitude and contact state in the driving parameters through a preset diffusion parameter mapping relationship. The preset diffusion parameter mapping relationship is a correspondence table between the motion amplitude range, contact state and diffusion radius and diffusion decay rate established in advance through experimental calibration. The correspondence table is established by recording the actual diffusion radius and diffusion decay rate of dust under different motion amplitudes and contact states at the cleaning operation site. The recorded motion amplitude is divided into several intervals, and each motion amplitude interval is combined with the corresponding contact state as a mapping entry. The diffusion radius and diffusion decay rate corresponding to the mapping entry are stored. The diffusion parameter mapping relationship defines the diffusion radius and diffusion decay rate corresponding to different motion amplitude ranges and different contact states. The larger the motion amplitude and the contact state is contact, the larger the values ​​of diffusion radius and diffusion decay rate, thus simulating the physical characteristics of dust diffusion range and slower decay. The dust diffusion sub-model takes the dust originating point as the center, calculates the degree of visual degradation of each pixel on the image plane due to dust occlusion according to the diffusion range and decay rate defined by the dust diffusion parameters, and outputs the dust degradation intensity value corresponding to each pixel.The illumination variation sub-model receives the mechanical component pose and bulkhead structural parameters as input. The mechanical component pose is determined by the attitude angle and spatial position of the auxiliary working parts of the cleaning machinery. The bulkhead structural parameters are a set of parameters obtained by pre-calibrating the images of the hull bottom captured by explosion-proof industrial cameras deployed at the hatch. The calibration process includes marking the intersection line of the bulkhead and the hull bottom plate, the top edge line of the bulkhead, and the pixel positions of each corner point on the bulkhead surface in the hull bottom image before the cleaning operation begins. The above pixel positions and the bulkhead height value determined by the physical dimensions of the bulkhead are stored as bulkhead structural parameters. The illumination variation sub-model calculates the area of ​​occlusion of the incident light from the hatch opening by the auxiliary working parts of the cleaning machinery based on the mechanical component pose and bulkhead structural parameters. In addition to the reflection area, the visual degradation degree of each pixel is calculated according to the preset light attenuation coefficient in the occluded area, and the visual degradation degree of each pixel is calculated according to the preset light enhancement coefficient in the reflection area. The preset light attenuation coefficient and the preset light enhancement coefficient are empirical values ​​set according to the actual light change range of the occlusion and reflection of the auxiliary working parts of the cleaning machinery under the natural lighting conditions of the hatch. The light attenuation coefficient is a value between 0.3 and 0.7, which represents the corresponding proportion of the pixel brightness in the occluded area to the original brightness. The light enhancement coefficient is a value between 1.2 and 1.8, which represents the corresponding multiple of the pixel brightness in the reflection area to the original brightness. The light degradation intensity value corresponding to each pixel is output.

[0038] Then, the dust degradation intensity values ​​of each pixel output by the dust diffusion sub-model and the illumination degradation intensity values ​​of each pixel output by the illumination change sub-model are obtained. For each pixel position on the image plane, the dust degradation intensity value and the illumination degradation intensity value corresponding to that pixel position are compared, and the larger of the two values ​​is selected as the visual degradation intensity value of that pixel position. After completing the above comparison and value selection operation for all pixel positions, the visual degradation intensity values ​​corresponding to all pixel positions are used to form a two-dimensional numerical matrix with the same size as the current frame image. This two-dimensional numerical matrix is ​​determined as a dynamic degradation distribution map. Each element in the dynamic degradation distribution map corresponds to a pixel position on the image plane, and the value of the element is the visual degradation intensity value of that pixel position. The higher the value, the more severe the impact of dust or illumination changes on that position, and the more difficult the visual recognition. The lower the value, the clearer the visual conditions at that position.

[0039] In this embodiment, reference Figure 3 As shown in the figure, this is a flowchart illustrating the determination of visual attenuation characteristics according to some embodiments of this application. In this embodiment, the evaluation of the visual attenuation characteristics of different spatial locations in the operational image sequence affected by dust and illumination changes based on the dynamic degradation distribution map can be achieved by the following steps: In step 1031, the image plane of the job image sequence is divided into several grid units; In step 1032, based on the dynamic degradation distribution map, the mean value of the visual degradation intensity of pixels in each grid cell is calculated, and the mean value is used as the visual attenuation feature of the corresponding spatial location.

[0040] The visual attenuation feature in this application is used to measure the decrease in visual recognizability of a certain spatial region in an image plane due to the combined effects of dust and changes in lighting.

[0041] In practical application, firstly, the image plane of the current frame in the image sequence is acquired, and the pixel width and pixel height of the image plane are determined. Based on a preset grid division resolution, the image plane is divided at equal intervals along both the width and height directions, forming a rectangular grid arrangement with several rows and columns. The grid division resolution is set according to the typical size of personnel targets and the influence range of cleaning machinery operations in the cabin cleaning scenario, ensuring that the side length of a single grid unit corresponds to an area of ​​approximately half a meter to one meter in the actual physical space of the cabin bottom on the image plane. The specific method for setting the grid division resolution is as follows: using the transformation relationship between the image pixel coordinate system in the camera calibration parameters and the world coordinate system of the cabin operation space, the preset target side length in the actual physical space of the cabin bottom is converted into pixel dimensions on the image plane. The converted pixel dimensions are then used as the side length of the grid units for equal-interval division. After division, the image plane is completely covered by several non-overlapping grid units. Each grid unit has a defined row and column index in the image plane, which together identify the spatial position of the grid unit in the image plane.

[0042] Then, for each grid cell, the pixel range covered by the grid cell in the image plane is determined. The visual degradation intensity value corresponding to each pixel position within the pixel range is read from the dynamic degradation distribution map. The arithmetic mean of the visual degradation intensity values ​​of all pixel positions in the grid cell is calculated to obtain the mean visual degradation intensity of the grid cell. The mean visual degradation intensity is determined as the visual attenuation feature of the corresponding spatial position of the grid cell. After performing statistical operations on all grid cells one by one, each grid cell obtains a corresponding visual attenuation feature value. The higher the value, the more severe the impact of dust and illumination changes on the corresponding spatial position in the current frame, and the lower the degree of visual information recognition.

[0043] In step 104, the confidence level of the cooperative relationship between each candidate target for clearing personnel and candidate target for clearing machinery is calculated, with the spatial position of the candidate target for clearing the cabin as a reference and the visual attenuation feature as a spatial condition.

[0044] In this embodiment, using the spatial location of the candidate targets for the cleaning machinery as a reference and the visual attenuation feature as a spatial condition, the confidence level of the cooperative relationship between each candidate target for cleaning personnel and the candidate target for cleaning machinery is calculated, specifically including: Calculate the spatial distance between the cleaning machinery and the cleaning personnel based on their spatial locations and the spatial locations of the candidate cleaning machinery and personnel. Based on the motion trajectories of the cleaning machinery and personnel between consecutive frames, determine the degree of consistency in the relative motion directions between the cleaning machinery and personnel. The consistency between the spatial distance and the relative motion direction is used as an explicit spatiotemporal correlation feature. Based on the explicit spatiotemporal correlation feature, explicit semantic reasoning is performed on the collaborative operation relationship between the cleaning machinery and the cleaning personnel to obtain an explicit collaboration index that characterizes the degree of spatiotemporal collaboration that can be directly observed. The visual attenuation features corresponding to the spatial location of each candidate target for clearing the cabin are obtained as implicit environmental features. Based on the implicit environmental features, implicit semantic reasoning is performed on the visual perception attenuation caused by dust and changes in light to obtain implicit environmental indicators that characterize the degree of impact of environmental degradation on personnel detection. By performing semantic reasoning on the explicit collaborative indicators and the implicit environmental indicators, and combining them with the preset safety rules for collaborative cleaning operations, the confidence level of the collaborative relationship between each candidate target for cleaning personnel and candidate target for cleaning machinery is determined.

[0045] It should be noted that the explicit collaboration index in this application is used to measure the credibility of the collaborative operation relationship between the cleaning personnel and the cleaning machinery inferred from directly observable spatiotemporal correlation features; the implicit environmental index is used to measure the impact of environmental degradation inferred from the degree of visual perception attenuation caused by dust and light changes on the detection results of the cleaning personnel; and the collaboration relationship confidence is used to measure the credibility of the collaborative operation relationship between the candidate targets of the cleaning personnel and the candidate targets of the cleaning machinery after association reasoning based on explicit spatiotemporal collaboration evidence and environmental degradation evidence.

[0046] It should also be noted that the safety rules for collaborative cleaning operations in this application refer to a set of constraints pre-established based on the safety procedures for cleaning operations and the actual operational characteristics of human-machine collaborative operations, used to determine whether a safe collaborative working relationship exists between the cleaning machinery and the cleaning personnel. Specifically, these constraints include: The spatial distance between the cleaning machinery and the cleaning personnel is less than the preset maximum safe distance for collaborative operations; The degree of consistency in the relative motion direction between the cleaning machinery and the cleaning personnel between consecutive frames is higher than the preset direction consistency threshold. When the visual attenuation characteristic value of the spatial location of the candidate target for cleaning personnel is higher than the preset visual attenuation threshold, the spatial distance between the candidate target for cleaning personnel and the candidate target for cleaning machinery should still meet the above-mentioned maximum safe distance constraint for collaborative operation. When the cleaning machinery is currently in the digging or scraping state, the cleaning personnel should be located within a pre-set safety area outside the working range of the cleaning machinery's auxiliary operating components.

[0047] In practical applications, the spatial distance between cleaning machinery and personnel can be calculated based on their spatial positions. This can be achieved as follows: The three-dimensional spatial position of the cleaning machinery candidate target within the ship's working space is obtained, and this position is used as the reference point for the cleaning machinery. The three-dimensional spatial position of each cleaning personnel candidate target within the ship's working space is also obtained. The three-dimensional coordinates of the midpoint of the bottom edge of the bounding box of each cleaning personnel candidate target, after conversion using camera calibration parameters, are used as the reference point for the cleaning personnel. For each cleaning personnel candidate target, the Euclidean distance between the cleaning machinery reference point and the cleaning personnel reference point is calculated, and this Euclidean distance is determined as the spatial distance between the cleaning machinery and the cleaning personnel.

[0048] In practical applications, the consistency of the relative motion directions between the cleaning machinery and personnel can be determined based on their motion trajectories across consecutive frames. This can be achieved as follows: Obtain the spatial position sequence of candidate cleaning machinery targets in several consecutive frames, and use the displacement direction of the cleaning machinery's spatial position between adjacent frames as the frame-by-frame motion direction of the cleaning machinery. Similarly, obtain the spatial position sequence of candidate cleaning personnel targets in the same several consecutive frames, and use the displacement direction of the cleaning personnel's spatial position between adjacent frames as the frame-by-frame motion direction of the cleaning personnel. For each pair of adjacent frames, calculate the angle between the frame-by-frame motion direction of the cleaning machinery and the frame-by-frame motion direction of the cleaning personnel. Compare this angle with a preset direction consistency angle threshold of 45 degrees. When the angle is less than or equal to the direction consistency angle threshold, the motion directions of the cleaning machinery and personnel in that adjacent frame are considered consistent. Count the number of adjacent frame pairs whose motion directions are determined to be consistent across several consecutive frames. Divide this number by the total number of adjacent frame pairs across several consecutive frames; the resulting ratio is the consistency of the relative motion directions between the cleaning machinery and personnel.

[0049] In practical applications, the consistency between the spatial distance and the relative motion direction is used as an explicit spatiotemporal correlation feature. Based on this explicit spatiotemporal correlation feature, explicit semantic reasoning is performed on the collaborative operation relationship between the cleaning machinery and the cleaning personnel to obtain an explicit collaboration index that characterizes the directly observable degree of spatiotemporal collaboration. This can be achieved in the following way: Obtain the spatial distance between the cleaning machinery and the cleaning personnel, as well as the consistency between their relative motion directions, and use the consistency between the spatial distance and the relative motion direction as an explicit spatiotemporal correlation feature; normalize the spatial distance by dividing it by a preset maximum reference distance for collaborative operation to obtain a normalized spatial distance. The normalized spatial distance ranges from 0 to 1, with the value closer to 1 indicating that the spatial positions of the cleaning machinery and the cleaning personnel are closer. The preset maximum reference distance for collaborative operations is set according to the maximum permissible distance for human-machine collaborative operations as specified in the safety regulations for cabin cleaning operations, with a value ranging from 3 to 5 meters. The normalized spatial distance and the consistency of relative motion direction are weighted and fused. A first weight is assigned to the normalized spatial distance, and a second weight is assigned to the consistency of relative motion direction. The sum of the first and second weights is 1. The first weight is set to 0.6, and the second weight to 0.4, ensuring that the contribution of spatial distance to the explicit collaborative index is slightly higher than the contribution of the consistency of relative motion direction. The product of the normalized spatial distance and the first weight, and the product of the consistency of relative motion direction and the second weight are calculated. The two products are added together to obtain the weighted fusion result. This weighted fusion result is the explicit collaborative index characterizing the directly observable spatiotemporal collaborative degree. The larger the value of the explicit collaborative index, the higher the credibility of judging a collaborative operational relationship between cabin cleaning machinery and personnel from the perspective of directly observable spatiotemporal correlation characteristics.

[0050] In practical applications, the visual attenuation features corresponding to the spatial locations of each candidate target for personnel clearance are obtained as implicit environmental features. Based on these implicit environmental features, implicit semantic reasoning is performed on the visual perception attenuation caused by dust and changes in illumination to obtain implicit environmental indicators characterizing the degree of impact of environmental degradation on personnel detection. This can be achieved in the following way: Visual attenuation features of each spatial location in the current frame are obtained, and these features are stored in grid cells. For each candidate target for personnel clearance, the grid cell containing the midpoint of the bottom edge of the bounding box of that candidate target in the image plane is determined. The method reads the visual attenuation feature value corresponding to the grid cell from the visual attenuation feature, determines the visual attenuation feature value as the visual attenuation feature corresponding to the spatial location of the candidate target of the clearing personnel, and directly uses the visual attenuation feature as the implicit environmental feature; then performs implicit semantic reasoning on the implicit environmental feature, and inputs the implicit environmental feature into a preset implicit environmental index mapping function. The specific expression of the implicit environmental index mapping function is: the implicit environmental index is equal to the implicit environmental feature, that is, the identity mapping method is used to directly use the original value of the implicit environmental feature as the value of the implicit environmental index. For example, when the visual attenuation feature value corresponding to the grid cell containing a candidate personnel for clearing the cabin is 0.6, the implicit environmental index value corresponding to that candidate personnel is also 0.6. In another implementation, the implicit environmental index mapping function can also be a linear mapping function, specifically expressed as: the implicit environmental index equals the implicit environmental feature multiplied by a scaling factor plus an offset, where the scaling factor and offset are preset parameters. By adjusting the scaling factor and offset, the value range of the implicit environmental feature is linearly mapped to the interval between 0 and 1. The mapped value is the implicit environmental index characterizing the degree of influence of environmental degradation on personnel detection. The larger the value of the implicit environmental index, the more severely the spatial location of the candidate personnel for clearing the cabin is affected by dust and changes in lighting, the lower the visual recognizability of the personnel, and the higher the degree of influence of environmental degradation on the personnel detection results.

[0051] Preferably, in some embodiments, the confidence level of the collaborative relationship between each candidate target for cleaning personnel and candidate target for cleaning machinery can be determined by performing semantic reasoning on the explicit collaborative indicators and the implicit environmental indicators, combined with preset safety rules for collaborative cleaning operations, using the following steps: By using the explicit collaborative indicators and the implicit environmental indicators as two independent sources of evidence, an evidence fusion framework for determining collaborative relationships is constructed. When the explicit coordination indicator suggests a spatiotemporal coordination relationship between the cleaning machinery and the cleaning personnel, and the original detection confidence level of the candidate personnel is lower than a preset confidence threshold, the implicit environmental indicator is introduced for evidence conflict analysis: If the collaborative behavior pattern indicated by the explicit collaborative indicator conforms to the human-machine collaborative operation behavior characteristics defined in the preset cabin clearing collaborative operation safety rules, and the visual attenuation intensity indicated by the implicit environmental indicator is higher than the preset attenuation threshold, then the evidence conflict between the explicit collaborative indicator and the original detection confidence is determined to be caused by environmental degradation. A confidence redistribution strategy is adopted to suppress the weight of the original detection confidence in the collaborative relationship determination, and the collaborative confidence calculated based on the explicit collaborative indicator is used as the collaborative relationship confidence.

[0052] The evidence fusion framework in this application refers to an inference structure that uses explicit synergy indicators and implicit environmental indicators as independent sources of evidence to determine the confidence level of synergy relationships.

[0053] In practical application, firstly, explicit coordination indicators characterizing the directly observable degree of spatiotemporal coordination and implicit environmental indicators characterizing the impact of environmental degradation on personnel detection are obtained. These explicit and implicit indicators are then used as independent sources of evidence to construct an identification framework that includes propositions regarding the establishment and non-establishment of coordination relationships. In this framework, the proposition of established coordination relationship indicates that there is a coordinated operational relationship between candidate personnel and candidate machinery for clearing the cabin, while the proposition of non-establishment indicates that there is no coordinated operational relationship between these two candidates. Initial trust levels are assigned to both the explicit and implicit environmental indicator evidence sources, with an initial trust level of 0.5 for both, ensuring that both sources have equal initial weight in determining the coordination relationship. The explicit collaboration indicator evidence source generates a basic probability allocation value for the proposition that the collaboration relationship is valid based on the value of the explicit collaboration indicator. Specifically, the value of the explicit collaboration indicator is directly used as the basic probability allocation value for the proposition that the collaboration relationship is valid, and 1 minus the value of the explicit collaboration indicator is used as the basic probability allocation value for the proposition that the collaboration relationship is invalid. The implicit environmental indicator evidence source generates a basic probability allocation value for the proposition that the collaboration relationship is invalid based on the value of the implicit environmental indicator. Specifically, the value of the implicit environmental indicator is directly used as the basic probability allocation value for the proposition that the collaboration relationship is invalid, and 1 minus the value of the implicit environmental indicator is used as the basic probability allocation value for the proposition that the collaboration relationship is valid. The sum of the basic probability allocation values ​​of each evidence source for the two propositions in the identification framework is 1. The basic probability allocation values ​​corresponding to the explicit collaboration indicator evidence source and the implicit environmental indicator evidence source are then combined to form the evidence fusion framework for determining the collaboration relationship.

[0054] Then, within the constructed evidence fusion framework, the value of the explicit coordination indicator is first checked to see if it is higher than a preset spatiotemporal coordination judgment threshold, which is 0.5. If it is higher, the explicit coordination indicator indicates that there is a spatiotemporal coordination relationship between the cleaning machinery and the cleaning personnel. Simultaneously, the original detection confidence level of the candidate cleaning personnel target is obtained in the target detection step, and it is checked whether this original detection confidence level is lower than a preset detection confidence level threshold, which is 0.5. When the explicit coordination indicator indicates the existence of a spatiotemporal coordination relationship and the original detection confidence level is lower than the preset detection confidence level threshold, the evidence conflict analysis procedure is initiated. In the evidence conflict analysis procedure, implicit environmental indicators are introduced for conflict resolution: It is determined whether the consistency of the spatial distance and relative movement direction between the cleaning machinery and personnel, as indicated by the explicit coordination indicators, conforms to the human-machine collaborative operation behavior characteristics defined in the preset safety rules for collaborative cleaning operations. These characteristics are: the spatial distance between the cleaning machinery and personnel is less than the maximum permissible distance for collaborative operations, and the consistency of their relative movement directions is higher than 0.5. Simultaneously, it is determined whether the value of the implicit environmental indicators is higher than a preset attenuation threshold of 0.5. If both conditions are met, the evidence conflict between the explicit coordination indicators and the original detection confidence is considered to be caused by environmental degradation, rather than by the actual absence of the cleaning personnel target in the scenario. In this case, a trust redistribution strategy is adopted to apply a trust discount to the basic probability allocation value of the original detection confidence in determining the collaborative relationship, reducing the negative contribution of the original detection confidence to the proposition of the establishment of the collaborative relationship. The collaborative trust calculated based on the explicit coordination indicators is then determined as the confidence of the collaborative relationship between the candidate targets of the cleaning personnel and the candidate targets of the cleaning machinery. If either of the two conditions is not met, it is determined that the evidence conflict may be caused by the lack of a true target for the personnel clearing the cabin. Instead of adopting a trust redistribution strategy, the confidence of the collaboration relationship is determined directly based on the original detection confidence, and the final determined confidence value is output as the confidence of the collaboration relationship.

[0055] Preferably, in some embodiments, the trust redistribution strategy includes: Based on the visual attenuation intensity indicated by implicit environmental indicators, a confidence discount factor is determined for the original detection confidence level. The visual attenuation intensity and the confidence discount factor have a preset positive correlation mapping relationship. The trust discount factor is used to discount the basic probability allocation value of the original detection confidence in the determination of collaborative relationship, thereby reducing the trust contribution of the original detection confidence in evidence fusion. The trust shares released by the discount process are redistributed to the propositions supporting the validity of cooperative relationships supported by explicit cooperative indicators, resulting in a modified basic probability allocation. Based on the corrected basic probability allocation, the confidence level of the cooperative relationship between each candidate target of the cleaning personnel and the candidate target of the cleaning machinery is calculated, so that the confidence level of the cooperative relationship is dominated by explicit cooperative indicators under the interference of environmental degradation.

[0056] The trust discount factor in this application is a coefficient used to measure the degree of trust decay adjustment of the basic probability assignment value of the original detection confidence within the evidence fusion framework.

[0057] In practical application, when the conflict of evidence is determined to be caused by environmental degradation, a trust redistribution strategy is initiated: First, the value of the implicit environmental indicator is obtained, which is the visual attenuation intensity. Based on the preset positive correlation mapping relationship between visual attenuation intensity and trust discount factor, the trust discount factor corresponding to the current visual attenuation intensity is determined. This positive correlation mapping relationship is a piecewise linear mapping function, dividing the range of visual attenuation intensity from 0 to 1 into multiple intervals, each interval corresponding to a linear mapping segment, so that the value of the trust discount factor increases with the increase of visual attenuation intensity. The trust discount factor, which monotonically increases and ranges from 0 to 1, indicates a higher visual attenuation intensity, signifying more severe environmental degradation. A larger trust discount factor value indicates a greater attenuation of trust in the original detection confidence level. Secondly, using the determined trust discount factor, the basic probability allocation value of the original detection confidence level in determining synergistic relationships is discounted. The basic probability allocation value of the original detection confidence level for the proposition that synergistic relationships do not exist within the evidence fusion framework is the value of the original detection confidence level itself, while the basic probability allocation value for the proposition that synergistic relationships exist is 1 minus the value of the original detection confidence level. Multiplying the basic probability allocation value of the original detection confidence level for the proposition that synergistic relationships do not exist by 1 and subtracting the trust discount factor yields the discounted basic probability allocation value for the proposition that synergistic relationships do not exist. After discounting, the trust contribution of the original detection confidence level to the proposition that synergistic relationships do not exist is reduced. Then, the trust share released due to the discounting is calculated. The released trust share is equal to the original basic probability allocation value of the original detection confidence level for the proposition that synergistic relationships do not exist minus the discounted basic probability allocation value for the proposition that synergistic relationships do not exist. All released trust shares are redistributed to propositions supporting the validity of cooperative relationships supported by explicit cooperative indicators. This involves adding the released trust shares to the basic probability allocation value of the explicit cooperative indicator evidence sources for the propositions supporting the validity of cooperative relationships, resulting in a revised basic probability allocation value for the propositions supporting the validity of cooperative relationships. The revised basic probability allocation value for propositions not supporting cooperative relationships equals 1 minus the revised basic probability allocation value for propositions supporting the validity of cooperative relationships. The revised basic probability allocation values ​​for propositions supporting and not supporting cooperative relationships are used as the revised basic probability allocation. Finally, based on the revised basic probability allocation, the confidence level of the cooperative relationship between each candidate target for clearing personnel and candidate target for clearing machinery is calculated, and the revised basic probability allocation value for propositions supporting the validity of cooperative relationships is directly output as the confidence level of the cooperative relationship.

[0058] In step 105, the detection confidence of the candidate targets for clearing personnel is corrected according to the confidence of the cooperative relationship, and after the cooperative relationship is verified, the multi-target recognition result containing the cooperative operation unit and its confidence is output.

[0059] In this embodiment, the detection confidence of the candidate targets for clearing personnel is corrected based on the confidence of the cooperative relationship, and after verification of the cooperative relationship, the output of the multi-target recognition result containing the cooperative operation unit and its confidence can be achieved by the following steps: The original detection confidence of the candidate targets for clearing personnel is replaced with the confidence of the cooperative relationship to obtain the corrected personnel detection results; The candidate targets for cleaning personnel whose confidence level is higher than the preset confidence threshold after correction are associated with the candidate targets for cleaning machinery whose spatial location is closest to the candidate targets for cleaning personnel, and candidate collaborative operation units are constructed. The candidate collaborative operation units are verified to check the collaborative relationship. The relative motion patterns of the candidate targets for clearing personnel and the candidate targets for clearing machinery between consecutive frames are verified to meet the preset logic constraints for collaborative operation. The candidate collaborative operation units that pass the verification are determined as formal collaborative operation units. The output includes the multi-target recognition result containing the collaborative work unit and the confidence level of the collaborative work unit.

[0060] It should be noted that the collaborative operation unit in this application refers to the basic identification unit formed by associating a candidate target for cleaning personnel and a candidate target for cleaning machinery after the collaborative relationship has been verified and confirmed to constitute a collaborative operation relationship. It includes personnel identification, machinery identification and confidence level.

[0061] In practical application, firstly, the confidence scores of the collaborative relationships corresponding to each candidate target of the personnel clearing operation, calculated by the semantic reasoning step, are obtained. For each candidate target, its original detection confidence score is replaced with the confidence score of the collaborative relationship to obtain the corrected personnel detection result. Secondly, candidate targets of the personnel clearing operation with a corrected confidence score higher than 0.5 are selected from the corrected personnel detection results. For each selected candidate target, the spatial distance between it and all candidate targets of the machinery clearing operation in the current frame is calculated. The candidate target of the machinery clearing operation with the smallest spatial distance is selected for association pairing. If there are multiple candidate targets of the machinery clearing operation with the smallest spatial distance, the candidate target of the machinery clearing operation with the highest original detection confidence score is selected for association pairing to form a candidate collaborative operation unit. Then, the collaborative relationship is verified for each candidate collaborative operation unit. The spatial position sequence of the candidate targets of the personnel clearing operation and the candidate targets of the machinery clearing operation in the candidate collaborative operation unit within 5 to 10 consecutive frames is obtained. The last frame in the consecutive frame count is taken as the current frame, and the process is repeated frame by frame. Calculate the angle between the motion directions of candidate targets for cleaning personnel and candidate targets for cleaning machinery in each adjacent frame pair. Adjacent frame pairs with an angle less than or equal to 45 degrees are considered to have the same motion direction. Calculate the ratio of the number of adjacent frame pairs with the same motion direction to the total number of all adjacent frame pairs as the relative motion direction consistency of the frame sequence. Check whether the spatial distance between the two targets in each frame is always less than 5 meters and whether the relative motion direction consistency is always higher than 0.5. If both conditions are met, the candidate collaborative operation unit is determined as a formal collaborative operation unit. If not, the candidate collaborative operation unit is discarded. Finally, combine the current frame number, the bounding box coordinates and category labels of all candidate targets for cleaning machinery and the original detection confidence, the bounding box coordinates and category labels of all candidate targets for cleaning personnel and the corrected confidence, and the personnel and machinery identifiers and collaborative operation unit confidence of all formal collaborative operation units into a structured multi-target recognition result and output it. The confidence of the collaborative operation unit is taken as the corrected confidence of the candidate targets for cleaning personnel in the collaborative operation unit.

[0062] In addition, this application also provides a computer device, which includes a memory and a processor. The memory stores code, and the processor is configured to acquire the code and execute the above-described multi-target recognition method for bulk carrier cleaning operations based on visual intelligent analysis.

[0063] In some embodiments, reference Figure 4 The figure is a schematic diagram of the structure of a computer device implementing a multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis, according to some embodiments of this application. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis in the above embodiments can... Figure 4The computer device shown is used to implement this, and the computer device 400 includes at least one processor 401, a communication bus 402, a memory 403, and at least one communication interface 404.

[0064] Processor 401 can be a general-purpose central processing unit (CPU) or an application-specific integrated circuit (ASIC).

[0065] The communication bus 402 can be used to transmit information between the aforementioned components.

[0066] The memory 403 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 403 may exist independently and be connected to the processor 401 via the communication bus 402. The memory 403 may also be integrated with the processor 401.

[0067] The memory 403 stores program code for executing the solution of this application, and its execution is controlled by the processor 401. The processor 401 executes the program code stored in the memory 403. The program code may include one or more software modules. In the above embodiments, the multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis can be implemented by the processor 401 and one or more software modules in the program code in the memory 403.

[0068] Communication interface 404 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0069] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0070] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0071] In addition, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned multi-target recognition method for bulk carrier cleaning operations based on visual intelligent analysis.

[0072] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0073] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis, characterized in that, Includes the following steps: Acquire operational image sequences of the bilge area of ​​a bulk carrier; Target detection is performed on the operation image sequence to obtain candidate targets for cleaning machinery and personnel, and the operation posture, action state and spatial position of the candidate targets for cleaning machinery are extracted. Based on the working posture and action state of the candidate cleaning machinery, a pre-constructed visual degradation field prediction model is used to predict the dynamic degradation distribution map of dust and illumination changes caused by the cleaning operation on the image plane. Then, the visual attenuation characteristics of different spatial locations in the operation image sequence affected by dust and illumination changes are evaluated based on the dynamic degradation distribution map. Using the spatial location of the candidate targets for the cleaning machinery as a reference and the visual attenuation feature as a spatial condition, calculate the confidence level of the cooperative relationship between each candidate target for cleaning personnel and the candidate target for cleaning machinery. The detection confidence of candidate targets for clearing personnel is corrected based on the confidence of the cooperative relationship, and after verification of the cooperative relationship, the multi-target recognition result containing the cooperative operation unit and its confidence is output.

2. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis as described in claim 1, characterized in that, By using an explosion-proof industrial camera, continuous image frames of the bilge cleaning operation are captured at a preset frame rate to obtain a sequence of operation images of the bilge area of ​​a bulk carrier.

3. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis as described in claim 1, characterized in that, Target detection is performed on the operation image sequence to obtain candidate targets for cleaning machinery and personnel, specifically including: The current frame in the operation image sequence is input into a pre-trained deep convolutional neural network detector to extract candidate regions of the cleaning machinery and personnel in the image. Each candidate region is classified and bounding box regressed, and the bounding boxes, category labels, and detection confidence scores of the candidate targets for cleaning machinery and personnel are output.

4. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis as described in claim 1, characterized in that, Extract the operational attitude, motion status, and spatial location of candidate mechanical targets for cabin cleaning, specifically including: Joint point detection is performed on the image region corresponding to the candidate target of the cleaning machinery to obtain the joint point position of the auxiliary working parts of the cleaning machinery; The attitude angle of the auxiliary working components is calculated based on the position of the joint points, and the current action state of the cleaning machinery is determined according to the displacement changes of the joint points between consecutive frames and the preset action state discrimination rules. By combining camera calibration parameters, the pixel coordinates of the candidate targets for cleaning machinery in the image are converted into three-dimensional spatial positions within the ship's working space.

5. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis as described in claim 1, characterized in that, Based on the operational posture and motion state of the candidate cleaning machinery, a pre-constructed visual degradation field prediction model is used to predict the dynamic degradation distribution map of dust and illumination changes caused by the cleaning operation on the image plane, specifically including: Extract the motion amplitude and direction of the auxiliary working parts of the cleaning machinery in the current operating state, as well as the contact state between the auxiliary working parts of the cleaning machinery and the bottom plate of the ship, as driving parameters of the visual degradation field; The driving parameters are input into a pre-constructed visual degradation field prediction model, which includes a dust diffusion sub-model and an illumination change sub-model. The dust diffusion sub-model takes the dust origin location and dust diffusion parameters as input and outputs a dust degradation intensity value that characterizes the degree of visual degradation caused by dust to each pixel of the image. The illumination change sub-model takes the mechanical component pose and bulkhead structure parameters as input and outputs an illumination degradation intensity value that characterizes the degree of visual degradation caused by illumination change to each pixel of the image. By combining the dust degradation intensity value and the illumination degradation intensity value, a visual degradation intensity value that characterizes the degree of visual degradation caused by dust or illumination changes to image pixels is determined, and the dynamic degradation distribution map is generated.

6. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis as described in claim 1, characterized in that, The visual attenuation characteristics of the operational image sequence at different spatial locations affected by dust and illumination changes are evaluated based on the dynamic degradation distribution map, specifically including: The image plane of the job image sequence is divided into several grid units; Based on the dynamic degradation distribution map, the mean value of the visual degradation intensity of pixels in each grid cell is calculated, and the mean value is used as the visual attenuation feature of the corresponding spatial location.

7. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis as described in claim 1, characterized in that, Using the spatial location of the candidate cleaning machinery targets as a reference and the visual attenuation feature as a spatial condition, the confidence level of the cooperative relationship between each candidate cleaning personnel target and the candidate cleaning machinery target is calculated, specifically including: Calculate the spatial distance between the cleaning machinery and the cleaning personnel based on their spatial locations and the spatial locations of the candidate cleaning machinery and personnel. Based on the motion trajectories of the cleaning machinery and personnel between consecutive frames, determine the degree of consistency in the relative motion directions between the cleaning machinery and personnel. The consistency between the spatial distance and the relative motion direction is used as an explicit spatiotemporal correlation feature. Based on the explicit spatiotemporal correlation feature, explicit semantic reasoning is performed on the collaborative operation relationship between the cleaning machinery and the cleaning personnel to obtain an explicit collaboration index that characterizes the degree of spatiotemporal collaboration that can be directly observed. The visual attenuation features corresponding to the spatial location of each candidate target for clearing the cabin are obtained as implicit environmental features. Based on the implicit environmental features, implicit semantic reasoning is performed on the visual perception attenuation caused by dust and changes in light to obtain implicit environmental indicators that characterize the degree of impact of environmental degradation on personnel detection. By performing semantic reasoning on the explicit collaborative indicators and the implicit environmental indicators, and combining them with the preset safety rules for collaborative cleaning operations, the confidence level of the collaborative relationship between each candidate target for cleaning personnel and candidate target for cleaning machinery is determined.

8. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis as described in claim 7, characterized in that, The explicit collaboration indicators and the implicit environmental indicators are correlated through semantic reasoning. Combined with preset safety rules for collaborative cleaning operations, the confidence level of the collaboration relationship between candidate targets for cleaning personnel and candidate targets for cleaning machinery is determined. Specifically, this includes: By using the explicit collaborative indicators and the implicit environmental indicators as two independent sources of evidence, an evidence fusion framework for determining collaborative relationships is constructed. When the explicit coordination indicator suggests a spatiotemporal coordination relationship between the cleaning machinery and the cleaning personnel, and the original detection confidence level of the candidate personnel is lower than a preset confidence threshold, the implicit environmental indicator is introduced for evidence conflict analysis: If the collaborative behavior pattern indicated by the explicit collaborative indicator conforms to the human-machine collaborative operation behavior characteristics defined in the preset cabin clearing collaborative operation safety rules, and the visual attenuation intensity indicated by the implicit environmental indicator is higher than the preset attenuation threshold, then the evidence conflict between the explicit collaborative indicator and the original detection confidence is determined to be caused by environmental degradation. A confidence redistribution strategy is adopted to suppress the weight of the original detection confidence in the collaborative relationship determination, and the collaborative confidence calculated based on the explicit collaborative indicator is used as the collaborative relationship confidence.

9. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis as described in claim 8, characterized in that, The trust redistribution strategy includes: Based on the visual attenuation intensity indicated by implicit environmental indicators, a confidence discount factor is determined for the original detection confidence level. The visual attenuation intensity and the confidence discount factor have a preset positive correlation mapping relationship. The trust discount factor is used to discount the basic probability allocation value of the original detection confidence in the determination of collaborative relationship, thereby reducing the trust contribution of the original detection confidence in evidence fusion. The trust shares released by the discount process are redistributed to the propositions supporting the validity of cooperative relationships supported by explicit cooperative indicators, resulting in a modified basic probability allocation. Based on the corrected basic probability allocation, the confidence level of the cooperative relationship between each candidate target of the cleaning personnel and the candidate target of the cleaning machinery is calculated, so that the confidence level of the cooperative relationship is dominated by explicit cooperative indicators under the interference of environmental degradation.

10. The multi-target recognition method for bulk carrier tank cleaning operations based on visual intelligent analysis as described in claim 1, characterized in that, The detection confidence of candidate targets for clearing personnel is corrected based on the confidence of the cooperative relationship, and after verification of the cooperative relationship, a multi-target recognition result containing the cooperative work units and their confidence is output, specifically including: The original detection confidence of the candidate targets for clearing personnel is replaced with the confidence of the cooperative relationship to obtain the corrected personnel detection results; The candidate targets for cleaning personnel whose confidence level is higher than the preset confidence threshold after correction are associated with the candidate targets for cleaning machinery whose spatial location is closest to the candidate targets for cleaning personnel, and candidate collaborative operation units are constructed. The candidate collaborative operation units are verified to check the collaborative relationship. The relative motion patterns of the candidate targets for clearing personnel and the candidate targets for clearing machinery between consecutive frames are verified to meet the preset logic constraints for collaborative operation. The candidate collaborative operation units that pass the verification are determined as formal collaborative operation units. The output includes the multi-target recognition result containing the collaborative work unit and the confidence level of the collaborative work unit.

Images