A method and device for identifying the intention of a tower crane operation worker in a human-machine collaborative manner

Abstract

This invention belongs to the field of intelligent manufacturing technology. It discloses a method and device for recognizing the intention of tower crane operators in human-machine collaboration. The steps are as follows: (1) Obtain sample data, which includes frame images extracted from operation videos and corresponding visual features. Generate corresponding scene semantic descriptions based on a large language model and fuse visual and semantic information to obtain visual-semantic representations; (2) Construct a vector knowledge base containing safety rules and operation cases. Use visual-semantic features as query vectors and obtain related knowledge features through cosine similarity retrieval; (3) Perform multimodal fusion of visual features, visual-semantic representations and knowledge features to output the predicted probability of the worker's operation intention; (4) Input the related knowledge features, the worker's operation intention and its corresponding predicted probability into a multimodal large model to generate interpretable natural language output. This invention solves the problem that tower crane collaborative control systems have difficulty recognizing the worker's operation intention.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing technology, and more specifically, relates to a method and equipment for recognizing the intentions of tower crane operators in human-machine collaboration. Background Technology

[0002] With the trend towards intelligent and unmanned construction, tower cranes, as core construction equipment, are gradually transforming from traditional manual operation to intelligent control systems. In recent years, the new generation of "intelligent tower cranes" has gradually eliminated the operator's cab, adopting ground or remote control systems to achieve automatic navigation and path planning during the hoisting process, thereby significantly improving hoisting efficiency and the level of intelligence in on-site management.

[0003] However, in actual construction, tower crane operations inevitably involve direct coordination between humans and machines, especially in the hooking and unhooking processes. Rigging workers must work precisely with the tower crane to connect and release the hook. These processes are characterized by frequent human-machine interaction, complex dynamic changes, and a strong reliance on on-site decision-making. Even slight misjudgments can lead to falling loads, equipment collisions, or personal injury. Therefore, these processes are a high-risk area for tower crane operation safety accidents.

[0004] Traditional tower crane collaborative control systems primarily rely on image recognition technologies, such as target detection and worker tracking, to analyze the scene. However, this traditional system has limited effectiveness in reducing safety accidents. Therefore, improving the safety of tower crane operations has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and equipment for recognizing the intention of tower crane operators in human-machine collaboration, which aims to solve the problem that tower crane collaborative control systems have difficulty recognizing the intention of operators.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for recognizing the intention of tower crane operators in human-machine collaboration is provided, comprising the following steps: (1) Visual-semantic representation generation: acquire sample data, including sample frame images extracted from the work video and corresponding visual features, input the sample frame images into the large language model to generate the corresponding scene semantic description, and fuse the visual features and scene semantic description to obtain visual-semantic representation. (2) Knowledge-enhanced semantic retrieval: Construct a vector knowledge base containing safety rules and work cases, use visual-semantic features as query vectors, and retrieve related knowledge features related to the current work scenario from the vector knowledge base through cosine similarity; (3) Intention reasoning: Multimodal fusion of visual features, visual-semantic representations and associated knowledge features to obtain the predicted probability of workers' work intentions; (4) Semantic interpretation: The uncertainty of the prediction probability is quantified to obtain the confidence level of the prediction probability. After filtering the prediction efficiency based on the obtained confidence level, the associated knowledge features, worker's work intention and its corresponding prediction probability are input into the multimodal large model to generate the tower crane operator's intention, and the tower crane operator's intention is expressed in interpretable natural language.

[0007] Furthermore, sample frame images are selected based on the difference in grayscale values ​​and the difference in image similarity among frame images in historical operation videos under tower crane operating conditions; then, the sample data is determined through the sample frame images.

[0008] Furthermore, based on the fact that the inter-frame scoring function of the frame image is greater than a first threshold, the frame image is determined to be the sample frame image; No. The inter-frame scoring function for a frame image is: ,in, , , These are weight parameters that are greater than 0 and less than or equal to 1; No. The grayscale difference of a frame image is represented as follows: ;in, and These are the height and width of the frame image, respectively. Indicates the first The coordinates in the frame image are The grayscale of the points, Indicates the first The coordinates in the frame image are The grayscale of the points; No. The structural similarity difference of frame images is represented as follows: Where SSIM represents the first Frame image and the first Structural similarity of frame images; No. The perceptual hash differential representation of a frame image is as follows: That is, the perceptual hash difference of the t-th frame image is divided into the t-th frame. Perceptual hash of frame image and the first The Hamming distance between the perceptual hashes of the frame images; where, For the first Frame image, For the first Frame image; for The resulting grayscale image.

[0009] Furthermore, if the inter-frame scoring function of the frame image is greater than a first threshold, and the time interval between the frame image and the previous sample frame image is greater than a preset time interval, then the frame image is determined to be the sample frame image.

[0010] Furthermore, the visual-semantic representation is generated by a large language model based on sample data. During the inference stage, the large language model is used to perform scene semantic understanding on the sample frame images corresponding to the task video, and fuses the visual features of the frame images with the corresponding scene semantic descriptions to obtain the visual-semantic representation.

[0011] Furthermore, a vector knowledge base containing safety rules and work cases is constructed; using the visual-semantic representation as the query vector, knowledge information related to the current work scenario is retrieved from the vector knowledge base through semantic similarity calculation to obtain associated knowledge features.

[0012] Furthermore, in the intention reasoning process, the visual features, visual-semantic representations, and associated knowledge features obtained through semantic retrieval are jointly input into the intention reasoning module of the multimodal large model. The intention reasoning module generates the predicted probability of the worker's work intention through multimodal fusion. Specifically, the intention reasoning module employs a multimodal fusion method: Let the visual features be... Visual-semantic representation is The characteristics of related knowledge are ,in, , , Both represent the feature representations after feature alignment, and the fused features satisfy: The weighting coefficients satisfy the following conditions: Furthermore, each weight is obtained by normalizing the cosine similarity between the visual-semantic representation and the corresponding feature; the predicted probability is obtained through... Received, among which For the intention prediction function, This is the predicted probability vector.

[0013] Furthermore, after obtaining the predicted probability of the worker's work intention, the predicted probability is quantified for uncertainty, and a prompt word template is constructed by combining the intention prediction result and the associated knowledge features. The prompt word template is then input into a multimodal large model to generate interpretable natural language output containing the tower crane operation intention, confidence information, and reasoning basis. The uncertainty quantification includes: calculating the variance index based on the predicted probability vector from N random forward propagations. With entropy index Based on comprehensive indicators Characterize the prediction confidence information, where , , These are the weighting coefficients.

[0014] The present invention also provides a tower crane operator intention recognition system for human-machine collaboration. The system includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it performs the tower crane operator intention recognition method for human-machine collaboration as described above.

[0015] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the human-machine collaboration-oriented tower crane operator intent recognition method as described above.

[0016] In summary, compared with the prior art, the method and equipment for recognizing the intentions of tower crane operators oriented towards human-machine collaboration provided by the present invention have the following beneficial effects: 1. This invention integrates the visual language understanding capabilities of a multimodal large model, a structured knowledge reasoning mechanism, and a semantic vector alignment method to achieve worker intent recognition in complex construction site scenarios. It can realize intent-level recognition based on language understanding, breaking through the limitations of previous image classification or state estimation methods.

[0017] 2. Sample frame images are selected based on the differences in grayscale values ​​and image similarity between frame images in historical operation videos under tower crane operating conditions; then, the sample data is determined using these sample frame images. This approach improves upon the problems of high cost, excessive redundant information, and detrimental effects on model training associated with traditional full-frame annotation. Furthermore, by combining inter-frame scoring functions (integrating differences in grayscale values, structural similarity, and perceptual hashing), sample frames are selected, thereby improving sample data quality and model training efficiency from multiple dimensions.

[0018] 3. This invention uses a dynamic thresholding method to determine the inter-frame difference threshold, which allows the inter-frame difference threshold to be adaptively adjusted according to changes in scene lighting, motion amplitude, and background, thereby improving the robustness of the sample frame selection process and enhancing the applicability of the method in different working environments.

[0019] 4. Construct a vector knowledge base based on security rules and historical operation cases, and use visual-semantic representations as query vectors for similarity retrieval. Introduce domain knowledge during the reasoning process to compensate for implicit security information that is difficult to reflect in visual or semantic features, thereby improving the reliability and semantic integrity of intent recognition.

[0020] 5. Multimodal fusion of visual features, visual-semantic representations and knowledge features is carried out. By using attention mechanisms, feature alignment or weighted fusion and other methods, the correlation between multi-source information is established, so that the model can simultaneously process worker action semantics, scene context and related safety rules, thereby improving the accuracy and robustness of work intention prediction.

[0021] 6. An uncertainty quantification mechanism is introduced. By calculating the degree of change in the predicted distribution through multiple random forward propagations, an uncertainty value reflecting the reliability of the model is obtained. This can effectively identify prediction risks, assist operators in judging prediction results, and improve the reliability of the system in safety-critical scenarios.

[0022] 7. Based on the intention prediction results, uncertainty indicators, and related knowledge features, a prompt word template is constructed and input into a multimodal big data model to generate a natural language explanation. The output results include: reasoning basis, risk warnings, and relevant safety rules, which can increase the likelihood that operators trust the worker's work intentions output by the multimodal big data model. Attached Figure Description

[0023] Figure 1 This is a flowchart of a method for recognizing the intentions of tower crane operators in a human-machine collaborative manner, provided by an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] With the development of intelligent and unmanned tower cranes, the traditional model relying on the tower crane operator's visual judgment of worker actions to complete coordinated operations is gradually being replaced. In high-risk processes such as hooking and unhooking, tower crane operations increasingly rely on automatic perception of worker behavior and safety decision support. However, due to the complex working environment, diverse worker actions, and ambiguous context, existing methods struggle to accurately understand the worker's actual operational intentions, easily leading to misoperation or safety accidents. Therefore, this embodiment proposes a worker intention recognition method for intelligent tower crane human-machine collaboration scenarios. This method acquires operational video through a vision device installed on the top of the tower crane hook or other locations, and uses this as a basis for keyframe extraction, worker behavior description generation, multimodal knowledge modeling, and semantic reasoning, thereby achieving interpretable worker intention recognition driven by a visual language model. This method elevates traditional video analysis from "action recognition" to "intention understanding," achieving structured reasoning of the worker's operational purpose by integrating behavior descriptions generated by a large model and soft knowledge rules, thus providing safer and more intelligent collaborative judgment support for the tower crane and improving operational safety.

[0026] This invention provides a method for recognizing the intentions of tower crane operators in a human-machine collaborative manner. First, the following explanations are provided for several terms used in the embodiments of this invention: Visual-Semantic Representation refers to the feature vector obtained by fusing visual features with textual semantic descriptions. Visual features reflect the spatial structure, action postures, and object relationships in an image; semantic descriptions express the intent of actions, scene relationships, and potential safety implications. By aligning and jointly modeling visual features and semantic information, a unified representation with both perceptual and semantic understanding capabilities is obtained, enabling a more comprehensive understanding of worker behavior.

[0027] Multimodal fusion refers to the alignment, integration, and joint modeling of heterogeneous information from different sources, such as visual features, visual-semantic representations, and retrieved related knowledge features, to form a high-dimensional fused feature capable of comprehensively representing a scene. Visual features mainly reflect worker actions and image structure information; visual-semantic representations provide semantic-level scene understanding; and related knowledge features supplement knowledge in areas such as safety rules. This method, by jointly combining features from multiple sources, enables the model to simultaneously utilize perceptual information, semantic descriptions, and external knowledge to improve the accuracy of work intention reasoning.

[0028] Uncertainty quantification refers to estimating the cognitive and accidental uncertainties that may exist in the prediction results after obtaining the intended prediction results, by evaluating the stability and reliability of the model output. This method involves performing multiple random forward propagations on the model, introducing a random deactivation mechanism to obtain the degree of change in the prediction results, and calculating the entropy value or confidence interval of the results accordingly. This mechanism can indicate potential risk scenarios, enabling the model to have a higher judgment ability when facing ambiguous actions or abnormal situations.

[0029] Retrieval-Augmented Generation (RAG) is an integrated AI reasoning mechanism that combines a generative model with an external knowledge base or vector database. This allows the model to reference relevant information retrieved from external sources when generating text or task intent results, thereby improving the accuracy and contextual relevance of the generated results. When processing input information (such as behavioral descriptions generated from task frame images), the RAG model first retrieves highly relevant information (such as historical behavioral intent vectors) from a pre-built knowledge base. Then, it uses the input and retrieval results together as generation conditions and feeds them into a multimodal large language model, outputting reasoning results or language explanations. This method is widely used in multimodal tasks for knowledge augmentation, improving reasoning interpretability, and context completion.

[0030] The identified method can improve safety during tower crane lifting operations, and it mainly includes the following steps: Step 1, Visual-Semantic Representation Generation: Obtain sample data, which includes sample frame images extracted from the job video and corresponding visual features. Input the sample frame images into a large language model to generate corresponding scene semantic descriptions. Fuse the visual features and scene semantic descriptions to obtain visual-semantic representations.

[0031] Specifically, sample frame images are selected based on the grayscale value difference and image similarity difference of frame images in historical operation videos under tower crane operating conditions; then, the sample data is determined through the sample frame images. This approach improves upon the problems of high cost, excessive redundant information, and adverse effects on model training associated with traditional full-frame annotation, thereby enhancing data quality and training efficiency.

[0032] A frame image is determined to be a sample frame image based on the fact that its inter-frame scoring function is greater than a first threshold; the inter-frame scoring function for the t-th frame image is: ,in, , , These are weight parameters that are greater than 0 and less than or equal to 1; The difference in grayscale values ​​of the t-th frame image can be represented as: .in, and These are the height and width of the frame image, respectively. Indicates the first The coordinates in the frame image are The grayscale of the points, Indicates the first The coordinates in the frame image are The grayscale of the points.

[0033] No. The structural similarity difference of a frame image can be represented as: Where SSIM represents the first... Frame image and the first Structural similarity of frame images.

[0034] No. The perceptual hash difference of a frame image can be represented as: That is, the first The perceptual hash difference of the frame image is divided into the first... Perceptual hash of frame image and the first The Hamming distance between the perceptual hashes of the frame images. For the first Frame image, For the first Frame image; for The resulting grayscale image.

[0035] In one implementation, a frame image is determined to be a sample frame image based on the fact that the inter-frame scoring function of the frame image is greater than a first threshold and the time interval between the frame image and the previous frame image is greater than a preset time interval. Since the change in a worker's work intention is a gradual process, having at least a preset time interval between sample frame images allows for the acquisition of comprehensive sample data while reducing the amount of sample data.

[0036] The first threshold is an inter-frame difference threshold determined by a dynamic thresholding method. This increases the accuracy of the first threshold and improves the quality of the sample data.

[0037] Furthermore, after determining the sample frame image, the method further includes: extracting visual features from the sample frame image to obtain visual features that reflect the worker's posture, the working environment, and the relationship between objects; the visual features can be obtained through a visual encoder or other image feature extraction models.

[0038] Furthermore, the sample frame image is input into a large language model to generate a corresponding scene semantic description; the scene semantic description is used to characterize the work actions, scene relationships and potential safety semantic information contained in the sample frame image.

[0039] Furthermore, visual and semantic features are aligned and fused to construct a visual-semantic representation, thereby providing a unified multimodal input for subsequent knowledge enhancement and intent reasoning.

[0040] Step 2, Knowledge-enhanced Semantic Retrieval: Construct a vector knowledge base containing safety rules and job cases. Using visual-semantic features as query vectors, retrieve relevant knowledge features related to the current job scenario from the vector knowledge base through cosine similarity.

[0041] Specifically, the vector knowledge base includes: inputting safety rule texts and historical operation case texts related to tower crane operations into the text embedding model, generating corresponding vector representations, and storing them in the vector knowledge base.

[0042] Furthermore, the visual-semantic representation obtained in step one is used as a query vector, and the similarity between the query vector and each knowledge vector in the vector knowledge base is calculated to obtain the most relevant associated knowledge features to the current frame image. The associated knowledge features include the corresponding safety rule fragments, job case descriptions and their text semantic vectors, which are used as knowledge features to supplement the implicit safety semantics that are difficult to reflect in the visual-semantic representation.

[0043] Step 3, Intent Reasoning: Multimodal fusion of visual features, visual-semantic representations, and associated knowledge features is performed to obtain the predicted probability of the worker's work intention.

[0044] Specifically, visual features Visual-semantic representation With knowledge characteristics By inputting the same multimodal fusion module, a unified representation of multi-source information is obtained through feature alignment, weighted fusion, attention mechanisms, and other methods. .in The multimodal fusion function is as follows: The weighting coefficients satisfy the following conditions: Furthermore, each weight is obtained by normalizing the cosine similarity between the visual-semantic representation and the corresponding feature. The fused comprehensive representation... Key elements used to reflect a worker's intentions in the current frame include: action semantics, environmental context, and relevant safety knowledge.

[0045] Furthermore, the fused comprehensive representation is input into the intention prediction module, and the predicted work intention of the worker in the current frame is obtained through classification probability output. The prediction process can be represented as follows: ,in For intention prediction function; Let be the predicted probability vector, where This represents the total number of task intent categories. This indicates that the current worker belongs to the [number]th [group]. The probability of a task intention. The task intention includes various task states such as hooking, unhooking, preparing, and completing.

[0046] Step 4, Semantic Interpretation: The uncertainty of the prediction probability is quantified to obtain the confidence level of the prediction probability. After filtering the prediction efficiency based on the obtained confidence level, the associated knowledge features, worker operation intentions and their corresponding prediction probabilities are input into the multimodal large model to generate the tower crane operation worker intentions, which are then expressed in interpretable natural language.

[0047] Specifically, uncertainty quantification includes: performing multiple random forward propagations on the intention prediction module, calculating the degree of change in the prediction results across these multiple inferences, and obtaining an uncertainty index representing the model's reliability. The worker's work intention probability prediction vector is... ,in, The number of candidate job intentions. This indicates that the worker in the current frame belongs to the first... Predict the probability of task intent; while keeping the model structure unchanged, perform the intent reasoning module... After several random forward propagations, N sets of prediction probability vectors are obtained: Therefore, the expected vector of the predicted probability can be calculated: .

[0048] Furthermore, the variance of the prediction distribution across multiple inference processes can be calculated to obtain an uncertainty index: Simultaneously, the entropy value can be calculated by combining the predicted probability distribution as a supplementary indicator, which, together with the variance indicator, constitutes a comprehensive uncertainty score. ,in The expected vector The elements in the index; this index can be used to characterize the predictive stability of the model in the current working scenario, and help to determine whether the prediction results have high risk or ambiguity. A larger value indicates greater fluctuation in the prediction result and lower reliability. A higher value indicates a more dispersed distribution of prediction results and a weaker ability to distinguish intentions.

[0049] Furthermore, a prompt word template is constructed based on the intent prediction results, uncertainty indicators, and retrieved related knowledge features. The prompt word template is then input into a multimodal model to generate a natural language explanation of the current work scenario. The explanation includes the reasoning basis for the work intent, risk warnings, and references to relevant knowledge.

[0050] The present invention will be further described in detail below with reference to specific embodiments.

[0051] Please see Figure 1 The worker intent identification method may include the following steps: S100, Visual-Semantic Representation Generation: Acquire sample data, which includes sample frame images extracted from the job video and corresponding visual features. Input the sample frame images into a large language model to generate corresponding scene semantic descriptions. Fuse the visual features and scene semantic descriptions to obtain visual-semantic representations.

[0052] In this embodiment, the operation video comes from a monitoring camera deployed at a suitable location, such as the top of the tower crane hook or above the operation area. The operation video content includes the interaction between the worker and the hoisted object during the operation. The video format is a standard RGB image sequence with a frame rate of 25 to 30 frames per second. The duration of a single operation video is approximately 20 to 40 seconds, recording the interaction between the worker and the hoisted object during the operation.

[0053] In tower crane operation videos, adjacent frames are often visually similar. Directly processing all frames would generate a large amount of redundant information. Therefore, this embodiment introduces a sample frame extraction strategy that is aware of inter-frame changes, selecting frames with significant semantic changes from the original operation video to construct a high-quality sample dataset.

[0054] Specifically, an inter-frame scoring function is constructed for adjacent frame images, and the inter-frame scoring function is used to quantize the first frame. Frame image and the first The degree of change between frames is expressed as: ,in , , These can be weight parameters that are greater than 0 and less than or equal to 1, used to balance the influence of different difference indices.

[0055] At this time, the The grayscale difference of a frame image can be represented as: .in, and These are the height and width of the frame image, respectively. Indicates the first The coordinates in the frame image are The grayscale of the points, Indicates the first The coordinates in the frame image are The grayscale of the points.

[0056] No. The structural similarity difference of a frame image can be represented as: Where SSIM represents the first... Frame image and the first Structural similarity of frame images.

[0057] No. The perceptual hash difference of a frame image can be represented as: That is, the first The perceptual hash difference of the frame image is divided into the first... Perceptual hash of frame image and the first The Hamming distance between the perceptual hashes of the frame images. For the first Frame image, For the first Frame image; for The resulting grayscale image.

[0058] When the Inter-frame score corresponding to the frame image If the value is greater than the first threshold and the time interval between the current image and the previous copy frame is greater than a preset time interval, then the first frame is determined to be... The frame image is a sample frame image.

[0059] After determining the sample frame images, a sample dataset can be constructed based on the sample frame images. The sample data includes frame images extracted from the work video and their corresponding visual features. The visual features may include the original RGB pixels of the frame images, preprocessed normalized image data, cropped images of the worker's target area, and intermediate visual features extracted based on the visual encoder. Furthermore, the sample data can be screened through manual review or automatic quality detection mechanisms to remove blurry, unobstructed, or inadequate frame images that fail to reflect the work scene, thereby ensuring the validity and stability of the sample dataset.

[0060] Therefore, based on the sample frame images after keyframe filtering, high-quality sample data for visual-semantic representation generation can be constructed, providing reliable input for subsequent multimodal semantic modeling, knowledge-enhanced retrieval, and intent reasoning.

[0061] Furthermore, sample frame images and their corresponding visual features are obtained from the work video, and the frame images are input into a visual encoder to extract the visual features of the images; in some embodiments, visual feature extraction can be represented as... ,in For the first Frame sample image, It is a visual encoder.

[0062] Furthermore, the sample frame images are input into a large language model to generate corresponding scene semantic descriptions, and the semantic descriptions are converted into semantic features by a text encoder; in some implementations, visual feature extraction can be represented as... ,in For the first Frame sample image, For text encoders.

[0063] Furthermore, visual and semantic features are fused through attention mechanisms or feature weighting to obtain a visual-semantic representation containing multi-source information.

[0064] In some implementations, the fusion of visual and semantic features can be achieved through weighted fusion or feature concatenation, for example: (Weighted fusion), or (Splicing and blending), among which Visual features extracted by the visual encoder. The semantic features transformed by the text encoder These are learnable weights.

[0065] S200. Knowledge-enhanced semantic retrieval: Construct a vector knowledge base containing safety rules and work cases, use visual-semantic features as query vectors, and retrieve related knowledge features related to the current work scenario from the vector knowledge base through cosine similarity.

[0066] After obtaining the visual-semantic representation, a vector knowledge base can be established based on knowledge text related to tower crane operations. This vector knowledge base provides structured and searchable semantic support for subsequent intent reasoning. The vector knowledge base includes safety rule texts, historical operation case texts, and their corresponding vector representations.

[0067] Furthermore, textual content such as tower crane operation safety rules, accident cases, and standard operating procedures (SOPs) can be input into the text embedding model to obtain corresponding semantic vectors, which can then be stored and managed through a vector knowledge base. The text embedding model can be a Transformer-based text encoder, such as Sentence-BERT, E5, or BGE, capable of mapping structured or unstructured text data from different sources to a unified vector space.

[0068] Furthermore, the obtained visual-semantic features are used as query vectors, and the similarity between the query vector and each knowledge vector in the vector knowledge base is calculated to retrieve the knowledge entries most relevant to the current job frame. The retrieval results may include safety rule fragments, descriptions of relevant job cases, and their semantic embedding vectors, which are used to characterize the potential safety semantics and operational specifications involved in the current frame image.

[0069] In some implementations, the retrieval similarity calculation can be expressed as: ,in For visual-semantic representation, For the first in the vector knowledge base A set of knowledge vectors is generated, and a group of vectors with similarity higher than a set threshold is selected, or a clustering method is used to select the center vector of the corresponding category. Then, based on the above similar vector set, vector fusion strategies such as weighted averaging and attention mechanisms can be used to generate associated knowledge features representing the current work scenario. Thus, these associated knowledge features, while retaining safety rules and work experience information highly relevant to the current visual-semantic representation, serve as one of the inputs to the intent reasoning module, constraining and enhancing the worker's reasoning process regarding work intent.

[0070] S300, Intent Reasoning: Multimodal fusion of visual features, visual-semantic representations and associated knowledge features to obtain the predicted probability of workers' work intentions.

[0071] After acquiring visual-semantic features and associated knowledge features, the worker's work intention can be predicted based on a multimodal fusion mechanism. The intention reasoning module takes visual features, visual-semantic representations, and associated knowledge features as input, and generates a comprehensive semantic vector that reflects the current work state through feature alignment and fusion strategies. This comprehensive semantic vector is used to describe the worker's action information, semantic context information, and corresponding safety rule knowledge in the current frame image.

[0072] The visual features can be obtained by convolution of frame images using a visual encoder or by Transformer encoding. These visual features reflect worker posture, actions, positional relationships, and interaction information between the worker and the suspended load. The visual-semantic representation is formed by fusing visual features with semantic description features generated by a large language model. This representation reflects implicit information that is difficult to infer solely from visual signals, such as scene semantics, action descriptions, and temporal context. The associated knowledge features are derived from semantic vectors retrieved from a vector knowledge base based on cosine similarity. These associated knowledge features provide a regularized semantic structure for operations, experiential operational patterns, and corresponding safety knowledge prompts.

[0073] Furthermore, the three types of features are aligned and fused. First, feature alignment is performed: visual features, visual-semantic representations, and related knowledge features are mapped to the same-dimensional semantic space through a fully connected layer. Let the visual features be... Visual-semantic representation is Knowledge characteristics are ,in, , , If each represents a feature representation after feature alignment, then the fused feature can be represented as: In some implementations, the feature alignment can be achieved through a fully connected layer or an adaptive average pooling layer, uniformly mapping visual features, visual-semantic representations, and knowledge features to... Semantic space (e.g.) or This is to ensure the feasibility of subsequent integration.

[0074] in, The multimodal fusion function is implemented using a weighted fusion method based on attention weights, specifically as follows: Since visual-semantic representations can simultaneously reflect scene semantics and task context information, they are used as fusion guiding features to determine the fusion weights of each modality feature. The weight coefficients satisfy the following conditions: Furthermore, each weight is obtained by normalizing the cosine similarity between the visual-semantic representation and the corresponding feature: ,in The cosine similarity function is used. , The calculation method is the same.

[0075] In some implementations, attention-guided weighted fusion can be achieved by calculating the attention weights of visual-semantic representations and each feature. Implementation, in which For visual-semantic representation, , These are joint features composed of visual features and knowledge features, respectively. For feature dimensions.

[0076] Furthermore, in the multimodal fusion process, visual-semantic representation is used as the guiding feature for fusion, and visual features and associated knowledge features are weighted and modeled. By using the scene semantics and action description information contained in the visual-semantic representation, the model is guided to focus on visual features related to the current task stage and safety rule knowledge consistent with the current semantic context during the fusion process, thereby reducing the interference of redundant information unrelated to the task intent on the inference results.

[0077] Furthermore, the fused feature representation The input intent prediction module jointly discriminates multiple candidate work intents. This module maps the fused features and outputs the predicted probability of the worker's work intent in the current frame. The prediction process can be represented as follows: .

[0078] In the formula, This represents the intent prediction function, used to map fused features to the job intent space; The predicted probability vector for the worker's work intention is as follows: ,in, This represents the total number of job intention categories, which may include various job states such as hooking, unhooking, preparing, and completing. Let represent the probability that the current worker belongs to the i-th type of job intention, and satisfy . This is used to characterize the confidence distribution of the current worker across multiple candidate job intention categories. In some implementations, the intention prediction function... It can be implemented by a fully connected layer with a Softmax activation function, a lightweight Transformer decoder, or a multilayer perceptron (MLP). By performing a non-linear mapping on the fused features, it outputs the predicted probability of each job intent category.

[0079] Therefore, through multimodal fusion, it is possible to identify intent categories that cannot be inferred from visual features alone but can be clearly judged under the joint constraints of semantic description and external knowledge. Thus, even in the presence of visual occlusion, incomplete actions, or ambiguous scene semantics, it can still output stable intent prediction results.

[0080] S400, Semantic Interpretation: The uncertainty of the prediction probability is quantified to obtain the confidence level of the prediction probability. After filtering the prediction efficiency based on the obtained confidence level, the associated knowledge features, worker's work intention and its corresponding prediction probability are input into the multimodal large model to generate the tower crane operator's intention, which is then expressed in interpretable natural language.

[0081] This step is used to provide an interpretable semantic representation of the prediction results. After obtaining the worker's work intention prediction results, uncertainty quantification is performed on the prediction results. Uncertainty quantification is used to measure the reliability of the model's judgment on the current frame image, and can be achieved through temperature calibration, entropy calculation, confidence interval estimation, or a Bayesian approximation method based on random forward propagation. After obtaining the credibility index, the accuracy and interpretability of intent recognition are improved by introducing a retrieval-enhanced generation (RAG) mechanism, which can further generate interpretable language output.

[0082] In this embodiment, the worker's work intention probability prediction vector is: In the formula, The number of candidate job intentions. This indicates that the worker in the current frame belongs to the first... Predicted probability of the type of task intention.

[0083] Furthermore, uncertainty quantification can be achieved through multiple random forward propagations. While maintaining the model structure unchanged, the intent reasoning module is executed... After one random forward propagation, the corresponding... Group prediction probability vector: .

[0084] Therefore, the expected vector of the predicted probability can be calculated: The number of random forward propagations. It can be adjusted according to the actual scenario (e.g.) Specifically, this can be achieved through Monte Carlo dropout.

[0085] Furthermore, a comprehensive uncertainty index can be calculated: ,in, , These are the weighting coefficients. To predict the variance distributed across multiple inference processes, it can be used to characterize the uncertainty of the model's judgment of the worker's work intention in the current frame image. The larger the value, the greater the fluctuation of the prediction result in multiple inferences, and the lower the corresponding prediction reliability. Reflects the degree of dispersion of the predicted probability distribution: When the prediction results are highly concentrated in a certain job intention category, the entropy value is low; when the prediction results are scattered across multiple intention categories, the entropy value is high.

[0086] Furthermore, obtain uncertainty indicators Subsequently, the prediction results, uncertainty indicators, visual-semantic representations, and associated knowledge features are used together for interpretation generation. Visual-semantic representations reflect the semantics of worker actions, scene context, and detailed information in the current frame image; retrieved associated knowledge features provide external reasoning support for the interpretation process, including safety rules, work process cases, or historical experience knowledge related to the current scene; uncertainty indicators guide the annotation and adjustment of the confidence level of the prediction results during the interpretation stage. Thus, a semantic input describing the current work state can be constructed, providing a semantic foundation for interpretation generation based on a retrieval-enhanced generation mechanism.

[0087] In some implementations, the prompt word template may be represented as: "The probability of predicting the worker's work intention in the current frame is..." The model prediction uncertainty is The associated security rule is {the retrieved security rule text}. Please combine the above information to generate a natural language explanation that includes the operational intent, prediction confidence level, risk level, and reasoning basis. Furthermore, prompt word templates in natural language form can be provided. Understandably, a prompt word template is a structured expression framework that guides the AI ​​model to generate answers through predefined semantic patterns, improving the efficiency and accuracy of information interaction. The prompt word template may include instructional content for the model, question-based guidance content, and semantically structured prompt content. Understandably, in embodiments of the invention, designing question templates allows the output worker's work intention label to include an explanation of why the model made such a decision, i.e., the reasoning process for deriving the worker's intention based on worker behavior.

[0088] For example, the prompt template could be "Please output the worker's intention based on the description of the worker's work behavior, and at the same time output the reasoning process." Another example is "Please output the worker's intention based on the description of the worker's work behavior, and at the same time output why this worker's intention is derived from the description of the worker's work behavior."

[0089] Furthermore, visual-semantic representations, knowledge features, intent prediction results, and prompts are input into a multimodal large model to generate structured, natural, and highly readable explanatory text. In a typical generated result, the model can output the following information: the worker is currently approaching the hook and facing it, therefore the predicted work intent is to hook the object; due to occlusion in the current view, the model indicates high uncertainty and reminds the operator to be aware of possible misjudgments. In addition, the model can also cite retrieved safety rules, such as "hooking operations must ensure the suspended object is stationary" or "operators should remain within visual range to give instructions."

[0090] Therefore, the system can output a natural language explanation containing multiple pieces of information for each predicted intent: including the worker's action basis, scene semantics, referenced safety rules, prediction confidence level, and potential risk factors. In practical applications, this explanation can be used to assist human-machine collaborative decision-making and also to provide intuitive safety prompts for regulatory personnel.

[0091] Understandably, tower crane operators also judge the worker's intentions based on their own observations and descriptions of the worker's work behavior. When the worker's intention output by the method of this embodiment is inconsistent with the operator's own judgment, the operator may insist on their own judgment, even if the judgment is incorrect. However, by using question templates, and when the worker's intention output by the artificial intelligence model includes the reasoning process of deriving the worker's intention from the worker's behavior, the likelihood of the operator trusting the worker's work intention label output by the artificial intelligence model can be increased, which helps the operator to perform safe operations and improves operational safety.

[0092] This invention achieves worker intent recognition in complex construction site scenarios by integrating the visual language understanding capabilities of a multimodal large model, a structured knowledge reasoning mechanism, and a semantic vector alignment method. Compared with traditional target detection or behavior recognition methods, the embodiments of this invention have the following significant advantages: (1) It can achieve intent-level recognition based on language understanding, breaking through the limitations of previous image classification or state estimation methods; (2) It has interpretability, outputting the reasoning path of worker intent through a generative thought chain; (3) It supports dynamic updates of the knowledge base and rule induction, and is suitable for diverse construction scenarios; (4) It is conducive to deep integration with the control logic of intelligent tower cranes, providing prior judgment support for unmanned hoisting operations.

[0093] This method is applicable to the collaborative control system for lifting and hoisting in the new generation of intelligent construction sites, and can be widely used in scenarios such as unmanned tower crane cruise hoisting, building industrialization, and intelligent construction management.

[0094] The present invention also provides a tower crane operator intention recognition system for human-machine collaboration. The system includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it performs the tower crane operator intention recognition method for human-machine collaboration as described above.

[0095] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the human-machine collaboration-oriented tower crane operator intent recognition method as described above.

[0096] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for recognizing worker intention in a tower crane operation, the method being oriented to human-machine collaboration, characterized in that, The steps are as follows: (1) Visual-semantic representation generation: acquire sample data, including sample frame images extracted from the work video and corresponding visual features, input the sample frame images into the large language model to generate the corresponding scene semantic description, and fuse the visual features and scene semantic description to obtain visual-semantic representation. (2) Knowledge-enhanced semantic retrieval: Construct a vector knowledge base containing safety rules and work cases, use visual-semantic features as query vectors, and retrieve related knowledge features related to the current work scenario from the vector knowledge base through cosine similarity; (3) Intention reasoning: Multimodal fusion of visual features, visual-semantic representations and associated knowledge features to obtain the predicted probability of workers' work intentions; (4) Semantic interpretation: The uncertainty of the prediction probability is quantified to obtain the confidence level of the prediction probability. After filtering the prediction efficiency based on the obtained confidence level, the associated knowledge features, worker's work intention and its corresponding prediction probability are input into the multimodal large model to generate the tower crane operator's intention, and the tower crane operator's intention is expressed in interpretable natural language.

2. The tower crane operator intent recognition method for human-machine collaboration as described in claim 1, characterized in that: Sample frame images are selected based on the difference in grayscale values ​​and the difference in image similarity between frame images in historical operation videos under tower crane operating conditions; then the sample data is determined through the sample frame images.

3. The tower crane operator intent recognition method for human-machine collaboration as described in claim 2, characterized in that: The frame image is determined to be the sample frame image based on the fact that the inter-frame scoring function of the frame image is greater than a first threshold. No. The inter-frame scoring function for a frame image is: ,in, , , These are weight parameters that are greater than 0 and less than or equal to 1; No. The grayscale difference of a frame image is represented as follows: ;in, and These are the height and width of the frame image, respectively. Indicates the first The coordinates in the frame image are The grayscale of the points, Indicates the first The coordinates in the frame image are The grayscale of the points; No. The structural similarity difference of frame images is represented as follows: Where SSIM represents the first Frame image and the first Structural similarity of frame images; No. The perceptual hash differential representation of a frame image is as follows: That is, the perceptual hash difference of the t-th frame image is divided into the t-th frame. Perceptual hash of frame image and the first The Hamming distance between the perceptual hashes of the frame images; where, For the first Frame image, For the first Frame image; for The resulting grayscale image.

4. The tower crane operator intent recognition method for human-machine collaboration as described in claim 3, characterized in that: If the inter-frame scoring function of the frame image is greater than a first threshold, and the time interval between the frame image and the previous sample frame image is greater than a preset time interval, then the frame image is determined to be the sample frame image.

5. The tower crane operator intent recognition method for human-machine collaboration as described in claim 1, characterized in that: The visual-semantic representation is generated by a large language model based on sample data. During the inference stage, the large language model is used to perform scene semantic understanding on the sample frame images corresponding to the task video, and fuses the visual features of the frame images with the corresponding scene semantic descriptions to obtain the visual-semantic representation.

6. The method according to any one of claims 1-5, characterized in that: Knowledge-enhanced semantic retrieval includes: constructing a vector knowledge base containing safety rules and job cases; using the visual-semantic representation as a query vector, retrieving knowledge information related to the current job scenario from the vector knowledge base through semantic similarity calculation, and obtaining associated knowledge features.

7. The tower crane operator intent recognition method for human-machine collaboration as described in any one of claims 1-5, characterized in that: In the intention reasoning process, the visual features, visual-semantic representations, and associated knowledge features obtained through semantic retrieval are jointly input into the intention reasoning module of the multimodal large model. The intention reasoning module generates a predicted probability of the worker's work intention through multimodal fusion. Specifically, the intention reasoning module employs a multimodal fusion approach: Let the visual features be... Visual-semantic representation is The characteristics of related knowledge are ,in, , , Both represent the feature representations after feature alignment, and the fused features satisfy: The weighting coefficients satisfy the following conditions: Furthermore, each weight is obtained by normalizing the cosine similarity between the visual-semantic representation and the corresponding feature; the predicted probability is obtained through... Received, among which For the intention prediction function, This is the predicted probability vector.

8. The tower crane operator intent recognition method for human-machine collaboration as described in any one of claims 1-5, characterized in that: After obtaining the predicted probability of the worker's work intention, the uncertainty of the predicted probability is quantified, and a prompt word template is constructed by combining the intention prediction result and the associated knowledge features. The prompt word template is input into a multimodal large model to generate interpretable natural language output containing the tower crane operation intention, confidence information, and reasoning basis. The uncertainty quantification includes: calculating the variance index based on the predicted probability vector of N random forward propagations. With entropy index Based on comprehensive indicators Characterize the prediction confidence information, where , , These are the weighting coefficients.

9. A tower crane operator intent recognition system for human-machine collaboration, characterized in that: The system includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it performs the tower crane operator intent recognition method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the human-machine collaboration-oriented tower crane operator intent recognition method according to any one of claims 1-8.

Images