A tower crane control method and a tower crane controller based on a multi-mode control model

Abstract

The embodiment of the application provides a tower crane control method and a tower crane controller based on a multi-mode control model, the method comprising: obtaining multi-mode data of a current site of the tower crane, the multi-mode data comprising environment state data, position state data, environment image data and natural language operation instructions; a multi-mode control model of the tower crane extracts feature vectors of each data in the multi-mode data respectively, and fuses into a global feature vector; the multi-mode control model outputs a control instruction for controlling movement actions of each shaft of the tower crane according to the global feature vector; and the multi-mode control model comprises at least two selectable movement action generation strategies. In the embodiment of the application, the environment image, the operation instruction, the environment state and the position state of the tower crane are fused by the multi-mode control model to generate a global feature vector, and the tower crane is intelligently controlled by pure vision according to at least two selectable strategies, thereby adapting to various working condition scenes.

Description

Technical Field

[0001] This application relates to the field of automatic control technology, and in particular to a tower crane control method and tower crane controller based on a multi-mode control model. Background Technology

[0002] Currently, there is no dedicated VLA model for tower cranes in the field of tower crane automation control. General VLA models are designed for service robots, robotic arms, and passenger vehicles, and are not adapted to the three-dimensional motion of tower cranes (luffing / slewing / lifting), hook visual features, construction commands, and high-safety, high-real-time operation characteristics, making them unusable directly. In other words, existing technology cannot achieve dedicated, purely vision-based intelligent control for tower crane scenarios. Summary of the Invention

[0003] In view of this, the present application provides a tower crane control method and tower crane controller based on a multi-mode control model. The technical solution of the present application integrates the environmental image, operation instructions, environmental status and position status of the tower crane through a multi-mode control model to generate a global feature vector, and implements intelligent control of the tower crane with at least two selectable strategies accordingly, adapting to various working conditions and scenarios, and the pure vision control does not require laser data.

[0004] In a first aspect, embodiments of this application provide a tower crane control method based on a multi-mode control model, comprising: obtaining multi-mode data of the tower crane at the current site, the multi-mode data including environmental state data, position state data, environmental image data, and natural language operation instructions; the tower crane's multi-mode control model extracting feature vectors from each data in the multi-mode data and fusing them into a global feature vector; the multi-mode control model outputting control instructions for controlling the motion of each axis of the tower crane based on the global feature vector; the multi-mode control model including at least two selectable motion generation strategies.

[0005] Based on the above, by fusing the environmental images, operation instructions, environmental status and position status of the tower crane through a multi-mode control model, a global feature vector is generated, and based on this, the tower crane can be intelligently controlled with at least two selectable strategies to adapt to various working conditions and scenarios. Moreover, pure vision control does not require laser data.

[0006] In one possible implementation of the first aspect, the multi-modal control model includes a multi-modal model and a multi-strategy control model. The multi-modal control model extracts feature vectors from each data point in the multi-modal data and fuses them into a global feature vector. Based on the global feature vector, the multi-modal control model outputs control commands for controlling the current motion of each axis of the tower crane. This includes: the multi-modal model normalizing the standard deviation of the environmental state data and the position state data to obtain a state feature vector that fuses the tower crane's environment and position; the multi-modal model extracting feature vectors from the environmental image data and the natural language operation commands and concatenating them into an image-text feature vector that fuses the current environmental image of the tower crane and the tower crane's operation commands; the multi-modal model concatenating the image-text feature vector and the state feature vector to obtain the current global feature vector of the tower crane; and the multi-strategy control model outputting control commands for controlling the current motion of each axis of the tower crane based on the global feature vector.

[0007] As described above, by concatenating the state feature vectors of the environmental state and position state with the image text feature vectors representing the environmental image features and natural language operation instructions, a global feature vector that integrates all features of multi-mode data is generated, which facilitates intelligent control of tower cranes based on multi-mode data.

[0008] In one possible implementation of the first aspect, when the multi-strategy control model is a deterministic regression model, the multi-strategy control model outputs control commands for the motion of each axis of the tower crane based on the global feature vector, including: the multi-strategy control model outputs control feature vectors for the motion of each axis of the tower crane based on the global feature vector, wherein the confidence level of the control feature vectors is higher than a set confidence threshold; and obtains control commands for the motion of each axis of the tower crane based on the control feature vectors. When the multi-strategy control model is a probabilistic generation model, the multi-strategy control model outputs control commands for the motion of each axis of the tower crane based on the global feature vector, including: the multi-strategy control model outputs control feature vectors and their variances for the motion of each axis of the tower crane based on the global feature vector; and when the variance is less than a set variance threshold, obtains control commands for the motion of each axis of the tower crane based on the control feature vectors.

[0009] As described above, the deterministic regression model outputs control feature vectors with confidence levels higher than a set confidence threshold, which are used for precise control scenarios. The probabilistic generative model outputs control feature vectors and their variances for controlling the motion of each axis of the tower crane. Using a generalization scenario, the control feature vector with the smallest variance is selected.

[0010] In one possible implementation of the first aspect, the natural language operation instructions include the current position and final target position of each axis of the tower crane, and control requirements, the control requirements including at least the requirement to avoid obstacles; the method further includes: obtaining the natural language operation instructions from the language of user interaction during each round of hoisting; and automatically generating the natural language operation instructions based on the final target position and current position of each axis of the tower crane during each subsequent control operation of the hoisting round using the multimodal model.

[0011] Based on the final target position and current position of each axis of the tower crane, natural language operation instructions are automatically generated. The natural language operation instructions are aligned to facilitate processing by the multi-mode control model.

[0012] In one possible implementation of the first aspect, it further includes: unifying the data range of each dimension in the environmental state data and the location state data; and / or performing at least one of the following preprocessing on the environmental image data: cropping the environmental image with the crane hook as the center and including a region with a set side length, compressing the environmental image to meet the pixel requirements of the multimodal model, and performing color transformation on the environmental image; and / or performing at least one of the following preprocessing on the natural language job instructions: placeholder standardization, tokenization, and sequence length alignment.

[0013] As described above, by preprocessing the environmental and location state data, the data range of each dimension of the environmental and location state data is unified, preventing large numbers from consuming smaller ones. Preprocessing the environmental images not only highlights the color contrast between the surrounding image of the hook and key obstacles and the environment color, but also adapts to the pixel requirements of the multimodal model, reducing computational load. Preprocessing the natural language operation instructions as described above facilitates standardized natural language processing and the fusion of image feature vectors.

[0014] In one possible implementation of the first aspect, the method further includes: classifying the environmental state data, the position state data, the environmental image data, the natural language operation instruction data, and the control instruction data during the automatic tower crane operation process, wherein the control instruction data is action category data, the environmental state data, the position state data, and the environmental image data are observation category data, and the natural language operation instruction data is language category data; the data in each category is stored based on the control cycle.

[0015] Based on the above, multimodal data is divided into observation category data, language category data, and action category data, which facilitates the storage and management of multimodal data.

[0016] In one possible implementation of the first aspect, when the absolute value of the wind speed in the environmental state data is greater than a set threshold, the multi-strategy control model is a probabilistic generation model; otherwise, it is a deterministic regression model.

[0017] Based on the wind speed data in the environmental condition data, the type of multi-strategy control model is determined so that the multi-strategy control model most adapted to the environment can be used.

[0018] In one possible implementation of the first aspect, it further includes: acquiring the global feature vector of each control cycle during the completed automatic driving process of the tower crane, and the control feature vector obtained after unifying and normalizing the range of the control command data, to form a training sample; wherein the global feature vector of each control cycle is the input, and the control feature vector is the output; keeping the parameters of the multimodal model unchanged, and training the multi-strategy control model with the training sample.

[0019] As shown above, by keeping the parameters of the multimodal model unchanged, the multi-policy control model is trained. The parameters of the multimodal model do not increase and the parameters of the multi-policy control model are small, which facilitates lightweight deployment.

[0020] In one possible implementation of the first aspect, when the multi-strategy control model is a deterministic regression model, the loss function during training is the mean squared error loss function; the process of obtaining the control feature vector includes: unifying the data range and normalizing the standard deviation of the control commands for each axis to obtain the control feature vector; when the multi-strategy control model is a probabilistic generation model, the loss function during training is the Gaussian negative log-likelihood loss function; the process of obtaining the control feature vector includes: unifying the data range and linearly normalizing the control commands for each axis to obtain the control feature vector.

[0021] Based on the above, the loss function during training is determined according to the type of the multi-policy control model, which facilitates the convergence of training for each type of multi-policy control model. The control command normalization method is also determined according to the type of multi-policy control model to match the application scenario of that type of policy control model.

[0022] In one possible implementation of the first aspect, the multimodal model is an LLaVA model pre-trained with matching tower crane scenarios; and / or the deterministic regression model is a multilayer perceptron regression network structure, and the probabilistic generation model is a two-branch MLP network structure.

[0023] As shown above, the LLaVA model is a lightweight multi-mode module. Combined with an even lighter multi-strategy control model, it forms a multi-mode control model that can be deployed locally on the tower crane controller. Furthermore, the deterministic regression model uses a multilayer perceptron regression network structure to achieve accurate predictive control actions; the probabilistic generative model uses a two-branch MLP network structure to achieve probabilistic predictive control actions.

[0024] Secondly, this application provides a tower crane controller, which locally deploys a multi-mode control model for the tower crane, including: a data acquisition module for acquiring multi-mode data of the tower crane at the current site, the multi-mode data including environmental state data, position state data, environmental image data, and natural language operation instructions; a feature extraction module for extracting feature vectors of each data in the multi-mode data by the multi-mode control model and fusing them into a global feature vector; and an intelligent control module for outputting the current control instructions of the tower crane by the multi-mode control model based on the global feature vector, wherein the multi-mode control model includes at least two selectable motion generation strategies.

[0025] Based on the above, by fusing the environmental images, operation instructions, environmental status and position status of the tower crane through a multi-mode control model, a global feature vector is generated, and based on this, the tower crane can be intelligently controlled with at least two selectable strategies to adapt to various working conditions and scenarios. Moreover, pure vision control does not require laser data.

[0026] In one possible implementation of the second aspect, the feature extraction module is specifically used to include: the multimodal model normalizes the standard deviation of the environmental state data and the position state data to obtain a state feature vector that integrates the tower crane environment and position; the multimodal model extracts feature vectors from the environmental image data and the natural language operation instructions respectively, and concatenates them into an image-text feature vector that integrates the current environmental image of the tower crane and the tower crane operation instructions; the multimodal model concatenates the image-text feature vector and the state feature vector to obtain the current global feature vector of the tower crane; the intelligent control module is specifically used to output control instructions for controlling the current motion of each axis of the tower crane based on the global feature vector by the multi-strategy control model.

[0027] As described above, by concatenating the state feature vectors of the environmental state and position state with the image text feature vectors representing the environmental image features and natural language operation instructions, a global feature vector that integrates all features of multi-mode data is generated, which facilitates intelligent control of tower cranes based on multi-mode data.

[0028] In one possible implementation of the second aspect, when the multi-strategy control model is a deterministic regression model, the intelligent control module is specifically used to include: the multi-strategy control model outputs control feature vectors for controlling the motion of each axis of the tower crane based on the global feature vector, wherein the confidence level of the control feature vectors is higher than a set confidence threshold; and obtaining control commands for controlling the motion of each axis of the tower crane based on the control feature vectors. When the multi-strategy control model is a probabilistic generation model, the intelligent control module is specifically used to include: the multi-strategy control model outputs control feature vectors and their variances for controlling the motion of each axis of the tower crane based on the global feature vector; and when the variance is less than a set variance threshold, obtaining control commands for controlling the motion of each axis of the tower crane based on the control feature vectors.

[0029] As described above, the deterministic regression model outputs control feature vectors with confidence levels higher than a set confidence threshold, which are used for precise control scenarios. The probabilistic generative model outputs control feature vectors and their variances for controlling the motion of each axis of the tower crane. Using a generalization scenario, the control feature vector with the smallest variance is selected.

[0030] In one possible implementation of the second aspect, the natural language operation instructions include the current position and final target position of each axis of the tower crane, and control requirements, the control requirements including at least the requirement to avoid obstacles; the method further includes: obtaining the natural language operation instructions from the language of user interaction during each round of hoisting; and automatically generating the natural language operation instructions based on the final target position and current position of each axis of the tower crane during each subsequent control operation of the hoisting round using the multimodal model.

[0031] Based on the final target position and current position of each axis of the tower crane, natural language operation instructions are automatically generated. The natural language operation instructions are aligned to facilitate processing by the multi-mode control model.

[0032] In one possible implementation of the second aspect, the feature extraction module is further specifically used to include: unifying the data range of each dimension in the environmental state data and the location state data; and / or performing at least one of the following preprocessing on the environmental image data: cropping the environmental image with the crane hook as the center and including a region with a set side length, compressing the environmental image to meet the pixel requirements of the multimodal model, and performing color transformation on the environmental image; and / or performing at least one of the following preprocessing on the natural language job instructions: placeholder standardization, tokenization, and sequence length alignment.

[0033] As described above, by preprocessing the environmental and location state data, the data range of each dimension of the environmental and location state data is unified, preventing large numbers from consuming smaller ones. Preprocessing the environmental images not only highlights the color contrast between the surrounding image of the hook and key obstacles and the environment color, but also adapts to the pixel requirements of the multimodal model, reducing computational load. Preprocessing the natural language operation instructions as described above facilitates standardized natural language processing and the fusion of image feature vectors.

[0034] In one possible implementation of the second aspect, it further includes: a data storage module for classifying the environmental state data, the position state data, the environmental image data, the natural language operation instruction data, and the control instruction data during the tower crane automatic driving process, wherein the control instruction data is action category data, the environmental state data, the position state data, and the environmental image data are observation category data, and the natural language operation instruction data is language category data; the data in each category is stored based on the control cycle.

[0035] Based on the above, multimodal data is divided into observation category data, language category data, and action category data, which facilitates the storage and management of multimodal data.

[0036] In one possible implementation of the second aspect, when the absolute value of the wind speed in the environmental state data is greater than a set threshold, the multi-strategy control model is a probabilistic generation model; otherwise, it is a deterministic regression model.

[0037] Based on the wind speed data in the environmental condition data, the type of multi-strategy control model is determined so that the multi-strategy control model most adapted to the environment can be used.

[0038] In one possible implementation of the second aspect, a model training module is further included, used to acquire the global feature vector of each control cycle during the completed automatic driving process of the tower crane, and the control feature vector obtained after unifying and normalizing the range of the control command data, to form training samples; wherein, the global feature vector of each control cycle is the input, and the control feature vector is the output; keeping the parameters of the multimodal model unchanged, the multi-strategy control model is trained with the training samples.

[0039] As shown above, by keeping the parameters of the multimodal model unchanged, the multi-policy control model is trained. The parameters of the multimodal model do not increase and the parameters of the multi-policy control model are small, which facilitates lightweight deployment.

[0040] In one possible implementation of the second aspect, when the multi-strategy control model is a deterministic regression model, the loss function during training is the mean squared error loss function; the process of obtaining the control feature vector includes: unifying the data range and normalizing the standard deviation of the control commands for each axis to obtain the control feature vector; when the multi-strategy control model is a probabilistic generation model, the loss function during training is the Gaussian negative log-likelihood loss function; the process of obtaining the control feature vector includes: unifying the data range and linearly normalizing the control commands for each axis to obtain the control feature vector.

[0041] Based on the above, the loss function during training is determined according to the type of the multi-policy control model, which facilitates the convergence of training for each type of multi-policy control model. The control command normalization method is also determined according to the type of multi-policy control model to match the application scenario of that type of policy control model.

[0042] In one possible implementation of the second aspect, the multimodal model is an LLaVA model pre-trained with matching tower crane scenarios; and / or the deterministic regression model is a multilayer perceptron regression network structure, and the probabilistic generation model is a two-branch MLP network structure.

[0043] As shown above, the LLaVA model is a lightweight multi-mode module. Combined with an even lighter multi-strategy control model, it forms a multi-mode control model that can be deployed locally on the tower crane controller. Furthermore, the deterministic regression model uses a multilayer perceptron regression network structure to achieve accurate predictive control actions; the probabilistic generative model uses a two-branch MLP network structure to achieve probabilistic predictive control actions.

[0044] Thirdly, embodiments of this application provide a computing device, including,

[0045] bus;

[0046] A communication interface, which is connected to the bus;

[0047] At least one processor connected to the bus; and

[0048] At least one memory is connected to the bus and stores program instructions that, when executed by the at least one processor, cause the at least one processor to perform the method described in any embodiment of the first aspect of this application.

[0049] Fourthly, embodiments of this application provide a computer-readable storage medium having program instructions stored thereon, which, when executed by a computer, cause the computer to perform the method described in any embodiment of the first aspect. Attached Figure Description

[0050] Figure 1This is a flowchart illustrating an embodiment of a tower crane control method based on a multi-mode control model according to this application.

[0051] Figure 2 This is a flowchart illustrating a second embodiment of a tower crane control method based on a multi-mode control model according to this application.

[0052] Figure 3 This is a schematic diagram of the organization and hierarchical structure of the dataset in the tower crane controller in Embodiment 2 of the tower crane control method based on a multi-mode control model of this application;

[0053] Figure 4 This is a flowchart illustrating an embodiment of a training method for a multi-mode control model of a tower crane according to this application.

[0054] Figure 5 This is a flowchart illustrating a second embodiment of a training method for a multi-mode control model of a tower crane according to this application.

[0055] Figure 6 This is a structural schematic diagram of an embodiment of a tower crane controller according to this application;

[0056] Figure 7 This is a schematic diagram of the structure of a model training module according to an embodiment of a tower crane controller of this application;

[0057] Figure 8 This is a schematic diagram of the computing device of this application. Detailed Implementation

[0058] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0059] In the following description, the terms “first, second, third, etc.” or module A, module B, module C, etc. are used not only to distinguish similar objects or different embodiments, but also do not represent a specific ordering of objects. It is understood that a specific order or sequence may be interchanged where permitted so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0060] In the following description, the labels of the steps, such as S110, S120, etc., do not necessarily mean that the steps will be executed in this way. The order of the steps can be interchanged or executed simultaneously if permitted.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] This application provides a tower crane control method and controller based on a multi-mode control model. The method includes: obtaining multi-mode data of the tower crane at the current site, the multi-mode data including environmental state data, position state data, environmental image data, and natural language operation instructions; the tower crane's multi-mode control model extracts feature vectors from each data in the multi-mode data and fuses them into a global feature vector; the multi-mode control model outputs control instructions for controlling the motion of each axis of the tower crane based on the global feature vector; the multi-mode control model includes at least two selectable motion generation strategies.

[0063] In this embodiment, a multi-mode control model is used to fuse the environmental image, operation instructions, environmental status and position status of the tower crane to generate a global feature vector. Based on this, the tower crane can be intelligently controlled with at least two selectable strategies to adapt to various working conditions. Moreover, pure vision control does not require laser data.

[0064] The embodiments of this application are described below with reference to the accompanying drawings. First, the terminology of this application will be introduced.

[0065] VLA, short for Vision-Language-Action model, is a type of multimodal AI model for the field of embodied intelligence in robots. Its core capability is to directly output executable robot actions based on visual input and natural language commands, making it one of the core directions of current robot intelligence research. VLA needs to meet two core conditions: ① using vision and language as input; ② building upon a large-scale pre-trained vision / language model to ultimately generate executable robot actions.

[0066] LLaVA (Large Language and Vision Assistant) is an open-source, end-to-end large-scale multimodal model that focuses on general visual and language understanding capabilities. It is one of the most influential open-source models in the field of multimodal AI. LLaVA consists of a visual encoder (CLIP-ViT), a text encoder, and a multimodal projection layer. It uses a lightweight projection module to align visual and linguistic features, accepting both image and text inputs and outputting text results. It supports general multimodal understanding and generation tasks. Its core architecture is very simple, with low training costs, allowing individuals and research teams to develop customized domain-specific models based on it.

[0067] HDF5 (Hierarchical Data Format version 5) is a cross-platform, open binary scientific data storage format and accompanying software library. It is designed for storing and managing large-scale, complex data and is currently one of the mainstream storage formats in the fields of scientific computing, big data, and AI. It adopts a tree-like structure similar to a file system for management. The core objects are divided into "Groups (similar to folders)" and "Datasets (essentially multidimensional arrays)". It can also store metadata through attributes to achieve self-description of data and parse content without additional documentation.

[0068] The dual-branch MLP model is a deep learning architecture based on an improved multilayer perceptron (MLP). The core idea is to design two independent and parallel MLP branches, which extract features from different dimensions and learn heterogeneous patterns respectively. Finally, the results are fused to solve the problem that a single MLP cannot simultaneously take into account multiple feature dependencies. It is currently widely used in tasks such as recommendation systems and time series prediction.

[0069] The following is combined Figure 1 This paper introduces an embodiment of a tower crane control method based on a multi-mode control model.

[0070] Figure 1 The flowchart of a tower crane control method based on a multi-mode control model is shown, including steps S110 to S130.

[0071] S110: Obtain the current environmental status data, location status data, environmental image data, and natural language operation instructions of the tower crane.

[0072] In some embodiments of this example, the environmental state data of the tower crane includes wind speed, radial tilt angle, and tangential tilt angle, so that the control of the tower crane takes into account the effects of wind speed and the tilt of the tower crane; the position state data of the tower crane includes the position coordinates and velocity of the tower crane on each axis, so that the control of the tower crane takes into account the influence of the tower crane's motion parameters on the control of the tower crane, and the axes of the tower crane include the lifting axis, the luffing axis, and the slewing axis; the natural language operation instructions of the tower crane are constructed in a defined language format, including the current position and final target position of each axis of the tower crane, as well as the control requirements for avoiding obstacles. The natural language operation instructions constructed in a standard format are easy for multimodal models to process and are unambiguous.

[0073] In some implementations of this embodiment, the user sends natural language operation instructions in natural language during each round of hoisting; thereafter, during each control of that round of hoisting, natural language operation instructions are automatically generated by the multi-mode control model of the tower crane based on the final target position and current position of each axis of the tower crane. The automatic generation of natural language operation instructions for tower crane control is convenient for multi-mode model processing.

[0074] In some embodiments of this example, during lifting, the natural language operation instructions sent by the user are first translated into a set language by a multi-mode control model. The multi-mode control model automatically generates natural language operation instructions in the set language as well. For example, the set language is Chinese.

[0075] S120: The multi-mode control model of the tower crane extracts the feature vectors of each data in the multi-mode data and merges them into a global feature vector.

[0076] The global feature vector integrates features from the tower crane's environmental status data, location status data, environmental image data, and natural language operation instructions.

[0077] In some implementations of this embodiment, the multi-mode control model includes a multimodal model and a multi-strategy control model. The multimodal model is used to extract global feature vectors from multimodal data, and the multi-strategy control model is used to output control commands for controlling the current motion of each axis of the tower crane. The two models are trained separately and independently, which facilitates control over the number of parameters in the entire multimodal control model, making it a lightweight VLA model that is easy to deploy locally.

[0078] In some implementations of this embodiment, the multimodal model first normalizes the standard deviation of the environmental state data and the location state data to obtain a state feature vector that integrates the tower crane's environment and location, thus eliminating the influence of parameter dimensions in the tower crane's environment and location. Then, the multimodal model utilizes its image and text multimodal processing capabilities to extract feature vectors from the tower crane's environmental image data and natural language operation instructions, respectively, and concatenates them into an image-text feature vector that integrates the current environmental image of the tower crane and the tower crane's operation instructions, making it easier to concatenate with other feature vectors. Finally, the multimodal model concatenates the obtained image-text feature vector and the state feature vector to obtain the current global feature vector of the tower crane, thus integrating features from the tower crane's environmental state data, location state data, environmental image data, and natural language operation instructions.

[0079] In some implementations of this embodiment, the data ranges for each dimension of the environmental state data and location state data are unified to prevent the problem of large numbers consuming small numbers.

[0080] In some embodiments of this example, the environmental image data of the tower crane is preprocessed in at least one of the following ways: cropping the image with the tower crane hook as the center and including a set variable length region to highlight the image around the hook; compressing the environmental image of the tower crane to meet the pixel requirements of the multimodal model to reduce the amount of computation; and performing color transformation on the environmental image of the tower crane to highlight the contrast between the color of key obstacles and the color of the environment.

[0081] In some embodiments of this example, the natural language operation instructions of the tower crane are preprocessed in at least one of the following ways: placeholder standardization, tokenization, and sequence length alignment. This not only enables standardized natural language processing but also facilitates the fusion of image feature vectors.

[0082] S130: The multi-mode control model outputs control commands for the movement of each axis of the tower crane based on the current global feature vector.

[0083] The multimodal control model includes at least two selectable motion generation strategies, namely a deterministic regression model and a probabilistic generation model.

[0084] In some implementations of this embodiment, when the absolute value of wind speed in the environmental state data is greater than a set threshold, the multi-strategy control model is a probabilistic generation model; otherwise, it is a deterministic regression model.

[0085] In some implementations of this embodiment, when the multi-strategy control model is a deterministic regression model, the multi-strategy control model outputs control feature vectors for controlling the motion of each axis of the tower crane based on the current global feature vector. The confidence level of the control feature vectors is higher than a set confidence threshold, which is suitable for precise control scenarios. Control commands for controlling the motion of each axis of the tower crane are obtained based on the control feature vectors, and these control commands achieve precise control.

[0086] In some embodiments of this example, when the multi-strategy control model is a probabilistic generation model, the multi-strategy control model outputs control feature vectors and their variances for controlling the motion of each axis of the tower crane based on the current global feature vectors. When the variance is less than a set variance threshold, control commands for controlling the motion of each axis of the tower crane are obtained based on the control feature vectors. This control strategy is suitable for scenarios where wind speed has a significant impact.

[0087] In some implementations of this embodiment, when the absolute value of wind speed in the environmental state data is greater than a set threshold, the multi-strategy control model is a probabilistic generation model; otherwise, it is a deterministic regression model.

[0088] In some implementations of this example, the multimodal model is an LLaVA model pre-trained to match the tower crane scenario. The model parameters are controllable, making it a lightweight multimodal module. Combined with an even lighter multi-strategy control model, it forms a multimodal control model that can be deployed locally on the tower crane controller. The deterministic regression model uses a multilayer perceptron regression network structure, and the probabilistic generation model uses a two-branch MLP network structure. The multi-strategy control model has even fewer parameters, resulting in a lightweight VLA model for easy local deployment.

[0089] In summary, the first embodiment of a tower crane control method based on a multi-mode control model integrates the tower crane's environmental image, operation instructions, environmental state, and position state through a multi-mode control model to generate a global feature vector. Based on this vector, the tower crane can be intelligently controlled using at least two selectable strategies, adapting to various working conditions. Furthermore, pure vision control does not require laser data.

[0090] The following is combined Figure 2 and Figure 3 This paper introduces a second embodiment of a tower crane control method based on a multi-mode control model.

[0091] Embodiment 2 of a tower crane control method based on a multi-mode control model is a detailed implementation of Embodiment 1 of a tower crane control method based on a multi-mode control model, and has all its advantages.

[0092] In this embodiment, the multi-mode control model of the tower crane includes a multi-modal model and a multi-policy control model, deployed on the tower crane's controller. The multi-modal model is an LLaVA model pre-trained to match the tower crane scenario. The multi-policy control model is the Policy Head of the multi-modal control model. The multi-policy control model can switch between a deterministic regression model (ACT model) and a probabilistic generative model (Diffusion model). The ACT model is a multilayer perceptron regression network structure, and the Diffusion model is a two-branch MLP network structure. The LLaVA model and the Policy Head are trained independently. The LLaVA model is pre-trained first, and its model parameters are fixed before training the Policy Head.

[0093] In this embodiment, environmental state data, position state data, environmental image data, natural language operation instruction data, and control instruction data during the tower crane's automatic driving process are classified. Control instruction data is action category data, environmental state data, position state data, and environmental image data are observation category data, and natural language operation instruction data is language category data. Data in each category is stored based on the control cycle. For example, the tower crane's environmental state data, position state data, environmental image data, natural language operation instructions, and control instructions are stored as HDF5 files for the above classification, and the data is aligned in the control cycle to facilitate the fusion of features from the tower crane's data within the same control cycle.

[0094] Figure 2 The flowchart of a second embodiment of a tower crane control method based on a multi-mode control model is shown, including steps S210 to S240. In this embodiment, each round of tower crane control includes several control cycles.

[0095] S210: Acquire multi-mode data for the current control cycle, including location status, environmental status, environmental image data, and natural language operation instructions.

[0096] Figure 3 This illustrates the organization and hierarchical structure of the dataset in the tower crane controller of this embodiment. It includes:

[0097] 1. Top-level structure of the dataset: The dataset is managed in the form of folders, which contain multiple independent HDF5 files. Each HDF5 file corresponds to a complete round of operation (the hoisting process from the start point to the end point), recording all interaction, observation and control data within that round.

[0098] 2. The internal structure of a single HDF5 file: Each HDF5 file contains 3 top-level groups, each carrying different types of data.

[0099] (1) Observations group (environmental and state observation data). This group is further divided into the state subgroup and the images subgroup.

[0100] The `state` subgroup carries numerical data on the position and environmental conditions of the tower crane, totaling 9 items. The position conditions include: lifting position, luffing position, slewing position, lifting speed, luffing speed, and slewing speed. The environmental conditions include wind speed, radial tilt angle, and tangential tilt angle.

[0101] The images subgroup contains environmental image data and contains only one item, namely the hook image.

[0102] It is important to emphasize that during each round of hoisting, the user sends natural language operation instructions in natural language; in each subsequent control cycle, the natural language operation instructions for that control cycle are automatically generated based on the final target position and current position of each axis of the tower crane through the LLaVA model.

[0103] (2) The `language_raw` group (natural language task instruction data) records the initial natural language task instruction for this round of operations. The format is descriptive text containing the current status and objective, such as: "The hook is located at [lifting position, luffing position, slewing position], please move the hook to the designated position [end point], and be careful to avoid obstacles." This instruction provides a high-level task objective for the entire round of operations and corresponds one-to-one with the observation and action data of subsequent time steps.

[0104] (3) Actions group (control instruction data) records the control instructions for each time step in this round of operation, including gear information in three dimensions: lifting gear, luffing gear, and slewing gear.

[0105] 3. In terms of data organization logic, actions, observations (state + images) and language_raw are strictly aligned in the control cycle, and each control cycle step shares the same index.

[0106] S220: Preprocess the current multimodal data using the LLaVA model.

[0107] Among them, the preprocessing of the current control cycle's position and environment status is as follows: the data range is unified to prevent the problem of large numbers consuming small numbers.

[0108] The preprocessing of natural language operation instructions for the current control cycle involves translating each instruction into a specified language, tokenizing it, and ensuring a uniform sequence length. Tower crane operation instructions are standardized lifting task descriptions, requiring the elimination of textual ambiguity to ensure the model accurately resolves 3D positioning and obstacle avoidance requirements. Natural language operation instructions are standardized into a fixed paradigm that includes initial 3D position, target 3D position, and obstacle avoidance constraints, strengthening the semantic expression of tower crane-specific tasks.

[0109] In this context, the environmental image of the tower crane has the hook as the core effective area. However, the hardware resources at the edge are limited, and high-resolution images will significantly increase inference latency. Therefore, the environmental image of the current control cycle is preprocessed to meet the requirements of the LLaVA model. This includes performing routine operations such as JPEG decoding, color space conversion, pixel normalization, and tensor format conversion. In addition, the image is uniformly scaled to a resolution of 640×480 to reduce the data dimension and reduce the computational overhead at the edge while fully preserving the hook features.

[0110] S230: Using the LLaVA model, feature vectors of each data point in the preprocessed multi-modal data are extracted and fused into a global feature vector.

[0111] The tower crane position status and environmental status include multi-dimensional heterogeneous physical quantities with large differences in dimensions. The preprocessed tower crane position status and environmental status are normalized based on standard deviation using the LLaVA model to obtain the state feature vector. This includes: calculating the mean and standard deviation of each dimension of the state, performing Z-score standardization on each dimension of the state, and pruning the lower limit of the standard deviation to avoid division by zero anomalies.

[0112] Among them, the pre-trained LLaVA model serves as the backbone for feature extraction. It sequentially performs image feature extraction and projection, text feature extraction, and image-text feature fusion (inserting the projected image features into the special image token positions of the text features) on the pre-processed environmental image and language instructions to obtain the image-text feature vector of the current control cycle.

[0113] The LLaVA model concatenates the image-text feature vector and state vector of the current control cycle to obtain the global feature vector of the current control cycle. This process includes: linearly projecting the state feature vector of the tower crane's current control cycle onto the same feature space, and then concatenating it into the image-text feature vector sequence of the current control cycle. The LLaVA model extracts features from the sequence... <cls>The corresponding features are labeled as global feature vectors. <cls>The model is labeled as a multimodal model to represent the global feature vector, which comprehensively reflects visual perception, language commands, and tower crane mechanical status information.

[0114] S240: Based on the current global feature vector, the multi-strategy model outputs control commands for the movement of each axis of the tower crane.

[0115] Specifically, the global feature vector of the current control cycle is input into the Policy Head. The policy corresponding to the Policy Head is determined based on the wind speed. When the wind speed is less than a set threshold, the control command for the current control cycle is obtained through the ACT model; when the wind speed is greater than or equal to the set threshold, the control command for the current control cycle is obtained through the Diffusion model.

[0116] The control commands include the control positions of each axis of the tower crane, and each control position is subject to mechanical constraints.

[0117] In this embodiment, the control instructions for each control cycle are stored according to the classification method described in step S210, and are saved in the corresponding HDF5 file, and are time-aligned with other multi-mode data of the same cycle in terms of control cycle.

[0118] The following is combined Figure 4 This paper introduces an embodiment of a training method for a multi-mode control model of a tower crane.

[0119] This embodiment is used to train the multi-strategy control model in Embodiment 1 of a tower crane control method based on a multi-mode control model.

[0120] Figure 4 The flowchart of a training method for a tower crane multi-mode control model is shown in Embodiment 1, including steps S310 to S320.

[0121] S310: Obtain the global feature vector of each control cycle during the completed hoisting process of the tower crane, and the control feature vector obtained by normalizing the control commands, to form training samples.

[0122] Among them, the global feature vector of each control cycle in the completed hoisting process is the sample input variable, and the control feature vector is the output variable.

[0123] The method for obtaining the global feature vector for each control cycle during the hoisting process is described in Embodiment 1 of a tower crane control method based on a multi-mode control model.

[0124] In some embodiments of this example, the control commands for each control cycle during the completed hoisting process are unified and normalized to obtain the control feature vector for that control cycle. When the multi-strategy control model of the tower crane is a deterministic regression model, the normalization of the control feature vector of the tower crane is based on standard deviation, i.e., Z-score normalization. Standard deviation normalization avoids the influence of outlier values, resulting in higher model computational stability and improving model prediction accuracy and performance stability. When the multi-strategy control model of the tower crane is a probabilistic generation model, the normalization of the control feature vector of the tower crane is linear normalization, i.e., Min-Max normalization to the [-1,1] interval. Linear normalization preserves the feature distribution of the most realistic scenario, which is convenient for use in probabilistic scenarios.

[0125] S320: Keep the parameters of the multimodal model constant, and train each model in the multi-policy control model with training samples.

[0126] In this approach, the parameters of the multimodal model are kept constant during training, and joint gradient descent is not calculated. This not only reduces the computational cost during training but also does not increase the number of parameters in the multimodal model, and keeps the multi-policy control model with a low number of parameters.

[0127] In some implementations of this embodiment, when the multi-strategy control model is a deterministic regression model, the loss function during training is the mean squared error loss function; when the multi-strategy control model is a probabilistic generation model, the loss function during training is the Gaussian negative log-likelihood loss function.

[0128] In some embodiments of this example, the multi-mode control model of the tower crane includes a multi-modal model and a multi-policy control model, which are deployed on the tower crane's controller. The multi-modal model is an LLaVA model pre-trained to match the tower crane scenario; the multi-policy control model is the Policy Head of the multi-mode control model; the LLaVA model and the Policy Head are trained independently, with the LLaVA model pre-trained first, its model parameters fixed, and then the Policy Head trained.

[0129] In some embodiments, the global feature vector and control feature vector of each control cycle in the completed hoisting process are fused together to obtain the sample vector at that time. The sample vector set is divided into a training sample set and a test sample set. The training sample set is used to train the multi-strategy control model and the test sample set is used to test the trained model.

[0130] In summary, the first embodiment of a training method for a tower crane multi-mode control model keeps the multi-mode model unchanged during training, does not increase the number of parameters of the multi-mode model, and reduces the number of parameters of the multi-strategy control model, so that the multi-mode control model composed of the multi-mode model and the multi-strategy control model can be deployed on the edge controller of the tower crane.

[0131] The following is combined Figure 5 This paper presents a second embodiment of a training method for a multi-mode control model of a tower crane.

[0132] Example 2 of a training method for a tower crane multi-mode control model is used to train the multi-strategy control model in Example 2 of a tower crane control method based on a multi-mode control model, and has all its advantages.

[0133] In this embodiment, the multi-mode control model of the tower crane includes a multi-modal model and a multi-policy control model, deployed on the tower crane's controller. The multi-mode control model is an LLaVA model pre-trained to match the tower crane scenario. The multi-policy control model is the Policy Head of the multi-mode control model, which switches between a deterministic regression model (ACT model) and a probabilistic generation model (Diffusion model). The ACT model is a multilayer perceptron regression network structure, and the Diffusion model is a two-branch MLP network structure. The LLaVA model and the Policy Head are trained independently. The LLaVA model is pre-trained first, and its model parameters are fixed before training the Policy Head.

[0134] In this embodiment, the environmental status data, position status data, environmental image data, natural language operation instructions, and control instructions of the tower crane during the lifting process are stored using HDF5 files. For the storage structure of the above data in the HDF5 file, please refer to Embodiment Two of a Tower Crane Control Method Based on a Multi-Mode Control Model.

[0135] Figure 5 The flowchart of a training method for a multi-strategy control model of an intelligent tower crane is shown in Embodiment 2, including steps S410 to S440.

[0136] S410: Obtain the environmental status, position status, environmental images, natural language operation instructions, and control instructions for each control cycle during the completed hoisting process.

[0137] For the description of environmental status, location status, environmental image, and control commands, please refer to Embodiment 2 of a tower crane control method based on a multi-mode control model.

[0138] S420: Obtain the global feature vector and control feature vector for each control cycle during the completed hoisting process.

[0139] In this process, a global feature vector for each control cycle is generated based on the environmental state, position state, and environmental image. For detailed methods, please refer to Steps S210, S220, and S230 of Embodiment 2 of a tower crane control method based on a multi-mode control model.

[0140] The control feature vector for each control cycle is obtained by unifying and normalizing the range of the control commands for that cycle. Specifically, when the multi-strategy control model for the tower crane is a deterministic regression model, the normalization of the control feature vector is based on standard deviation. Standard deviation normalization avoids the influence of outliers, resulting in higher model computational stability and improving both prediction accuracy and performance stability. When the multi-strategy control model for the tower crane is a probabilistic generation model, the normalization of the control feature vector is linear normalization. Linear normalization preserves the most realistic feature distribution of the scenario, facilitating its use in probabilistic scenarios.

[0141] S430: Use the global feature vector and control feature vector of each control cycle during the completed lifting process of the tower crane as samples, and classify the training samples based on training and testing.

[0142] In this system, the global feature vector for each control cycle is the input variable, and the control feature vector for each control cycle is the output sample. The training samples are divided into two subsets: 80% and 20%. The deterministic regression model and the probabilistic generative model are trained using the 80% subset, respectively, and the trained models are tested using the two 20% subsets.

[0143] S440: Keep the LLaVA model unchanged and train a multi-policy control model using training samples.

[0144] The training process freezes the LLaVA backbone parameters throughout, and only updates the parameters of the lightweight multi-policy model (PolicyHead parameters) to reduce computing power consumption and adapt to the deployment conditions of industrial-grade tower cranes on edge controllers.

[0145] The training process employs the Adam optimizer with targeted learning rate and weight decay coefficient configurations, and selects a loss function that matches the tower crane control target based on the strategy type: the ACT model uses mean squared error loss to enhance the accurate regression of the three-dimensional control gears of amplitude change, slewing, and lifting; the Diffusion model uses Gaussian negative log-likelihood loss to optimize the reliability of motion distribution under uncertain conditions.

[0146] The training employs a fixed batch size and learning rate decay strategy for iteration, guided by verification of loss convergence to prevent overfitting and preserve the optimal strategy head weights. The training terminates when a preset number of rounds are reached or when continuous convergence of the loss is verified. The final output is a lightweight model that meets the requirements for real-time tower crane control (5Hz-10Hz) and can be directly deployed on edge controllers, ensuring the safety and stable control of heavy construction machinery operations.

[0147] This step includes the following process:

[0148] 1. Configure training parameters.

[0149] The Adam optimizer was used with a learning rate of 1e-4 and a weight decay of 1e-5. The corresponding loss function was selected based on the PolicyHead type: the ACT strategy used mean squared error loss (MSE Loss) to measure the numerical difference between predicted and actual actions, while the Diffusion strategy used Gaussian negative log-likelihood loss to measure the degree of matching between the predicted and actual action distributions. The number of training epochs (num_epochs) was set to 50k, the batch size (batch_size) was set to 32, and the learning rate decay strategy used was StepLR with a step size of 10 and a decay coefficient of 0.9 to avoid overfitting in the later stages of training.

[0150] 2. Training cycle (executed in rounds).

[0151] The training phase includes:

[0152] Samples are read from the training set in batches, with each batch containing 32 samples, and then transferred to the training device (GPU / CPU).

[0153] Input the global feature vector of the multimodal model into the Policy Head and output the action prediction result (ACT policy outputs deterministic action vector, Diffusion policy outputs mean + variance).

[0154] The prediction results of the Policy Head are compared with the corresponding control feature vectors in the training set samples, and the loss value (MSE Loss or Gaussian negative log-likelihood loss) is calculated.

[0155] Backpropagation is performed on the loss value, updating only the parameters of the Policy Head while keeping the LLaVA model parameters unchanged; then, the gradients are cleared, and the next batch of training begins.

[0156] The verification phase (eval mode) includes:

[0157] Samples are read from the validation set in batches and transferred to the training device;

[0158] Using the same method as in the training phase, multimodal fusion features are extracted and input into the Policy Head to obtain the prediction results; gradients are not calculated.

[0159] Calculate the loss value between the prediction result and the action_norm of the validation set samples to obtain the average loss value of the validation set;

[0160] The average loss value of the validation set is compared with the current best loss value. If the current validation loss value is smaller, the parameters (weight file) of the current Policy Head are saved, the best loss value is updated, and the saved model is ensured to have the best generalization performance.

[0161] When the training rounds reach a preset 50k rounds, or the validation set loss value no longer decreases (converges) for 10 consecutive rounds, training stops and the trained Policy Head is output.

[0162] 3. Output Results

[0163] The trained multi-policy model (Policy Head) includes a model weight file, which is divided into two types:

[0164] (1) ACT policy: The trained ACT type policy head can receive multimodal fusion features and output control feature vectors of deterministic actions, and the prediction accuracy meets the robot control requirements;

[0165] (2) Diffusion strategy: The trained Diffusion type policy head can receive multimodal fusion features and output the distribution parameters (mean + variance) of the control feature vector. The generalization of the generated features meets the robot control requirements.

[0166] The following is combined Figure 6 An embodiment of a tower crane controller is introduced.

[0167] An embodiment of a tower crane controller implements the method described in Embodiment 1 of a tower crane control method based on a multi-mode control model, which has all its advantages.

[0168] Figure 6 The structure of a control device embodiment for an intelligent tower crane is shown, including: a data acquisition module 610, a feature extraction module 620, and an intelligent control module 630.

[0169] The data acquisition module 610 is used to acquire the current environmental status data, position status data, environmental image data, and natural language operation instructions of the tower crane. For its working principle and advantages, please refer to step S110 of Embodiment 1 of a tower crane control method based on a multi-mode control model.

[0170] The feature extraction module 620 is used to extract the feature vectors of each data point in the multi-mode control model of the tower crane and fuse them into a global feature vector. For its working principle and advantages, please refer to step S120 of Embodiment 1 of a tower crane control method based on a multi-mode control model.

[0171] The intelligent control module 630 is used by the multi-mode control model to output control commands for the movement of each axis of the tower crane based on the current global feature vector. For its working principle and advantages, please refer to step S130 of Embodiment 1 of a tower crane control method based on a multi-mode control model.

[0172] The following is combined Figure 7 This paper introduces a model training module for an embodiment of a tower crane controller.

[0173] The model training module of one embodiment of a tower crane controller executes the method described in Embodiment 1 of a training method for a multi-strategy control model of an intelligent tower crane, which has all its advantages. The model training module described in this embodiment can be located in the controller of one embodiment of a tower crane controller, or in the host computer of the controller of one embodiment of a tower crane controller.

[0174] Figure 7 The structure of a model training module of an embodiment of a tower crane controller is shown, including: a sample acquisition module 710 and a model training module 720.

[0175] The sample acquisition module 710 is used to acquire the global feature vector of each control cycle during the completed hoisting process of the tower crane, and the control feature vector obtained by normalizing the control commands, to form training samples. For its working principle and advantages, please refer to step S310 of Example 1 of a training method for a multi-strategy control model of an intelligent tower crane.

[0176] The model training module 720 is used to keep the parameters of the multimodal model constant and train each model in the multi-strategy control model with training samples. For its working principle and advantages, please refer to step S320 of Example 1 of a training method for a multi-strategy control model of an intelligent tower crane.

[0177] This application also provides a computing device, which will be described below in conjunction with... Figure 8 Detailed introduction.

[0178] The computing device 800 includes a processor 810, a memory 820, a communication interface 830, and a bus 840.

[0179] It should be understood that the communication interface 830 in the computing device 800 shown in the figure can be used to communicate with other devices.

[0180] The processor 810 can be connected to the memory 820. The memory 820 can be used to store the program code and data. Therefore, the memory 820 can be a storage unit inside the processor 810, an external storage unit independent of the processor 810, or a component that includes both the storage unit inside the processor 810 and the external storage unit independent of the processor 810.

[0181] Optionally, the computing device 800 may also include a bus 840. The memory 820 and communication interface 830 can be connected to the processor 810 via the bus 840. The bus 840 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 840 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one line is used in this figure, but this does not mean that there is only one bus or one type of bus.

[0182] It should be understood that in the embodiments of this application, the processor 810 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. Alternatively, the processor 810 may employ one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0183] The memory 820 may include read-only memory and random access memory, and provides instructions and data to the processor 810. A portion of the processor 810 may also include non-volatile random access memory. For example, the processor 810 may also store device type information.

[0184] When the computing device 800 is running, the processor 810 executes computer execution instructions stored in the memory 820 to perform the operation steps of each method embodiment.

[0185] It should be understood that the computing device 800 according to the embodiments of this application can correspond to the corresponding subject in executing the methods according to the various embodiments of this application, and the above and other operations and / or functions of each module in the computing device 800 are respectively for implementing the corresponding processes of the methods of this embodiment. For the sake of brevity, they will not be described in detail here.

[0186] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0187] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0191] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to perform the operation steps of the various method embodiments.

[0193] The computer storage medium in this application embodiment can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0194] Computer-readable signal media may include data signals transmitted in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which can send, transmit, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0195] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0196] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0197] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, all of which fall within the scope of protection of this application.< / cls> < / cls>

Claims

1. A tower crane control method based on a multi-mode control model, characterized in that, include: Obtain multi-mode data of the current site of the tower crane, including environmental status data, location status data, environmental image data, and natural language operation instructions; The multi-mode control model of the tower crane extracts the feature vectors of each data in the multi-mode data and merges them into a global feature vector; The multi-mode control model outputs control commands for the motion of each axis of the tower crane based on the global feature vector; the multi-mode control model includes at least two selectable motion generation strategies.

2. The method according to claim 1, characterized in that, The multi-mode control model includes a multi-modal model and a multi-strategy control model. The multi-mode control model extracts feature vectors from each data point in the multi-mode data and merges them into a global feature vector. Based on the global feature vector, the multi-mode control model outputs control commands for the current motion of each axis of the tower crane, including: The multimodal model normalizes the standard deviation of the environmental state data and the location state data to obtain a state feature vector that integrates the tower crane's environment and location. The multimodal model extracts feature vectors from the environmental image data and the natural language operation instructions, and concatenates them into an image-text feature vector that integrates the current environmental image of the tower crane and the tower crane operation instructions; The multimodal model concatenates the image text feature vector and the state feature vector to obtain the current global feature vector of the tower crane; The multi-strategy control model outputs control commands for the current motion of each axis of the tower crane based on the global feature vector.

3. The method according to claim 2, characterized in that, When the multi-strategy control model is a deterministic regression model, the multi-strategy control model outputs control commands for the motion of each axis of the tower crane based on the global feature vector, including: The multi-strategy control model outputs control feature vectors for the motion of each axis of the tower crane based on the global feature vector, wherein the confidence level of the control feature vectors is higher than a set confidence threshold; control commands for the motion of each axis of the tower crane are obtained based on the control feature vectors; when the multi-strategy control model is a probabilistic generation model, the multi-strategy control model outputs control commands for the motion of each axis of the tower crane based on the global feature vector, including: The multi-strategy control model outputs control feature vectors and their variances for controlling the motion of each axis of the tower crane based on the global feature vectors; when the variance is less than a set variance threshold, control commands for controlling the motion of each axis of the tower crane are obtained based on the control feature vectors.

4. The method according to claim 1, characterized in that, The natural language operation instructions include the current position and final target position of each axis of the tower crane, as well as control requirements, the control requirements including at least the requirement to avoid obstacles; The method further includes obtaining the natural language operation instructions from the language of the user interaction during each round of hoisting; During each subsequent control operation following the initial lifting operation, the natural language operation instructions are automatically generated using the multimodal model based on the final target position and current position of each axis of the tower crane.

5. The method according to claim 1, characterized in that, Also includes: The data ranges for each dimension in the environmental state data and the location state data are unified; and / or The environmental image data undergoes at least one of the following preprocessing steps: cropping the environmental image with the crane hook as the center and including a region of a set side length; compressing the environmental image to meet the pixel requirements of the multimodal model; and performing color transformation on the environmental image; and / or The natural language job instructions shall be preprocessed by at least one of the following: placeholder normalization, tokenization, and sequence length alignment.

6. The method according to claim 1, characterized in that, Also includes: The environmental state data, position state data, environmental image data, natural language operation instruction data, and control instruction data during the automatic tower crane operation process are classified, wherein the control instruction data is action category data, the environmental state data, position state data, and environmental image data are observation category data, and the natural language operation instruction data is language category data; Data in each category is stored based on a control cycle.

7. The method according to claim 3, characterized in that, When the absolute value of wind speed in the environmental state data is greater than a set threshold, the multi-strategy control model is a probabilistic generation model; otherwise, it is a deterministic regression model.

8. The method according to claim 3, characterized in that, Also includes: The global feature vector of each control cycle during the completed automatic driving process of the tower crane and the control feature vector obtained after unifying and normalizing the range of the control command data are used to form training samples; wherein, the global feature vector of each control cycle is the input and the control feature vector is the output. Keeping the parameters of the multimodal model unchanged, the multi-policy control model is trained using the training samples.

9. The method according to claim 8, characterized in that, When the multi-strategy control model is a deterministic regression model, the loss function during training is the mean squared error loss function; obtaining the control feature vector includes: The control commands for each axis are standardized in terms of data range and standard deviation to obtain the control feature vector. When the multi-strategy control model is a probabilistic generation model, the loss function during training is the Gaussian negative log-likelihood loss function; the control feature vector is obtained by: The control commands for each axis are unified in data range and linearly normalized to obtain the control feature vector.

10. The method according to claim 3, characterized in that, The multimodal model is an LLaVA model pre-trained to match tower crane scenarios; and / or The deterministic regression model is a multilayer perceptron regression network structure, and the probabilistic generation model is a two-branch MLP network structure.

11. A tower crane controller, characterized in that, A multi-mode control model for locally deployed tower cranes, including: The data acquisition module is used to obtain multi-mode data of the current site of the tower crane. The multi-mode data includes environmental status data, location status data, environmental image data, and natural language operation instructions. The feature extraction module is used to extract the feature vectors of each data in the multi-mode data from the multi-mode control model and fuse them into a global feature vector. The intelligent control module is used to output the current control command of the tower crane based on the global feature vector by the multi-mode control model. The multi-mode control model includes at least two selectable motion generation strategies.

12. A computing device, characterized in that, include, bus; A communication interface, which is connected to the bus; At least one processor is connected to the bus; as well as At least one memory connected to the bus and storing program instructions that, when executed by the at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, It stores program instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 10.

Images