Unmanned tower crane intelligent voice driving system based on large language model

Abstract

The invention discloses an unmanned tower crane intelligent voice driving system based on a large language model, which is characterized in that a single voice instruction is received through a control terminal, a structured instruction containing lifting and target positions is generated through the large language model, and the real-time position of a lifting hook and the environment information of obstacle distribution are combined; and a path planning module integrating an improved A * algorithm and an RRT algorithm is adopted to generate a motion path, dual detection and local path re-planning are performed on position deviation and newly added obstacles based on a dynamic window algorithm, and loading and unloading confirmation is sent after the lifting hook autonomously moves to a target position. Full-process control is completed through single voice input, repeated instruction interaction is not needed, operation efficiency is effectively improved, safety is enhanced through real-time path correction and dynamic obstacle avoidance, manual intervention is reduced, and unmanned intelligent transportation of the tower crane is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent construction, and particularly relates to an unmanned tower crane intelligent voice driving system based on a large language model. Background Art

[0002] In the current application of the prior art, there are disadvantages such as low transportation efficiency, long hook positioning time, and high labor cost. The main reasons are that traditional tower cranes are manually operated, making it difficult to position the hook. The positioning of the tower crane also requires a ground commander to give commands, which is time-consuming and labor-intensive. When using a voice control system to control the tower crane, there is one worker in the tower crane cab and one in the ground command post to give commands, and there is a fixed language instruction library. Recognition can only be successful when the language matches the instruction library, greatly reducing convenience. Existing tower crane intelligent voice control systems also have a language instruction library, and continuous voice commands need to be sent to control the operation of the tower crane. It is difficult to complete a single hoisting in one voice, and it is difficult to judge the height and angle of the hook in actual operation, requiring a high level of operator skill. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to propose an unmanned tower crane intelligent voice driving system based on a large language model to solve the problems of insufficient efficiency and high labor cost in the prior art.

[0004] To achieve the above technical objectives, the technical solution adopted in this application is as follows:

[0005] An unmanned tower crane intelligent voice driving system based on a large language model, including a tower crane body, a hook assembly, and a control terminal. The system further includes:

[0006] Receiving a voice control command through the control terminal and inputting it into a large language model processing module to generate a structured control command, where the structured control command includes the hoisting position information and target position information of the hook assembly;

[0007] Obtaining environmental information, where the environmental information includes the first position information of the hook assembly and the first obstacle distribution information;

[0008] Inputting the structured control command and the environmental information into a path planning module to obtain a hook movement path and control the movement of the hook assembly. The path planning module is configured to be constructed based on an improved A* algorithm and an RRT algorithm, and the hook movement path includes the step position information of the hook assembly;

[0009] Real-time obtaining the second position information of the hook assembly and comparing the first position deviation value between the second position information and the step position information;

[0010] Judge whether the first position deviation value is within the range of the first preset deviation threshold. If not, perform local path replanning on the hook assembly through the dynamic window algorithm to obtain the first local planning path, and update the first local planning path to the hook movement path;

[0011] And, obtain the second obstacle distribution information within the preset detection area in real time, and match the first obstacle distribution information with the second obstacle distribution information;

[0012] Judge whether the second obstacle distribution information coincides with the first obstacle distribution information. If not, perform local path replanning on the hook assembly through the dynamic window algorithm to obtain the second local planning path, and update the second local planning path to the hook movement path;

[0013] After the hook assembly reaches the target position information, send a loading and unloading confirmation instruction through the control terminal.

[0014] In some embodiments, the control terminal receives a voice control instruction and inputs it into the large language model processing module to generate a structured control instruction, including:

[0015] Perform endpoint detection and noise suppression on the voice control instruction to obtain a denoised audio frame sequence;

[0016] Extract Mel spectrum features from the denoised audio frame sequence through an acoustic encoder to obtain an acoustic feature vector;

[0017] Input the acoustic feature vector into the trained large language model to generate output information with preset semantic slots, and the preset semantic slots include a material type field, a lifting area field, and a target floor field;

[0018] Convert the output information into three-dimensional space coordinates according to the preset coordinate mapping rule to generate a lifting position information and a target position information;

[0019] Generate a structured control instruction including a safety confirmation identifier according to the lifting position information and the target position information.

[0020] In some embodiments, the path planning module is configured to be constructed based on the improved A* algorithm and the RRT algorithm, including:

[0021] Initialize the search tree and convert the kinematic constraint information of the hook assembly into a state space sampling rule;

[0022] Introduce a dynamic weight factor into the heuristic function of the improved A* algorithm, and the dynamic weight factor is configured to be adjusted according to the wind speed information;

[0023] Execute the improved A* algorithm according to the state space sampling rule and the heuristic function to obtain an initial movement path;

[0024] The initial motion path is subjected to spline smoothing by the BRT algorithm to obtain a sequence of stepping position information, which is the motion path of the lifting hook.

[0025] In some embodiments, the dynamic window algorithm performs local path replanning on the lifting hook assembly, and the obtained first local planning path includes:

[0026] A first dynamic window velocity space is established. The first dynamic window velocity space includes first local velocity information corresponding to multiple positions within a unit time. Each first local velocity information is represented by formula (1), and formula (1) is as follows:

[0027] V di ={(v, w)|v ∈ [v min , v max , w ∈ [w min , w max};

[0028] In formula (1), v is the first linear velocity of the sling in the lifting hook assembly, w is the first rotational angular velocity of the lifting hook assembly, V di is the first local velocity information of the lifting hook assembly at the i-th unit time, i is the count of unit time within the first dynamic window velocity space, T is the total time used by the first dynamic window velocity space, v max is the first maximum planned velocity of the first linear velocity of the sling in the lifting hook assembly, v min is the first minimum planned velocity of the first linear velocity of the sling in the lifting hook assembly, w max is the first maximum planned velocity of the first rotational angular velocity of the lifting hook assembly, w min is the first minimum planned velocity of the first rotational angular velocity of the lifting hook assembly;

[0029] Calculate the first evaluation function of each first local velocity information, which is represented by formula (2), and formula (2) is as follows:

[0030] G(V di ) = ξ1·align(v, w) + ξ2·dist(v, w) + ξ3·vel(v);

[0031] In formula (2), G(V di ) is the first evaluation value of the first evaluation function, ξ1 is the first evaluation weight, ξ2 is the second evaluation weight, ξ3 is the third evaluation weight, is the direction alignment term, is the minimum obstacle avoidance distance term, is the velocity efficiency term, and the weight coefficients satisfy ξ1 + ξ2 + ξ3 = 1. The default configuration is ξ1 = 0.4, ξ2 = 0.4, ξ3 = 0.2;

[0032] Select the first local velocity information with the highest first evaluation value in each unit time and generate a first local planned path, which is represented by formula (3). Formula (3) is as follows:

[0033]

[0034] In formula (3), P local is the first local planned path, p t is the first local path segment, and p0 is the path within the range of the first local planned path in the current hook movement path, denoted as the first initial hook path. is the unit vector of the current movement direction, R is the rotation matrix, and Δr is the hook swing compensation amount.

[0035] In some embodiments, the second obstacle distribution information within a preset detection area is obtained in real time, and matching the first obstacle distribution information with the second obstacle distribution information includes:

[0036] Extract the first obstacle spatio-temporal features from the first obstacle distribution information;

[0037] Construct the second obstacle distribution information through a UWB locator and a vision device, and extract the second obstacle spatio-temporal features;

[0038] Perform the Hungarian algorithm matching on the first obstacle features and the second obstacle features at adjacent timestamps to establish an obstacle trajectory association network;

[0039] When there are unmatched second obstacle features, trigger the second obstacle distribution information update flag.

[0040] In some embodiments, the dynamic window algorithm performs local path replanning on the hook assembly, and obtaining the second local planned path includes:

[0041] Establish a second dynamic window velocity space. The second dynamic window velocity space includes second local velocity information corresponding to multiple positions within a unit time. Each second local velocity information is represented by formula (4). Formula (4) is as follows:

[0042] V′ di′ ={(v′, w′)|v′ ∈ [v′ min , v′ max , w′ ∈ [w′ min , w′ max};

[0043] In formula (4), v′ is the second linear velocity of the sling in the hook assembly, w′ is the second rotational angular velocity of the hook assembly, and V′ di′is the second local velocity information of the hook assembly at the i'-th unit time, where i' is the count of unit time in the second dynamic window velocity space, T' is the total time of the second dynamic window velocity space, and v' max is the second maximum planned velocity of the second linear velocity of the sling in the hook assembly, v' min is the second minimum planned velocity of the second linear velocity of the sling in the hook assembly, w' max is the second maximum planned velocity of the second rotational angular velocity of the hook assembly, w' mib is the second minimum planned velocity of the second rotational angular velocity of the hook assembly;

[0044] Calculate the second evaluation function for each second local velocity information, which is represented by formula (5), and formula (5) is as follows:

[0045] G(V′ di′ ) = ξ4·align(v′, w′) + ξ5·dist(v′, w′) + ξ6·vel(v′) + θ;

[0046] In formula (5), G(V′ di′ ) is the second evaluation value of the second evaluation function, ξ4 is the fourth evaluation weight, ξ5 is the fifth evaluation weight, ξ6 is the sixth evaluation weight, K is the emergency avoidance weight, is the direction alignment term, is the minimum obstacle avoidance distance term, is the speed efficiency term, and the weight coefficients satisfy ξ4 + ξ5 + ξ6 = 1. By default, ξ4 = 0.4, ξ5 = 0.4, ξ6 = 0.2, and θ is the second obstacle feature that is not matched;

[0047] Select the second local velocity information with the highest second evaluation value in each unit time and generate the second local planning path, which is represented by formula (6), and formula (6) is as follows:

[0048]

[0049] In formula (6), P′ local is the second local planning path, p′ t is the second local path segment, p′0 is the path in the range of the second local planning path in the current hook movement path, denoted as the second initial hook path, is the unit vector of the current movement direction, R is the rotation matrix, and Δr′ is the hook swing compensation amount.

[0050] In some embodiments, the environmental information is configured to be obtained through the following steps:

[0051] Generate depth point clouds through the binocular camera on the hook assembly and perform ICP registration with the lidar data;

[0052] Fuse the UWB positioning data and encoder readings to construct the hook pose state equation, which is represented by formula (7). Formula (7) is as follows:

[0053]

[0054] In formula (7), is the optimal estimated value of the hook state at time e, F e is the state transition matrix, x e-1 is the hook state vector at time e-1, B e is the control input matrix, u e is the motor control input vector, E e is the Kalman gain matrix, z e is the sensor observation vector, H e is the observation matrix;

[0055] Solve the hook pose state equation to obtain the environmental information.

[0056] In some embodiments, after the hook assembly reaches the target position information, the loading and unloading confirmation instruction sent by the control terminal includes:

[0057] Verify whether the load change rate of the hook assembly is less than the preset loading and unloading threshold range through a pressure sensor;

[0058] And, detect the contact state of the material with the target area corresponding to the target position information through a vision device, and the contact state is configured as the proportion of the contact area;

[0059] When both the load change rate is less than the preset loading and unloading threshold range and the proportion of the contact area is greater than the preset contact threshold range are satisfied, trigger the sending of the loading and unloading confirmation instruction by the control terminal.

[0060] In some embodiments, the first preset deviation threshold is configured to be dynamically adjusted:

[0061] When the hook load weight is 80% of the preset load weight, reduce the upper limit value of the first preset deviation threshold;

[0062] When the real-time wind speed of the hook assembly is 70% of the preset safe wind speed, increase the lower limit value of the first preset deviation threshold.

[0063] Adopting the above technical solutions, compared with the prior art, the beneficial effects of the present invention are:

[0064] The above technical solution enables the control terminal to receive a single voice command, generate a structured command containing the lifting and target positions through a large language model, combine the real-time position of the hook and the environmental information of the obstacle distribution, use a path planning module that fuses the improved A* algorithm and the RRT algorithm to generate a motion path, and perform dual detection and local path replanning on the position deviation and newly added obstacles based on the dynamic window algorithm, so as to realize the sending of loading and unloading confirmation after the hook autonomously moves to the target position. The present invention completes the full-process control through a single voice input, without repeated command interaction, effectively improves the operation efficiency, enhances the safety through real-time path correction and dynamic obstacle avoidance, reduces manual intervention, and realizes the unmanned intelligent transportation of tower cranes. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0066] Figure 1 It is a step schematic diagram of steps S101 to S107 of the unmanned tower crane intelligent voice driving system described in the specific implementation manner;

[0067] Figure 2 It is a step schematic diagram of steps S201 to S205 of the unmanned tower crane intelligent voice driving system described in the specific implementation manner. SPECIFIC IMPLEMENTATION MANNER

[0068] The following will further describe the present invention in detail in conjunction with the drawings and embodiments. It should be specifically pointed out that the following embodiments are only used to illustrate the present invention, but do not limit the scope of the present invention. Similarly, the following embodiments are only some embodiments of the present invention rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0069] Please refer to Figure 1 , this embodiment provides an unmanned tower crane intelligent voice driving system based on a large language model, including a tower crane body, a hook assembly and a control terminal. The system further includes:

[0070] S101. Receive a voice control command through the control terminal and input it into the large language model processing module to generate a structured control command. The structured control command includes the lifting position information and target position information of the hook assembly;

[0071] S102. Obtain environmental information, where the environmental information includes the first position information of the hook assembly and the first obstacle distribution information;

[0072] S103. Input the structured control instruction and the environmental information into the path planning module to obtain the hook movement path and control the movement of the hook assembly. The path planning module is configured to be constructed based on the improved A* algorithm and the RRT algorithm. The hook movement path includes the step position information of the hook assembly;

[0073] S104. Real-time obtain the second position information of the hook assembly and compare the first position deviation value between the second position information and the step position information;

[0074] S105. Determine whether the first position deviation value is within the range of the first preset deviation threshold. If not, perform local path replanning on the hook assembly through the dynamic window algorithm to obtain the first local planning path, and update the first local planning path to the hook movement path;

[0075] And, real-time obtain the second obstacle distribution information within the preset detection area, and match the first obstacle distribution information and the second obstacle distribution information;

[0076] S106. Determine whether the second obstacle distribution information coincides with the first obstacle distribution information. If not, perform local path replanning on the hook assembly through the dynamic window algorithm to obtain the second local planning path, and update the second local planning path to the hook movement path;

[0077] S107. After the hook assembly reaches the target position information, send a loading and unloading confirmation instruction through the control terminal.

[0078] In steps S101 and S102, after the system receives the instruction, it first obtains the data collected by the image acquisition devices and sensors installed on the tower crane arm and the hook. The image acquisition devices include ultrasonic sensors, lidar, infrared sensors, cameras, and depth cameras. The sensors include height sensors, distance sensors, speed sensors, and weight sensors. Preferably, use the point cloud + image fusion algorithm to process the data. Among them, the preprocessing of the point cloud data includes point cloud stitching, denoising, and filtering, and the image is stitched and corrected. Combining with the BIM model, generate a real-time environmental map to obtain the three-dimensional coordinates of the lifting point (i.e., the lifting position information), the three-dimensional coordinates of the landing point (i.e., the target position information), the first position information of the hook assembly, and the first obstacle distribution information.

[0079] In step S103, the path planning algorithm adopts an improved A* algorithm and an RRT (Rapidly-exploring Random Tree) fusion model. Preferably, the parameter preset includes: a safety distance threshold (the minimum interval between the hook and the obstacle); a dynamic weight coefficient (the weight of the path smoothness is adjusted in real time according to the data of the wind speed sensor); map initialization requirements: in a new environment, a 2D grid map needs to be generated through the SLAM algorithm and the 3D structure data provided by the BIM model is superimposed, and the map update frequency is 1 s / time. Finally, a hook movement sequence is generated, which is the movement path of the hook.

[0080] In step S104, the second position information is the real-time position information of the hook assembly when it moves under the instruction. Preferably, a height sensor and a distance sensor are used to obtain the second position information of the hook assembly, and the obtained second position information is compared with the previously planned step position information of the hook assembly to obtain the first position deviation value.

[0081] In step S105, preferably, the first preset deviation threshold is 0.3 m. According to the comparison result, when the first position deviation value exceeds 0.3 m, the dynamic window algorithm is triggered to perform local path replanning, a correction instruction is generated to obtain the first local planning path, and the error generated during the operation is corrected in real time to ensure that the real-time transportation path is consistent with the planned transportation path. Specifically, the large language model processes the data by calling the large language model API, sends the obtained information to the large language model, sets some prompt words (such as: help me judge whether the implementation path is consistent with the planned path), and allows the large model to compare the obtained position information with the previously planned path planning information. According to the comparison result, it is judged whether the path needs to be replanned. When the deviation between the planned path and the actual coordinates exceeds 0.3 m, local path replanning is performed; a correction instruction is generated to correct the error generated during the operation in real time to ensure that the real-time transportation path is consistent with the planned transportation path.

[0082] An image acquisition device is used to collect images of the environment around the tower crane and the positions and paths passed by the transported materials to obtain the second obstacle distribution information.

[0083] In step S106, the second obstacle distribution information is the obstacle distribution information obtained in real time, and the first obstacle distribution information is the initial obstacle distribution information in the environment. The inconsistency between the second obstacle distribution information and the first obstacle distribution information indicates that the obstacle distribution in the environment has changed, and there may be obstacles on the current hook movement path. The path should be adjusted to avoid the obstacles on the current planned path. Preferably, a multi-sensor fusion algorithm combined with the SLAM algorithm, the perception algorithm, the action prediction algorithm, and the balance calculation modification algorithm can be used for intelligent obstacle avoidance, and a new transportation path is recalculated.

[0084] In step S107, after the materials are transported to the destination, a notice is sent for the next operation to facilitate manual loading and unloading of materials.

[0085] In this embodiment, a structured control instruction including lifting and target position is generated by receiving a single voice instruction, a real-time map is constructed by combining environmental data collected by multiple sensors, a global path is planned based on an improved A* and RRT fusion algorithm, and double detection of position deviation and newly added obstacles and local path replanning are performed through a dynamic window algorithm. Finally, after the hook assembly autonomously reaches the target position, a loading and unloading confirmation is sent. This method can complete the full-process control through a single voice input without repeated instruction interaction, significantly improving the operation efficiency; adopting multi-sensor fusion and real-time map update technology, combined with path planning and dynamic obstacle avoidance mechanisms, effectively coping with position offsets and dynamic changes of obstacles during the lifting process, ensuring transportation safety; through intelligent environmental perception, path generation and autonomous deviation correction functions, reducing the need for manual monitoring and intervention, and realizing unmanned and precise transportation of tower cranes.

[0086] Please refer to Figure 2 , in some embodiments, the control terminal receives a voice control instruction and inputs it to the large language model processing module to generate a structured control instruction, including:

[0087] S201. Perform endpoint detection and noise suppression on the voice control instruction to obtain a denoised audio frame sequence;

[0088] S202. Extract Mel spectrum features from the denoised audio frame sequence through an acoustic encoder to obtain an acoustic feature vector;

[0089] S203. Input the acoustic feature vector into the trained large language model to generate output information with preset semantic slots, and the preset semantic slots include a material type field, a lifting area field, and a target floor field;

[0090] S204. Convert the output information into three-dimensional space coordinates according to a preset coordinate mapping rule to generate a lifting position information and a target position information;

[0091] S205. Generate a structured control instruction including a safety confirmation flag according to the lifting position information and the target position information.

[0092] In steps S201 and S202, the Mel spectrum is a frequency domain feature designed based on the auditory characteristics of the human ear, which is generated by framing the audio signal and then performing Fourier transform and filtering by a Mel filter bank, representing the frequency distribution of the voice signal; the acoustic feature vector is a high-dimensional numerical vector output by the acoustic encoder after encoding the Mel spectrum, used to represent the voice semantic information.

[0093] In step S203, the voice is converted into text information by a speech recognition model and sent to a large language model. The speech recognition model is an end-to-end ASR model based on Transformer. The training data includes construction industry terms, speech in noisy environments, and multi-dialect speech libraries, and the output is a text sequence. The large language model is fine-tuned based on the deepseek architecture. The training data includes tower crane operation instruction sets, construction specification texts, and safety protocols, and the output is a structured instruction JSON. After parsing by the large language model, a confirmation voice is generated.

[0094] Preferably, the speech recognition and the large language model adopt a joint training model, using the SpeechT5 architecture. The input is the speech mel spectrogram, and the output is directly mapped to the semantic vector space. The instruction text and operation intention are generated through decoding, and a confirmation voice is output.

[0095] In step S204, the preset coordinate mapping rule refers to the conversion rule that associates the lifting area field and the target floor field with the three-dimensional coordinate system in the construction drawing or BIM model. For example, according to the floor plan and the lifting area coding table, the semantic information is mapped to the spatial coordinates of the starting and ending points of the hook.

[0096] In this embodiment, through the endpoint detection, noise suppression, and acoustic feature extraction of the voice instruction, combined with the semantic parsing ability of the large language model, the natural language instruction is accurately converted into a structured control instruction containing three-dimensional coordinates, avoiding manual repeated input; through the preset coordinate mapping rule, the automation conversion from semantics to spatial position is realized, reducing the human coordinate calculation error and improving the lifting positioning accuracy; based on the generation mechanism of the safety confirmation identifier, the integrity and executability of the instruction are ensured, further guaranteeing the safety and reliability of the unmanned transportation process, and significantly reducing the human dependence.

[0097] In some embodiments, the path planning module is configured to construct based on the improved A* algorithm and the RRT algorithm, including:

[0098] Initialize the search tree and convert the kinematic constraint information of the hook assembly into a state space sampling rule;

[0099] Introduce a dynamic weight factor into the heuristic function of the improved A* algorithm, and the dynamic weight factor is configured to be adjusted according to the wind speed information;

[0100] According to the state space sampling rule and the heuristic function, execute the improved A* algorithm and obtain an initial motion path;

[0101] Smooth the initial motion path through the BRT algorithm to obtain a sequence of stepping position information, which is the hook motion path.

[0102] In this embodiment, the conversion of the kinematic constraint information of the hook assembly into a state space sampling rule is represented by the following formula:

[0103]

[0104] Among them, is the maximum allowable acceleration, and ψ is the sling length.

[0105] The introduction of a dynamic weight factor in the heuristic function of the improved A* algorithm is represented by the following formula:

[0106] h(n) = w d · Euclidean distance(n,g) + w c · path curvature(n);

[0107] Among them, w d is dynamically adjusted according to the wind speed sensor data in the range of 0.8 to 1.2.

[0108] The initial path generated by A is smoothed by the B-spline using the RRT algorithm to generate the sequence of the stepping position information that satisfies the continuous acceleration constraint.

[0109] The path planning module of this embodiment ensures that the hook path conforms to the mechanical dynamics limit by integrating the improved A algorithm and the RRT algorithm and combining the state space sampling rules converted by the kinematic constraints; the improved A algorithm introduces a dynamic weight factor to adjust the weight coefficients of the Euclidean distance and the path curvature in the heuristic function in real time according to the wind speed sensor data, improving the path smoothness and the anti-wind disturbance ability; the RRT algorithm performs B-spline smoothing on the initial path to generate a sequence of stepping positions that satisfies the continuous acceleration constraint, avoiding sudden stops or jitters. Finally, the generated hook motion path has both global optimality, dynamic adaptability and motion continuity, significantly improving the accuracy and safety of unmanned tower crane transportation.

[0110] In some embodiments, the dynamic window algorithm performs local path replanning on the hook assembly, and the obtained first local planning path includes:

[0111] A first dynamic window velocity space is established. The first dynamic window velocity space includes first local velocity information corresponding to multiple positions within a unit time. Each first local velocity information is represented by formula (1), and formula (1) is as follows:

[0112] V di = {(v, w)|v ∈ [v min , v max , w ∈ [w min , w max};

[0113] In formula (1), v is the first linear velocity of the sling in the hook assembly, w is the first rotational angular velocity of the hook assembly, and V diis the first local velocity information of the hook assembly at the i-th unit time, where i is the count of unit time in the first dynamic window velocity space, T is the total time of the first dynamic window velocity space, and v max is the first maximum planned velocity of the first linear velocity of the sling in the hook assembly, v min is the first minimum planned velocity of the first linear velocity of the sling in the hook assembly, w max is the first maximum planned velocity of the first rotational angular velocity of the hook assembly, w min is the first minimum planned velocity of the first rotational angular velocity of the hook assembly;

[0114] Calculate the first evaluation function of each first local velocity information, which is represented by formula (2), and formula (2) is as follows:

[0115] G(V di ) = ξ1·align(v, w) + ξ2·dist(v, w) + ξ3·vel(v);

[0116] In formula (2), G(V di ) is the first evaluation value of the first evaluation function, ξ1 is the first evaluation weight, ξ2 is the second evaluation weight, ξ3 is the third evaluation weight, is the direction alignment term, is the minimum obstacle avoidance distance term, is the velocity efficiency term, and the weight coefficients satisfy ξ1 + ξ2 + ξ3 = 1. The default configuration is ξ1 = 0.4, ξ2 = 0.4, ξ3 = 0.2;

[0117] Select the first local velocity information with the highest first evaluation value in each unit time and generate the first local planning path, which is represented by formula (3), and formula (3) is as follows:

[0118]

[0119] In formula (3), P local is the first local planning path, p t is the first local path segment, p0 is the path in the current hook movement path within the range of the first local planning path, denoted as the first initial hook path, is the unit vector of the current movement direction, R is the rotation matrix, and Δr is the hook swing compensation amount.

[0120] In this embodiment, the v step size is 0.1 m / s, the w step size is 0.5 ° / s. Calculate the first evaluation function of each first local velocity information, select the optimal velocity pair to generate the local path segment, and splice P local continuously with the global path to generate the first local planning path.

[0121] The dynamic window algorithm of this embodiment generates a first set of local velocity information that meets kinematic constraints by constructing a first dynamic window velocity space and combining the constraint ranges of the first linear velocity and the first rotational angular velocity of the hook assembly; constructs a first evaluation function based on the direction alignment term, the minimum obstacle avoidance distance term, and the velocity efficiency term, and balances the path direction accuracy, the obstacle avoidance efficiency, and the motion speed optimization through default weight coefficients to ensure the dynamic adaptability of the local path; performs a refined search on the velocity space through the step size parameter, selects the first local velocity information with the highest first evaluation value, and generates a continuous and smooth first local path segment in combination with the rotation matrix and the hook swing compensation amount to achieve seamless stitching with the global path. This mechanism effectively solves the problems of dynamic obstacle interference and path deviation, improves the real-time obstacle avoidance ability and motion continuity of the hook motion trajectory, and ensures the stability and safety of the unmanned hoisting process.

[0122] In some embodiments, obtaining the second obstacle distribution information within the preset detection area in real time, and matching the first obstacle distribution information with the second obstacle distribution information includes:

[0123] Extracting the first obstacle spatio-temporal features from the first obstacle distribution information;

[0124] Constructing the second obstacle distribution information through the UWB locator and the vision device, and extracting the second obstacle spatio-temporal features;

[0125] Performing the Hungarian algorithm matching on the first obstacle features and the second obstacle features with adjacent timestamps to establish an obstacle trajectory association network;

[0126] When there are unmatched second obstacle features, triggering the second obstacle distribution information update flag.

[0127] In this embodiment, the UWB locator is used to obtain the three-dimensional coordinates of the starting point of the transported materials being lifted and the three-dimensional coordinates of the target, and timestamp alignment and Kalman filtering are used to eliminate the time difference and noise of the sensor data; the obstacle contour point cloud is extracted from the data collected by the lidar, and the material safety area is generated according to the outer contour range of the point cloud + 1 meter. The first obstacle features and the second obstacle features are obtained by identifying the obstacle material (such as steel bars, wooden boards) through the depth camera to predict the collision risk. The description of the obstacle spatio-temporal features includes the point cloud density, the RGB texture features, and the motion vector.

[0128] In this embodiment, by fusing the multi-source perception data of the UWB locator and the vision device, combining timestamp alignment and Kalman filtering to eliminate sensor noise, the positioning accuracy of the three-dimensional coordinates of the obstacle is improved; using lidar and depth cameras to extract the contour point cloud, material type, and RGB texture features of the obstacle, constructing a multi-dimensional spatio-temporal feature description, and enhancing the robustness of dynamic obstacle recognition; based on the Hungarian algorithm, matching the first obstacle feature and the second obstacle feature, establishing an obstacle trajectory association network, realizing continuous tracking of the motion vector and collision risk prediction; the unmatched feature triggers the update mechanism of the second obstacle distribution information, ensuring the real-time and integrity of obstacle detection, providing a basis for dynamic obstacle avoidance decision-making in unmanned hoisting, and effectively reducing the collision risk.

[0129] In some embodiments, the dynamic window algorithm performs local path replanning on the hook assembly, and the obtained second local planning path includes:

[0130] A second dynamic window velocity space is established. The second dynamic window velocity space includes second local velocity information corresponding to multiple positions within a unit time. Each second local velocity information is represented by formula (4), and formula (4) is as follows:

[0131] V′ di′ ={(v′,w′)|v′∈[v′ min ,v′ max ,w′∈[w′ min ,w′ max};

[0132] In formula (4), v′ is the second linear velocity of the sling in the hook assembly, w′ is the second rotational angular velocity of the hook assembly, V′ di′ is the second local velocity information of the hook assembly at the i′-th unit time, i′ is the count of unit time in the second dynamic window velocity space, T′ is the total time used in the second dynamic window velocity space, v′ max is the second maximum planned velocity of the second linear velocity of the sling in the hook assembly, v′ min is the second minimum planned velocity of the second linear velocity of the sling in the hook assembly, w′ max is the second maximum planned velocity of the second rotational angular velocity of the hook assembly, w′ min is the second minimum planned velocity of the second rotational angular velocity of the hook assembly;

[0133] Calculate the second evaluation function of each second local velocity information, which is represented by formula (5), and formula (5) is as follows:

[0134] G(V′ di′ )=ξ4·align(v′,w′)+ξ5·dist(v′,w′)+ξ6·vel(v′)+θ;

[0135] In formula (5), G(V′ di′ ) is the second evaluation value of the second evaluation function, ξ4 is the fourth evaluation weight, ξ5 is the fifth evaluation weight, ξ6 is the sixth evaluation weight, K is the emergency avoidance weight, is the direction alignment term, is the minimum obstacle avoidance distance term, is the speed efficiency term, and the weight coefficients satisfy ξ4 + ξ5 + ξ6 = 1. By default, ξ4 = 0.4, ξ5 = 0.4, ξ6 = 0.2, and θ is the un-matched second obstacle feature;

[0136] Select the second local velocity information with the highest second evaluation value in each unit time and generate the second local planning path, which is represented by formula (6). Formula (6) is as follows:

[0137]

[0138] In formula (6), P′ local is the second local planning path, p′ t is the second local path segment, and p′0 is the path in the range of the second local planning path in the current hook movement path, denoted as the second initial hook path. is the unit vector of the current movement direction, R is the rotation matrix, and Δr′ is the hook swing compensation amount.

[0139] In this embodiment, the cubic spline interpolation method is used to ensure the continuity of the jerk of the obstacle avoidance path;

[0140] The feasibility of the second local planning path is verified by the hoisting swing stability verification module.

[0141] An example of the obstacle avoidance process is as follows:

[0142] Perception stage: The lidar detects the temporarily stacked materials 2 meters ahead.

[0143] Pre-judgment stage: Combining with the BIM model, the LSTM predicts that the material stacking may expand, triggering path re-planning.

[0144] Decision-making stage: The A* algorithm generates a detour path, which is smoothed by RRT and the new hoisting path task is the second local planning path.

[0145] Execution stage: The frequency converter control board adjusts the motor control command according to the second local planning path.

[0146] In this embodiment, the second linear velocity and the second rotational angular velocity of the hook are constrained by constructing a second dynamic window velocity space, and the second evaluation function is optimized by combining the unmatched second obstacle feature items, enhancing the dynamic adaptability to sudden obstacles; the cubic spline interpolation method ensures the continuity of the jerk of the obstacle avoidance path and avoids sudden changes in the hoisting motion; the hoisting swing stability verification module conducts a feasibility check on the second local planning path to ensure that the path complies with the mechanical dynamics constraints; the collaborative planning mechanism that integrates the LSTM to predict the material expansion trend and the A* algorithm and the RRT algorithm realizes the closed-loop response from obstacle perception to path replanning; the frequency converter control board accurately adjusts the motor commands according to the second local planning path, coordinates the data of the lidar and the depth camera, improves the real-time performance and execution reliability of the hook obstacle avoidance in complex scenarios, and strengthens the dynamic safety protection ability of the unmanned hoisting system.

[0147] In some embodiments, the environmental information is configured to be obtained through the following steps:

[0148] Generate depth point clouds through the binocular camera on the hook assembly and perform ICP registration with the lidar data;

[0149] Fuse the UWB positioning data and the encoder readings to construct the hook pose state equation, which is represented by formula (7), and formula (7) is as follows:

[0150]

[0151] In formula (7), is the optimal estimated value of the hook state at time e, F e is the state transition matrix, x e-1 is the hook state vector at time e-1, B e is the control input matrix, u e is the motor control input vector, E r is the Kalman gain matrix, z e is the sensor observation vector, H e is the observation matrix;

[0152] Solve the hook pose state equation to obtain the environmental information.

[0153] In this embodiment, includes the three-dimensional position and the three-dimensional attitude, B e is associated with the motor control quantity, u e includes the swing / luffing / lifting motor commands, E e is used for sensor weight allocation, z e is the fused measurement value of UWB and vision, H e is the mapping from the state to the observation.

[0154] After the path task work order is formed, it is automatically sent to the frequency converter control board. After receiving the path work order, the frequency converter control board decomposes it into motor control instructions:

[0155] Slewing motor: S-curve acceleration and deceleration control;

[0156] Luffing motor: PID closed-loop control;

[0157] Hoisting motor: multi-speed vector control.

[0158] The frequency converter control board controls the tower crane to transport materials according to the transportation steps.

[0159] In this embodiment, the depth point cloud ICP registration of the binocular camera and the lidar is used to enhance the environmental perception accuracy. The UWB positioning data and the encoder readings are fused to construct the hook pose state equation, and the Kalman gain matrix is combined to optimize the sensor weight allocation to improve the robustness of the three-dimensional pose estimation. The path task work order is automatically distributed to the frequency converter control board and decomposed into the S-curve acceleration and deceleration of the slewing motor, the PID closed-loop of the luffing motor, and the multi-speed vector control instructions of the hoisting motor to achieve precise coordination of multi-degree-of-freedom motion. Based on the dynamic mapping mechanism of the state transition matrix and the observation matrix, it ensures the efficient fusion of the hook pose state and the sensor measurement values, strengthens the stability and response speed of the hoisting and transportation under complex working conditions, and guarantees the safety and control reliability of the entire process of material transportation.

[0160] In some embodiments, after the hook assembly reaches the target position information, the loading and unloading confirmation instruction sent by the control terminal includes:

[0161] Verify whether the load change rate of the hook assembly is less than the preset loading and unloading threshold range through the pressure sensor;

[0162] And, detect the contact state between the material and the target area corresponding to the target position information through the vision device, and the contact state is configured as the contact area ratio;

[0163] When both the load change rate is less than the preset loading and unloading threshold range and the contact area ratio is greater than the preset contact threshold range are satisfied, trigger the loading and unloading confirmation instruction sent by the control terminal.

[0164] In this embodiment, the preset contact threshold is preferably 80%. Detect whether the load change rate is less than the preset loading and unloading threshold range through the pressure sensor, and combine the vision device to verify whether the contact area ratio between the material and the target area meets the preset contact threshold range. The double conditions cooperate to trigger the loading and unloading confirmation instruction, accurately determine the completion state of the hook unloading, avoid the risks of false triggering or missed judgment, ensure the stability of the material placement and the safety of unloading, improve the automation accuracy and reliability of the loading and unloading process, and reduce the dependence on manual confirmation.

[0165] In some embodiments, the first preset deviation threshold is configured to be dynamically adjusted:

[0166] When the hook load weight is 80% of the preset load weight, the upper limit value of the first preset deviation threshold is decreased;

[0167] When the real-time wind speed of the hook assembly is 70% of the preset safe wind speed, the lower limit value of the first preset deviation threshold is increased.

[0168] In this embodiment, when the hook load weight is 80% of the preset load weight, the first preset deviation threshold is tightened to 0.2 m; when the detected wind speed ≥ 8 m / s, the first preset deviation threshold is relaxed to 0.4 m; in other cases, the default threshold of 0.3 m is maintained.

[0169] Based on the hook load weight and the real-time wind speed, this embodiment dynamically adjusts the first preset deviation threshold, adaptively balances the positioning accuracy and anti-interference ability, enhances the control safety under high loads and the operation stability under strong wind conditions, avoids the misjudgment or failure risks caused by a single threshold, optimizes the adaptability to different working conditions during the hoisting process, and ensures the coordinated improvement of operation safety and efficiency.

[0170] Adopting the above technical solution, compared with the prior art, the beneficial effects of the present invention are as follows:

[0171] The present invention provides an unmanned tower crane intelligent voice driving system based on a large language model. The voice command is parsed into a structured control command including the hoisting and target positions by the large language model processing module, and a global motion path is generated by combining the path planning module that fuses the improved A* algorithm and the RRT algorithm, realizing the full-process control with a single voice input and significantly reducing the need for manual interaction; based on the dynamic window algorithm, double detection of position deviation and newly added obstacles and local path replanning are performed, and combined with the multi-sensor fusion mechanism of the hook pose state equation, the motion trajectory is corrected in real time and obstacles are dynamically avoided, ensuring the transportation safety and path adaptability under complex working conditions; the loading and unloading state is collaboratively verified by the pressure sensor and the vision device, and combined with the dynamically adjusted first preset deviation threshold, the positioning accuracy and anti-interference ability of high loads and strong winds are adaptively balanced, realizing the full-closed-loop intelligent control of the hoisting process. Through the coordinated action of environment perception, path planning, dynamic obstacle avoidance and self-correction, the system effectively improves the automation level, operation efficiency and safety reliability of the unmanned operation of the tower crane, and reduces the dependence on manual intervention.

[0172] In addition, in each embodiment of the present invention, each functional unit can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0173] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods according to various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0174] The foregoing are only partial embodiments of the present invention, and thus do not limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. An unmanned tower crane intelligent voice driving system based on a large language model, characterized in that, The system includes a tower crane body, a hook assembly, and a control terminal. The system further includes: Receiving a voice control instruction through the control terminal and inputting it into a large language model processing module to generate a structured control instruction, where the structured control instruction includes the lifting position information and target position information of the hook assembly; Obtaining environmental information, where the environmental information includes the first position information of the hook assembly and the first obstacle distribution information; Inputting the structured control instruction and the environmental information into a path planning module to obtain a hook movement path and controlling the movement of the hook assembly. The path planning module is configured to be constructed based on an improved A* algorithm and an RRT algorithm. The hook movement path includes the step position information of the hook assembly; Obtaining the second position information of the hook assembly in real time and comparing the first position deviation value between the second position information and the step position information; Judging whether the first position deviation value is within the range of a first preset deviation threshold. If not, performing local path replanning on the hook assembly through a dynamic window algorithm to obtain a first local planning path and updating the first local planning path to the hook movement path; And obtaining the second obstacle distribution information within a preset detection area in real time and matching the first obstacle distribution information with the second obstacle distribution information; Judging whether the second obstacle distribution information coincides with the first obstacle distribution information. If not, performing local path replanning on the hook assembly through a dynamic window algorithm to obtain a second local planning path and updating the second local planning path to the hook movement path; After the hook assembly reaches the target position information, sending a loading and unloading confirmation instruction through the control terminal.

2. The unmanned tower crane intelligent voice driving system based on the large language model according to claim 1, wherein, Receiving a voice control instruction through the control terminal and inputting it into a large language model processing module to generate a structured control instruction, including: Performing endpoint detection and noise suppression on the voice control instruction to obtain a denoised audio frame sequence; Extracting Mel spectrogram features from the denoised audio frame sequence through an acoustic encoder to obtain an acoustic feature vector; Inputting the acoustic feature vector into a trained large language model to generate output information with preset semantic slots, where the preset semantic slots include a material type field, a lifting area field, and a target floor field; Converting the output information into three-dimensional space coordinates according to a preset coordinate mapping rule to generate the lifting position information and target position information; Generating a structured control instruction including a safety confirmation identifier according to the lifting position information and target position information.

3. The unmanned tower crane intelligent voice driving system based on the large language model according to claim 1, characterized in that, The path planning module is configured to be constructed based on an improved A* algorithm and an RRT algorithm, including: Initializing a search tree and converting the kinematic constraint information of the hook assembly into a state space sampling rule; Introducing a dynamic weight factor into the heuristic function of the improved A* algorithm, where the dynamic weight factor is configured to be adjusted according to wind speed information; Executing the improved A* algorithm according to the state space sampling rule and the heuristic function to obtain an initial movement path; Performing spline smoothing processing on the initial movement path through a BRT algorithm to obtain a sequence of step position information, which is the hook movement path.

4. The unmanned tower crane intelligent voice driving system based on the large language model according to claim 1, wherein, The dynamic window algorithm performs local path replanning on the hook assembly, and the obtained first local planning path includes: A first dynamic window velocity space is established. The first dynamic window velocity space includes first local velocity information corresponding to multiple positions within a unit time. Each piece of the first local velocity information is represented by formula (1), and formula (1) is as follows: V di = {(v, w) | v ∈ [v min , v max , w ∈ [w min , w max}; In formula (1), v is the first linear velocity of the sling in the hook assembly, w is the first rotational angular velocity of the hook assembly, V di is the first local velocity information of the hook assembly in the i-th unit time, i is the count of unit time in the first dynamic window velocity space, T is the total time used in the first dynamic window velocity space, v max is the first maximum planned velocity of the first linear velocity of the sling in the hook assembly, v min is the first minimum planned velocity of the first linear velocity of the sling in the hook assembly, w max is the first maximum planned velocity of the first rotational angular velocity of the hook assembly, w min is the first minimum planned velocity of the first rotational angular velocity of the hook assembly; Calculate the first evaluation function of each piece of the first local velocity information, which is represented by formula (2), and formula (2) is as follows: G(V di ) = ξ1·align(v, w) + ξ2·dist(v, w) + ξ3·vel(v); In formula (2), G(V di ) is the first evaluation value of the first evaluation function, ξ1 is the first evaluation weight, ξ2 is the second evaluation weight, ξ3 is the third evaluation weight, is the direction alignment term, is the minimum obstacle avoidance distance term, is the speed efficiency term, and the weight coefficients satisfy ξ1 + ξ2 + ξ3 = 1. By default, ξ1 = 0.4, ξ2 = 0.4, and ξ3 = 0.2; Select the first local velocity information with the highest first evaluation value in each unit time and generate a first local planning path, which is represented by formula (3), and formula (3) is as follows: In formula (3), P local is the first local planning path, p t is the first local path segment, and p0 is the path within the range of the first local planning path in the current hook movement path, denoted as the first initial hook path. is the unit vector of the current movement direction, R is the rotation matrix, and Δr is the hook swing compensation amount.

5. The unmanned tower crane intelligent voice driving system based on the large language model according to claim 1, characterized in that, Obtain the second obstacle distribution information within the preset detection area in real time, and match the first obstacle distribution information with the second obstacle distribution information, including: Extract the first obstacle spatio-temporal characteristics from the first obstacle distribution information; Construct the second obstacle distribution information through a UWB locator and a vision device, and extract the second obstacle spatio-temporal characteristics; Perform Hungarian algorithm matching on the first obstacle characteristics and the second obstacle characteristics at adjacent time stamps to establish an obstacle trajectory association network; When there are unmatched second obstacle characteristics, trigger the second obstacle distribution information update flag.

6. The unmanned tower crane intelligent voice driving system based on a large language model according to claim 5, characterized in that, The dynamic window algorithm performs local path replanning on the hook assembly, and the obtained second local planning path includes: A second dynamic window velocity space is established. The second dynamic window velocity space includes second local velocity information corresponding to multiple positions within a unit time. Each piece of the second local velocity information is represented by formula (4), and formula (4) is as follows: V′ di′ ={(v′, w′)|v′ ∈ [v′ min , v′ max , w′ ∈ [w′ min , w′ max}; In formula (4), v′ is the second linear velocity of the sling in the hook assembly, w′ is the second rotational angular velocity of the hook assembly, V′ di′ is the second local velocity information of the hook assembly at the i′-th unit time, i′ is the count of unit time in the second dynamic window velocity space, T′ is the total time used in the second dynamic window velocity space, v′ max is the second maximum planned velocity of the second linear velocity of the sling in the hook assembly, v′ min is the second minimum planned velocity of the second linear velocity of the sling in the hook assembly, w′ max is the second maximum planned velocity of the second rotational angular velocity of the hook assembly, w′ min is the second minimum planned velocity of the second rotational angular velocity of the hook assembly; Calculate the second evaluation function of each piece of the second local velocity information, which is represented by formula (5), and formula (5) is as follows: G(V′ di′ ) = ξ4·align(v′, w′) + ξ5·dist(v′, w′) + ξ6·vel(v′) + θ; In formula (5), G(V′ di′ ) is the second evaluation value of the second evaluation function, ξ4 is the fourth evaluation weight, ξ5 is the fifth evaluation weight, ξ6 is the sixth evaluation weight, K is the emergency avoidance weight, is the direction alignment term, is the minimum obstacle avoidance distance term, is the speed efficiency term, and the weight coefficients satisfy ξ4 + ξ5 + ξ6 = 1. By default, ξ4 = 0.4, ξ5 = 0.4, ξ6 = 0.2, and θ is the second obstacle feature that is not matched; Select the second local velocity information with the highest second evaluation value in each unit time and generate a second local planning path, which is represented by formula (6), and formula (6) is as follows: In formula (6), P′ local is the second local planned path, p′ t is the second local path segment, and p′0 is the path within the range of the second local planned path in the current hook movement path, denoted as the second initial hook path. is the unit vector of the current movement direction, R is the rotation matrix, and Δr′ is the hook swing compensation amount.

7. The unmanned tower crane intelligent voice driving system based on the large language model according to claim 1, characterized in that, The environmental information is configured to be obtained through the following steps: Generate depth point clouds through a binocular camera on the hook assembly and perform ICP registration with lidar data; Fuse UWB positioning data and encoder readings to construct a hook pose state equation, which is represented by formula (7), and formula (7) is as follows: In formula (7), is the optimal estimated value of the hook state at time e, F e is the state transition matrix, x e-1 is the hook state vector at time e-1, B e is the control input matrix, u e is the motor control input vector, E e is the Kalman gain matrix, z e is the sensor observation vector, H e is the observation matrix; Solve the hook pose state equation to obtain the environmental information.

8. The unmanned tower crane intelligent voice driving system based on a large language model according to claim 1, characterized in that, After the hook assembly reaches the target position information, send a loading and unloading confirmation instruction through the control terminal, including: Verify whether the load change rate of the hook assembly is less than the preset loading and unloading threshold range through a pressure sensor; And detect the contact state between the material and the target area corresponding to the target position information through a vision device. The contact state is configured as the contact area ratio; When both the load change rate is less than the preset loading and unloading threshold range and the contact area ratio is greater than the preset contact threshold range are satisfied, trigger the sending of the loading and unloading confirmation instruction through the control terminal.

9. The unmanned tower crane intelligent voice driving system based on the large language model according to claim 1, characterized in that, The first preset deviation threshold is configured to be dynamically adjusted: When the hook load weight is 80% of the preset load weight, reduce the upper limit value of the first preset deviation threshold; When the real-time wind speed of the hook assembly is set at 70% of the preset safe wind speed, increase the lower limit value of the first preset deviation threshold.

Images