Methods, systems, equipment and storage media for assisted driving of tire cranes

Abstract

This application provides a method, system, device, and storage medium for assisted driving of a tire-mounted crane. The method includes: acquiring images of the environment where the tire-mounted crane is located; preprocessing the environmental images to generate a surround view image; based on the surround view image, recognizing environmental semantics and constructing a dynamic semantic map using an image big data language model, wherein the image big data language model uses an attention mechanism to output spatial perception features, and fusing the spatial perception features with equipment status information for semantic understanding; inputting features including the surround view image, dynamic semantic map, and equipment status information into a hybrid expert model, outputting control commands including continuous control quantities; converting the control commands into execution signals for assisted driving, and adjusting and optimizing in real time during the driving process. This application uses an image big data language model to perceive and understand the semantics of complex scenes, and adopts an end-to-end architecture to make decisions based on the perceived information, obtaining accurate assisted driving control commands adapted to the tire-mounted crane scenario.

Description

Technical Field

[0001] This invention relates to the field of automated control technology for heavy port equipment, and in particular to a method, system, device and storage medium for assisted driving of a tire-mounted crane. Background Technology

[0002] As the core equipment for container loading and unloading at ports, the efficiency and safety performance of rubber-tired cranes directly determine the operational benefits of ports.

[0003] In related technologies, traditional environmental perception systems often employ a combination of LiDAR and RTK satellite positioning. However, this approach is susceptible to obstruction from high-altitude infrastructure, leading to reduced satellite positioning accuracy. Furthermore, LiDAR systems account for approximately 50% of the total equipment cost, making them prohibitively expensive. Site subsidence can cause the trolley's satellite antenna to tilt to one side, resulting in satellite positioning projections not near the trolley's tires and thus out-of-range positioning. In addition, driving operation relies heavily on human experience; novice operators experience a 40% increase in error rate after four hours of continuous operation in complex conditions. Moreover, the real-time point cloud data generated by LiDAR has low resolution, limiting the available environmental data. Obstacle avoidance judgments are limited to the fixed installation direction of the trolley's travel, leading to misjudgments of distant objects. Additionally, site subsidence causes pitch and tilt of the trolley, making distant ground objects easily identified as obstacles. End-to-end control in the field of drone piloting can improve dynamic decision-making capabilities, but current solutions are only designed for road scenarios and cannot adapt to scenarios with obstructions from the spreader, dense stacking of containers, and heavy-load trolley starts—the specific operating environments of tire-mounted cranes.

[0004] Based on the above analysis of the development status of this technology field, the existing technologies lack an assisted driving solution that uses stitched surround view images to obtain all-round information, employs image big language models for sufficient environmental perception, and then outputs control commands through an end-to-end hybrid expert model to adapt to the tire crane scenario. Summary of the Invention

[0005] The purpose of this invention is to provide a tire-mounted crane assisted driving method, system, device and storage medium, which aims to solve the above-mentioned problems in the prior art.

[0006] According to a first aspect of the present invention, a tire-mounted assisted driving method is provided, comprising: Collect images of the environment where the tire crane is located, and preprocess the environmental images to generate a surround view image; Based on the panoramic image, the image big language model is used to identify environmental semantics and construct a dynamic semantic map. The image big language model uses an attention mechanism to output spatial perception features, and integrates spatial perception features with device status information for semantic understanding. The hybrid expert model is input with features including a surround view image, a dynamic semantic map, and equipment status information, and outputs control commands including continuous control quantities. Control commands are converted into execution signals to assist driving, and adjustments and optimizations are made in real time during the driving process.

[0007] According to a second aspect of the present invention, a tire-mounted crane assisted driving system is provided, comprising: The acquisition module is used to acquire images of the environment where the tire crane is located, and to preprocess the environmental images to generate a surround view image; The perception module is used to identify environmental semantics and construct a dynamic semantic map based on the panoramic image through an image big language model. The image big language model uses an attention mechanism to output spatial perception features, which are then fused with device status information for semantic understanding. The decision module is used to input features including surround view images, dynamic semantic maps and equipment status information into the hybrid expert model, and output control commands including continuous control quantities. The control module is used to convert control commands into execution signals to assist driving and make adjustments and optimizations in real time during the driving process.

[0008] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the tire-mounted crane assisted driving method provided in the first aspect of the present disclosure.

[0009] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which an implementation program for information transmission is stored, which, when executed by a processor, implements the steps of the tire-mounted crane assisted driving method provided in the first aspect of the present disclosure.

[0010] The technical solution provided by the embodiments of the present invention has the following beneficial effects: it uses an image-based large language model to perceive the semantic understanding of complex scenes, and enhances the scene understanding ability by acquiring equipment status information, thereby improving the adaptability to severe weather and reducing the hardware cost of lidar equipment; it adopts an end-to-end architecture to make decisions based on the perceived information, eliminates the cumulative error of multiple links, meets the requirements of high-density stacking, and obtains accurate assisted driving control commands adapted to the tire crane trolley scenario.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a tire-mounted crane assisted driving method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of camera installation and deployment according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the image-based large language model structure according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the overall frame of the assisted driving system according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the complete processing flow of an embodiment of the present invention; Figure 6 This is a schematic diagram of a tire-mounted crane assisted driving system according to an embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0015] Method Example 1 According to an embodiment of the present invention, a tire-mounted crane-assisted driving method is provided. Figure 1 This is a flowchart of a tire-mounted crane assisted driving method according to an embodiment of the present invention, such as... Figure 1 As shown, the tire-mounted crane-assisted driving method according to an embodiment of the present invention specifically includes: In step S110, an environmental image of the tire crane is acquired, and the environmental image is preprocessed to generate a surround view image, specifically including: Initial environmental images are simultaneously acquired by industrial-grade global shutter cameras installed in the front, rear, left, and right directions of the wheeled crane's gantry legs. Figure 2 This is a schematic diagram of the camera installation and deployment according to an embodiment of the present invention, as shown below. Figure 2 The diagram shows an approximate representation of the hardware deployment.

[0016] The environmental image is sequentially subjected to distortion correction, image enhancement and panoramic stitching based on Zhang's calibration method to generate a panoramic image. Zhang's calibration method uses a camera projection model to adjust the position and image size, and adjusts the local shape of the image through distortion correction formula.

[0017] Formula 1 represents the camera projection model of Zhang's calibration method, and Formula 2 represents the distortion correction formula of Zhang's calibration method: Formula 1; Formula 2; in, Indicates the scaling factor. Represents the x-coordinate of pixels. Represents the pixel ordinate. Indicates the focal length in the horizontal direction. Indicates the focal length in the vertical direction. Indicates the horizontal center of the image. Indicates the vertical center of the image, ( , , () represents the three-dimensional position of an object in reality; , Represents the radial distortion coefficient. This represents the tangential distortion coefficient.

[0018] The camera projection model is used to adjust the image size and position, the distortion correction formula is used for local image adjustment, and the reprojection error of Zhang's calibration method is ≤0.5 pixels; In this embodiment of the invention, image enhancement employs adaptive histogram equalization to improve contrast in low-light environments, and panoramic stitching uses feature point matching combined with bundle adjustment, with a stitching error ≤ 3 pixels.

[0019] In step S120, based on the panoramic image, environmental semantics are identified and a dynamic semantic map is constructed using an image-based large language model. The image-based large language model employs an attention mechanism to output spatial awareness features, which are then fused with device status information for semantic understanding. Specifically, this includes: The LLaVA-NeXT architecture, consisting of a visual encoder, a spatial relation reasoning module, and a text decoder, is used as the image-based large language model, i.e., the improved LLaVA-NeXT architecture. To adapt to port scenarios, the original LLaVA-NeXT architecture has been improved, including the addition of a port target detection head, the introduction of a spatial inference module, and the fusion of equipment status information and target distance estimation.

[0020] The surround view image is input into the visual encoder of the newly added port target detection head, and the output is the identified target. The detection network is optimized for 12 core target categories, including containers (recognition accuracy ≥99.5%), trucks (recognition accuracy ≥99%), workers (recognition accuracy ≥98.5%), and container stacking area (recognition accuracy ≥99%). A spatial reasoning module is introduced to learn the relative positional relationships between targets by extracting content from the Transformer encoder, such as "the truck is stopped 3 meters directly below the spreader". The spatial reasoning model uses an attention mechanism to calculate and identify the relative positions between targets, which are used as spatial perception features. Formula 3 is used to calculate the relative positions between targets using an attention mechanism: Formula 3; in, These represent the query matrix, key matrix, and value matrix, respectively. Indicates the feature dimension; For example, when querying the feature vector of a spreader, the key includes container features, truck features, etc. Attention is used to find that the spreader is most relevant to the truck, and the value is the final output result.

[0021] Based on spatial perception features and with equipment status information as prompts, the system obtains the following through a status encoder: In this embodiment of the invention, parameters such as the real-time speed of the tire crane, the height of the spreader, and the steering angle are used as conditional inputs to enhance scene understanding. A text decoder combines the two types of information to output a more accurate environmental semantic result.

[0022] Figure 3 This is a schematic diagram of the image-based large language model structure according to an embodiment of the present invention, such as... Figure 3 As shown, the connection between the visual encoder, spatial relationship reasoning module, and text decoder is illustrated. Each module can be replaced with other models.

[0023] Preferably, Formula 4 is further used to estimate the target distance. This indicates the distance between the device and the screen: Formula 4; in, Indicates the camera's focal length. Indicates the actual height of the target. This indicates the pixel height of the target in the image.

[0024] Constructing dynamic semantic maps based on environmental semantics specifically includes: The layout of the container stacking area, the driving lanes, and the boundaries of the work area are set as a static layer. The risk level and prohibited work areas are marked in the static layer. The collision risk level is divided into high, medium, and low. The moving target consisting of trucks and personnel in the environmental semantics is used as the initial result of the dynamic layer. The position and speed of the moving target are labeled, and the motion trend is predicted by Kalman filtering to generate a dynamic semantic map. In this embodiment of the invention, the error of motion trend prediction is ≤0.5 meters within 3 seconds.

[0025] Kalman filter prediction uses the following method: State prediction:

[0026] Covariance prediction:

[0027] Kalman gain:

[0028] Status Update:

[0029] Covariance update:

[0030] in, Represents the state transition matrix. Represents the control input matrix. Represents the control input vector. Represents the covariance matrix. Represents the process noise covariance matrix. Represents the observation matrix. This represents the measurement noise covariance. Represents the measurement vector. This indicates that information from the previous time step is being used in the current time step. This indicates that information from the previous time step was used in the previous time step.

[0031] In step S130, features including a surround view image, a dynamic semantic map, and equipment status information are input into the hybrid expert model, and control commands including continuous control quantities are output, specifically including: Obtaining a hybrid expert model (MoE) consisting of an input layer, a gating network, an expert network, and an output layer is equivalent to obtaining an end-to-end control model. The combined features are obtained by inputting the panoramic image features (512×512×3), dynamic semantic map features (1024-dimensional), and device status information features (16-dimensional) into the input layer. In this embodiment of the invention, eight expert network branches are set up for different operating scenarios, including scenarios such as empty driving, heavy-load movement, and truck alignment. Expert weights are assigned based on comprehensive features through a single gated network, that is, each weight is output through softmax. Through the expert network, corresponding control commands are output for their respective operation scenarios, such as "Given speed of the tire crane on both land and sea sides: 0-100%, braking pressure: 0-16MPa, and correction superposition speed: ±5%"; The final control command is obtained by fusing the results of each expert network based on expert weights in the output layer. The hybrid expert fusion result is calculated using Formula 5: Formula 5; in, This represents the weights of the gating network output. Indicates the first The output of the expert network satisfies .

[0032] Hybrid expert models are trained using a combination of supervised pre-training and reinforcement learning. In this embodiment of the invention, supervised pre-training uses 5,000 hours of manual driving data, including data from 30 typical working conditions, for supervised learning. Reinforcement learning optimization is continuously performed in the digital twin environment using the PPO algorithm, with a cumulative training step count ≥ 1e8. Equation 6 represents the objective function of the PPO algorithm. : Formula 6; in, Expressing expectations, Indicates the probability ratio. This represents the advantage estimation function. This indicates a limited probability ratio. This represents the clipping hyperparameter. .

[0033] Preferably, the contrastive learning method is used to eliminate the differences between different port scenarios, which improves the generalization ability by 40%.

[0034] In step S140, control commands are converted into execution signals to assist driving, and adjustments and optimizations are made in real time during the driving process, specifically including: During the start-up or braking phase, the acceleration and deceleration curves should be adjusted according to the operating status of the tire crane. For example, when under heavy load, the starting acceleration should be controlled at ≤0.1m / s², and the braking distance should be ≤2 meters.

[0035] By identifying ground markings, such as magnetic nails or road markings, to determine if the driver has deviated from the path, MPC (Model Predictive Control) is used for path tracking and correction. This is achieved by solving an optimization objective function under constraints. The optimization objective of MPC is: The constraints are: ,in, Represents the weight matrix. Indicates the prediction time domain, Represents the state vector. This represents the control vector.

[0036] Determine if the driver is beyond a safe distance from surrounding objects to avoid obstacles, and calculate the safe distance using Formula 7: Formula 7; in, Indicates a safe distance. Represents relative velocity. Indicates system response time. Indicates the maximum braking acceleration. Indicates a safety margin; It adopts a layered obstacle avoidance strategy: the early warning layer (distance > 15 meters) provides audible and visual alarms and deceleration prompts; the intervention layer (distance ≤ 15 meters) automatically decelerates to a safe speed (≤ 1 km / h); and the emergency layer (distance ≤ 5 meters) triggers emergency braking with a braking response time ≤ 0.3 seconds. The method further includes: In step S150, remote monitoring is performed by displaying the surround view image, environmental semantics, equipment status information, and control commands in real time via the touchscreen of the human-machine interface. Specifically, this includes: The device features a 15.6-inch touchscreen and transmits critical data wirelessly to the central control room for remote monitoring. It also supports remote diagnostics and control with a latency of ≤50ms.

[0037] Preferably, embodiments of the present invention additionally provide safety redundancy functions, including hardware redundancy, algorithm redundancy, and human-machine intervention. Hardware redundancy employs a dual-set design for key sensors such as encoders or cameras; algorithm redundancy involves synchronously running a rule-based backup control algorithm to verify the consistency of end-to-end model output results; and human-machine intervention is equipped with an emergency stop button and a trolley control master handle, switching to manual mode within 100ms upon detection of human intervention.

[0038] In this embodiment of the invention, the hardware layer depends on an image processing unit, a network transmission unit, a computing unit, and an execution unit; The image processing unit contains four industrial global shutter cameras with a resolution of 3840×2160, at least greater than 1920×1080, a frame rate of 25fps, and a dynamic range of 120dB. They are installed in the front, rear, left, and right directions of the tire crane's gantry leg to capture approximately 360-degree environmental images. The lenses are 8mm fixed-focus lenses with a horizontal field of view of 78° and are fixed by shock-absorbing brackets to eliminate the effects of mechanical vibration. The network transmission unit adopts a 2.5G industrial Ethernet switch, supports IEEE 802.1p priority scheduling, and realizes bidirectional transmission of camera data (total bandwidth of about 5Gbps) and control commands through a fiber optic ring network, with an end-to-end transmission delay of ≤10ms; The computing unit is configured with dual Intel Xeon Platinum 8480+ processors, 16×NVIDIA H100 GPUs, a total of 1024GB of video memory, and 4TB NVMe SSDs to form a neural network computing workstation. It supports FP8 precision computing and has a total computing power of ≥5PFlops. The actuator unit consists of a modified hydraulic proportional valve group, a control accuracy of 0.5%FS, a variable frequency drive system with a speed range of 0-5km / h, a speed control accuracy of ±0.02km / h, and a positioning accuracy of ±5mm absolute encoder.

[0039] The workflow can be summarized as follows: (1) System initialization, camera calibration, model loading, and equipment self-check, taking ≤30 seconds; (2) Four cameras simultaneously collect environmental images and transmit them to the computing workstation via a switch, with a transmission delay of ≤10ms; (3) Perception generates a surround view image, identifies environmental semantics through an image big language model, and constructs a dynamic map, with a processing time of ≤40ms; (4) End-to-end model integrates environmental information and equipment status, generates control commands for decision-making, with an inference time of ≤20ms; (5) The commands are converted into execution signals for control, driving the tire crane to complete the corresponding actions, with a response time of ≤50ms; (6) Real-time collection of equipment feedback data is used for model closed-loop optimization and safety monitoring.

[0040] The above technical solutions of the embodiments of the present invention will be illustrated with reference to the following accompanying drawings.

[0041] Figure 4 This is a schematic diagram of the overall framework of the assisted driving system according to an embodiment of the present invention, as shown below. Figure 4 As shown, the application structure corresponding to the workflow is illustrated, including the perception layer, decision reasoning layer, control layer, and execution layer.

[0042] Figure 5 This is a schematic diagram of the complete processing flow of an embodiment of the present invention, as shown below. Figure 5 As shown, the process from environmental input to control output is demonstrated. The image big language model perceives environmental semantics by fusing spatial perception features and device status information. In the subsequent process of outputting control commands by the hybrid expert model, it makes real-time diffusion predictions over time.

[0043] In summary, addressing the existing problems, this invention presents a method for assisted driving of rubber-tired gantry cranes. It employs an image-based large language model to perceive and understand the semantics of complex scenes, adds a port target detection head adapted to the crane's operating scenarios, and enhances scene understanding by acquiring equipment status information. This improves adaptability to adverse weather conditions by 60% and reduces the hardware cost of lidar equipment by 15%-20%. An end-to-end architecture is adopted for decision-making based on perceived information, eliminating accumulated errors across multiple stages, meeting the requirements of high-density container stacking, and obtaining accurate assisted driving control commands adapted to the crane's operating scenarios. Real-time adjustments and optimizations are made during driving, and automatic start / stop and correction functions shorten single-container operation time by 15%-20%, improve continuous operation stability, and reduce the operational error rate by 60%. Overall, a complete data-driven model solution is proposed, which can be quickly deployed under different port layouts, climate conditions, and operational processes, shortening the deployment cycle.

[0044] Method Example 2 According to an embodiment of the present invention, a specific configuration of a tire-mounted crane assisted driving method is provided.

[0045] In this embodiment of the invention, the hardware deployment details are as follows: The cameras are installed in the following positions: Front camera: installed at the center of the upper edge of the front door frame, with a lens tilt of 15°, covering an area of ​​5-50 meters in front; Rear camera: installed at the center of the lower edge of the rear door frame, with a lens tilt of 12°, covering an area of ​​5-40 meters behind; Side cameras: one on each side, installed in the middle of the side door frame, with the lens tilted outwards horizontally by 5°, covering an area of ​​3-30 meters on both sides. All cameras are equipped with IP67 protective housings and support operating temperatures from -30°C to +70°C.

[0046] The network architecture adopts a star topology, with cameras powered via PoE+ and supporting the IEEE 802.3af standard. The switches are configured with port mirroring to enable simultaneous data transmission to the computing unit and redundant monitoring unit. Control command transmission uses UDP protocol + CRC check to ensure real-time performance and reliability.

[0047] The computing unit is configured with an operating system of Ubuntu 22.04 LTS, equipped with NVIDIA CUDA 12.1 and TensorRT 8.6 acceleration library; model deployment uses TensorRT quantization (INT8 precision), which improves inference speed by more than 3 times; it is equipped with a UPS uninterruptible power supply, which supports safe shutdown 10 minutes after power failure.

[0048] In this embodiment of the invention, the model settings are as follows: Improvements to the image-based large language model include a visual encoder using an improved ViT-L / 14 architecture, pre-trained on a port dataset (1 million labeled images), a text decoder using the LLaMA-2-7B model, fine-tuned with LoRA to adapt to port terminology, and the introduction of a spatial attention mechanism to enhance feature extraction capabilities for close-range targets (≤10 meters).

[0049] During the training of the end-to-end control model, the dataset is constructed to include weather conditions such as sunny / rainy / foggy days, load states such as empty / full load, and driving modes such as straight / turning. The loss function is designed as: L1 loss (control error) + cosine loss (direction consistency) + risk penalty term (collision risk). The deployment optimization adopts model pruning technology to reduce the number of parameters by 30% while keeping the performance loss <5%.

[0050] In this embodiment of the invention, the software algorithm parameters and functional module parameters are set as follows: The camera calibration parameter built-in matrix is ​​set as follows: (pixel unit); The distortion coefficient is .

[0051] MPC controller parameters are set to prediction time domain. Control time domain Weight matrix , The constraint is the steering angle. Steering rate .

[0052] The safety distance parameter is set to the system response time. Maximum braking acceleration Safety margin When speed hour, .

[0053] Functional parameters include automatic start / stop, dynamic adjustment of acceleration (0.05-0.15m / s²) based on load weight (0-60 tons); path correction, fine-tuning initiated when deviating from the preset path by more than 3cm, with a maximum steering angle ≤5°; obstacle avoidance safety threshold, dynamically adjusted based on relative speed, with a relative speed of 0-5km / h corresponding to a safety distance of 2-8 meters.

[0054] In this embodiment of the invention, the system test indicators for verifying the assisted driving effect of the tire-mounted crane include: The perception performance includes: container recognition distance of 5-30 meters, accuracy ≥99.5%, recall rate ≥99%; and obstacle detection distance of 3-50 meters, with a minimum detectable target size of 30×30cm; Control performance includes: speed control with a set speed of 0-5 km / h and an actual error of ≤ ±0.02 km / h; positioning accuracy relative to ground markings with a positioning error of ≤ ±5 cm; and response time from obstacle detection to braking execution of ≤300 ms. Environmental adaptability includes: illumination conditions of 50-100,000 lux, supporting nighttime infrared mode; weather conditions of light rain (≤5mm / h), moderate snow, and fog (visibility ≥50 meters); and temperature range of -20℃ to +60℃, which is the operating environment of the computing unit.

[0055] System Implementation Examples According to an embodiment of the present invention, a tire-mounted crane assisted driving system is provided. Figure 6 This is a schematic diagram of a tire-mounted crane assisted driving system according to an embodiment of the present invention, as shown below. Figure 6 As shown, the tire-mounted crane assisted driving system according to an embodiment of the present invention specifically includes: The acquisition module 60 is used to acquire images of the environment where the tire crane is located, and to preprocess the environmental images to generate a surround view image. Specifically, it is used for: Initial environmental images are simultaneously acquired by industrial-grade global shutter cameras installed in the front, rear, left, and right directions of the wheeled crane's gantry legs. The environmental image is sequentially subjected to distortion correction, image enhancement and panoramic stitching based on Zhang's calibration method to generate a panoramic image. Zhang's calibration method uses a camera projection model to adjust the position and image size, and adjusts the local shape of the image through distortion correction formula.

[0056] Perception module 62 is used to identify environmental semantics and construct a dynamic semantic map based on the surround view image using an image-based large language model. The image-based large language model employs an attention mechanism to output spatial perception features, which are then fused with device status information for semantic understanding. Specifically, it is used for: The LLaVA-NeXT architecture, consisting of a visual encoder, a spatial relation inference module, and a text decoder, is used as a large language model for images. The surround view image is input into the visual encoder of the newly added port target detection head, which outputs the identified target. The relative position between the identified targets is calculated using an attention mechanism through a spatial reasoning model, which serves as a spatial perception feature. Based on the spatial perception feature and with device status information as a prompt, the environmental semantic result is output through a text decoder. Constructing dynamic semantic maps based on environmental semantics specifically includes: Set up the container stacking area layout, driving lanes, and work area boundaries as a static layer, and mark the risk level and prohibited work areas in the static layer; The moving targets consisting of trucks and personnel in the environmental semantics are used as the initial result of the dynamic layer. The positions and velocities of the moving targets are labeled, and Kalman filtering is used to predict the motion trend to generate a dynamic semantic map.

[0057] Decision module 64 is used to input features including surround view image, dynamic semantic map and equipment status information into the hybrid expert model, and output control commands including continuous control quantities, specifically for: Obtain a hybrid expert model consisting of an input layer, a gating network, an expert network, and an output layer; The features of the panoramic image, dynamic semantic map and equipment status information are input into the input layer to obtain comprehensive features. Expert weights are assigned based on the comprehensive features through a gating network. The expert network outputs corresponding control commands for their respective operation scenarios. The output layer fuses the results of each expert network based on the expert weights to obtain the final control command. The hybrid expert model is trained using a combination of supervised pre-training and reinforcement learning.

[0058] Control module 66 is used to convert control commands into execution signals to assist driving, and to make adjustments and optimizations in real time during the driving process. Specifically, it is used for: Adjust the acceleration and deceleration curves according to the operating status of the tire crane during the start-up or braking phase; By identifying ground markings to determine whether the driver has deviated, the MPC model predictive control method is used for path tracking and correction, which is achieved by solving the optimization objective function under constraints. Determine whether the driver is beyond a safe distance from the surroundings and take obstacle avoidance measures.

[0059] The system further includes: The visualization module 68 is used to display surround view images, environmental semantics, equipment status information and control commands in real time through the touch screen of the human-machine interface for remote monitoring.

[0060] In summary, addressing the existing problems, this invention, a tire-mounted crane assisted driving system, employs an image-based large language model to perceive and understand the semantics of complex scenes. It adds a port target detection head adapted to the operating scenarios of the tire-mounted crane and enhances scene understanding by acquiring equipment status information. This improves adaptability to adverse weather conditions by 60% and reduces the hardware cost of lidar equipment by 15%-20%. An end-to-end architecture is adopted for decision-making based on perceived information, eliminating accumulated errors from multiple stages, meeting the requirements of high-density container stacking, and obtaining accurate assisted driving control commands adapted to the tire-mounted crane scenario. Real-time adjustments and optimizations are made during the driving process, and automatic start-stop and correction functions shorten single-container operation time by 15%-20%, improve continuous operation stability, and reduce the operational error rate by 60%. Overall, a complete data-driven model solution is proposed, which can be quickly deployed under different port layouts, climatic conditions, and operational processes, shortening the deployment cycle.

[0061] Electronic device examples Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 700 may include at least one processor 710 and a memory 720. The processor 710 can execute instructions stored in the memory 720. The processor 710 is communicatively connected to the memory 720 via a data bus. In addition to the memory 720, the processor 710 can also be communicatively connected to an input device 730, an output device 740, and a communication device 750 via the data bus.

[0062] The processor 710 can be any conventional processor, such as a commercially available CPU. The processor may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0063] The memory 720 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0064] In this embodiment of the present disclosure, the memory 720 stores executable instructions, and the processor 710 can read the executable instructions from the memory 720 and execute the instructions to implement all or part of the steps of any of the above exemplary embodiments of the tire-mounted crane assisted driving method.

[0065] Computer-readable storage medium embodiments In addition to the methods and apparatus described above, exemplary embodiments of this disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product, the computer product including computer program instructions that can be executed by a processor to implement all or part of the steps described in any of the tire-mounted crane assisted driving methods in the exemplary embodiments described above.

[0066] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. Programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0067] Computer-readable storage media may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires; electrically erasable programmable read-only memory (EEPROM); erasable programmable read-only memory (EPROM); programmable read-only memory (PROM); read-only memory (ROM); magnetic storage; flash memory; magnetic disk or optical disk; or any suitable combination thereof.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for assisting driving of a tyre crane, characterized in that include: Acquire images of the environment in which the tire crane is located, and preprocess the environmental images to generate a surround view image; Based on the panoramic image, environmental semantics are identified and a dynamic semantic map is constructed through an image big language model. The image big language model uses an attention mechanism to output spatial perception features, and the spatial perception features are fused with device status information for semantic understanding. The features including the panoramic image, the dynamic semantic map, and the device status information are input into the hybrid expert model, and the output is a control command including continuous control quantities. The control commands are converted into execution signals to assist driving, and adjustments and optimizations are made in real time during the driving process.

2. The method of claim 1, wherein, The method further includes: Remote monitoring is achieved by displaying the surround view image, environmental semantics, device status information, and control commands in real time via a touchscreen human-machine interface.

3. The method according to claim 1, characterized in that, The process of acquiring environmental images of the tire crane and preprocessing these images to generate a surround view image specifically includes: Initial environmental images are simultaneously acquired by industrial-grade global shutter cameras installed in the front, rear, left, and right directions of the tire crane's gantry legs. The environmental image is sequentially subjected to distortion correction, image enhancement, and panoramic stitching based on Zhang's calibration method to generate the panoramic image. The Zhang's calibration method uses a camera projection model to adjust the position and image size, and adjusts the local shape of the image through distortion correction formula.

4. The method of claim 1, wherein, The step of recognizing environmental semantics and constructing a dynamic semantic map based on the panoramic image using an image-based large language model specifically includes: The LLaVA-NeXT architecture, consisting of a visual encoder, a spatial relation reasoning module, and a text decoder, is used as the image-based large language model. The panoramic image is input into the visual encoder of the newly added port target detection head, which outputs the identified target. The relative position between the identified targets is calculated by the attention mechanism through the spatial reasoning model and used as a spatial perception feature. Based on the spatial perception feature and with the device status information as a prompt, the environmental semantic result is output through the text decoder. A dynamic semantic map is constructed based on the aforementioned environmental semantics.

5. The method of claim 4, wherein, The construction of a dynamic semantic map based on the environmental semantics specifically includes: Set up a static layer with the layout of the container stacking area, the driving lanes and the boundaries of the work area, and mark the risk level and prohibited work areas in the static layer; The moving target consisting of trucks and personnel in the environmental semantics is used as the initial result of the dynamic layer. The position and velocity of the moving target are labeled, and the motion trend is predicted by Kalman filtering to generate the dynamic semantic map.

6. The method of claim 1, wherein, The feature input hybrid expert model, which includes the surround view image, the dynamic semantic map, and the device status information, and outputs control commands including continuous control quantities, specifically includes: Obtain a hybrid expert model consisting of an input layer, a gating network, an expert network, and an output layer; The features of the panoramic image, the dynamic semantic map, and the equipment status information are input into the input layer to obtain comprehensive features. Expert weights are assigned based on the comprehensive features through a gating network. The expert network outputs corresponding control commands for their respective operating scenarios. The output layer fuses the results of each expert network based on the expert weights to obtain the final control command. The hybrid expert model is trained using a combination of supervised pre-training and reinforcement learning.

7. The method of claim 1, wherein, The real-time adjustment and optimization during the driving process specifically includes: Adjust the acceleration and deceleration curves according to the operating status of the tire crane during the start-up or braking phase; By identifying ground markings to determine whether the driver has deviated, the MPC model predictive control method is used for path tracking and correction, which is achieved by solving the optimization objective function under constraints. Determine whether the driver is beyond a safe distance from the surroundings and take obstacle avoidance measures.

8. A tyre crane assisted driving system, characterized in that, include: The acquisition module is used to acquire images of the environment where the tire crane is located, and to preprocess the environmental images to generate a surround view image; The perception module is used to identify environmental semantics and construct a dynamic semantic map based on the surrounding image through an image big language model. The image big language model uses an attention mechanism to output spatial perception features, and integrates the spatial perception features with device status information for semantic understanding. The decision module is used to input features including the surround view image, the dynamic semantic map and the device status information into the hybrid expert model, and output control commands including continuous control quantities. The control module is used to convert the control commands into execution signals to assist driving and make adjustments and optimizations in real time during the driving process.

9. An electronic device, comprising: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the tire-mounted crane assisted driving method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the tire-mounted crane assisted driving method as described in any one of claims 1 to 7.

Images