Loader operation guidance system for non-line-of-sight remote control operation

Abstract

This application relates to a loader operation guidance system for non-line-of-sight remote control driving, belonging to the field of loader operation guidance technology. It includes an on-board terminal, an on-board environmental sensing module, a wireless communication module, a human-machine interface module, and a guidance control module. The solution uses the on-board terminal and the on-board environmental sensing module to collect vehicle data and on-board sensor data in real time. This data is then uploaded to the human-machine interface module via the wireless communication module to construct a guidance model. The system sets the loader's operation tasks and objectives, and sends these tasks and objectives to the loader's guidance control module to guide the loader in its operation, reducing the driver's workload. The human-machine interface module also collects and performs big data analysis on the loader's guidance operation data. Based on this data, a digital model is constructed, and the big data analysis results are used to simulate real-world loader operation scenarios. This reduces the impact of environmental factors within the driver's field of vision on their performance, alleviates driver fatigue, and improves the driving experience.

Description

Technical Field

[0001] This application belongs to the field of loader operation guidance technology, specifically relating to a loader operation guidance system for non-line-of-sight remote control driving. Background Technology

[0002] Metallurgy refers to the mining, beneficiation, and sintering of metal ores, followed by smelting and processing them into metallic materials. Due to the industry's unique nature, the work environments often involve manually operated specialized engineering machinery, resulting in highly dangerous tasks. Examples include slag removal under furnaces, coal stockpiling, coking asphalt treatment, and ship hull cleaning. These work environments present numerous challenges, including high dust levels, heat radiation, toxic and harmful gases, strong acids and corrosion, limited visibility, and the risk of personal injury.

[0003] Existing technologies typically address the issue of unmanned applications in such work areas by remotely controlling the vehicles over a network. However, existing remote-controlled driving systems require the driver to actively perform remote-controlled driving operations, which is labor-intensive and susceptible to driver fatigue due to limited field of vision. This results in a poor driving experience and consequently, low work efficiency. Summary of the Invention

[0004] Therefore, this application provides a loader operation guidance system for non-line-of-sight remote control driving, which helps to solve the problems of high operational intensity, low operation efficiency, and driver fatigue and poor driving experience when remotely controlling loading operations.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] This application provides a loader operation guidance system for non-line-of-sight remote control driving, including:

[0007] The vehicle-mounted terminal is used to collect vehicle data from the loader.

[0008] The vehicle-mounted ambient sensing module is used to collect data from the loader's onboard sensors.

[0009] The wireless communication module is used to establish a network for wireless communication with the human-machine interaction module in the cockpit. The wireless communication module sends vehicle sensor data and vehicle data to the human-machine interaction module and receives the operation tasks and operation targets from the human-machine interaction module.

[0010] The human-machine interaction module is used to build a guidance model based on vehicle sensor data and vehicle data, set work tasks and work objectives, collect loader guidance operation data and perform big data analysis on the guidance operation data, build a digital model based on the guidance operation data, simulate real loader operation scenarios in combination with big data analysis results, and evaluate the operation process based on the digital model.

[0011] The guidance and control module is used to receive the work tasks and work objectives from the human-machine interaction module and guide the loader to perform the work.

[0012] Furthermore, the human-computer interaction module specifically includes a touch screen, a computing unit, a sensor calibration unit, and a task interaction unit;

[0013] The touchscreen, connected to the computing unit, is used to display the loader's onboard sensor data, vehicle data, and loader operation scene, and to receive operating commands input by the driver.

[0014] The computing unit is connected to the touch screen, sensor calibration unit and task interaction unit respectively. It is used to establish a guidance model based on vehicle sensor data and vehicle data, using a multi-data composite superposition and conversion method, and to define the guidance model function.

[0015] The sensor calibration unit is used to pre-calibrate the vehicle-mounted ambient sensing module. The pre-calibration information includes: basic vehicle attribute information, attitude reference point information, obstacle classification information, and guide line scale information.

[0016] The task interaction module is used to pre-determine the loader's work tasks and objectives based on the guidance model function. The work tasks and objectives include the work task type, work task angle range, work task duration, work task level, and evaluation rules.

[0017] The computing unit is also used to collect and store loader guidance operation data, perform big data analysis on the guidance operation data, build digital models based on the guidance operation data, simulate real loader operation scenarios in combination with the big data analysis results, and evaluate the guidance operation process based on the digital model.

[0018] Furthermore, the vehicle-mounted ambient sensing module includes an inclinometer, an angle encoder, a lidar, an inertial measurement unit, an AI camera, a weighing valve block, and a positioning module;

[0019] Two tilt meters are installed, one at the loading boom and the other at the bucket, to collect attitude information of the boom and the bucket.

[0020] An angle encoder is installed at the hinge of the loader body to collect the relative attitude angles of the front and rear bodies of the loader;

[0021] There are three lidar units installed on the left, right and rear sides of the loader cab, respectively, to collect laser point cloud data on the left, right and rear sides of the loader.

[0022] The inertial measurement unit is installed at the headlight position on the front of the loader to collect the vehicle attitude information of the loader;

[0023] There are four AI cameras installed on the left, front, right and rear sides above the loader cab to collect 360-degree surround view image data around the loader.

[0024] The weighing valve block is installed on the hydraulic oil circuit of the loader tooling to collect the weight of the materials loaded and unloaded by the loader tooling;

[0025] The positioning module is installed on the top of the loader's cab to collect the loader's positioning data in open-air environments.

[0026] Furthermore, the vehicle-mounted sensor data includes boom tilt angle, bucket tilt angle, articulation angle, laser point cloud data, inertial measurement attitude data, surround view image data, material weight, positioning data, elevation and heading information; the vehicle data includes engine speed, ignition status, turn signal status, high and low beam status, parking status, gear status, windshield wiper status, horn status, cooling fan status, and transmission power status.

[0027] Furthermore, the wireless communication module is installed on the top of the loader cab and uses the TCP / IP protocol to establish a communication mechanism for wireless communication; the wireless communication module is a broadband self-organizing network device or a 4G / 5G wireless router.

[0028] Furthermore, the collection of loader guidance operation data and the big data analysis of the guidance operation data specifically include: during the loader guidance operation, the human-machine interaction module uses the vehicle-mounted environmental sensing module to collect real-time vehicle-mounted sensor data, and at the same time collects driver operation records through the touch screen, and establishes a time-series database to store vehicle-mounted sensor data and driver operation records.

[0029] Data mining algorithms were used to mine the vehicle sensor data and driver operation records stored in the time series database to establish a correspondence sequence between driver operation records and the operating status of the loader vehicle.

[0030] Furthermore, the construction of the digital model based on the guidance operation data specifically includes:

[0031] By utilizing data from onboard sensors, such as boom tilt angle, bucket tilt angle, articulation angle, laser point cloud data, inertial measurement attitude data, surround view image data, material weight and positioning data, and combining this with a high-definition 3D map of the loader's work surface, a 3D digital twin visualization model is constructed on the 3D map.

[0032] Furthermore, the guided model functions include one-click leveling, one-click lifting, material pile identification, route planning, blind spot detection, safety warning, autonomous loading and unloading, and operation statistics.

[0033] The application employs the above technical solution and has at least the following beneficial effects:

[0034] The loader operation guidance system for non-line-of-sight remote control driving includes: an on-board terminal for collecting vehicle data of the loader; an on-board environmental sensing module for collecting on-board sensor data of the loader; a wireless communication module for networking and wireless communication with the human-machine interface module in the cab, wherein the wireless communication module sends on-board sensor data and vehicle data to the human-machine interface module and receives operation tasks and operation targets from the human-machine interface module; the human-machine interface module is used to establish a guidance model based on on-board sensor data and vehicle data, set operation tasks and operation targets, collect loader guidance operation data and perform big data analysis on the guidance operation data, construct a digital model based on the guidance operation data, simulate real loader operation scenarios based on the big data analysis results, and evaluate the operation process based on the digital model; and a guidance control module for receiving operation tasks and operation targets from the human-machine interface module and guiding the loader to perform operations. In this system architecture, the proposed solution collects vehicle data and on-board sensor data in real time through an on-board terminal and an on-board environmental sensing module. This data is then uploaded to a human-machine interface (HMI) module via a wireless communication module to construct a guidance model. The driver uses this HMI module to set the loader's work tasks and objectives, which are then sent to the loader's guidance control module to control the loader during guided operations, thus reducing the driver's workload. Simultaneously, the HMI module collects loader guidance operation data in real time through the on-board terminal and on-board environmental sensing module and performs big data analysis on this data. Based on this data, a digital model is constructed, and the big data analysis results are combined to simulate real-world loader operation scenarios. This reduces the impact of environmental factors within the driver's field of vision during guided operations, alleviates driver fatigue, and improves the driving experience.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is an exemplary embodiment illustrating the architecture of a loader operation guidance system for non-line-of-sight remote control driving;

[0038] Figure 2 This is a flowchart illustrating the operation guidance system for a loader operation guidance system for non-line-of-sight remote control driving according to an exemplary embodiment;

[0039] Appendix Figure 1 In the middle: 1-Vehicle terminal, 2-Vehicle ambient sensing module, 3-Wireless communication module, 4-Human-machine interaction module, 5-Guidance control module. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be described in detail below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] Please see Figure 1 , Figure 1 This is an exemplary embodiment illustrating the architecture of a loader operation guidance system for non-line-of-sight remote control driving, such as... Figure 1 As shown, the system includes: an on-board terminal 1, an on-board ambient sensing module 2, a wireless communication module 3, a human-machine interaction module 4, and a guidance and control module 5. Among them,

[0042] The vehicle-mounted terminal 1 is used to collect vehicle data of the loader; the vehicle-mounted environmental sensing module 2 is used to collect vehicle sensor data of the loader; the wireless communication module 3 is used to establish a network and communicate wirelessly with the human-machine interaction module 4 in the cab. The wireless communication module 3 sends the vehicle sensor data and vehicle data to the human-machine interaction module 4 and receives the work tasks and work objectives from the human-machine interaction module 4; the human-machine interaction module 4 is used to establish a guidance model based on the vehicle sensor data and vehicle data, set work tasks and work objectives, collect loader guidance work data and perform big data analysis on the guidance work data, construct a digital model based on the guidance work data, simulate the real loader operation scenario based on the big data analysis results, and evaluate the guidance work process based on the digital model; the guidance control module 5 is used to receive the work tasks and work objectives from the human-machine interaction module 4 and guide the loader to perform the work.

[0043] Furthermore, in one embodiment, the vehicle-mounted terminal 1 uses the front-end equipment of an existing vehicle monitoring and management system, which integrates multiple functions such as positioning, communication, vehicle driving recorder, security alarm, wire cut alarm, and remote safety protection against fuel and power failure. This application utilizes the vehicle-mounted terminal 1 to collect various vehicle data from the loader, including engine speed, ignition status, turn signal status, high and low beam headlight status, parking status, gear status, wiper status, horn status, cooling fan status, and transmission power status. The guidance control module 5 can use an existing PLC controller to implement the guidance control of the loader.

[0044] Furthermore, in one embodiment, the vehicle-mounted ambient sensing module 2 includes an inclinometer, an angle encoder, a lidar, an inertial measurement unit (IMU), an AI camera, a weighing valve block, and a positioning module (GPS / RTK).

[0045] in,

[0046] Two tilt meters are installed, one at the loading boom and the other at the bucket, to collect attitude information of the boom and the bucket.

[0047] An angle encoder is provided and installed at the hinge of the loader body to collect the relative attitude angles of the front and rear bodies of the loader;

[0048] There are three lidar units installed on the left, right and rear sides of the loader cab, respectively, to collect laser point cloud data, i.e. point cloud models, from the left, right and rear sides of the loader.

[0049] One inertial measurement unit is installed at the headlight position on the front of the loader to collect the vehicle attitude information of the loader;

[0050] There are four AI cameras installed above the loader's cab on the left, front, right, and rear sides, respectively. They are used to collect 360-degree surround view image data around the loader and, through a deep learning artificial target detection algorithm, to achieve pedestrian and object classification and recognition functions. This expands the driver's field of vision and solves the problem of the driver's operation being affected by the field of vision.

[0051] The weighing valve block is installed on the hydraulic oil circuit of the loader tooling to collect the weight of the materials loaded and unloaded by the loader tooling;

[0052] The positioning module is installed on the top of the loader's cab to collect the loader's positioning data in open-air environments.

[0053] The AI ​​camera employs existing object recognition algorithms, such as the YOLOv5 algorithm based on a deep learning framework. YOLOv5 uses the PyTorch framework, offering good training speed and demonstrating good performance in both speed and accuracy during pedestrian and vehicle object detection. Alternatively, Fast R-CNN, R-CNN, and SSD algorithms can also be used for pedestrian and object classification and recognition, which will not be elaborated upon here.

[0054] Specifically, the vehicle-mounted sensor data collected by the aforementioned vehicle-mounted surround sensing module 2 includes boom tilt angle, bucket tilt angle, articulated angle, laser point cloud data, inertial measurement attitude data, surround view image data, material weight, positioning data, elevation and heading information.

[0055] Furthermore, in one embodiment, the human-machine interaction module 4 is located in the remote control cockpit. The module adopts an industrial GUI design style, displaying various data of the loader to the driver. The interactive information of the module includes video display, vehicle posture information, 3D digital visualization, vehicle status information, alarm information, and material statistics. The human-machine interaction module 4 specifically includes a touchscreen, a computing unit, a sensor calibration unit, and a task interaction unit. The touchscreen is connected to the computing unit and is used to display the loader's onboard sensor data, vehicle data, and loader operating scene, and to receive operating commands input by the driver.

[0056] The computing unit is connected to the touch screen, sensor calibration unit and task interaction unit respectively. It is used to establish a guidance model based on vehicle sensor data and vehicle data, using a multi-data composite superposition and conversion method, and to define the guidance model function.

[0057] The sensor calibration unit is used to pre-calibrate the vehicle-mounted ambient sensing module 2. The pre-calibration information includes: vehicle basic attribute information, attitude reference point information, obstacle classification information, and guide line scale information.

[0058] The task interaction module is used to pre-determine the loader's work tasks and objectives based on the guidance model function. The work tasks and objectives include the work task type, work task angle range, work task duration, work task level, and evaluation rules.

[0059] The computing unit is also used to collect and store loader guidance operation data, perform big data analysis on the guidance operation data, construct digital models based on the guidance operation data, simulate real-world loader operation scenarios based on the big data analysis results, and evaluate the guidance operation process based on the digital model. The computing unit can use an industrial computer to perform data processing.

[0060] Specifically, the guided model functions in this application include one-click leveling, one-click lifting, material pile identification, route planning, blind spot detection, safety warning, autonomous loading and unloading, and operation statistics.

[0061] The one-click leveling function is calculated using coupled data from the boom tilt angle and bucket tilt angle. The boom tilt angle is preset to a range of 0% to 100%, and the bucket tilt angle is preset to a range of -50% to 50%. When both the boom tilt angle and bucket tilt angle are within the 0-35% range, the bucket tilt angle is adjusted to 0% first, and then the boom tilt angle is adjusted to 0%.

[0062] One-button lifting is calculated using coupled data from the boom tilt angle and bucket tilt angle. The boom tilt angle is preset to a range of 0% to 100%, and the bucket tilt angle is preset to a range of -50% to 50%. When the boom tilt angle is gradually raised from 0% to 50%, the bucket tilt angle remains unchanged at 0%. When the boom tilt angle is gradually raised from 50% to 100%, the bucket tilt angle is gradually adjusted to reach 50%.

[0063] The data coupling calculation principle of one-click lifting and one-click leveling: First, preset the boom tilt angle and bucket tilt angle range, and within this range, mark and save a series of position points, and then alternately control the loader's bucket and boom to the corresponding position points.

[0064] Material pile identification employs deep learning neural network methods to identify the shape, slope, and height of the material pile. The identification process involves image preprocessing, semantic segmentation, image classification and annotation, and recognition. The material pile shape, slope, and height data are obtained through dual verification using image calibration and laser point cloud data. For the deep learning neural network method, existing deep learning techniques are used to achieve the identification, classification, and annotation of the material pile images. Deep learning methods include two types of algorithms: two-stage algorithms (such as the R-CNN series) and one-stage algorithms (such as YOLO, SSD, etc.). The main difference is that two-stage algorithms require first generating a proposal (a pre-selected bounding box that may contain the object to be detected) and then performing fine-grained object detection. One-stage algorithms, on the other hand, directly extract features from the network to predict object classification and location. The core of the region extraction algorithm in the two-stage algorithm is a convolutional neural network (CNN). It first uses the CNN backbone to extract features, then finds candidate regions, and finally uses a sliding window to determine the target category and location. The first-order algorithm performs feature extraction, target classification, and location regression throughout the convolutional network. It obtains the target location and category through a single backpropagation, achieving a significant speed improvement while maintaining slightly lower recognition accuracy than the two-stage target detection algorithm. This application primarily employs the YOLOv4 algorithm from the first-order algorithm to identify the shape, slope, and height of the material pile.

[0065] The route planning utilizes the results of perception computing to design autonomous motion planning routes, which are displayed on the touchscreen's human-computer interaction main page using green guide lines. This application's solution employs the existing DQN algorithm for autonomous motion planning, combining DQN with onboard surround-sense data for reinforcement learning and neural network training. Through training and optimization of the neural network model, the network's training parameters are obtained, resulting in a relatively accurate path output.

[0066] Blind zone detection employs a clustering and identification process for LiDAR point cloud data, ensuring safety in blind zones except for forward-facing areas. The point cloud clustering process in this application utilizes existing point cloud clustering algorithms, such as k-means, DBSCAN, and Euclidean algorithms. This application uses the Euclidean algorithm for point cloud clustering. In LiDAR point cloud data, the distance between two points within a cluster of points belonging to the same object is less than a certain value, while the distance between point cloud clusters belonging to different objects is greater than a certain value. The Euclidean clustering algorithm, based on this principle, merges points with an Euclidean distance less than a set threshold into one cluster, thus completing the clustering process.

[0067] The safety early warning system employs a combined detection and recognition process using images from AI cameras and LiDAR data, outputting detection results in real time. It can detect pedestrians and obstacles, provide autonomous safety warnings, and proactively intervene when necessary to ensure the safety of construction operations. This is primarily achieved by setting corresponding safety early warning rules, issuing warnings when these rules are triggered in the image and radar data.

[0068] The autonomous loading and unloading system employs a comprehensive approach integrating one-click leveling, one-click lifting, material pile identification, route planning, blind spot detection, and safety warnings. Based on material pile identification, the system guides the autonomous vehicle to the loading and unloading point via a planned route, enabling one-click leveling, autonomous loading, and one-click lifting operations. Blind spot detection ensures the safety boundaries of moving vehicles, while safety warnings ensure timely adjustments and avoidance maneuvers when dangerous situations are detected.

[0069] The work statistics use material weighing and counting feedback, work habits, work duration and work shifts, combined with big data in the background, to provide an evaluation of driving habits and work efficiency.

[0070] Furthermore, in one embodiment, the collection of loader guidance operation data and the big data analysis of the guidance operation data specifically include: during the loader's guidance operation, the human-machine interaction module 4 uses the vehicle-mounted ambient sensing module 2 to collect real-time vehicle-mounted sensor data, and simultaneously collects driver operation records through the touch screen, and establishes a time-series database to store the vehicle-mounted sensor data and driver operation records; then, data mining algorithms are used to perform data mining on the vehicle-mounted sensor data and driver operation records stored in the time-series database to establish a correspondence sequence between driver operation records and the loader's vehicle operating status. After a long period of accumulation, the best driver's operating habits and the vehicle's optimal operating status can be effectively analyzed, providing a reference for subsequently combining vehicle body and sensor data to autonomously control the engine's working rhythm, reduce fuel consumption, and improve machine operating efficiency.

[0071] Furthermore, in one embodiment, virtual modeling refers to the process of establishing a digital model. Through model establishment and big data analysis results, the working site status can be clearly displayed, the vehicle's operating status can be promptly warned, and maintenance information and the surrounding working environment during operations can be predicted in advance. The digital model uses boom tilt angle, bucket tilt angle, articulation angle, laser point cloud data, IMU (Inertial Measurement Unit) attitude data, camera video detection, material quality, GPS / RTK information, and a high-definition 3D map of the working surface to construct a 3D digital twin visualization platform, used to recreate the construction site and simulate real working scenarios. The construction of the digital model based on the guided operation data specifically includes: using boom tilt angle, bucket tilt angle, articulation angle, laser point cloud data, inertial measurement attitude data, surround view image data, material weight, and positioning data from vehicle-mounted sensor data, combined with a high-definition 3D map of the loader's working surface, to perform 3D coordinate transformation and construct a 3D digital twin visualization digital model on the 3D map.

[0072] The digital model in this application utilizes laser point cloud data registration, including temporal and spatial registration. Temporal registration primarily employs a unified GPS time system, achieved through a combination of hardware and software. Spatial registration utilizes the three-dimensional coordinates (X, Y, Z), laser reflection intensity (Intensity), and color information (RGB) of points on the target object's surface, leveraging position and attitude data provided by GPS / RTK and IMU (Inertial Measurement Unit) to convert the three-dimensional coordinates into a model. This application uses sensors to collect data such as boom tilt angle, bucket tilt angle, articulation angle, and material mass, importing the data collected from the actual vehicle into the digital model for analysis. By integrating GIS technology with digital twins, a high-precision 3D map covering the entire area is created, enabling three-dimensional visualization management of construction scenes, facilities, and location-based building structures. Simultaneously, high-precision positioning capabilities allow vehicles equipped with high-precision positioning devices to display their travel routes and location information on the 3D map. Based on the real-time vehicle location and material mass, operators can effectively and efficiently match loading and unloading point transportation needs, recreating the construction site, simulating real-world work scenarios, and efficiently and safely completing tasks.

[0073] Furthermore, in one embodiment, the process of setting work tasks and work objectives specifically includes: setting the work task type, work task angle range, work task duration, work task level, and evaluation rules. The task type can be categorized as site leveling, material stockpiling, material turnover, and material loading and unloading; the work task duration can be selected as 10 minutes, 15 minutes, 30 minutes, 45 minutes, and 60 minutes; the work task level can be selected as Level 1, Level 2, and Level 3 (Level 1 is the highest, with the most stringent constraints); the evaluation rules can be manual evaluation or automatic evaluation. Setting work tasks can involve multiple superimposed tasks forming a work task set.

[0074] Furthermore, in one embodiment, job evaluation refers to evaluating the compliance of the loader's operation process. By establishing a job evaluation system or setting corresponding job evaluation rules, it is convenient to select high-quality drivers, identify the optimal working condition of the loader, monitor the loader's operating status in real time, and provide feedback on job quality.

[0075] Furthermore, in one embodiment, the wireless communication module 3 is installed on the top of the loader cab and uses the TCP / IP protocol to establish a communication mechanism for wireless communication; the wireless communication module 3 is a broadband self-organizing network device or a 4G / 5G wireless router.

[0076] The human-machine interaction module 4, through the wireless communication module 3, issues work tasks and objectives. These tasks or task sets, edited in the cockpit, are byte sequences transmitted via a fixed protocol. The communication method for issuing tasks uses TCP communication rules and includes encryption.

[0077] Reference Figure 2 As shown, the workflow of the loader operation guidance system of this application includes: installing corresponding hardware units (sensors and vehicle-mounted terminal 1, etc.) on the engineering vehicle; acquiring hardware unit data and establishing communication; establishing a guidance model based on hardware unit data; setting operation tasks and objectives; issuing tasks and conducting guided operations; collecting operation datasets and performing big data analysis; performing virtual modeling based on operation data; and evaluating operations based on the digital model.

[0078] This application solution focuses on the driver's cab as the operating object. By installing several sensors on the loader, it achieves real-time monitoring of the loader's vehicle attitude information, collection of information about the vehicle's surrounding environment, and vehicle functional safety early warning and control output. By introducing a loader operation guidance system, it aims to optimize operational efficiency, maintain vehicle safety, and enhance the variety of operational tasks.

[0079] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0080] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "multiple" or "more" means at least two.

[0081] It should be understood that when an element is referred to as “fixed to” or “set on” another element, it may be directly on the other element or may be interposed with an intervening element; when an element is referred to as “connected to” another element, it may be directly connected to the other element or may be interposed with an intervening element. Furthermore, the term “connected” as used herein may include wireless connections; the word “and / or” as used includes any and all combinations of one or more of the associated listed items.

[0082] Any process or method description in the flowchart or otherwise herein can be understood as: representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0083] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0084] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0086] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0087] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0088] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A loader operation guidance system for non-line-of-sight remote control driving, characterized in that, include: The vehicle-mounted terminal is used to collect vehicle data from the loader. The vehicle-mounted ambient sensing module is used to collect data from the loader's onboard sensors. The wireless communication module is used to establish a network for wireless communication with the human-machine interaction module in the cockpit. The wireless communication module sends vehicle sensor data and vehicle data to the human-machine interaction module and receives the operation tasks and operation targets from the human-machine interaction module. The human-machine interaction module is used to build a guidance model based on vehicle sensor data and vehicle data, set work tasks and work objectives, collect loader guidance operation data and perform big data analysis on the guidance operation data, build a digital model based on the guidance operation data, simulate real loader operation scenarios in combination with big data analysis results, and evaluate the operation process based on the digital model. The guidance and control module is used to receive the work tasks and work objectives from the human-machine interaction module and guide the loader to perform the work. The human-computer interaction module specifically includes a touch screen, a computing unit, a sensor calibration unit, and a task interaction unit; The touchscreen, connected to the computing unit, is used to display the loader's onboard sensor data, vehicle data, and loader operation scene, and to receive operating commands input by the driver. The computing unit is connected to the touch screen, sensor calibration unit and task interaction unit respectively. It is used to establish a guidance model based on vehicle sensor data and vehicle data, using a multi-data composite superposition and conversion method, and to define the guidance model function. The sensor calibration unit is used to pre-calibrate the vehicle-mounted ambient sensing module. The pre-calibration information includes: basic vehicle attribute information, attitude reference point information, obstacle classification information, and guide line scale information. The task interaction unit is used to pre-determine the loader's work tasks and objectives based on the guidance model function. The work tasks and objectives include the work task type, work task angle range, work task duration, work task level, and evaluation rules. The computing unit is also used to collect and store loader guidance operation data, perform big data analysis on the guidance operation data, build a digital model based on the guidance operation data, simulate real loader operation scenarios in combination with the big data analysis results, and evaluate the guidance operation process based on the digital model. The collection of loader guidance operation data and the big data analysis of the guidance operation data specifically include: during the loader guidance operation, the human-machine interaction module uses the vehicle-mounted environmental sensing module to collect real-time vehicle-mounted sensor data, and at the same time collects the driver's operation records through the touch screen, and establishes a time-series database to store the vehicle-mounted sensor data and driver's operation records. Data mining algorithms were used to mine the vehicle sensor data and driver operation records stored in the time series database to establish a sequence of correspondences between driver operation records and the operating status of the loader vehicle. The construction of the digital model based on the guidance operation data specifically includes: By utilizing data from onboard sensors, such as boom tilt angle, bucket tilt angle, articulation angle, laser point cloud data, inertial measurement attitude data, surround view image data, material weight and positioning data, and combining this with a high-definition 3D map of the loader's work surface, a 3D digital twin visualization model is constructed on the 3D map.

2. The loader operation guidance system for non-line-of-sight remote control driving according to claim 1, characterized in that, The vehicle-mounted ambient sensing module includes an inclinometer, an angle encoder, a lidar, an inertial measurement unit, an AI camera, a weighing valve block, and a positioning module; Two tilt meters are installed, one at the loading boom and the other at the bucket, to collect attitude information of the boom and the bucket. An angle encoder is installed at the hinge of the loader body to collect the relative attitude angles of the front and rear bodies of the loader; There are three lidar units installed on the left, right and rear sides of the loader cab, respectively, to collect laser point cloud data on the left, right and rear sides of the loader. The inertial measurement unit is installed at the headlight position on the front of the loader to collect the vehicle attitude information of the loader; There are four AI cameras installed on the left, front, right and rear sides above the loader cab to collect 360-degree surround view image data around the loader. The weighing valve block is installed on the hydraulic oil circuit of the loader tooling to collect the weight of the materials loaded and unloaded by the loader tooling; The positioning module is installed on the top of the loader's cab to collect the loader's positioning data in open-air environments.

3. The loader operation guidance system for non-line-of-sight remote control driving according to claim 1, characterized in that, The vehicle-mounted sensor data includes boom tilt angle, bucket tilt angle, articulation angle, laser point cloud data, inertial measurement attitude data, surround view image data, material weight, positioning data, elevation and heading information; the vehicle data includes engine speed, ignition status, turn signal status, high and low beam headlight status, parking status, gear status, windshield wiper status, horn status, cooling fan status, and transmission power status.

4. The loader operation guidance system for non-line-of-sight remote control driving according to claim 1, characterized in that, The wireless communication module is installed on the top of the loader cab and uses the TCP / IP protocol to establish a communication mechanism for wireless communication; the wireless communication module is a broadband self-organizing network device or a 4G / 5G wireless router.

5. The loader operation guidance system for non-line-of-sight remote control driving according to claim 1, characterized in that, The guided model functions include one-click leveling, one-click lifting, material pile identification, route planning, blind spot detection, safety warning, autonomous loading and unloading, and operation statistics.

Images