A method and system for automatic control of equipment based on human-computer interaction

Abstract

This invention relates to the field of engineering equipment technology and discloses a human-computer interaction-based automated equipment control method, comprising: responding to a user's terrain loading command, loading terrain image data on a corresponding display interface, and receiving work area information selected by the user on the terrain image data; determining the equipment information of each device in the current area based on the received device connection signals, and displaying the equipment information of each device; responding to the user's operation command on the terrain image data, configuring corresponding device combinations for the work area information, and associating the device combinations with the work area information; determining corresponding task planning information based on the work area information and the device combinations, and determining the excavation operation path of each intelligent excavator and the transportation path of each transportation device based on the task planning information. The solution of this invention can achieve efficient user interaction, improve overall operation efficiency, and facilitate user management.

Description

Technical Field

[0001] This invention relates to the field of engineering equipment technology, and specifically to an automated control method and system for equipment based on human-computer interaction. Background Technology

[0002] Currently, with the rapid development of the construction machinery industry, the demand for automation in excavator operations is becoming increasingly prominent. Traditional excavator operation methods currently suffer from the following problems: First, existing automated control systems have complex interaction methods, requiring professional training to master, and are not intuitive or user-friendly, making them difficult to learn quickly. Second, the coordination efficiency between excavators and transport vehicles is low. Long waiting times for transport vehicles to load or for excavators to unload frequently occur, resulting in low equipment utilization.

[0003] In mining construction, transportation efficiency is crucial for the overall project schedule and cost control. Traditionally, the method relies primarily on manual visual inspection to determine if mining trucks are fully loaded. However, this method has several problems: First, human observation is susceptible to factors such as fatigue, lighting, and viewing angle, leading to occasional inaccurate judgments of truck fullness, resulting in some trucks departing unloaded and reducing transportation efficiency. Second, manual observation cannot achieve real-time, continuous monitoring, making it difficult to accurately grasp the loading status of each truck and hindering the optimization and scheduling of the overall transportation process. Furthermore, the traditional method lacks an effective excavator-truck coordination mechanism, resulting in poor communication between excavator operators and truck drivers, making it difficult to ensure that every truck departs fully loaded. Summary of the Invention

[0004] To address the aforementioned shortcomings, this invention discloses a device automation control method based on human-computer interaction, which enables efficient user interaction, improves overall operational efficiency, and facilitates user management.

[0005] The first aspect of this invention discloses a device automation control method based on human-computer interaction, comprising:

[0006] In response to the user's terrain loading command, the system loads terrain image data on the corresponding display interface and receives the working area information selected by the user on the terrain image data.

[0007] The device information of each device in the current area is determined based on the received device connection signals, and the device information of each device is displayed.

[0008] In response to user operation commands on terrain image data, a corresponding device combination is configured for the work area information, and the device combination is associated with the work area information, wherein the device combination includes device information of each device;

[0009] Based on the work area information and equipment combination, the corresponding task planning information is determined, and the excavation operation path of each intelligent excavator and the transportation path of each transportation device are determined based on the task planning information.

[0010] As an optional implementation, in the first aspect of the present invention, the device information includes location information, working status information, and device type information;

[0011] The display of equipment information for each device includes:

[0012] The system determines all available devices in the current area based on the working status information of each device, generates a corresponding list of available devices based on the available device information, and displays the list of available devices; or, it generates a corresponding display component based on the device type information of each device, wherein the display component is used to respond to various operation commands from the user.

[0013] Based on the location information of each device, its actual display position in the terrain image data is determined, and the corresponding display components are displayed.

[0014] As an optional implementation, in a first aspect of the present invention, the equipment type information includes intelligent excavators and transportation equipment; the equipment combination includes intelligent excavators and / or transportation equipment.

[0015] The process of determining corresponding task planning information based on the work area information and equipment combination, and determining the excavation operation path of each intelligent excavator and the transportation path of each transportation device based on the task planning information, includes:

[0016] Based on the location information, working area information, and preset working parameters of the selected intelligent excavator, a path planning algorithm is used to plan the optimal excavation path of the intelligent excavator.

[0017] The optimal parking position of each transport vehicle at the loading point is calculated based on the excavation position, working area information, and transport vehicle information of the intelligent excavator. The corresponding loading area is determined based on the optimal parking position. The optimal parking position is then sent to the corresponding transport vehicle. The transport vehicle information includes vehicle size information.

[0018] As an optional implementation, in the first aspect of the present invention, the control method further includes:

[0019] In response to the user's selected work area operation command, information is displayed for the work area. The displayed information includes a list of intelligent excavators, a list of transport vehicles, a list of work areas, and a work status display component. The work status display component includes progress information, remaining time information, and completed work. The progress information is color-coded, with different progress levels using different colors. The list of intelligent excavators, transport vehicles, and work areas uses charts to indicate equipment status, while the remaining time information and completed work are displayed numerically.

[0020] As an optional implementation, in the first aspect of the present invention, the automated control method further includes:

[0021] The system controls and coordinates the loading and transportation of intelligent excavators and transport vehicles; wherein the transport vehicles and the intelligent excavators communicate through a backend server.

[0022] The control and coordination of loading and transportation between the intelligent excavator and the transport vehicle includes:

[0023] The loading characteristic information of the transport vehicle's cargo bed is obtained by an environmental parameter acquisition device installed on the intelligent excavator. The loading characteristic information is then matched with preset loading conditions. If the preset loading conditions are met, a corresponding action signal is sent to the transport vehicle to remind the driver to leave the loading area.

[0024] As an optional implementation, in the first aspect of the present invention, the step of acquiring loading characteristic information at the truck bed of a transport vehicle through an environmental parameter acquisition device installed on an intelligent excavator, matching the loading characteristic information with preset loading conditions, and sending a corresponding action signal to the transport vehicle to remind the driver to leave the loading area if the preset loading conditions are met, includes:

[0025] The three-dimensional point cloud data of the transport vehicle's cargo bed is obtained by using a lidar installed on the intelligent excavator, and the image data of the transport vehicle's cargo bed is obtained by using an image acquisition device installed on the intelligent excavator.

[0026] Preprocessing operations are performed on the acquired 3D point cloud data and image data;

[0027] The preprocessed 3D point cloud data and image data are input into a deep learning model. The deep learning model adopts an architecture that combines convolutional neural networks and recurrent neural networks. The convolutional neural network is used to extract features from the image data, and the recurrent neural network is used to perform temporal feature analysis on the 3D point cloud data. The feature information extracted by the convolutional neural network and the recurrent neural network is fused in the deep learning model to obtain a comprehensive feature vector of the truck bed state.

[0028] Based on the extracted comprehensive feature vectors, the deep learning model uses the trained classifier to judge the state of the two truck beds of the transport vehicle to determine the identification result of the transport vehicle.

[0029] When the identification result indicates that the transport vehicle is full, the intelligent excavator sends a full signal to the transport vehicle through the wireless communication module, and at the same time sends the information that the transport vehicle is ready to depart to the dispatch system.

[0030] Once the transport vehicle receives a full loading signal, the driver confirms departure and leaves the loading area; if no full loading signal is received, the transport vehicle waits to continue loading.

[0031] As an optional implementation, in a first aspect of the present invention, receiving the working area information selected by the user on the terrain image data includes:

[0032] Receive information on multiple working areas selected by the user on the terrain image data.

[0033] A second aspect of this invention discloses an automated control system for equipment based on human-computer interaction, comprising:

[0034] Loading module: In response to the user's terrain loading command, it loads terrain image data in the corresponding display interface and receives the working area information selected by the user on the terrain image data;

[0035] Display module: Used to determine the device information of each device in the current area based on the received device connection signals, and to display the device information of each device;

[0036] Configuration module: In response to user operation commands on terrain image data, it configures a corresponding device combination for the work area information and associates the device combination with the work area information, wherein the device combination includes device information of each device;

[0037] Determining Module: Used to determine the corresponding task planning information based on the work area information and equipment combination, and to determine the excavation operation path of each intelligent excavator and the transportation path of each transportation device based on the task planning information.

[0038] A third aspect of the present invention discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the device automation control method based on human-computer interaction disclosed in the first aspect of the present invention.

[0039] A fourth aspect of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the device automation control method based on human-computer interaction disclosed in the first aspect of the present invention.

[0040] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0041] The equipment automation control method based on human-computer interaction in this invention plans tasks based on terrain image data and equipment information, and accurately determines the digging operation path of the intelligent excavator and the transportation path of the transport equipment. This enables the equipment to complete tasks more accurately during operation, reducing errors and deviations. For example, the intelligent excavator can dig precisely according to the planned path, avoiding over-digging or under-digging, and the transport equipment can also travel along the optimal path, improving transportation efficiency and accuracy.

[0042] The present invention presents terrain image data and equipment information in a graphical interface. Users can complete complex task configuration and planning through simple operation commands. The operation is convenient and quick, requiring no professional technical knowledge or experience, providing users with a good interactive experience and improving user work satisfaction. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the device automation control method based on human-computer interaction disclosed in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the process for displaying the device list as disclosed in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram illustrating the specific process of full-vehicle recognition disclosed in an embodiment of the present invention;

[0047] Figure 4 This is a system architecture diagram of human-computer interaction disclosed in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the human-computer interaction display page disclosed in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram illustrating the specific process of full-vehicle recognition disclosed in an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of the system architecture disclosed in an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the structure of an automated control system for equipment based on human-computer interaction provided in an embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] It should be noted that the terms "first," "second," "third," "fourth," etc., in the specification and claims of this invention are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, in the embodiments of this invention are intended to cover non-exclusive inclusion. Exemplarily, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0055] Currently, with the rapid development of the construction machinery industry, the demand for automation in excavator operations is becoming increasingly prominent. Traditional excavator operation methods currently suffer from the following problems: First, existing automated control systems have complex interaction methods, requiring professional training to master, lacking intuitiveness and user-friendliness, and making them difficult to quickly learn. Second, the coordination efficiency between excavators and transport vehicles is low. Long waiting times for transport vehicles to load or for excavators to unload frequently occur, resulting in low equipment utilization. Based on this, this invention discloses a human-machine interaction-based automated equipment control method, system, electronic device, and storage medium. It displays terrain image data and equipment information through a graphical interface, allowing users to complete complex task configurations and planning through simple operation commands. The operation is convenient and quick, requiring no professional technical knowledge or experience, providing users with a good interactive experience and improving user satisfaction.

[0056] Example 1

[0057] Please see Figure 1 , Figure 1This is a flowchart illustrating a human-computer interaction-based automated control method for devices disclosed in an embodiment of the present invention. The execution entity of the method described in this embodiment is an execution entity composed of software and / or hardware. This execution entity can receive relevant information via wired or / or wireless means and can send certain instructions. It may also have certain processing and storage functions. This execution entity can control multiple devices, such as remote physical servers or cloud servers and related software, or local hosts or servers and related software that perform related operations on devices located in a certain location. In some scenarios, multiple storage devices can also be controlled; these storage devices may be placed in the same location as the devices or in different locations. Figures 1 to 7 As shown, the device automation control method based on human-computer interaction includes the following steps:

[0058] S101: In response to the user's terrain loading command, load terrain image data on the corresponding display interface and receive the working area information selected by the user on the terrain image data;

[0059] S102: Determine the device information of each device in the current area based on the received device connection signal, and display the device information of each device;

[0060] S103: In response to the user's operation command on the terrain image data, configure a corresponding device combination for the work area information, and associate the device combination with the work area information, wherein the device combination includes the device information of each device;

[0061] S104: Determine the corresponding task planning information based on the work area information and equipment combination, and determine the excavation operation path of each intelligent excavator and the transportation path of each transportation device based on the task planning information.

[0062] This invention enables rapid loading of terrain image data in response to terrain loading commands, allowing users to intuitively select the work area. Compared to traditional manual planning methods, this significantly reduces the time required to determine the work area. Simultaneously, it automatically determines task planning information and equipment operation paths based on the work area and equipment combination, avoiding errors and tedious processes that may occur with manual planning and improving the efficiency of the entire workflow.

[0063] During implementation, device information is determined and displayed based on the device connection signal, allowing users to understand the status and location of devices in the current area in real time, facilitating unified management and monitoring. When a device malfunctions or malfunctions, users can quickly locate the problem and take appropriate measures, reducing downtime and improving device utilization. Furthermore, the combination of image and sensor technology enables a realistic and objective dynamic display.

[0064] Users can flexibly configure equipment combinations according to the needs of the work area and associate them with work area information. This ensures that equipment resources are allocated rationally in different work scenarios, maximizing equipment functionality and performance, avoiding waste of equipment resources, and reducing operating costs. This method automates many tasks that originally required manual labor, such as work area planning, equipment configuration, and task planning, through human-computer interaction. This reduces human intervention, thereby lowering the risk of errors caused by human factors and improving the reliability and stability of the entire operation process. This human-computer interaction-based automated equipment control method has high flexibility and scalability. When new equipment or functions need to be added, only corresponding configuration and programming in the system are required to bring the new equipment into the management and control scope, achieving smooth system upgrades and expansions to adapt to work scenarios of different scales and complexities.

[0065] The solution in this invention simplifies human-computer interaction, improves operator efficiency, and reduces operating costs for human operators, upgrading from one-to-one to one-to-many operations. Through convenient interaction, it drives the upgrade and application of automation technology, truly liberating human labor.

[0066] like Figure 4 As shown, the system of the present invention mainly includes the following modules: a human-machine interaction module including a touch display interface, an equipment selection and task allocation operation interface, and a real-time operation status monitoring and feedback interface, wherein the touch display interface is used to display the topographic map of the operation area and the equipment status; a task planning module including work area identification and division units, excavation operation path planning, and transport vehicle scheduling planning; an operation control module including an excavator automatic control system, a transport vehicle cooperative control system, and an operation progress monitoring system; and a data acquisition and processing module including terrain data acquisition and processing, equipment status data acquisition, and operation efficiency data statistics.

[0067] The workflow of this invention mainly includes:

[0068] During the initialization phase, the system starts up, loads terrain data, waits for device connections, checks device status, and displays a list of available devices.

[0069] During the task allocation phase, the operator selects a specific excavator by clicking, selects a designated work area on the topographic map, and selects and assigns transport vehicles.

[0070] During the automated operation phase, the system automatically plans the excavation operation path, calculates the optimal stopping position for transport vehicles, controls the excavator to excavate according to the planned path, and coordinates the transport vehicles for loading and transportation.

[0071] During the monitoring and adjustment phase, the work progress is monitored in real time, work parameters are dynamically adjusted, and abnormal situations are handled.

[0072] like Figure 4 As shown, the system of the present invention adopts a distributed architecture and mainly includes:

[0073] The central control server is divided into four core modules:

[0074] Global Map Management Module: Stores and updates the overall work area map; Work Area Planning and Management Module: Responsible for work area division and status management; Equipment Coordination and Task Allocation Module: Coordinates and schedules excavators and mining trucks; Data Storage and Analysis Module: Stores and analyzes work data.

[0075] The excavator control unit is divided into two major systems:

[0076] Sensor System: LiDAR for scene scanning and obstacle detection; GPS / IMU integrated navigation for precise positioning and attitude measurement; operating condition sensors to monitor excavator working status; HD cameras for environmental perception and loading monitoring; attitude sensors for bucket position and attitude measurement; computational control module: scene reconstruction to build a 3D map of the work area; path planning to plan the excavation path; motion control to control excavation actions; loading control to coordinate the loading process; safety monitoring to monitor operational safety in real time.

[0077] The mining truck control unit is also divided into two main systems:

[0078] Sensor System: LiDAR for obstacle detection; GPS / IMU integrated navigation for precise positioning and navigation; load sensor for monitoring loading status; surround view camera for all-around environmental perception; vehicle attitude sensor for monitoring vehicle status; Computation and Control Module: Positioning and navigation for real-time positioning and navigation; path planning for planning the driving route; docking control for precise docking position control; loading coordination for coordinating loading with the excavator; safety monitoring to ensure driving safety;

[0079] Communication connection: such as Figure 4 As shown, solid lines represent communication with the central server (control commands and data transmission), while dashed lines represent direct communication between the excavator and the mining truck (loading coordination).

[0080] like Figure 5As shown, the layout of the human-computer interaction main interface is as follows: the left side displays the device list and working status, and the center displays the topographic map of the work area and work area information. The interaction method is as follows: for excavators or mining trucks, click to select the device and drag to select the work area; for work areas, click to select the work area and view the excavators and mining trucks in operation, as well as the work progress; during specific implementation, you can also create new work areas or delete existing work areas; the status feedback method uses color coding to display the work progress, icons to indicate the device status, and numerical values ​​to display key parameters.

[0081] like Figure 2 and Figure 3 More preferably, the device information includes location information, working status information, and device type information; the display of device information for each device includes:

[0082] S1021: Determine all available device information in the current area based on the working status information of each device, generate a corresponding list of available devices based on the available device information, and display the list of available devices; or, generate a corresponding display component based on the device type information of each device, wherein the display component is used to respond to various operation commands of the user;

[0083] S1022: Determine the actual display position of each device in the terrain image data based on the corresponding location information of each device, and display the corresponding display components.

[0084] This invention identifies available devices by analyzing their operational status and generates a list for display. Users can quickly see which devices in the current area are immediately usable, avoiding the tedious process of checking the status of each device individually. This facilitates rapid task allocation decisions and improves the efficiency of device scheduling. A display component is generated based on device type information, allowing users to quickly locate the required type of device according to specific task needs. This facilitates unified management and operation of similar devices, improving the targeted nature and convenience of device management.

[0085] The display component in this invention can respond to various user operation commands, providing users with an intuitive and convenient interaction method. Users do not need to memorize complex operation commands or procedures; they can perform various operations on the device simply by interacting with the display component, such as clicking and dragging, reducing operational difficulty and improving user work efficiency and satisfaction. Based on location information, the actual display location of the device is shown in the terrain image data, allowing users to clearly see the specific distribution of the device in the work area. This helps users better plan the working range and path of the device, avoid mutual interference between devices, and also facilitates users to quickly locate the device's position when encountering problems, enabling timely maintenance and management.

[0086] This invention combines equipment location information, operational status information, and equipment type information, allowing users to comprehensively consider various factors and rationally arrange equipment and allocate tasks within the work area. For example, based on the location and availability of equipment, tasks can be assigned to equipment that is close to the work area and currently available, reducing equipment movement time and energy consumption, and improving task execution efficiency and quality. Accurately displaying equipment locations in terrain image data and interacting with operation commands through display components ensures greater accuracy in task planning and equipment control. Users can precisely formulate excavation and transportation paths based on the actual location and status of the equipment, avoiding task errors caused by unclear equipment locations or incorrect operation commands, thus improving the accuracy and reliability of the entire operation process.

[0087] More preferably, the equipment type information includes intelligent excavators and transportation equipment; the equipment combination includes intelligent excavators and / or transportation equipment;

[0088] The process of determining corresponding task planning information based on the work area information and equipment combination, and determining the excavation operation path of each intelligent excavator and the transportation path of each transportation device based on the task planning information, includes:

[0089] Based on the location information, working area information, and preset working parameters of the selected intelligent excavator, a path planning algorithm is used to plan the optimal excavation path of the intelligent excavator.

[0090] The optimal parking position of each transport vehicle at the loading point is calculated based on the excavation position, working area information, and transport vehicle information of the intelligent excavator. The corresponding loading area is determined based on the optimal parking position. The optimal parking position is then sent to the corresponding transport vehicle. The transport vehicle information includes vehicle size information.

[0091] This invention combines the location, working area, and working parameters of the intelligent excavator with a path planning algorithm to plan the optimal excavation path. This enables the excavator to excavate in the most efficient way, reducing unnecessary movement and excavation time, increasing the excavation volume per unit time, and improving overall work efficiency. Path planning based on the working area and preset parameters ensures that the excavator excavates accurately according to set requirements, ensuring that excavation depth, angle, etc., meet engineering standards, improving excavation quality, and reducing subsequent correction work.

[0092] This invention calculates the optimal parking position and loading area based on the excavation location of the intelligent excavator, the work area, and the information of the transport vehicle. This ensures accurate parking of the transport vehicle, facilitates rapid loading by the excavator, reduces waiting and adjustment time, optimizes the loading process, and improves loading efficiency. Considering vehicle size information to determine the optimal parking position and loading area allows for rational use of work area space, avoiding chaotic vehicle parking or excessive space occupation, improving space utilization, and ensuring orderly operation of the work area. Sending the optimal parking position to the transport vehicle enables collaborative operation between the intelligent excavator and the transport equipment, enhancing the coordination and stability of the entire automated control system, ensuring close integration of excavation, loading, and transportation processes, and improving overall work efficiency.

[0093] More preferably, the control method further includes:

[0094] In response to the user's selected work area operation command, information is displayed for the work area. The displayed information includes a list of intelligent excavators, a list of transport vehicles, a list of work areas, and a work status display component. The work status display component includes progress information, remaining time information, and completed work. The progress information is color-coded, with different progress levels using different colors. The list of intelligent excavators, transport vehicles, and work areas uses charts to indicate equipment status, while the remaining time information and completed work are displayed numerically.

[0095] When a user selects a work area operation command, a list of intelligent excavators, transport vehicles, work areas, and work status display components will be displayed, covering various aspects of information related to the work area, such as equipment and work status. This allows users to fully understand the overall situation of the work area without having to switch between multiple interfaces or systems, improving the efficiency and convenience of information acquisition.

[0096] In the work status display component, progress information uses color coding, with different colors corresponding to different progress levels. This allows users to quickly and intuitively judge the work progress; for example, green indicates good progress, while red indicates lagging progress. This helps users quickly identify problem areas and take timely measures to adjust work arrangements. Remaining time and completed work are displayed numerically, providing accurate and quantifiable data. This allows users to more precisely grasp the completion status and remaining workload, providing a reliable basis for subsequent resource allocation and task planning. In the specific design, this logical design approach can significantly improve the user's decision-making efficiency.

[0097] In this embodiment of the invention, the intelligent excavator list, transport vehicle list, and work area list use charts to identify equipment status, presenting the operating and idle states of the equipment in a graphical way, which is more vivid and intuitive than simple text descriptions. Users can see at a glance which equipment is in working condition and which is in idle condition, thereby rationally scheduling equipment use, improving equipment utilization, and avoiding waste of equipment resources.

[0098] By providing a comprehensive, intuitive, and clear display of information about the work area, users can manage tasks more effectively. For example, they can adjust task allocation and optimize workflows in a timely manner based on work progress and equipment status. Simultaneously, it provides rich and accurate information support for user decision-making, enabling them to make more scientific and rational decisions and improve overall work efficiency and quality. This information display method aligns with users' operating habits and cognitive patterns, offering simple and convenient operation with intuitive and easy-to-understand information presentation. This reduces user learning costs and operational difficulties, enhances user satisfaction and user experience, and strengthens the interactivity and usability between users and the system.

[0099] More preferably, the automated control method further includes:

[0100] S105: Control and coordinate the loading and transportation of the intelligent excavator and the transport vehicle; wherein the transport vehicle and the intelligent excavator communicate through a back-end server;

[0101] The control and coordination of loading and transportation between the intelligent excavator and the transport vehicle includes:

[0102] The loading characteristic information of the transport vehicle's cargo bed is obtained by an environmental parameter acquisition device installed on the intelligent excavator. The loading characteristic information is then matched with preset loading conditions. If the preset loading conditions are met, a corresponding action signal is sent to the transport vehicle to remind the driver to leave the loading area.

[0103] By acquiring real-time loading characteristic information of the transport vehicle's cargo bed through environmental parameter acquisition devices and matching it with preset loading conditions, the system can accurately determine whether loading is complete. Once the conditions are met, the driver is alerted to leave, avoiding overloading or underloading, reducing the vehicle's dwell time in the loading area, and enabling the intelligent excavator to quickly carry out the next loading operation, thereby improving overall loading and transportation efficiency.

[0104] The transport vehicles and intelligent excavators communicate through a backend server, and with the application of environmental parameter acquisition devices, they achieve close collaboration. The intelligent excavator can adjust its work process in real time according to the loading status of the transport vehicles, and the transport vehicles can also respond promptly to signals sent by the intelligent excavator. This makes the connection between loading and transportation smoother, reduces waiting time and idling time between equipment, and improves the collaborative work capability of the entire production system.

[0105] This automated control method reduces the need for manual judgment of loading conditions and directing vehicles to leave, thus lowering labor costs and the possibility of human error. At the same time, it reduces the workload of operators, allowing them to devote more energy to other important tasks and improving overall work efficiency.

[0106] More preferably, such as Figure 6 and Figure 7 As shown, the method involves acquiring loading characteristic information at the truck bed of a transport vehicle through an environmental parameter acquisition device installed on the intelligent excavator, matching the loading characteristic information with preset loading conditions, and sending a corresponding action signal to the transport vehicle to remind the driver to leave the loading area if the preset loading conditions are met.

[0107] S1051: Acquire three-dimensional point cloud data of the transport vehicle's cargo bed by using a lidar installed on the intelligent excavator and acquire image data of the transport vehicle's cargo bed by using an image acquisition device installed on the intelligent excavator.

[0108] S1052: Perform preprocessing operations on the acquired 3D point cloud data and image data;

[0109] S1053: Input the preprocessed 3D point cloud data and image data into a deep learning model. The deep learning model adopts an architecture combining convolutional neural networks and recurrent neural networks. The convolutional neural network is used to extract features from the image data, and the recurrent neural network is used to perform temporal feature analysis on the 3D point cloud data. The feature information extracted by the convolutional neural network and the recurrent neural network is fused in the deep learning model to obtain a comprehensive feature vector of the truck bed state.

[0110] S1054: Based on the extracted comprehensive feature vector, the deep learning model uses the trained classifier to judge the state of the two truck beds of the transport vehicle to determine the identification result of the transport vehicle.

[0111] S1055: When the identification result is that the transport vehicle is full, the intelligent excavator sends a full signal to the transport vehicle through the wireless communication module, and at the same time sends the information that the transport vehicle is ready to depart to the dispatch system.

[0112] S1056: After receiving a full-load signal, the driver of the transport vehicle confirms departure and leaves the loading area; if no full-load signal is received, the transport vehicle waits to continue loading.

[0113] The lidar in this invention can acquire high-precision three-dimensional point cloud data of the truck bed, accurately measure the volume and shape of the goods inside the truck bed, and has strong anti-interference ability and wide adaptability, and can work stably even in complex outdoor environments or under changing lighting conditions. The image acquisition device can provide visual image data of the truck bed, supplementing the details of the goods' appearance, texture, etc. The combination of the two can comprehensively and accurately reflect the loading characteristics of the truck bed.

[0114] Preprocessing 3D point cloud data and image data can remove noise, correct data deviations, and improve data quality and reliability, providing a good data foundation for subsequent analysis and processing and reducing the possibility of errors and misjudgments.

[0115] A deep learning model combining convolutional neural networks (CNNs) and recurrent neural networks (RNNs) is employed. CNNs excel at feature extraction from image data, capturing spatial features such as surface texture and edge shapes of goods. RNNs, on the other hand, analyze the temporal features of 3D point cloud data, handling dynamic changes during the loading process of the truck bed. By fusing the feature information extracted from both, a more comprehensive and accurate integrated feature vector of the truck bed's state can be obtained, allowing for deeper mining of useful information from the data and a better description of the truck bed's loading status.

[0116] Based on comprehensive feature vectors, a trained classifier is used to determine the status of the truck bed in transport vehicles. This accurately identifies whether the truck bed is full, offering higher accuracy and reliability compared to manual judgment or simple sensor detection, and reducing the probability of false positives and false negatives. When the identification result indicates that the truck bed is full, the intelligent excavator automatically sends a full signal to the transport vehicle via wireless communication and sends a departure preparation message to the dispatch system. Upon receiving the signal, the transport vehicle driver confirms departure and drives away. The entire process requires no frequent manual intervention, achieving automated collaborative operation between the intelligent excavator, transport vehicle, and dispatch system. This improves the efficiency of loading and transportation processes, reduces waiting time, and enhances overall production efficiency.

[0117] In deep learning models, convolutional neural networks extract features from image data, effectively capturing visual features such as texture and color. Recurrent neural networks perform temporal feature analysis on 3D point cloud data, uncovering the changing patterns of point cloud data over time. Combining these two approaches allows for comprehensive extraction of feature information about the truck bed's state from different perspectives, resulting in more representative and discriminative comprehensive feature vectors. This enhances the accuracy and reliability of the deep learning model's judgment of the truck bed's state. This combination of laser and image processing, along with the corresponding deep learning model architecture, better adapts to various complex real-world loading scenarios. For example, different types of transport vehicles have varying truck bed shapes and sizes, different types and arrangements of cargo, and varying lighting conditions and environmental backgrounds. By fusing feature information from LiDAR and image data, the model can more accurately learn the characteristic patterns of a full truck bed in different scenarios, improving the model's generalization ability and robustness. It can accurately determine the truck bed's state under various complex conditions, which is a key innovation of this technical solution.

[0118] Based on the extracted comprehensive feature vectors, the deep learning model uses the trained classifier to judge the state of the mining truck's cargo bed and determine whether the cargo bed is full.

[0119] The classifier training process involves using a large amount of labeled sample data of both full and partial vehicle states for training, continuously adjusting model parameters, and improving recognition accuracy.

[0120] The solutions of the embodiments of the present invention can achieve the following technical effects:

[0121] Improve transportation efficiency: By accurately identifying the full status of mining trucks, the situation of transporting trucks with insufficient capacity is avoided, the number of transport trips is reduced, and the amount of transportation per unit time is increased.

[0122] Improved loading accuracy: The application of deep learning models has improved the accuracy of full-load vehicle identification, ensuring that mining trucks can be loaded to the appropriate full load level and optimizing resource utilization.

[0123] Enhanced excavator-mining truck collaboration: Real-time communication and collaborative control between excavators and mining trucks enable seamless integration of loading and transportation processes, reducing waiting time and improving operational smoothness.

[0124] Reduce labor costs: It reduces reliance on human visual observation, lowers labor costs, and avoids losses caused by human judgment errors.

[0125] More preferably, receiving the working area information selected by the user on the terrain image data includes:

[0126] Receive information on multiple working areas selected by the user on the terrain image data.

[0127] Users can select multiple work areas on the terrain image data according to actual needs, and can make differentiated equipment configurations and task planning based on the characteristics and task requirements of different areas. For example, in a large construction site, there may be different construction stages or different types of construction tasks. By selecting multiple work areas, appropriate intelligent excavators and transportation equipment can be arranged for each area, achieving more precise and flexible construction arrangements.

[0128] The equipment configuration can be independently configured and optimized for different work areas. For areas with a large workload or high construction difficulty, more intelligent excavators and transportation equipment can be allocated; while for smaller or relatively simpler areas, equipment investment can be appropriately reduced. This allows for the rational allocation of resources based on the specific needs of each work area, avoiding equipment waste or shortages, and improving equipment utilization and production efficiency.

[0129] In practical implementation, the state of the excavation area can be determined directly by image recognition, and the image can be dynamically updated based on the acquired comprehensive information. The image recognition object can be directly image data from the terrain data set. By setting appropriate color display rules, the image can be combined for comprehensive recognition, improving overall coordination and processing capabilities.

[0130] The solution in this embodiment of the invention updates the equipment data every preset time interval to control and coordinate the loading and transportation between transport vehicles and intelligent excavators. In specific implementation, the working efficiency of the intelligent excavator can also be determined based on the number of loading operations performed.

[0131] Furthermore, if the working efficiency of the intelligent excavator is detected to be lower than the set value within a certain time range, the user will be reminded to optimize the control.

[0132] In practical implementation, equipment adjustments can be dynamically made across multiple work areas. This embodiment of the invention uses two work areas as an example for illustration. Specifically, based on the ore reserves, mining progress, and equipment capacity of the two work areas, excavation and transportation tasks are rationally allocated to avoid one work area being overburdened while the other is underutilized. For instance, if work area A has large ore reserves but low excavation efficiency, the allocation of transportation vehicles can be appropriately reduced, with some vehicles moved to work area B. Simultaneously, the operating time of excavators in work area A can be increased, or more excavators can be deployed.

[0133] The optimal route for transport vehicles can be planned by comprehensively considering the location, terrain, road conditions, and unloading point of both work areas. For example, if there is a shortcut between the two work areas, but the road is narrow and only suitable for small vehicles, small transport vehicles can be arranged to make short-distance transfers between the two work areas, while larger vehicles travel on the main road to improve overall transport efficiency.

[0134] The system dynamically adjusts the routes of transport vehicles based on real-time traffic conditions and unforeseen circumstances such as equipment malfunctions. For example, if the regular route from work area A to the unloading point is impassable due to road collapse, the system automatically plans a temporary route for the transport vehicles in that work area, from work area A through work area B to the unloading point.

[0135] In practical implementation, transport vehicles can be rationally allocated based on the workload and transport capacity of the two work areas. Mathematical models or intelligent algorithms, such as genetic algorithms and simulated annealing algorithms, can be used to calculate the required number and type of vehicles for each work area to maximize transport efficiency. For example, for work areas closer to the unloading point, smaller, more flexible vehicles can be allocated to facilitate travel on narrow mining roads; while for work areas farther away, larger transport vehicles can be allocated to reduce the number of transport trips.

[0136] The equipment automation control method based on human-computer interaction in this invention plans tasks based on terrain image data and equipment information, and accurately determines the digging operation path of the intelligent excavator and the transportation path of the transport equipment. This enables the equipment to complete tasks more accurately during operation, reducing errors and deviations. For example, the intelligent excavator can dig precisely according to the planned path, avoiding over-digging or under-digging, and the transport equipment can also travel along the optimal path, improving transportation efficiency and accuracy.

[0137] The present invention presents terrain image data and equipment information in a graphical interface. Users can complete complex task configuration and planning through simple operation commands. The operation is convenient and quick, requiring no professional technical knowledge or experience, providing users with a good interactive experience and improving user work satisfaction.

[0138] Example 2

[0139] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a human-computer interaction-based automated control system for equipment disclosed in an embodiment of the present invention. Figure 8 As shown, the human-computer interaction-based automated control system for equipment may include:

[0140] Loading module 21: In response to the user's terrain loading command, loads terrain image data on the corresponding display interface and receives the working area information selected by the user on the terrain image data;

[0141] Display module 22: Used to determine the device information of each device in the current area based on the received device connection signal, and to display the device information of each device;

[0142] Configuration module 23: In response to user operation commands on terrain image data, it configures a corresponding device combination for the work area information and associates the device combination with the work area information, wherein the device combination includes device information of each device;

[0143] Determining module 24: is used to determine the corresponding task planning information based on the work area information and equipment combination, and to determine the excavation operation path of each intelligent excavator and the transportation path of each transportation device based on the task planning information.

[0144] The equipment automation control method based on human-computer interaction in this invention plans tasks based on terrain image data and equipment information, and accurately determines the digging operation path of the intelligent excavator and the transportation path of the transport equipment. This enables the equipment to complete tasks more accurately during operation, reducing errors and deviations. For example, the intelligent excavator can dig precisely according to the planned path, avoiding over-digging or under-digging, and the transport equipment can also travel along the optimal path, improving transportation efficiency and accuracy.

[0145] The present invention presents terrain image data and equipment information in a graphical interface. Users can complete complex task configuration and planning through simple operation commands. The operation is convenient and quick, requiring no professional technical knowledge or experience, providing users with a good interactive experience and improving user work satisfaction.

[0146] Example 3

[0147] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. The electronic device can be a computer, a server, etc. Of course, in certain cases, it can also be a mobile phone, tablet computer, monitoring terminal, or other smart device, as well as an image acquisition device with processing capabilities. Figure 9 As shown, the electronic device may include:

[0148] Memory 510 storing executable program code;

[0149] Processor 520 coupled to memory 510;

[0150] The processor 520 calls the executable program code stored in the memory 510 to execute some or all of the steps in the human-computer interaction-based device automation control method in Embodiment 1.

[0151] This invention discloses a computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps in the human-computer interaction-based device automation control method of Embodiment 1.

[0152] This invention also discloses a computer program product, wherein when the computer program product is run on a computer, the computer executes some or all of the steps in the human-computer interaction-based device automation control method in Embodiment 1.

[0153] This invention also discloses an application publishing platform, which is used to publish computer program products. When the computer program products are run on a computer, the computer executes some or all of the steps in the human-computer interaction-based device automation control method in Embodiment 1.

[0154] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of the present invention.

[0158] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0159] Those skilled in the art will understand that some or all of the steps in the various methods of the embodiments described can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0160] The above provides a detailed description of the device automation control method, system, electronic device, and storage medium based on human-computer interaction disclosed in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for automated control of equipment based on human-computer interaction, characterized in that, include: In response to the user's terrain loading command, the system loads terrain image data on the corresponding display interface and receives the working area information selected by the user on the terrain image data. The device information of each device in the current area is determined based on the received device connection signals, and the device information of each device is displayed; the device information includes location information, working status information, and device type information; the device type information includes intelligent excavators and transportation equipment; The display of equipment information for each device includes: Based on the working status information of each device, determine all available device information in the current area, generate a corresponding list of available devices based on the available device information, and display the list of available devices; or, generate a corresponding display component based on the device type information of each device, wherein the display component is used to respond to various operation commands of the user; determine the actual display position of each device in the terrain image data based on the corresponding location information of each device, and display the corresponding display component. In response to user operation commands on terrain image data, a corresponding equipment combination is configured for the work area information, and the equipment combination is associated with the work area information. The equipment combination includes equipment information of each device; the equipment combination includes intelligent excavators and / or transportation equipment. Based on the work area information and equipment combination, corresponding task planning information is determined, and based on the task planning information, the excavation operation path of each intelligent excavator and the transportation path of each transportation device are determined; the process of determining corresponding task planning information based on the work area information and equipment combination, and determining the excavation operation path of each intelligent excavator and the transportation path of each transportation device based on the task planning information, includes: Based on the location information, working area information, and preset working parameters of the selected intelligent excavator, a path planning algorithm is used to plan the optimal excavation path of the intelligent excavator. The optimal parking position of each transport vehicle at the loading point is calculated based on the excavation position, working area information, and transport vehicle information of the intelligent excavator. The corresponding loading area is determined based on the optimal parking position. The optimal parking position is then sent to the corresponding transport vehicle. The transport vehicle information includes vehicle size information.

2. The equipment automation control method based on human-computer interaction as described in claim 1, characterized in that, The control method further includes: In response to the user's selected work area operation command, information is displayed for the work area. The displayed information includes a list of intelligent excavators, a list of transport vehicles, a list of work areas, and a work status display component. The work status display component includes progress information, remaining time information, and completed work. The progress information is color-coded, with different progress levels using different colors. The list of intelligent excavators, transport vehicles, and work areas uses charts to indicate equipment status, while the remaining time information and completed work are displayed numerically.

3. The equipment automation control method based on human-computer interaction as described in claim 1, characterized in that, The automated control method further includes: The system controls and coordinates the loading and transportation of intelligent excavators and transport vehicles; wherein the transport vehicles and the intelligent excavators communicate through a backend server. The control and coordination of loading and transportation between the intelligent excavator and the transport vehicle includes: The loading characteristic information of the transport vehicle's cargo bed is obtained by an environmental parameter acquisition device installed on the intelligent excavator. The loading characteristic information is then matched with preset loading conditions. If the preset loading conditions are met, a corresponding action signal is sent to the transport vehicle to remind the driver to leave the loading area.

4. The equipment automation control method based on human-computer interaction as described in claim 3, characterized in that, The method involves acquiring loading characteristic information at the truck bed of a transport vehicle through an environmental parameter acquisition device installed on the intelligent excavator, matching the loading characteristic information with preset loading conditions, and if the preset loading conditions are met, sending a corresponding action signal to the transport vehicle to remind the driver to leave the loading area. This includes: The three-dimensional point cloud data of the transport vehicle's cargo bed is obtained by using a lidar installed on the intelligent excavator, and the image data of the transport vehicle's cargo bed is obtained by using an image acquisition device installed on the intelligent excavator. Preprocessing operations are performed on the acquired 3D point cloud data and image data; The preprocessed 3D point cloud data and image data are input into a deep learning model. The deep learning model adopts an architecture that combines convolutional neural networks and recurrent neural networks. The convolutional neural network is used to extract features from the image data, and the recurrent neural network is used to perform temporal feature analysis on the 3D point cloud data. The feature information extracted by the convolutional neural network and the recurrent neural network is fused in the deep learning model to obtain a comprehensive feature vector of the truck bed state. Based on the extracted comprehensive feature vectors, the deep learning model uses the trained classifier to judge the state of the two truck beds of the transport vehicle to determine the identification result of the transport vehicle. When the identification result indicates that the transport vehicle is full, the intelligent excavator sends a full signal to the transport vehicle through the wireless communication module, and at the same time sends the information that the transport vehicle is ready to depart to the dispatch system. Once the transport vehicle receives a full loading signal, the driver confirms departure and leaves the loading area; if no full loading signal is received, the transport vehicle waits to continue loading.

5. The equipment automation control method based on human-computer interaction as described in claim 1, characterized in that, Receiving the work area information selected by the user on the terrain image data includes: Receive information on multiple working areas selected by the user on the terrain image data.

6. A device automation control system based on human-computer interaction, characterized in that, include: Loading module: In response to the user's terrain loading command, it loads terrain image data in the corresponding display interface and receives the working area information selected by the user on the terrain image data; Display module: Used to determine the device information of each device in the current area based on the received device connection signals, and to display the device information of each device; the device information includes location information, working status information, and device type information; the device type information includes intelligent excavators and transportation equipment; The display of equipment information for each device includes: Based on the working status information of each device, determine all available device information in the current area, generate a corresponding list of available devices based on the available device information, and display the list of available devices; or, generate a corresponding display component based on the device type information of each device, wherein the display component is used to respond to various operation commands of the user; determine the actual display position of each device in the terrain image data based on the corresponding location information of each device, and display the corresponding display component. Configuration module: In response to user operation commands on terrain image data, it configures corresponding equipment combinations for the work area information and associates the equipment combinations with the work area information. The equipment combinations include equipment information of each device; the equipment combinations include intelligent excavators and / or transportation equipment. The determining module is used to determine corresponding task planning information based on the work area information and equipment combination, and to determine the excavation operation path of each intelligent excavator and the transportation path of each transportation device based on the task planning information; the step of determining corresponding task planning information based on the work area information and equipment combination, and determining the excavation operation path of each intelligent excavator and the transportation path of each transportation device based on the task planning information, includes: Based on the location information, working area information, and preset working parameters of the selected intelligent excavator, a path planning algorithm is used to plan the optimal excavation path of the intelligent excavator. The optimal parking position of each transport vehicle at the loading point is calculated based on the excavation position, working area information, and transport vehicle information of the intelligent excavator. The corresponding loading area is determined based on the optimal parking position. The optimal parking position is then sent to the corresponding transport vehicle. The transport vehicle information includes vehicle size information.

7. An electronic device, characterized in that, include: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the device automation control method based on human-computer interaction as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program causes a computer to perform the human-computer interaction-based device automation control method according to any one of claims 1 to 5.

Images