Equipment automation control method and system based on human-computer interaction

Abstract

The embodiment of the invention relates to the technical field of engineering equipment, and discloses an equipment automation control method based on man-machine interaction, which comprises the following steps: loading topographic image data on a corresponding display interface in response to a topographic loading instruction of a user, and receiving work area information selected on the topographic image data by the user; determining device information of each device in the current area according to the received device connection signal, and displaying the device information of each device; in response to an operation instruction of a user on the topographic image data, configuring a corresponding equipment combination for the working area information, and performing information association on the equipment combination and the working area information; and determining corresponding task planning information according to the working area information and the equipment combination, and determining an excavation operation path of each intelligent excavator and a transportation path of each transportation equipment according to the task planning information. According to the scheme provided by the embodiment of the invention, efficient user interaction can be realized, the overall operation efficiency is improved, and a user can conveniently manage.

Description

Technical Field

[0001] The present invention relates to the technical field of engineering equipment, and in particular to an equipment automation control method and system based on human-computer interaction. Background Art

[0002] With the rapid development of the construction machinery sector, the demand for automated excavator operations is becoming increasingly prominent. Currently, traditional excavator operations present the following major challenges: First, existing automated control systems have complex interaction methods that require specialized training to master, lack intuitiveness and user friendliness, and are difficult to quickly master. Second, coordination between excavators and transport vehicles is inefficient. Transport vehicles often wait excessively for loading, or excavators frequently wait for unloading, resulting in low equipment utilization.

[0003] In mining construction, transportation efficiency is crucial to the overall project progress and cost control. Traditionally, this approach relies primarily on visual inspection to determine whether the truck bed is fully loaded. However, this approach presents numerous problems. First, human visual observation is susceptible to factors such as fatigue, lighting, and visual angle, resulting in occasional inaccurate judgments about the truck bed's fullness. This can cause some trucks to depart before being fully loaded, reducing transportation efficiency. Second, manual inspection cannot achieve real-time, continuous monitoring, making it difficult to accurately determine the loading status of each truck, hindering the optimization and scheduling of the overall transportation process. Furthermore, the traditional approach lacks an effective excavator-truck coordination mechanism, resulting in poor communication between excavator operators and truck drivers, making it difficult to ensure that each truck is fully loaded before departure. Summary of the Invention

[0004] In response to the above-mentioned defects, an embodiment of the present invention discloses an equipment automation control method based on human-computer interaction, which can achieve efficient user interaction, improve overall operation efficiency, and facilitate user management.

[0005] A first aspect of an embodiment of the present invention discloses a device automation control method based on human-computer interaction, comprising:

[0006] In response to a terrain loading instruction from a user, loading terrain image data on a corresponding display interface, and receiving information about a working area selected by the user on the terrain image data;

[0007] Determine the device information of each device in the current area based on the received device connection signal, and display the device information of each device;

[0008] In response to a user's operation instruction on the terrain image data, configuring a corresponding device combination for the work area information, and associating the device combination with the work area information, wherein the device combination includes device information of each device;

[0009] Corresponding task planning information is determined according to the work area information and the equipment combination, and the excavation operation path of each intelligent excavator and the transportation path of each transportation equipment are determined according to the task planning information.

[0010] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the device information includes location information, working status information and device type information;

[0011] The display of device information of each device includes:

[0012] Determining all available device information in the current area based on the working status information of each device, generating a corresponding available device list based on the available device information, and displaying the available device list; or generating a corresponding display component based on the device type information of each device, wherein the display component is used to respond to various user operation instructions;

[0013] The actual display position of each device in the terrain image data is determined according to the corresponding position information of each device, and the corresponding display component is displayed.

[0014] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the equipment type information includes an intelligent excavator and a transportation device; the equipment combination includes an intelligent excavator and / or a transportation device;

[0015] Determining corresponding task planning information according to the work area information and the equipment combination, and determining the excavation operation path of each intelligent excavator and the transportation path of each transportation equipment according to the task planning information, includes:

[0016] Based on the location information, working area information and preset working parameters of the selected intelligent excavator, a path planning algorithm is used to plan the optimal excavation path of the intelligent excavator;

[0017] The optimal docking position of each transport vehicle at the loading point is calculated based on the excavation position, work area information and transport vehicle information of the intelligent excavator, and the corresponding loading area is determined based on the optimal docking position; and the optimal docking position is sent to the corresponding transport vehicle, wherein the transport vehicle information includes vehicle size information.

[0018] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the control method further includes:

[0019] In response to the operation instruction of the work area clicked by the user, information of the work area is displayed, wherein the displayed information includes a smart excavator list, a transport vehicle list, a work area list and a work status display component, and the work status display component includes progress information, remaining time information and completed work; the progress information is color-coded, and different progress is coded with different colors; the smart excavator list, transport vehicle list and work area list use charts to identify the equipment status, and the remaining time information and completed work are displayed using numerical values.

[0020] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the automatic control method further includes:

[0021] Control and coordinate the loading and transportation of the intelligent excavator and the transport vehicle; wherein the transport vehicle and the intelligent excavator communicate through the background server;

[0022] The control coordinates the loading and transportation of the intelligent excavator and the transport vehicle, including:

[0023] The loading characteristic information of the transport vehicle's bucket is obtained by an environmental parameter acquisition device installed in the intelligent excavator, and the loading characteristic information is matched with the preset loading conditions. If the set loading conditions are met, a corresponding action signal is sent to the transport vehicle to remind the driver to leave the loading area.

[0024] As an optional implementation, in the first aspect of the embodiment of the present invention, the environmental parameter collection device provided on the intelligent excavator is used to obtain the loading characteristic information of the transport vehicle bucket, and the loading characteristic information is matched with the preset loading conditions. If the set loading conditions are met, a corresponding action signal is sent to the transport vehicle to remind the driver to leave the loading area, including:

[0025] Acquiring three-dimensional point cloud data of the transport vehicle bucket by using a laser radar installed on the intelligent excavator and acquiring image data of the transport vehicle bucket by using an image acquisition device installed on the intelligent excavator;

[0026] Perform preprocessing operations on the acquired three-dimensional point cloud data and image data;

[0027] The preprocessed 3D point cloud data and image data are input into a deep learning model. The deep learning model uses an architecture that combines a convolutional neural network with a recurrent neural network. The convolutional neural network is used to extract features from the image data, and the recurrent neural network is used to perform time series feature analysis on the 3D point cloud data. The feature information extracted by the convolutional neural network and the recurrent neural network is integrated into the deep learning model to obtain a comprehensive feature vector of the vehicle body state.

[0028] Based on the extracted comprehensive feature vector, the deep learning model uses the trained classifier to judge the status of the two truck beds to determine the identification result of the transport vehicle;

[0029] When the identification result shows that the transport vehicle is full, the intelligent excavator sends a full-load signal to the transport vehicle through the wireless communication module, and at the same time sends a message to the dispatching system that the transport vehicle is ready to depart;

[0030] After receiving the full-load signal, the transport vehicle driver confirms departure and leaves the loading area; if the full-load signal is not received, the transport vehicle waits to continue loading.

[0031] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the receiving the work area information selected by the user on the terrain image data includes:

[0032] Receive information about multiple working areas selected by a user on the terrain image data.

[0033] A second aspect of an embodiment of the present invention discloses an equipment automation control system based on human-computer interaction, comprising:

[0034] Loading module: used to load terrain image data on the corresponding display interface in response to the user's terrain loading instruction, and receive the working area information selected by the user on the terrain image data;

[0035] Display module: used to determine the device information of each device in the current area according to the received device connection signal, and display the device information of each device;

[0036] a configuration module configured to configure a corresponding device combination for the work area information in response to a user's operation instruction on the terrain image data, and to associate the device combination with the work area information, wherein the device combination includes device information of each device;

[0037] Determination module: used to determine corresponding task planning information according to the work area information and equipment combination, and determine the excavation operation path of each intelligent excavator and the transportation path of each transportation equipment according to the task planning information.

[0038] A third aspect of an embodiment of the present invention discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the device automation control method based on human-computer interaction disclosed in the first aspect of the embodiment of the present invention.

[0039] A fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the device automation control method based on human-computer interaction disclosed in the first aspect of an embodiment of the present invention.

[0040] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0041] The human-machine interaction-based automated equipment control method of the present invention uses terrain image data and equipment information to perform task planning and accurately determine the excavation path of the intelligent excavator and the transportation path of the transport equipment. This enables the equipment to complete tasks more accurately and reduce errors and deviations during operation. For example, the intelligent excavator can accurately excavate according to the planned path, avoiding over-excavation or under-excavation, and the transport equipment can also follow the optimal path, improving transportation efficiency and accuracy.

[0042] The solution of the present invention displays terrain image data and equipment information in a graphical interface. Users can complete complex task configuration and planning through simple operating instructions. The operation is convenient and fast, and no professional technical knowledge and experience are required. It provides users with a good interactive experience and improves their work satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is a flow chart of a method for automatic device control based on human-computer interaction disclosed in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of a device list display disclosed in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of a specific process of full vehicle identification disclosed in an embodiment of the present invention;

[0047] Figure 4 This is a system architecture diagram of human-computer interaction disclosed in an embodiment of the present invention;

[0048] Figure 5 is a schematic diagram of a display page for human-computer interaction disclosed in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of a specific process of full vehicle identification disclosed in an embodiment of the present invention;

[0050] Figure 7 is a schematic diagram of the system architecture disclosed in an embodiment of the present invention;

[0051] Figure 8 This is a schematic structural diagram of an equipment automation control system based on human-computer interaction provided by an embodiment of the present invention;

[0052] Figure 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0054] It should be noted that the terms "first," "second," "third," "fourth," etc. in the description and claims of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having," as well as any variations thereof, in the embodiments of the present invention, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0055] At present, with the rapid development of the field of engineering machinery, the demand for automation of excavator operations is becoming increasingly prominent. At present, the traditional excavator operation mode mainly has the following problems: First, the interaction mode of the existing automated control system is complex, which requires professional training to master. It is not intuitive and friendly enough and difficult to get started quickly. Second, the coordination efficiency between the excavator and the transport vehicle is low. It often takes too long for the transport vehicle to wait for loading, or for the excavator to wait for unloading, resulting in low equipment utilization. Based on this, the embodiment of the present invention discloses an equipment automation control method, system, electronic device and storage medium based on human-computer interaction, which displays terrain image data and equipment information in a graphical interface. Users can complete complex task configuration and planning through simple operating instructions. The operation is convenient and fast, and no professional technical knowledge and experience are required. It provides users with a good interactive experience and improves user job satisfaction.

[0056] Example 1

[0057] See also Figure 1 , Figure 1It is a flow chart of the device automation control method based on human-computer interaction disclosed in the embodiment of the present invention. Among them, the execution subject of the method described in the embodiment of the present invention is an execution subject composed of software and / or hardware, and the execution subject can receive relevant information by wired or / and wireless means, and can send certain instructions. Of course, it can also have certain processing functions and storage functions. The execution subject can control multiple devices, such as a remote physical server or cloud server and related software, or a local host or server and related software that performs related operations on a device placed somewhere. In some scenarios, multiple storage devices can also be controlled, and the storage devices can be placed in the same place or different places as the devices. For example Figures 1 to 7 As shown, the device automation control method based on human-computer interaction includes the following steps:

[0058] S101: In response to a terrain loading instruction from a user, loading terrain image data on a corresponding display interface, and receiving information about a working area selected by the user on the terrain image data;

[0059] S102: Determine device information of each device in the current area according to the received device connection signal, and display the device information of each device;

[0060] S103: In response to a user's operation instruction on the terrain image data, configuring a corresponding device combination for the work area information, and associating the device combination with the work area information, wherein the device combination includes device information of each device;

[0061] S104: Determine corresponding task planning information according to the work area information and the equipment combination, and determine the excavation operation path of each intelligent excavator and the transportation path of each transportation equipment according to the task planning information.

[0062] This embodiment of the present invention rapidly loads terrain image data in response to terrain loading commands, allowing users to intuitively select work areas. This significantly reduces the time required to determine work areas compared to traditional manual planning methods. Furthermore, task planning information and equipment operation paths are automatically determined based on the work area and equipment combination, avoiding the potential errors and tediousness of manual planning and improving the efficiency of the entire workflow.

[0063] During implementation, device information is determined and displayed based on device connection signals, allowing users to gain real-time insights into the status and location of devices within the current area, facilitating unified device management and monitoring. When a device malfunctions or anomalies occur, users can quickly locate the location and take appropriate measures, minimizing downtime and improving device utilization. The combination of imaging and sensing enables objective and dynamic display.

[0064] Users can flexibly configure equipment combinations based on the needs of the work area and associate them with work area information. This ensures that equipment resources can be reasonably allocated in different work scenarios, so that the functions and performance of the equipment can be fully utilized, avoiding waste of equipment resources and reducing operating costs. This method automates many tasks that originally required manual completion, such as work area planning, equipment configuration, task planning, etc., through human-computer interaction, reducing human intervention, thereby reducing the risk of errors caused by human factors and improving the reliability and stability of the entire operation process. This equipment automation control method based on human-computer interaction has high flexibility and scalability. When new equipment or functions need to be added, only the corresponding configuration and programming in the system is required to bring the new equipment into the management and control scope, realizing smooth upgrades and expansions of the system to adapt to work scenarios of different scales and complexities.

[0065] The solution of the embodiment of the present invention simplifies human-machine interaction, improves operator efficiency, and reduces the operating costs of human operators, upgrading from previous one-to-one operation to one-to-many operation. Through simple interaction, the upgrade and application of automation technology is driven, truly freeing up human labor.

[0066] like Figure 4 As shown, the system of the present invention mainly includes the following modules: the human-computer interaction module includes a touch display interface, an equipment selection and task allocation operation interface, and a real-time operation status monitoring and feedback interface, and the touch display interface is used to display the topographic map of the operation area and the equipment status; the task planning module includes a work area identification and division unit, an excavation operation path planning, and a transport vehicle scheduling planning; the operation control module includes an excavator automatic control system, a transport vehicle collaborative control system, and an operation progress monitoring system; the data acquisition and processing module includes terrain data acquisition and processing, equipment status data acquisition, and operation efficiency data statistics.

[0067] The workflow of the present invention mainly includes:

[0068] During the initialization phase, the system starts, loads terrain data, waits for device connection, detects device status, and displays a list of available devices;

[0069] During the task allocation phase, the operator selects a specific excavator by point-and-click, selects a designated work area on a topographic map, and selects and allocates a transport vehicle;

[0070] During the automatic operation phase, the system automatically plans the excavation path, calculates the optimal parking position for transport vehicles, controls the excavator to dig according to the planned path, and coordinates the transport vehicles for loading and transportation;

[0071] During the monitoring and adjustment phase, we monitor the progress of the operation in real time, dynamically adjust the operation parameters, and handle abnormal situations.

[0072] like Figure 4 As shown, the system of the present invention adopts a distributed architecture, mainly including:

[0073] The central control server is divided into four core modules:

[0074] Global map management module: stores and updates the overall work area map; work area planning and management module: is responsible for work area division and status management; equipment coordination and task allocation module: coordinates and dispatches excavators and mining trucks; data storage and analysis module: stores and analyzes work data

[0075] The excavator control unit is divided into two major systems:

[0076] Sensor system: LiDAR for scene scanning and obstacle detection; GPS / IMU integrated navigation for precise positioning and attitude measurement; working condition sensor for monitoring the excavator's working status; HD camera for environmental perception and loading monitoring; attitude sensor for bucket position and attitude measurement; computing and control module for scene reconstruction and building a 3D map of the work area; path planning for planning the excavation path; motion control for controlling the excavation action; loading control for coordinating the loading process; and safety monitoring for real-time monitoring of operational safety.

[0077] The mining truck control unit is also divided into two major systems:

[0078] Sensor system: LiDAR for obstacle detection; GPS / IMU combined navigation for precise positioning and navigation; load sensor for monitoring loading status; surround-view camera for all-round environmental perception; vehicle posture sensor for monitoring vehicle status; computing and control module: positioning and navigation for real-time positioning and navigation; path planning for planning driving paths; docking control for precise docking position control; loading coordination for collaborative loading with the excavator; safety monitoring to ensure driving safety;

[0079] Communication connection: such as Figure 4 As shown, the solid line represents the communication with the central server (control instructions and data transmission) and the dotted line represents the direct communication between the excavator and the mining truck (loading coordination)

[0080] like Figure 5As shown in the figure, the layout of the main interface of human-computer interaction is that the equipment list and working status are displayed on the left, and the topographic map of the working area and work area information are displayed in the center. The interactive operation method is to click on the excavator or mining truck to select the equipment and drag to select the work area; for the work area, click to select the work area to view the working excavators and mining trucks, as well as the progress of the work; when carrying out specific implementation, new work areas can be created or existing work areas can be deleted; the status feedback method is color-coded to display the work progress, icons to identify the equipment status, and numerical values to display key parameters.

[0081] like Figure 2 and Figure 3 More preferably, the device information includes location information, working status information, and device type information; and the display of the device information of each device includes:

[0082] S1021: Determine all available device information in the current area based on the working status information of each device, generate a corresponding available device list based on the available device information, and display the available device list; or generate a corresponding display component based on the device type information of each device, wherein the display component is used to respond to various user operation instructions;

[0083] S1022: Determine the actual display position of each device in the terrain image data according to its corresponding position information, and display the corresponding display component.

[0084] This embodiment of the present invention uses operating status information to identify available devices and generate a list for display. This allows users to quickly understand which devices in the current area are immediately available for use, eliminating the tedious process of checking device status one by one. This facilitates rapid task allocation decisions and improves device scheduling efficiency. Display components are generated based on device type information, allowing users to quickly locate the required device type based on specific task requirements. This facilitates unified management and operation of similar devices, improving the targeted and convenient nature of device management.

[0085] The display component of the embodiment of the present invention can respond to various user operation instructions, providing users with an intuitive and convenient interaction method. Users do not need to memorize complex operation commands or processes. They can perform various operations on the device by simply interacting with the display component, such as clicking and dragging, which reduces the difficulty of operation and improves user work efficiency and satisfaction. The actual display position of the device is displayed in the terrain image data based on the location information, allowing users to clearly see the specific distribution of the device in the work area. This helps users better plan the operating range and path of the equipment and avoid mutual interference between devices. It also makes it convenient for users to quickly locate the location of the device when encountering problems and perform maintenance and management in a timely manner.

[0086] The embodiments of the present invention combine the location information, working status information and equipment type information of the equipment, so that users can consider various factors more comprehensively, thereby rationally arranging and assigning tasks to the equipment in the work area. For example, based on the location and availability of the equipment, tasks are assigned to equipment that is closer to the work area and currently available, so as to reduce the moving time and energy consumption of the equipment and improve the efficiency and quality of task execution. Accurately displaying the location of the equipment in the terrain image data and interacting with the operating instructions through the display component can ensure that users are more accurate in planning tasks and controlling equipment. Users can accurately formulate excavation operation paths and transportation paths based on the actual location and status of the equipment, avoiding task errors caused by unclear equipment location or incorrect operating instructions, and improving the accuracy and reliability of the entire operation process.

[0087] More preferably, the equipment type information includes an intelligent excavator and a transport device; the equipment combination includes an intelligent excavator and / or a transport device;

[0088] Determining corresponding task planning information according to the work area information and the equipment combination, and determining the excavation operation path of each intelligent excavator and the transportation path of each transportation equipment according to the task planning information, includes:

[0089] Based on the location information, working area information and preset working parameters of the selected intelligent excavator, a path planning algorithm is used to plan the optimal excavation path of the intelligent excavator;

[0090] The optimal docking position of each transport vehicle at the loading point is calculated based on the excavation position, work area information and transport vehicle information of the intelligent excavator, and the corresponding loading area is determined based on the optimal docking position; and the optimal docking position is sent to the corresponding transport vehicle, wherein the transport vehicle information includes vehicle size information.

[0091] The present invention utilizes a path planning algorithm to plan the optimal excavation path based on the intelligent excavator's location, work area, and operating parameters. This allows the excavator to dig in the most efficient manner, reducing wasted movement and digging time, increasing the amount of excavation per unit time, and improving overall work efficiency. Planning the path based on the work area and preset parameters ensures the excavator digs precisely as required, ensuring that the excavation depth and angle meet engineering standards, improving excavation quality, and reducing subsequent corrections.

[0092] The embodiment of the present invention calculates the optimal docking position and loading area based on the excavation position, work area, and transport vehicle information of the intelligent excavator, allowing the transport vehicle to dock accurately, facilitating rapid loading of the excavator, reducing waiting and adjustment time, optimizing the loading process, and improving loading efficiency. Taking vehicle size information into account to determine the optimal docking position and loading area can rationally utilize the work area space, avoid chaotic vehicle docking or excessive space occupation, improve space utilization, and ensure orderly operation of the work area. The optimal docking position is sent to the transport vehicle, enabling the coordinated operation of the intelligent excavator and transport equipment, enhancing the coordination and stability of the entire equipment automation control system, ensuring close connection between excavation, loading, and transportation, and improving overall work efficiency.

[0093] More preferably, the control method further includes:

[0094] In response to the operation instruction of the work area clicked by the user, information of the work area is displayed, wherein the displayed information includes a smart excavator list, a transport vehicle list, a work area list and a work status display component, and the work status display component includes progress information, remaining time information and completed work; the progress information is color-coded, and different progress is coded with different colors; the smart excavator list, transport vehicle list and work area list use charts to identify the equipment status, and the remaining time information and completed work are displayed using numerical values.

[0095] When the user clicks on the work area operation instruction, the intelligent excavator list, transport vehicle list, work area list and work status display component can be displayed, covering various information such as equipment and work status related to the work area. This allows users to fully understand the overall situation of the work area without switching between multiple interfaces or systems, thereby improving the efficiency and convenience of information acquisition.

[0096] In the work status display component, progress information is color-coded, with different colors corresponding to different progress levels. This allows users to quickly and intuitively judge work progress. For example, green indicates good progress, while red indicates lagging progress. This allows users to quickly identify problem areas and take timely measures to adjust work schedules. Remaining time and completed work are displayed numerically, providing accurate and quantified data. This allows users to more precisely understand the completion status and remaining workload, providing a reliable basis for subsequent resource allocation and task planning. This logical design approach can significantly improve user decision-making efficiency during specific design phases.

[0097] In this embodiment of the present invention, the intelligent excavator list, transport vehicle list, and work area list use charts to identify equipment status. This graphical representation of equipment status, such as operational and idle, is more intuitive than simple text descriptions. Users can quickly identify which equipment is operational and which is idle, allowing them to rationally arrange equipment usage, improve utilization, and avoid wasting equipment resources.

[0098] By displaying comprehensive, intuitive, and clear information about the work area, users can manage tasks more effectively. For example, they can adjust task assignments and optimize workflows based on work progress and equipment status. This information display provides rich and accurate information support for user decision-making, enabling them to make more scientific and rational decisions, improving overall work efficiency and quality. This information display method aligns with user operating habits and cognitive patterns, offering simple and convenient operation and intuitive and easy-to-understand information. This reduces user learning costs and operational difficulty, improves user satisfaction and user experience, and enhances interactivity and usability between users and the system.

[0099] More preferably, the automatic control method further includes:

[0100] S105: Control and coordinate the loading and transportation of the intelligent excavator and the transport vehicle; wherein the transport vehicle and the intelligent excavator communicate through a background server;

[0101] The control coordinates the loading and transportation of the intelligent excavator and the transport vehicle, including:

[0102] The loading characteristic information of the transport vehicle's bucket is obtained by an environmental parameter acquisition device installed in the intelligent excavator, and the loading characteristic information is matched with the preset loading conditions. If the set loading conditions are met, a corresponding action signal is sent to the transport vehicle to remind the driver to leave the loading area.

[0103] The environmental parameter acquisition device captures real-time loading characteristics of the transport vehicle's bucket and matches them with pre-set loading conditions, accurately determining whether loading is complete. Once the conditions are met, the driver is alerted to depart, avoiding overloading or underloading, reducing vehicle dwell time in the loading area, and enabling the intelligent excavator to quickly proceed to the next loading operation, thereby improving overall loading and transportation efficiency.

[0104] Transport vehicles and intelligent excavators communicate via a backend server, utilizing an environmental parameter collection device to achieve close collaboration. The intelligent excavator can adjust its operating procedures in real time based on the transport vehicle's loading status, and the transport vehicle can promptly respond to signals from the intelligent excavator. This streamlines the loading and transport process, reduces waiting time and idling between devices, and improves the collaborative capabilities of the entire production system.

[0105] This automated control method reduces the need for manual evaluation of loading conditions and vehicle departure instructions, reducing labor costs and the potential for human error. It also reduces the workload of operators, allowing them to focus more on other important tasks, improving overall efficiency.

[0106] More preferably, Figure 6 and Figure 7 As shown, the environmental parameter acquisition device provided on the intelligent excavator is used to obtain the loading characteristic information of the transport vehicle bucket, and the loading characteristic information is matched with the preset loading conditions. If the set loading conditions are met, a corresponding action signal is sent to the transport vehicle to remind the driver to leave the loading area, including:

[0107] S1051: Acquire three-dimensional point cloud data of the transport vehicle bucket using a laser radar installed on the intelligent excavator, and acquire image data of the transport vehicle bucket using an image acquisition device installed on the intelligent excavator;

[0108] S1052: Preprocessing the acquired 3D point cloud data and image data;

[0109] S1053: Inputting the preprocessed 3D point cloud data and image data into a deep learning model, wherein the deep learning model adopts an architecture that combines a convolutional neural network and a recurrent neural network, wherein the convolutional neural network is used to extract features from the image data, and the recurrent neural network is used to perform time series feature analysis on the 3D point cloud data; the feature information extracted by the convolutional neural network and the recurrent neural network is integrated into the deep learning model to obtain a comprehensive feature vector of the truck bed state;

[0110] S1054: Based on the extracted comprehensive feature vector, the deep learning model uses the trained classifier to judge the status of the two truck beds of the transport vehicle to determine the transport vehicle identification result;

[0111] S1055: When the identification result indicates that the transport vehicle is fully loaded, the intelligent excavator sends a full load signal to the transport vehicle through the wireless communication module, and simultaneously sends a message to the dispatching system that the transport vehicle is ready to depart;

[0112] S1056: After receiving the full-load signal, the transport vehicle driver confirms departure and leaves the loading area; if the full-load signal is not received, the transport vehicle waits for further loading.

[0113] The lidar in this embodiment of the present invention can acquire high-precision three-dimensional point cloud data from the cargo bed of a transport vehicle, accurately measuring the volume and shape of the cargo within. It also features strong anti-interference capabilities and wide adaptability, operating stably even in complex outdoor environments or under varying lighting conditions. The image acquisition device provides visual image data of the cargo bed, supplementing it with detailed information such as the cargo's appearance and texture. The combination of the two provides a comprehensive and accurate representation of the cargo bed's loading characteristics.

[0114] Preprocessing of 3D point cloud data and image data can remove noise, correct data deviations, etc., improve the quality and reliability of the data, provide a good data foundation for subsequent analysis and processing, and reduce the possibility of errors and misjudgments.

[0115] The deep learning model combines convolutional neural networks with recurrent neural networks. Convolutional neural networks excel at extracting features from image data, capturing spatial features such as the surface texture and edge shape of cargo. Recurrent neural networks analyze the temporal characteristics of 3D point cloud data, processing dynamic changes during the loading process. By integrating the features extracted from these two methods, a more comprehensive and accurate feature vector of the cargo bed state is generated, enabling deeper mining of useful information within the data and a better description of the loading status of the cargo bed.

[0116] Based on comprehensive feature vectors, the trained classifier determines the status of the transport vehicle bucket, accurately identifying whether it is full. Compared to manual judgment or simple sensor detection, this method offers higher accuracy and reliability, reducing the probability of misjudgments and missed detections. When the intelligent excavator automatically transmits a full-load signal to the transport vehicle via the wireless communication module and sends a ready-to-depart message to the dispatch system. Upon receiving the signal, the transport vehicle driver confirms departure. This entire process eliminates the need for frequent human intervention, enabling automated collaborative operation between the intelligent excavator, transport vehicle, and dispatch system. This improves the efficiency of the loading and transport links, reduces waiting time, and enhances overall production efficiency.

[0117] In the deep learning model, a convolutional neural network extracts features from image data, effectively capturing visual features such as texture and color. A recurrent neural network analyzes temporal features of 3D point cloud data, exploring temporal patterns of point cloud data change. The combination of these two methods comprehensively extracts feature information about the truck bed status from different perspectives, making the extracted comprehensive feature vector more representative and discriminative, thereby improving the accuracy and reliability of the deep learning model's truck bed status assessment. This combination of laser and image data, and the corresponding deep learning model architecture, allows for better adaptation to a variety of complex real-world loading scenarios. For example, truck bed shapes and sizes vary across different types of transport vehicles, as do the types and placement of loads, and lighting conditions and environmental backgrounds can also vary. By integrating the feature information from lidar and image data, the model can more accurately learn the characteristic patterns of truck bed filling in different scenarios, improving the model's generalization and robustness, enabling accurate assessment of truck bed status in a variety of complex situations. This is a key demonstration of the innovative nature of this technical solution.

[0118] Based on the extracted comprehensive feature vector, the deep learning model uses the trained classifier to judge the status of the mining truck bucket and determine whether the bucket is full.

[0119] The classifier training process involves using a large amount of sample data labeled with full and half car states for training, continuously adjusting model parameters, and improving recognition accuracy.

[0120] The solution of the embodiment of the present invention can achieve the following technical effects:

[0121] Improve transportation efficiency: By accurately identifying the full status of mining trucks, the situation of transporting incomplete trucks is avoided, the number of transport trips is reduced, and the transportation volume per unit time is increased.

[0122] Improving loading accuracy: The application of deep learning models improves the accuracy of full truck identification, ensuring that mining trucks can be loaded to the appropriate fullness and optimizing resource utilization.

[0123] Enhanced collaboration between excavators and trucks: Real-time communication and collaborative control between excavators and trucks enable a close connection between loading and transportation, reducing waiting time and improving operational fluidity.

[0124] Reduce labor costs: Reduce dependence on manual visual observation, reduce labor costs, and avoid losses caused by human judgment errors.

[0125] More preferably, the receiving of the work area information selected by the user on the terrain image data includes:

[0126] Receive information about multiple working areas selected by a user on the terrain image data.

[0127] Users can select multiple work areas on the terrain image data based on their actual needs, enabling differentiated equipment configuration and task planning based on the characteristics and task requirements of each area. For example, a large construction site may have different construction phases or different types of construction tasks. By selecting multiple work areas, appropriate intelligent excavators and transport equipment can be assigned to each area, achieving more precise and flexible construction scheduling.

[0128] Equipment combinations can be independently configured and optimized for different work areas. Areas with heavy workloads or higher construction difficulty can be allocated more intelligent excavators and transport equipment, while smaller or simpler areas can be allocated less equipment. This allows for the rational allocation of resources based on the specific needs of each work area, avoiding equipment waste or shortages and improving equipment utilization and production efficiency.

[0129] In practice, image recognition can be used to determine the state of the excavation area, and the image can be dynamically updated based on the comprehensive information obtained. The image recognition object can be directly the image data in the terrain data. By setting corresponding color display rules, the image can be combined for comprehensive recognition, improving the overall coordination and processing capabilities.

[0130] The solution of the embodiment of the present invention updates the data of the equipment every preset time, controls and coordinates the loading and transportation of the transport vehicle and the intelligent excavator. During the specific implementation, the working efficiency of the intelligent excavator can also be determined based on the number of loading times of the intelligent excavator.

[0131] If the working efficiency of the intelligent excavator is detected to be lower than the set value within a certain time range, the user will be reminded to optimize the control;

[0132] During implementation, equipment adjustments can be made dynamically across multiple work areas. This embodiment of the present invention uses two work areas as an example. Specifically, excavation and transportation tasks are rationally allocated based on the ore reserves, mining progress, and equipment capacity of the two work areas, avoiding a situation where one work area is overloaded while the other has idle resources. For example, if work area A has large ore reserves but low excavation efficiency, the number of transport vehicles allocated there can be appropriately reduced, with some vehicles being transferred to work area B. Meanwhile, the excavator operating time in work area A can be increased, or more excavators can be deployed.

[0133] The location, terrain, road conditions, and unloading point of the two work areas can be comprehensively considered to plan the optimal global route for the transport vehicles. For example, if there is a shortcut between the two work areas, but the road is narrow and only suitable for small vehicles, small transport vehicles can be arranged to make short-distance transfers between the two work areas, while large vehicles can travel on the main road to improve overall transportation efficiency.

[0134] Dynamically adjust transport vehicle routes based on real-time traffic conditions and emergencies such as equipment failures. For example, if the regular route from work area A to the unloading point is impassable due to road collapse, the system will automatically plan a temporary route for transport vehicles from work area A to the unloading point via work area B.

[0135] During implementation, transport vehicles can be rationally allocated based on the workload and transport capacity of the two work areas. Mathematical models or intelligent algorithms, such as genetic algorithms or simulated annealing, can be used to calculate the number and type of vehicles required for each work area to maximize transport efficiency. For example, work areas close to the unloading point can be assigned small, flexible vehicles that are easy to navigate narrow mine roads; while work areas farther away can be assigned larger transport vehicles to reduce the number of trips.

[0136] The human-machine interaction-based automated equipment control method of the present invention uses terrain image data and equipment information to perform task planning and accurately determine the excavation path of the intelligent excavator and the transportation path of the transport equipment. This enables the equipment to complete tasks more accurately and reduce errors and deviations during operation. For example, the intelligent excavator can accurately excavate according to the planned path, avoiding over-excavation or under-excavation, and the transport equipment can also follow the optimal path, improving transportation efficiency and accuracy.

[0137] The solution of the present invention displays terrain image data and equipment information in a graphical interface. Users can complete complex task configuration and planning through simple operating instructions. The operation is convenient and fast, and no professional technical knowledge and experience are required. It provides users with a good interactive experience and improves their work satisfaction.

[0138] Example 2

[0139] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of the equipment automation control system based on human-computer interaction disclosed in the embodiment of the present invention. Figure 8 As shown, the equipment automation control system based on human-computer interaction may include:

[0140] Loading module 21: for responding to a terrain loading instruction from a user, loading terrain image data on a corresponding display interface, and receiving information about a working area selected by the user on the terrain image data;

[0141] Display module 22: used to determine the device information of each device in the current area according to the received device connection signal, and display the device information of each device;

[0142] Configuration module 23: configured to configure a corresponding device combination for the work area information in response to a user's operation instruction on the terrain image data, and associate the device combination with the work area information, wherein the device combination includes device information of each device;

[0143] The determination module 24 is configured to determine corresponding task planning information according to the work area information and the equipment combination, and determine the excavation operation path of each intelligent excavator and the transportation path of each transportation equipment according to the task planning information.

[0144] The human-machine interaction-based automated equipment control method of the present invention uses terrain image data and equipment information to perform task planning and accurately determine the excavation path of the intelligent excavator and the transportation path of the transport equipment. This enables the equipment to complete tasks more accurately and reduce errors and deviations during operation. For example, the intelligent excavator can accurately excavate according to the planned path, avoiding over-excavation or under-excavation, and the transport equipment can also follow the optimal path, improving transportation efficiency and accuracy.

[0145] The solution of the present invention displays terrain image data and equipment information in a graphical interface. Users can complete complex task configuration and planning through simple operating instructions. The operation is convenient and fast, and no professional technical knowledge and experience are required. It provides users with a good interactive experience and improves their work satisfaction.

[0146] Example 3

[0147] See also Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. The electronic device can be a computer, a server, etc. Of course, in certain circumstances, it can also be a smart device such as a mobile phone, a tablet computer, and a monitoring terminal, as well as an image acquisition device with processing functions. Figure 9 As shown, the electronic device may include:

[0148] A memory 510 storing executable program code;

[0149] a processor 520 coupled to the memory 510;

[0150] The processor 520 calls the executable program code stored in the memory 510 to execute part or all of the steps in the device automation control method based on human-computer interaction in the first embodiment.

[0151] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute some or all of the steps in the device automation control method based on human-computer interaction in the first embodiment.

[0152] An embodiment of the present invention further discloses a computer program product, wherein when the computer program product is run on a computer, the computer is caused to execute some or all of the steps in the device automation control method based on human-computer interaction in the first embodiment.

[0153] An embodiment of the present invention also discloses an application publishing platform, wherein the application publishing platform is used to publish a computer program product. When the computer program product runs on a computer, the computer executes some or all of the steps in the device automation control method based on human-computer interaction in embodiment one.

[0154] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the processes does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0155] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of this embodiment.

[0156] In addition, the functional units in the embodiments of the present invention may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The integrated unit may be implemented in the form of hardware or software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a memory and includes several requests for causing a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the method described in each embodiment of the present invention.

[0158] In the embodiments provided herein, it should be understood that "B corresponding to A" means that B is associated with A and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0159] Those skilled in the art will appreciate that some or all of the steps in the various methods of the embodiments may be performed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0160] The above is a detailed introduction to the device automation control method, system, electronic device and storage medium based on human-computer interaction disclosed in the embodiments of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A device automation control method based on human-computer interaction, characterized in that: include: In response to a terrain loading instruction from a user, loading terrain image data on a corresponding display interface, and receiving information about a working area selected by the user on the terrain image data; Determine the device information of each device in the current area based on the received device connection signal, and display the device information of each device; In response to a user's operation instruction on the terrain image data, configuring a corresponding device combination for the work area information, and associating the device combination with the work area information, wherein the device combination includes device information of each device; Corresponding task planning information is determined according to the work area information and the equipment combination, and the excavation operation path of each intelligent excavator and the transportation path of each transportation equipment are determined according to the task planning information.

2. The device automation control method based on human-computer interaction according to claim 1, characterized in that: The device information includes location information, working status information and device type information; The display of device information of each device includes: Determining all available device information in the current area based on the working status information of each device, generating a corresponding available device list based on the available device information, and displaying the available device list; or generating a corresponding display component based on the device type information of each device, wherein the display component is used to respond to various user operation instructions; The actual display position of each device in the terrain image data is determined according to the corresponding position information of each device, and the corresponding display component is displayed.

3. The device automation control method based on human-computer interaction according to claim 2, characterized in that: The equipment type information includes an intelligent excavator and a transport device; the equipment combination includes an intelligent excavator and / or a transport device; The determining of corresponding task planning information according to the work area information and the equipment combination, and determining the excavation operation path of each intelligent excavator and the transportation path of each transportation equipment according to the task planning information, includes: Based on the location information, working area information and preset working parameters of the selected intelligent excavator, a path planning algorithm is used to plan the optimal excavation path of the intelligent excavator; The optimal docking position of each transport vehicle at the loading point is calculated based on the excavation position, work area information and transport vehicle information of the intelligent excavator, and the corresponding loading area is determined based on the optimal docking position; and the optimal docking position is sent to the corresponding transport vehicle, wherein the transport vehicle information includes vehicle size information.

4. The device automation control method based on human-computer interaction according to claim 2, characterized in that: The control method further includes: In response to the operation instruction of the work area clicked by the user, information of the work area is displayed, wherein the displayed information includes a smart excavator list, a transport vehicle list, a work area list and a work status display component, and the work status display component includes progress information, remaining time information and completed work; the progress information is color-coded, and different progress is coded with different colors; the smart excavator list, transport vehicle list and work area list use charts to identify the equipment status, and the remaining time information and completed work are displayed using numerical values.

5. The device automation control method based on human-computer interaction according to claim 1, characterized in that: The automated control method further includes: Control and coordinate the loading and transportation of the intelligent excavator and the transport vehicle; wherein the transport vehicle and the intelligent excavator communicate through the background server; The control coordinates the loading and transportation of the intelligent excavator and the transport vehicle, including: The loading characteristic information of the transport vehicle's bucket is obtained by an environmental parameter acquisition device installed in the intelligent excavator, and the loading characteristic information is matched with the preset loading conditions. If the set loading conditions are met, a corresponding action signal is sent to the transport vehicle to remind the driver to leave the loading area.

6. The device automation control method based on human-computer interaction according to claim 5, characterized in that: The environmental parameter acquisition device provided on the intelligent excavator is used to obtain the loading characteristic information of the transport vehicle bucket, and the loading characteristic information is matched with the preset loading conditions. If the preset loading conditions are met, a corresponding action signal is sent to the transport vehicle to remind the driver to leave the loading area, including: Acquiring three-dimensional point cloud data of the transport vehicle bucket by using a laser radar installed on the intelligent excavator and acquiring image data of the transport vehicle bucket by using an image acquisition device installed on the intelligent excavator; Perform preprocessing operations on the acquired three-dimensional point cloud data and image data; The preprocessed 3D point cloud data and image data are input into a deep learning model. The deep learning model uses an architecture that combines a convolutional neural network with a recurrent neural network. The convolutional neural network is used to extract features from the image data, and the recurrent neural network is used to perform time series feature analysis on the 3D point cloud data. The feature information extracted by the convolutional neural network and the recurrent neural network is integrated into the deep learning model to obtain a comprehensive feature vector of the vehicle body state. Based on the extracted comprehensive feature vector, the deep learning model uses the trained classifier to judge the status of the two truck beds to determine the identification result of the transport vehicle; When the identification result shows that the transport vehicle is full, the intelligent excavator sends a full-load signal to the transport vehicle through the wireless communication module, and at the same time sends a message to the dispatching system that the transport vehicle is ready to depart; After receiving the full-load signal, the transport vehicle driver confirms departure and leaves the loading area; if the full-load signal is not received, the transport vehicle waits to continue loading.

7. The device automation control method based on human-computer interaction according to claim 1, characterized in that: The receiving the work area information selected by the user on the terrain image data includes: Receive information about multiple working areas selected by a user on the terrain image data.

8. An equipment automation control system based on human-computer interaction, characterized in that: include: Loading module: used to load terrain image data on the corresponding display interface in response to the user's terrain loading instruction, and receive the working area information selected by the user on the terrain image data; Display module: used to determine the device information of each device in the current area according to the received device connection signal, and display the device information of each device; a configuration module configured to configure a corresponding device combination for the work area information in response to a user's operation instruction on the terrain image data, and to associate the device combination with the work area information, wherein the device combination includes device information of each device; Determination module: used to determine corresponding task planning information according to the work area information and equipment combination, and determine the excavation operation path of each intelligent excavator and the transportation path of each transportation equipment according to the task planning information.

9. An electronic device, characterized in that: include: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the equipment automation control method based on human-computer interaction according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the device automation control method based on human-computer interaction according to any one of claims 1 to 7.

Images