Intelligent Self-Loading and Unloading Management Method, Device, Electronic Device and Storage Medium

Abstract

The present invention discloses an intelligent self-loading and unloading management method, device, electronic device and storage medium, which are used to solve the problem of low loading and unloading efficiency in the current self-loading and unloading related technologies. Obtain the images of goods to be loaded and unloaded, the loading and unloading logs of the goods, and the monitoring videos of the self-loading and unloading platform; perform goods information matching according to the images of the goods to be loaded and unloaded and the loading and unloading logs of the goods to obtain the goods attribute information of each good, and further generate the loading and unloading priority of each good in combination with the loading and unloading logs of the goods; perform loading and unloading path recognition on the monitoring videos, and construct a dynamic loading and unloading model; perform available loading and unloading path analysis according to the dynamic loading and unloading model, optimize the loading and unloading time sequence decision based on each loading and unloading priority, and construct a time sequence optimization path for goods loading and unloading; perform multi-task collaborative control calculation based on the time sequence optimization path, obtain multi-task collaborative real-time control parameters and operation efficiency values, and perform dynamic loading and unloading management optimization according to the real-time control parameters and operation efficiency values to construct an intelligent loading and unloading management model.

Description

Technical Field

[0001] The present invention relates to the technical field of handling management, and particularly to an intelligent self-handling management method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of modern industrial automation technology, intelligent self-handling platforms are increasingly widely used in various production lines and logistics systems, becoming an important tool for improving production efficiency, reducing labor costs, and optimizing resource allocation. The self-handling platform can complete the handling tasks of items through an automated system, realizing the efficient flow and collaborative operation between equipment and items. Especially in the fields of high-speed logistics, warehouse management, and robot applications, it has broad application prospects.

[0003] However, with the complexity of the platform operation environment and the diversity of item types, traditional static item handling management methods often cannot cope with complex dynamic requirements. These methods are usually based on fixed paths, preset tasks, or simple rules, and cannot respond in real time to platform environment changes, item state fluctuations, and external interferences. As a result, the handling efficiency is low, and problems such as operation errors and item damage may even occur. Especially in the case where multiple items need to be scheduled in real time, it is difficult to handle the dynamic relationships and scheduling conflicts between items using traditional methods, and the accuracy and timeliness of cargo handling cannot be effectively guaranteed.

[0004] To address the above challenges, there is an urgent need for a more intelligent item handling management solution. Summary of the Invention

[0005] The present invention provides an intelligent self-handling management method, device, electronic device and storage medium, which are used to solve or partially solve the problems that current self-handling related technologies cannot respond in real time to platform environment changes, item state fluctuations, and external interferences, resulting in low handling efficiency and even problems such as operation errors and item damage.

[0006] The present invention provides an intelligent self-handling management method, and the method includes:

[0007] Obtain the images of goods to be handled, the goods handling logs, and the monitoring videos of the self-handling platform;

[0008] Perform edge recognition-based goods information matching according to the images of goods to be handled and the goods handling logs to obtain the goods attribute information of each good;

[0009] Combine the goods handling logs and the goods attribute information of each good to calculate the dynamic handling priority based on handling risk assessment, and generate the handling priority of each good;

[0010] Identify the handling path of the monitoring video and construct a dynamic handling model;

[0011] Perform an available loading and unloading path analysis according to the dynamic loading and unloading model, and optimize the loading and unloading time sequence decision based on each loading and unloading priority to construct an optimized time sequence path for cargo loading and unloading.

[0012] Perform multi-task collaborative control calculation based on the optimized time sequence path to obtain real-time control parameters and operation efficiency values for multi-task collaboration, and optimize dynamic loading and unloading management according to the real-time control parameters and the operation efficiency values to construct an intelligent loading and unloading management model, which is used for real-time decision-making and management of cargo loading and unloading.

[0013] The present invention also provides an intelligent self-loading and unloading management device, including:

[0014] A data acquisition unit for acquiring images of goods to be loaded and unloaded, cargo loading and unloading logs, and monitoring videos of the self-loading and unloading platform;

[0015] A cargo information matching unit for performing cargo information matching based on edge recognition according to the images of goods to be loaded and unloaded and the cargo loading and unloading logs to obtain the cargo attribute information of each cargo;

[0016] A loading and unloading priority generation unit for performing dynamic loading and unloading priority calculation based on loading and unloading risk assessment by combining the cargo loading and unloading logs and the cargo attribute information of each cargo to generate the loading and unloading priority of each cargo;

[0017] A dynamic loading and unloading model construction unit for identifying the loading and unloading path of the monitoring video and constructing a dynamic loading and unloading model;

[0018] A loading and unloading path planning unit for performing an available loading and unloading path analysis according to the dynamic loading and unloading model, and optimizing the loading and unloading time sequence decision based on each loading and unloading priority to construct an optimized time sequence path for cargo loading and unloading;

[0019] An intelligent loading and unloading management unit for performing multi-task collaborative control calculation based on the optimized time sequence path to obtain real-time control parameters and operation efficiency values for multi-task collaboration, and optimizing dynamic loading and unloading management according to the real-time control parameters and the operation efficiency values to construct an intelligent loading and unloading management model, which is used for real-time decision-making and management of cargo loading and unloading.

[0020] The present invention also provides an electronic device, which includes a processor and a memory:

[0021] The memory is used for storing program codes and transmitting the program codes to the processor;

[0022] The processor is used for executing the intelligent self-loading and unloading management method as described in any one of the above according to the instructions in the program codes.

[0023] The present invention also provides a computer-readable storage medium for storing program code for executing the intelligent self-loading and unloading management method described in any one of the above.

[0024] As can be seen from the above technical solutions, the present invention has the following advantages:

[0025] An intelligent self-loading and unloading management method is provided. The first step: Obtain the images of goods to be loaded and unloaded, the loading and unloading logs of the goods, and the monitoring videos of the self-loading and unloading platform; perform edge-recognition-based goods information matching according to the images of goods to be loaded and unloaded and the loading and unloading logs of the goods to obtain the goods attribute information of each good. Thus, through image edge segmentation technology, the shape and appearance features of each good can be accurately extracted, eliminating the risk of misrecognition. The precise comparison with the loading and unloading logs can ensure that the goods attribute information obtained by the system is consistent with the actual goods, thereby providing accurate basic data for subsequent operations. The second step: Combine the loading and unloading logs of the goods and the goods attribute information of each good to calculate the dynamic loading and unloading priorities based on the loading and unloading risk assessment, and generate the loading and unloading priorities of each good. Thus, by comprehensively evaluating the physical characteristics of the goods, the system can automatically identify high-risk goods, ensure special treatment measures are taken during loading and unloading, and avoid breakage and accidents. As the environment, platform load, and operation conditions change, the system can adjust the loading and unloading priorities in real time. Some goods need to be processed preferentially under high-load conditions, and the system will dynamically adjust the task priorities according to real-time feedback, improving the overall operation efficiency and flexibility of the platform. The third step: Identify the loading and unloading paths from the monitoring videos and construct a dynamic loading and unloading model. Thus, based on the constructed digital twin model, the system continuously optimizes and adjusts the operation strategies according to the real-time feedback data to ensure the smooth progress of the entire loading and unloading process. In subsequent processing flows, the system can analyze the optimal loading and unloading paths of each good according to the loading and unloading priorities of the goods. The fourth step: Analyze the available loading and unloading paths according to the dynamic loading and unloading model, and optimize the decision-making of the loading and unloading time sequence based on each loading and unloading priority to construct an optimized time sequence path for the loading and unloading of the goods. Thus, through the dynamic optimization path planning of the loading and unloading of the goods, conflicts and intersections between paths can be avoided, ensuring that the robotic arm and other equipment can smoothly execute tasks during subsequent loading and unloading. The fifth step: Perform multi-task collaborative control calculation based on the optimized time sequence path to obtain the real-time control parameters and operation efficiency values of multi-task collaboration. Thus, by comprehensively considering multiple factors such as the loading and unloading priorities of the goods, path planning, and load status, the system can not only complete tasks in the shortest time, significantly improving the overall efficiency and accuracy of the loading and unloading operations, but also coordinate the collaborative work of multiple robotic arms in different loading and unloading tasks, avoiding conflicts, waiting, and ineffective operations between the robotic arms. The sixth step: Optimize the dynamic loading and unloading management according to the real-time control parameters and operation efficiency values, and construct an intelligent loading and unloading management model for real-time decision-making and management of the loading and unloading of the goods. Thus, through the dynamic optimization based on the efficiency values, the system can adjust the operation strategies according to the feedback data of each operation, construct an intelligent loading and unloading management model, so as to facilitate the system to make better real-time decisions and management for the loading and unloading of the goods. Description of the Drawings

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0027] Figure 1 It is a flowchart of the steps of an intelligent self-loading and unloading management method;

[0028] Figure 2 It is a flowchart of the steps of matching cargo information based on edge recognition;

[0029] Figure 3 It is a flowchart of the steps of calculating dynamic loading and unloading priorities based on loading and unloading risk assessment;

[0030] Figure 4 It is a flowchart of the steps of identifying the loading and unloading path;

[0031] Figure 5 It is a flowchart of the steps of constructing an optimized path for the time sequence of cargo loading and unloading;

[0032] Figure 6 It is a flowchart of the steps of multi-task collaborative control calculation;

[0033] Figure 7 It is a flowchart of the steps of optimizing dynamic loading and unloading management;

[0034] Figure 8 It is a structural block diagram of an intelligent self-loading and unloading management device. Specific embodiments

[0035] The embodiments of the present invention provide an intelligent self-loading and unloading management method, device, electronic device and storage medium, which are used to solve or partially solve the problems that the current self-loading and unloading related technologies cannot respond to the changes in the platform environment, the fluctuations in the item status and external interferences in real time, resulting in low loading and unloading efficiency, and even operation errors, item damage and other problems.

[0036] To make the invention purpose, features and advantages of the present invention more obvious and understandable, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0037] As an example, with the increasing complexity of the platform operating environment and the diversity of item types, traditional static item loading and unloading management methods often fail to cope with complex dynamic requirements. These methods are usually based on fixed paths, preset tasks, or simple rules, and are unable to respond in real time to changes in the platform environment, fluctuations in item status, and external interferences. As a result, loading and unloading efficiency is low, and problems such as operation errors and item damage may occur. Especially in the case of real-time scheduling of multiple items, it is difficult to handle the dynamic relationships and scheduling conflicts between items using traditional methods, and the accuracy and timeliness of cargo loading and unloading cannot be effectively guaranteed.

[0038] Therefore, one of the core inventive points of the embodiments of the present invention lies in: aiming at the deficiencies of the current technology, a method for intelligent cargo loading and unloading management is proposed. (1) Through image edge segmentation technology, the shape and appearance features of each cargo can be accurately extracted, eliminating the risk of misidentification. The accurate comparison with the loading and unloading log can ensure that the cargo attribute information obtained by the system is consistent with the actual cargo, thus providing accurate basic data for subsequent operations. (2) By comprehensively evaluating the physical characteristics of the cargo, the system can automatically identify high-risk cargo and ensure that special handling measures are taken during loading and unloading to avoid breakage and accidents. As the environment, platform load, and operation conditions change, the system can adjust the loading and unloading priority in real time. When certain cargo needs to be processed first under high load conditions, the system will dynamically adjust the task priority according to real-time feedback to improve the overall operation efficiency and flexibility of the platform. (3) Based on the constructed dynamic twin model, the system continuously optimizes and adjusts the operation strategy according to real-time feedback data to ensure the smooth progress of the entire loading and unloading process. In subsequent processing flows, the system can analyze the best loading and unloading path for each cargo according to the loading and unloading priority of the cargo. (4) Through dynamic optimization path planning for cargo loading and unloading, conflicts and intersections between paths can be avoided, ensuring that the robotic arm and other equipment can smoothly execute tasks during subsequent loading and unloading. (5) By comprehensively considering multiple factors such as the loading and unloading priority of the cargo, path planning, and load status, the system can not only complete tasks in the shortest time, significantly improving the overall efficiency and accuracy of loading and unloading operations, but also coordinate the collaborative work of multiple robotic arms in different loading and unloading tasks, avoiding conflicts, waiting, and ineffective operations between robotic arms. (6) Through dynamic optimization based on efficiency values, the system can adjust the operation strategy according to the feedback data of each operation, constructing an intelligent loading and unloading management model to facilitate the system's better real-time decision-making and management of cargo loading and unloading.

[0039] The intelligent self-loading and unloading management method proposed by the present invention is applicable to, including but not limited to, mechanical equipment equipped with a self-loading and unloading platform, data processing platforms, cloud server nodes, and network uploading devices. These can all be regarded as general computing nodes of the present invention. The data processing platform includes but is not limited to at least one of an audio and image management system, an information management system, and a cloud data management system.

[0040] Refer to Figure 1 , which shows the step flow chart of an intelligent self-loading and unloading management method provided by an embodiment of the present invention. Specifically, it may include the following steps:

[0041] Step 101, obtain the images of goods to be loaded and unloaded, the goods loading and unloading log, and the monitoring video of the self-loading and unloading platform;

[0042] A high-resolution camera (such as a DSLR (Digital Single Lens Reflex) or an industrial camera) can be used to collect the images of goods to be loaded and unloaded. Set the resolution to at least 1920x1080 pixels to make the details of the collected images clearly distinguishable. Select a suitable lens to obtain clear images at different shooting distances and angles. A wide-angle lens can be used to cover a larger field of view. For the purpose of comprehensively covering the loading and unloading area, cameras are arranged in the goods loading and unloading area. At the same time, to make the illumination uniform and avoid the influence of shadows and reflections on the image quality, shooting parameters such as aperture, shutter speed, and ISO (the sensitivity setting of the camera sensor to light) can be set.

[0043] Record the goods loading and unloading log, including information such as goods ID, loading and unloading time, operators, loading and unloading methods (such as manual or mechanical). And use a spreadsheet system (such as Excel) or dedicated logistics management software for recording. By recording, the log information can be made consistent with the time stamp of the images for subsequent matching and analysis.

[0044] In terms of the collection of the monitoring video, a suitable high-definition industrial camera can be selected. Such as a camera with a resolution of 1920x1080. This camera supports real-time video recording at 30 frames per second to capture clear and smooth images. At the same time, the selected camera should have good low-light performance to meet the monitoring requirements under different lighting conditions. Install cameras at key positions on the self-loading and unloading platform to cover the entire loading and unloading area. The field of view of the camera should be able to capture all the loading and unloading paths and platform components. Set the angle and height of the camera according to the actual site to avoid blind spots. To ensure the stability of the monitoring video, an anti-shake bracket and a protective cover can be used for protection. Obtain the monitoring video stream in real time through video acquisition software (such as OpenCV or dedicated industrial camera software). And set the video encoding format (such as H.264) to optimize the storage and transmission efficiency. Monitor the frame rate and resolution of the real-time video to stably obtain high-quality video streams under different conditions. To further ensure the security and accessibility of the data, the real-time video stream can be saved to a local server or cloud storage and backed up regularly to prevent data loss. At the same time, a storage policy can also be set. For example, save a video segment every hour for subsequent analysis and processing.

[0045] Step 102: Perform cargo information matching based on edge recognition according to the to-be-loaded / unloaded cargo image and the cargo loading / unloading log to obtain the cargo attribute information of each cargo.

[0046] In some embodiments, in combination with Figure 2 the step flowchart shown, the implementation process of performing cargo information matching based on edge recognition according to the to-be-loaded / unloaded cargo image and the cargo loading / unloading log to obtain the cargo attribute information of each cargo can be achieved by executing the following sub-steps S01 to S05:

[0047] Step S01: Segment the edge contours of the to-be-loaded / unloaded cargo image to extract the edge contour lines of each cargo.

[0048] In this step, the OpenCV library is used to process the collected image. First, convert the color image to a grayscale image to simplify subsequent edge detection processing. For example, it can be achieved by cv2.cvtColor(image, cv2.COLOR_BGR2GRAY). Apply Gaussian blur (such as a 5×5 Gaussian kernel) to reduce the influence of image noise. It can be processed by cv2.GaussianBlur(gray_image, (5, 5), 0).

[0049] Use the Canny edge detection algorithm to extract the edges of the cargo. Set low and high thresholds (such as 50 and 150) to detect the edges in the image respectively. Specifically, it can be achieved by cv2.Canny(blurred_image, 50, 150). Through edge detection, a binary image can be generated. Among them, the edge part is white and the other parts are black.

[0050] Step S02: Perform geometric shape analysis on each edge contour line to generate the geometric shape features of each cargo.

[0051] Use the cv2.findContours function to extract the edge contours and return the contour list and hierarchical information. In the case of only extracting external contours, the contour retrieval mode can be set to cv2.RETR_EXTERNAL. Process each extracted contour, calculate its area and perimeter to filter out small noise contours. Specifically, for the edge contour of each cargo, cv2.contourArea can be used to calculate its area, and cv2.arcLength can be used to calculate the perimeter.

[0052] Use the cv2.moments function to obtain the moments of the contour, and calculate the centroid position and shape features (such as rectangularity, roundness, etc.). Then, the extracted geometric shape features (such as area, perimeter, centroid coordinates, rectangularity, roundness, etc.) can be stored in a dictionary or data frame for subsequent analysis and matching.

[0053] Step S03: Perform cargo surface texture recognition on the image of the cargo to be loaded and unloaded, and extract the cargo surface texture features;

[0054] To ensure data consistency, set the unit for each geometric feature (e.g., area is calculated in square centimeters). Use the Local Binary Pattern (LBP) or Gray-Level Co-occurrence Matrix (GLCM) method for texture feature extraction. Select appropriate parameters (such as neighborhood radius, direction) to extract effective texture information. For example, use the cv2.calcHist function in OpenCV to calculate the histogram of the image to obtain the texture distribution characteristics. For each cargo image, apply the gray-level co-occurrence matrix to calculate texture features (such as contrast, correlation, energy, and uniformity, etc.). Exemplarily, set the directions to 0, 45, 90, and 135 degrees to calculate texture features in multiple directions. Store the extracted cargo surface texture features in the same data structure as the geometric features for subsequent processing.

[0055] Step S04: Perform depth vision recognition based on each geometric feature and cargo surface texture feature respectively to generate the depth vision representation of each cargo;

[0056] Select a deep learning model for vision recognition. For example, a commonly used Convolutional Neural Network (CNN) can be selected. For model training, set the input layer as the combination of the extracted geometric features and cargo surface texture features, and the output layer as the class label of the cargo. Combine the geometric features and cargo surface texture features of each cargo into a feature vector, and together with the corresponding label, form a training dataset. To ensure the balance of the number of samples in each category, training parameters can be set, such as batch size (32) and learning rate (0.001), and use the cross-entropy loss function for training. During the training process, the performance of the model can be monitored through the validation set, and the accuracy and loss values can be recorded. Use a confusion matrix to evaluate the performance of the model on different categories.

[0057] Input the geometric features and cargo surface texture features extracted in the previous steps into the trained model for depth vision recognition, and the depth vision representation of each cargo can be output.

[0058] Thus, by processing the cargo image through the edge contour segmentation algorithm, the shape, size, and edge features of each cargo can be accurately recognized, ensuring the accuracy and reliability of image processing. In this way, the physical characteristics of the cargo are extracted in real time, providing accurate data for subsequent task scheduling and path planning.

[0059] Step S05: Match the cargo information with the cargo handling log according to each depth visual representation to obtain the cargo attribute information of each cargo.

[0060] Perform cargo information matching by using the depth visual representation (such as feature vector) and the information in the cargo handling log. For example, use measurement methods such as cosine similarity or Euclidean distance to evaluate the similarity between feature vectors. Set a matching threshold (such as 0.8), and determine that the matching is successful when the similarity between the two is higher than the matching threshold. Record the successfully matched cargo attribute information (such as cargo ID, type, weight, size, etc.) in the database for subsequent query and analysis. In addition, a matching report can also be generated to record the correspondence between the depth visual representation of each cargo and the information in the handling log.

[0061] Thus, through the matching of the cargo handling log and combined with the attribute information of each cargo (such as weight, fragility, etc.), the system can comprehensively understand the characteristics of the items. This provides basic data support for subsequent risk assessment, priority ranking, and handling path planning, and avoids handling errors or operation accidents caused by misidentification or data loss.

[0062] Step 103, calculate the dynamic handling priority based on the handling risk assessment by combining the cargo handling log and each piece of cargo attribute information, and generate the handling priority of each cargo;

[0063] In some embodiments, in combination with Figure 3 the shown step flow chart, the implementation process of calculating the dynamic handling priority based on the handling risk assessment by combining the cargo handling log and each piece of cargo attribute information to generate the handling priority of each cargo can be implemented by performing the following sub-steps S11 to S15:

[0064] Step S11: Perform handling risk assessment on each piece of cargo attribute information respectively to generate the handling risk assessment value of each cargo;

[0065] Identify the key factors affecting the risk during the handling process. These include cargo type, weight, volume, handling method (such as manual or mechanical), environmental conditions (such as weather, temperature), etc. Classify these factors and set the weight of each factor. For example, weight and volume have a greater impact on risk and are given higher weights (such as 0.4 and 0.3). While environmental conditions are given a lower weight (such as 0.2). Set risk assessment indicators according to the identified risk factors.

[0066] Extract the required data from the goods attribute information, including goods type, weight, volume, etc., and format them into computable numerical values. Use data processing tools to clean and transform the data to ensure its accuracy and consistency. Apply the set risk assessment model to each good, calculate the handling risk assessment value for each good, and store the results in the database for subsequent query and analysis.

[0067] Step S12: Mine the goods flow direction based on the goods handling log to generate the flow direction data for each good;

[0068] Extract relevant data from the goods handling log (such as goods ID, goods type, handling time, handling method, etc.). Through data cleaning and transformation (such as duplicate removal, formatting), ensure the accuracy and integrity of the data. Set a time window (such as 24 hours) to analyze the goods flow direction within this time period. Use techniques such as frequency analysis and path analysis to extract the patterns of the goods flow direction. Statistically analyze the frequency of a specific good arriving at another location from a certain location within a specific time period. Store the generated flow direction data in the database for subsequent query and analysis.

[0069] Step S13: Calculate the final placement position for each good respectively according to each flow direction data to generate the final placement position for each good;

[0070] Identify the factors affecting the final placement position of the goods, including flow direction data, warehouse layout, goods type, handling sequence, etc. Set the weight of each factor to consider the influence of different factors in the calculation. Select a suitable algorithm for position calculation. For example, rule-based algorithms and optimization algorithms (such as genetic algorithms or particle swarm optimization) can be used.

[0071] Set the objective function, such as minimizing the handling time and path length. Input the flow direction data and other relevant factors into the position calculation model, ensuring that all data formats are consistent for calculation. Use, for example, the SciPy library in Python or other optimization tools to calculate the placement position. After generating the final placement position for each good, the results can be stored in the database for subsequent use.

[0072] Step S14: Conduct real-time platform handling load analysis based on the goods handling log to obtain real-time handling load data;

[0073] Set up a visualization tool to display the placement positions of goods to assist operators in understanding and adjustment. Define the calculation criteria for handling loads, including the quantity, weight, and volume of goods handled each time, etc. According to the information in the goods handling log, set the time window for real-time load analysis (such as using one hour as the time window). Extract the relevant data for each handling from the goods handling log (such as goods ID, goods type, goods weight, quantity of goods, etc.), and improve the data accuracy through data cleaning and filtering. Use data processing tools to calculate the total load within each time window. And generate real-time handling load data based on the calculation results, including information such as the current load, maximum load, and remaining load, etc.

[0074] Step S15: Dynamically calculate the handling priorities for the handling risk assessment values and final placement positions of each good according to the real-time handling load data, and generate the handling priority for each good.

[0075] An alarm mechanism can be set up. When the real-time load approaches or exceeds the maximum load, immediately notify the relevant personnel. First, determine the factors affecting the handling priority, including the handling risk assessment value, the flow direction of goods, the real-time handling load data, and the placement position of goods, etc. Set the weight for each factor to ensure that the priority calculation can reflect the actual situation. Generate the handling priority for each good according to the influencing factor weighting method and sort them by priority. For example, if the influencing factor weight values of a certain good's handling risk assessment value, flow direction of goods, real-time handling load data, and placement position are w1, w2, w3, and w4 respectively, then the calculation result of its handling priority is w1 + w2 + w3 + w4. The larger the value, the higher the handling priority. Store the calculated results in the database for subsequent handling operations. During the actual handling process, the handling priority information calculated can be used to guide the operators in handling, so that high-risk goods and key goods can be given priority treatment.

[0076] Thus, through the dynamic analysis of the attribute information of each good (such as fragile, heavy, large volume, etc.), the handling risk of each good can be intelligently evaluated. High-risk items (such as fragile items, heavy goods, etc.) will be marked as high priority to ensure that these items can receive priority attention and treatment during handling to avoid damage or accidents. Combining the real-time load and the platform operating conditions, the system can dynamically calculate and adjust the handling priority of goods. In the embodiments of the present invention, the handling priority of items not only depends on the preset static rules but also is intelligently adjusted according to real-time data to ensure the optimal scheduling of resources and efficient operation.

[0077] Step 104, identify the handling path from the monitoring video and construct a dynamic handling model;

[0078] In some embodiments, in combination with Figure 4The following step flowchart can identify the loading and unloading paths of the monitoring video and construct the implementation process of the dynamic loading and unloading model, which can be achieved by executing the following sub-steps S21 to S25:

[0079] Step S21: Perform visual recognition of the loading and unloading paths on the monitoring video to obtain multiple loading and unloading paths;

[0080] First, preprocess the obtained monitoring video, including noise removal, image enhancement, and grayscale conversion, to improve the accuracy of subsequent analysis. Through Gaussian blur and histogram equalization techniques in OpenCV. Process the video frame by frame, extract and analyze each frame of the image, and set the processing time for each frame of the image (such as 100 ms) to ensure real-time performance.

[0081] Based on each frame of the image extracted, identify the loading and unloading paths through computer vision algorithms (such as Hough transform or Canny edge detection). Specifically, by detecting the straight lines and edges in the image, extract multiple loading and unloading paths. Set relevant parameters (such as line length and angle threshold) to ensure that the detected paths meet the requirements of actual loading and unloading operations. Mark the multiple identified loading and unloading paths and store them as a path data structure, including information such as the starting point, ending point, and path length of the path. Subsequently, a path visualization graph can be generated based on the loading and unloading paths to help operators understand and confirm the recognition results.

[0082] Thus, through the monitoring video data, the platform can obtain the changes in the working environment in real time, including the position of obstacles, the working space, and the path planning. And based on the visual recognition technology, the system accurately identifies and analyzes the loading and unloading paths, ensuring the correctness of path selection and avoiding misoperations caused by visual obstacles or environmental changes.

[0083] Step S22: Analyze the platform components in the monitoring video to obtain multiple platform components;

[0084] Use a deep learning model (such as YOLO (You Only Look Once, a target detection algorithm) or SSD (Single Shot MultiBox Detector, a single-shot multi-box detector / target detection algorithm)) to identify and extract the platform components. Specifically, select a pre-trained model to improve the recognition efficiency and perform fine-tuning to adapt to the specific application scenario. Set the detection threshold (such as 0.5) to ensure that the identified platform components have a certain confidence level. For each identified platform component, extract its feature information, including component type, size, position, and status, etc. Extract the edge information of the component through image processing (such as contour detection) and store it in a structured database for subsequent query and analysis.

[0085] Step S23: Monitor path obstacles based on multiple platform components and mark each path obstacle node;

[0086] Define path obstacles, including any objects that obstruct the loading and unloading path, such as unmarked goods, tools, various platform components, or other equipment, etc. Use image segmentation (such as GrabCut or deep learning segmentation models) to identify path obstacles in video frames. Mark nodes for each identified path obstacle and record its position and size information in the video. Store the marked obstacle information in the database, including the coordinates, type, and relevant loading and unloading paths of the obstacles.

[0087] Step S24: Locate the spatial position of each of the path obstacle nodes respectively to generate spatial position coordinates;

[0088] Use computer vision techniques (such as stereo vision or depth cameras) to locate the spatial position of obstacles. Combine camera calibration at the same time to ensure the accuracy of position detection. Specifically, set the internal and external parameters of the camera to optimize the spatial positioning process. For each path obstacle node, calculate its position coordinates (X, Y, Z) in three-dimensional space. Predict the obstacle position through triangulation or deep learning models. Store the calculated spatial position coordinates in the database for subsequent analysis and calculation.

[0089] Step S25: Perform three-dimensional topological modeling based on multiple loading and unloading paths and each spatial position coordinate to construct a dynamic loading and unloading model.

[0090] Select a suitable three-dimensional modeling tool to construct a twin model of the dynamic loading and unloading platform (simply referred to as the dynamic loading and unloading model in this invention). The three-dimensional modeling tool used needs to support importing coordinate data and path information. Generate a three-dimensional model through a point cloud-based modeling method, combining the spatial position coordinates of path obstacle nodes and multiple loading and unloading path information.

[0091] Import the loading and unloading path information and the spatial position coordinates of path obstacle nodes into the three-dimensional modeling tool to generate the three-dimensional topological structure of the dynamic loading and unloading platform. Optimize the generated three-dimensional model to ensure that it can reflect the dynamic changes of the platform in real time (such as the goods loading and unloading status, path changes, etc.). At the same time, through model verification means (such as comparing the spatial data measured on-site with the spatial data of the model), ensure that the model accurately reflects the layout and status of the actual loading and unloading platform. Apply the dynamic loading and unloading model to the intelligent loading and unloading management system, which can provide support for loading and unloading decision-making and management.

[0092] Specifically, the implementation process of performing three-dimensional topological modeling based on multiple loading and unloading paths and each spatial position coordinate to construct a dynamic loading and unloading model can be achieved by executing the following sub-steps S251 to S259:

[0093] Step S251: Identify the current location of the goods placement;

[0094] Use a high-definition camera and a Light Detection and Ranging (LiDAR) sensor to collect real-time images and depth data of the goods placement location. Among them, to ensure capturing real-time goods information, the camera resolution can be set to 1920x1080, and the sampling frequency of the LiDAR can be set to 10 Hz. Calibrate the camera to ensure its coordinate system is consistent with that of the LiDAR for subsequent data fusion.

[0095] Identify the edges of the goods through image processing algorithms (such as edge detection and contour extraction), and perform 3D positioning by combining depth information. Specifically, the Canny edge detection in OpenCV and the cv2.findContours function can be used to extract the contours of the goods. Combine the image data and the point cloud data to generate the 3D coordinates of the goods, record the placement location of each good, and store it in the database.

[0096] Step S252: Calculate the platform operation space based on the current goods placement location and each spatial position coordinate to obtain the platform operation space characteristics;

[0097] Define the operation space as the area that the loading and unloading robotic arm can reach when performing tasks. Consider the goods placement location and the obstacle location to construct the boundary of the operation space. Set the calculation of the operation space based on the 3D coordinates of the goods (i.e., the current goods placement location) and the obstacle coordinates (i.e., the spatial position coordinates of the path obstacle nodes), and determine the effective area of the space through geometric calculation methods. Use a space partitioning algorithm (such as the Voronoi diagram) to calculate the operation space.

[0098] First, establish a spatial model based on the coordinates of the goods and the obstacles, and calculate the reachable area. Set calculation parameters, such as the minimum separation distance, to ensure that the robotic arm maintains a safe distance from the obstacles during movement. Then extract the platform operation space characteristics, including the volume of the effective operation area, the boundary shape, and reachability, etc. Store the extracted characteristics in the database for subsequent analysis and application.

[0099] Step S253: Analyze the spatial layout of multiple platform components to generate the spatial layout data of the platform components;

[0100] Determine the spatial layout characteristics of platform components, including component types, dimensions, relative positions, and layout methods. Platform components include robotic arms, sensors, cargo storage areas, etc. Set a reference coordinate system for layout analysis to facilitate subsequent spatial calculations. Process the spatial layout data of platform components through a structured data analysis tool. Store the dimensions and positions of each platform component as a data frame for subsequent analysis. Adopt a rule-based analysis method to check the rationality of the platform component layout, including spacing and arrangement. Generate the spatial layout data of each platform component, including the starting position, bounding box, and occupied space of the platform component. Store this data in a database for subsequent query and analysis.

[0101] Step S254: Conduct a three-dimensional topological structure analysis of the platform based on the spatial layout data to obtain three-dimensional topological structure characteristics;

[0102] Define the three-dimensional topological structure as the spatial relationship between platform components, including relative position, distance, and connection method. Set the analysis of the three-dimensional topological structure based on the spatial layout data of platform components. The basic concepts of graph theory can be adopted, regarding platform components as nodes, and the connecting lines between nodes represent mutual relationships. Adopt graph algorithms (such as Dijkstra or A* algorithm) to analyze the connection relationships between platform components, and calculate the optimal path and relative position relationships.

[0103] Set a minimum connection distance to ensure reasonable connections between platform components. Extract the three-dimensional topological structure characteristics, including the number of nodes, number of edges, connectivity, and network density, etc. Store the three-dimensional topological structure characteristics in a database for subsequent analysis and query. In addition, the topological structure can also be visually displayed through a visualization tool to help operators understand the structural relationships.

[0104] Step S255: Obtain the state parameters of the loading and unloading robotic arm of the self-loading and unloading platform;

[0105] Install a variety of sensors on the robotic arm, including position sensors, force sensors, and angle sensors, to monitor the state parameters of the robotic arm in real time. Specifically, set the sensor sampling frequency (such as 10 times per second) to ensure the accuracy and timeliness of the state data. Obtain the sensor data through a data acquisition system (such as a PLC (Programmable Logic Controller) or a single-chip microcomputer), and record the state parameters of the robotic arm in real time, including position, speed, acceleration, and load, etc. Store the collected data in a database for subsequent analysis.

[0106] Step S256: Conduct an operating state analysis of the state parameters of the loading and unloading robotic arm to obtain the operating state characteristics of the robotic arm;

[0107] Analyze the operating state of the robotic arm to identify its characteristics such as working efficiency, load conditions, and motion trajectories. Set analysis metrics, including the working cycle, average load, and motion smoothness. Extract key features from the state parameters as the operating state features of the robotic arm, such as average speed, maximum load, operating time, and cycle frequency. Calculate these features through statistical analysis methods. At the same time, graphical tools can be used to visualize these data to help identify the changing trends of the state features.

[0108] Step S257: Calculate the loading and unloading activity range of the robotic arm based on the operating state features of the robotic arm, and extract the loading and unloading activity range of the robotic arm;

[0109] Based on the extracted operating state features of the robotic arm, classify the operating state of the robotic arm through a machine learning model (such as Support Vector Machine (SVM) or Random Forest) to identify states such as normal, overloaded, and faulty. Set the accuracy threshold of the model (such as 85%) to ensure the reliability of the classification results.

[0110] Determine the loading and unloading activity range of the robotic arm, including the spatial area where the robotic arm can operate effectively. Consider the working radius and swing angle of the robotic arm. Determine the geometric shape of the activity range according to the model and motion parameters of the robotic arm (such as joint angles, telescopic lengths, etc.), usually a cone or a sphere. To ensure the accuracy of the calculation results, the kinematic model of the robotic arm (such as the Denavit-Hartenberg parameter method) can be used for the calculation of the activity range. Set the calculation accuracy (such as 0.01 meters) to ensure the detail level of the activity range. Display the activity range of the robotic arm through a 3D visualization tool to help the operator understand and adjust the working area of the robotic arm.

[0111] Step S258: Perform 3D point cloud modeling on the 3D topological structure features according to the platform operation space features to construct a 3D loading and unloading platform model;

[0112] Generate a 3D model of the operating range to facilitate subsequent dynamic rendering and analysis. Specifically, according to the operating environment of the platform (such as the position of obstacles and the spatial layout) and the operating range of the robotic arm, extract the characteristics of the operating space, and identify the operable area and the restricted area. Set a feature extraction algorithm (such as the Random Sample Consensus (RANSAC) algorithm) to analyze the point cloud data in the operating space. To ensure the density of the point cloud data (such as at least 10 points per square meter) to guarantee the accuracy of the model, a lidar or a depth camera can be used to obtain the point cloud data of the operating space. Then, filter and denoise the point cloud data, and use the filtering algorithm in the Point Cloud Library (PCL) to remove the outlier points. Build a 3D loading and unloading platform model based on the processed point cloud data. At the same time, apply the Delaunay triangulation or Poisson reconstruction algorithm to generate the surface model.

[0113] Step S259: According to the operating range of the robotic arm for loading and unloading and multiple loading and unloading paths, perform dynamic loading and unloading behavior rendering on the 3D loading and unloading platform model to build a dynamic loading and unloading model.

[0114] In this step, by refining and optimizing the model, ensure the authenticity and operability of the 3D loading and unloading platform model. Specifically, an appropriate dynamic rendering tool can be selected to simulate and render the loading and unloading behavior. The selected dynamic rendering tool needs to support physical simulation and real-time rendering. Use a physics engine to perform dynamic simulation of the model to ensure the realism of the loading and unloading behavior. According to the operating range of the robotic arm for loading and unloading and multiple loading and unloading paths, model the loading and unloading behavior of the robotic arm, including actions such as grasping, moving, and releasing. Set action parameters (such as speed, acceleration, etc.) to ensure the smoothness and coherence of the simulation results. Integrate the dynamic behavior of the robotic arm with the 3D loading and unloading platform model, and display the loading and unloading process through real-time simulation. At the same time, record the key data (such as time, energy consumption, etc.) during the loading and unloading process to provide a basis for subsequent optimization.

[0115] Thus, by building a dynamic twin model of the self-loading and unloading platform, the virtual model can be synchronized with the actual operating platform in real time, reflecting the actual operating state of the platform. This twin model provides high-precision and real-time updated virtual environment data for subsequent path planning, task scheduling, equipment coordination, etc., further improving the decision-making accuracy and reaction speed of the system.

[0116] Step 105, perform available loading and unloading path analysis according to the dynamic loading and unloading model, and make an optimization decision on the loading and unloading timing sequence based on each loading and unloading priority to build an optimized timing path for cargo loading and unloading;

[0117] In some embodiments, in combination with Figure 5The shown step flowchart analyzes the available loading and unloading paths according to the dynamic loading and unloading model, optimizes the loading and unloading time sequence decision based on each loading and unloading priority, and constructs the implementation process of the time sequence optimization path for cargo loading and unloading, which can be realized by executing the following sub-steps S31 to S35:

[0118] Step S31: Analyze the available loading and unloading paths for each cargo one by one according to the dynamic loading and unloading model, and generate the available loading and unloading paths for each cargo;

[0119] According to the previously constructed dynamic loading and unloading model, it can accurately reflect the real situation of the loading and unloading environment, including cargo, robotic arm and path information. First, determine the path analysis algorithm, such as the A* algorithm or Dijkstra algorithm. These algorithms can effectively find the shortest path and consider obstacles. For each cargo, use the selected path algorithm to analyze the available loading and unloading paths from the current placement position to the target placement position.

[0120] Set the cost function of the path, including factors such as time, distance and obstacle influence. In path calculation, unreasonable paths can be excluded by setting thresholds (such as the maximum allowable path length). Mark the available loading and unloading paths of each cargo found and store them in the database, including information such as the starting point, ending point, passed nodes and path cost of the path. At the same time, a visualization graph can be generated to help operators understand the available loading and unloading paths of each cargo.

[0121] Step S32: Conduct cross-prediction of cargo transportation for each available loading and unloading path, and mark the loading and unloading path intersections;

[0122] Define the intersections, including the nodes where the loading and unloading paths of different cargos intersect. The identification of intersections is crucial for optimizing the loading and unloading process. The intersection detection algorithm in graph theory, such as the edge crossing detection method based on the graph, can be used to detect the intersections of the available loading and unloading paths of each cargo, record the intersection nodes between the paths, and calculate the influence degree of the intersections (such as traffic flow). Set the marking rules for intersections. For example, mark when there are multiple paths at the intersection and the traffic flow exceeds the set threshold. Store the marked intersection information in the database, including the coordinates of the intersections, the involved paths and the cross-traffic flow, etc. At the same time, a visualization graph can be generated to identify all intersections to help operators consider potential cross-conflicts when making decisions.

[0123] Step S33: Conduct potential path obstacle detection for each available loading and unloading path, and identify potential path obstacles;

[0124] Use computer vision technology (such as deep learning models) to detect obstacles in the available loading and unloading paths. For example, through object detection algorithms such as YOLO or SSD, identify potential obstacles in the available loading and unloading paths. To ensure that the identified obstacles have sufficient credibility, the detection accuracy can be set (such as 95%). Conduct real-time video analysis on the available loading and unloading paths of each cargo to detect obstacles existing on the paths. Set a time window (such as processing 10 frames per second) to improve real-time performance. Mark the detected obstacles, including information such as the type, size, and location of the obstacles. Store the information of the identified potential path obstacles in a database for subsequent analysis and processing.

[0125] Step S34: Conduct available path avoidance analysis based on the potential path obstacles and the intersection points of the loading and unloading paths to generate an avoidance path for each cargo;

[0126] Set avoidance strategies, including dynamic path adjustment and static obstacle avoidance. An improved version of the A* algorithm can be adopted, considering the influence of obstacles and intersection points on the path at the same time. Specifically, set the parameters for calculating the avoidance path, such as the minimum safety distance and the maximum turning angle, to ensure the safety and feasibility of the avoidance path. After detecting potential obstacles and intersection points, use the path replanning algorithm to generate a new avoidance path. Ensure that the new path does not conflict with obstacles and minimize the path cost. For each cargo, generate an avoidance path according to its specific available loading and unloading path and mark it. Store the information of the generated avoidance path in the database, including information such as the starting point, ending point, passing nodes, and avoidance cost of the path.

[0127] Step S35: Make loading and unloading timing optimization decisions for each avoidance path according to each loading and unloading priority respectively, and construct an optimized loading and unloading timing path for the cargo.

[0128] Generate a visualization graph to identify the avoidance path. Optimize the timing of the avoidance path for each cargo according to its loading and unloading priority. The timing optimization strategy is: consider both the loading and unloading priority of the cargo and the avoidance path (such as minimizing the avoidance cost and minimizing the passing nodes), and ensure that high-priority cargo is loaded and unloaded first under the lowest risk. Generate an optimized loading and unloading timing path through a scheduling algorithm (such as shortest job first or priority scheduling) to ensure the efficiency and safety of the operation.

[0129] The order and timing optimization of cargo loading and unloading are crucial. In this step, based on the loading and unloading priority of the cargo, optimize the analysis of the loading and unloading path of each cargo, and propose multiple feasible path plans. By evaluating the intersection points, obstacles, and resource load conditions between different paths, the system accurately selects the optimal loading and unloading path, thereby improving the loading and unloading efficiency and avoiding resource waste.

[0130] Step 106: Based on the timing optimization path, perform multi-task collaborative control calculation to obtain real-time control parameters and operation efficiency values for multi-task collaboration, and optimize dynamic loading and unloading management according to the real-time control parameters and the operation efficiency values, and construct an intelligent loading and unloading management model, where the intelligent loading and unloading management model is used for real-time decision-making and management of cargo loading and unloading.

[0131] In some embodiments, in combination with Figure 6 the step flowchart shown, the implementation process of performing multi-task collaborative control calculation based on the timing optimization path to obtain real-time control parameters and operation efficiency values for multi-task collaboration can be realized by executing the following sub-steps S41 to S45:

[0132] Step S41: According to the timing optimization path, perform collaborative operation control of the platform robotic arm, and collect real-time control parameters for multi-task collaboration;

[0133] In the robotic arm control system, a multi-task collaborative control architecture can be designed first. Specifically, to ensure that multiple robotic arms can perform collaborative operations according to the loading and unloading timing optimization path, real-time control algorithms (such as PID control (Proportional-Integral-Derivative Control) or fuzzy control) can be used for precise control. Set control parameters, such as the control frequency (e.g., 50Hz), to achieve real-time response. Construct a task scheduling mechanism so that each robotic arm can perform task allocation according to priority and available paths. Adopt a priority-based scheduling algorithm (such as EDF scheduling (Earliest Deadline First)) to arrange loading and unloading tasks. Set scheduling parameters (such as task response time and maximum waiting time) to improve system efficiency.

[0134] During the collaborative control process, collect real-time control parameters of the robotic arm, including position, speed, acceleration, load, temperature, etc. And record the data through a data acquisition system. Configure the data acquisition frequency to ensure real-time performance (such as 100 times per second). Store the collected real-time control parameters in a database using a structured format (such as JSON or CSV) for subsequent analysis. In addition, a data backup strategy can be set to ensure the security and integrity of real-time data.

[0135] Step S42: Calculate the loading and unloading execution time for the real-time control parameters to generate the loading and unloading execution time for each cargo;

[0136] Define the loading and unloading execution time, including the total time from the start to the completion of the robotic arm's loading and unloading operations. During the calculation of the loading and unloading execution time, factors such as action time, waiting time, and delays need to be considered. Extract relevant data from the real-time control parameters and calculate the loading and unloading execution time for each cargo. By setting a threshold (such as the maximum allowable execution time), the validity of the data can be ensured. At the same time, store the calculated loading and unloading execution time in the database, including the ID of each cargo, the execution time, and related parameters.

[0137] Step S43: Obtain the operation simulation data of multi-task collaboration and conduct statistics on loading and unloading conflicts for the operation simulation data to identify the time points of loading and unloading conflicts.

[0138] Define loading and unloading conflicts, including multiple robotic arms simultaneously attempting to access the same loading and unloading path or operating on the same cargo at the same time. Through an event-driven model, monitor the status of loading and unloading tasks to identify potential conflicts. By setting the operation simulation data, count the time points when conflicts occur and set a conflict identification algorithm (such as time series analysis) to monitor the status changes of tasks. Set a conflict detection time window (such as once per second) to ensure that conflict events can be captured in a timely manner. Record the identified time points of loading and unloading conflicts in the database, including conflict time, information on the involved cargo and robotic arms. Generate a visualization graph to display the time series of conflict occurrences to assist operators in analyzing the causes of conflicts.

[0139] Step S44: Conduct mining on the distribution of conflict paths for the time points of loading and unloading conflicts to obtain the distribution data of loading and unloading conflict paths.

[0140] Determine the definition of conflict paths, including the paths between conflict time points, information on the robotic arms and cargo involved in the conflicts. Conduct path distribution analysis through data mining algorithms (such as clustering analysis or association rule algorithms). Extract relevant information from the data of loading and unloading conflict time points and analyze the distribution of conflict paths. Set the parameters of the mining algorithm (such as minimum support and confidence). Count the occurrence frequency of each conflict path and identify high-frequency conflict paths. Store the mined distribution data of loading and unloading conflict paths in the database, including path information, frequency, and information on related cargo and robotic arms. Generate a visualization graph to display the distribution of conflict paths to help optimize path planning.

[0141] Step S45: Based on the distribution data of loading and unloading conflict paths and each loading and unloading execution time, conduct a quantitative calculation of collaboration efficiency to generate an operation efficiency value for multi-task collaboration.

[0142] Define the efficiency of multi-task collaborative operation, including factors such as loading and unloading execution time, conflict path distribution, and task completion rate. Specifically, it can be set as: E(collab) = T(exec) + C(path) / T(total). Among them, E(collab) is the collaborative efficiency (the operation efficiency value of multi-task collaboration), T(total) is the total task time, C(path) is the influence coefficient of the conflict path, and T(exec) is the loading and unloading execution time. Using data analysis tools, by integrating the loading and unloading execution time and the loading and unloading conflict path distribution data, calculate the multi-task collaborative operation efficiency value of each cargo. Set the efficiency evaluation criteria (such as an efficiency value greater than 0.8 is considered efficient) to ensure the reliability of the evaluation results. Store the calculated operation efficiency value of multi-task collaboration in the database, including the ID, efficiency value, and related parameters of each cargo.

[0143] By calculating the loading and unloading time sequence of each cargo and dynamically adjusting according to priorities, path conflicts, robotic arm workload, etc., the system can generate an efficient and safe loading and unloading time sequence for each cargo, thereby improving the overall operation efficiency and shortening the operation cycle. In the case of simultaneous execution of multiple tasks, the system can coordinate the working order and load distribution of each robotic arm to avoid conflicts or excessive waiting between robotic arms. Through multi-task collaborative control, the actions of each robotic arm can be synchronized with the actions of other robotic arms to ensure maximum operation efficiency. And through real-time monitoring and quantitative calculation, the system can evaluate and generate a multi-task collaborative operation efficiency value based on indicators such as the working efficiency and task completion degree of the robotic arm. This efficiency value provides a basis for subsequent optimization and adjustment, helping the platform continuously improve the effect of collaborative operations.

[0144] In some embodiments, in combination with Figure 7 the shown step flow chart, perform dynamic loading and unloading management optimization according to real-time control parameters and operation efficiency values, and the implementation process of constructing an intelligent loading and unloading management model can be achieved by executing the following sub-steps S51 to S54:

[0145] Step S51: Predict loading and unloading failures according to real-time control parameters and generate loading and unloading failure prediction data;

[0146] In practical applications, a prediction model can be pre-trained for loading and unloading failure prediction.

[0147] First, collect multiple historical control parameters of the robotic arm during multi-task collaboration. To ensure the stability of model training, it is necessary to preprocess the collected data. This includes denoising, standardization, and missing value processing to obtain a dataset for model training. Select a suitable fault prediction model, such as a machine learning-based classification model (e.g., random forest, support vector machine) or a deep learning model (e.g., Long Short-Term Memory (LSTM)) for time series prediction. For the selected model, hyperparameters can be set (e.g., the number of decision trees in the random forest, the depth of the long short-term memory network, etc.) to optimize the model performance. Divide the prepared dataset into a training set and a test set according to actual needs (e.g., 70% for training and 30% for testing). Use the training set to train the model and evaluate the prediction accuracy of the trained model through the test set (the prediction target is above 95%). Cross-validation (e.g., k-fold cross-validation) can also be used to further improve the robustness and accuracy of the model to ensure that the model can effectively identify potential faults.

[0148] Based on the trained loading and unloading fault prediction model, perform loading and unloading fault prediction on real-time control parameters to generate loading and unloading fault prediction data. The output prediction results include the predicted fault type, occurrence probability, and recommended preventive measures. Store the prediction results in the database for subsequent analysis and fault traceability positioning.

[0149] Step S52: Perform fault traceability positioning on the loading and unloading fault prediction data to obtain the fault traceability points from the loading and unloading platform;

[0150] Define the standards and processes for fault traceability, which can specifically include identifying the causes of faults from the prediction data. Fault traceability positioning can be achieved through Fault Tree Analysis (FTA) or a causal relationship model. Set the key parameters for traceability, such as the time window and scope of influence of the fault occurrence, to ensure the effectiveness of the traceability process.

[0151] According to the loading and unloading fault prediction data, analyze the real-time parameters related to the fault, identify the time point of the fault occurrence and the affected equipment. Then, based on a pre-set risk threshold (e.g., the fault occurrence probability is greater than 0.7), filter out the high-risk parameters and equipment. Record the determined fault traceability points in the database, including the traced parameters, occurrence time, related equipment, and predicted fault type. In addition, a traceability report can also be generated to help operators understand the root cause of the fault and provide a basis for subsequent fault diagnosis.

[0152] Step S53: Perform immediate fault diagnosis on the fault traceability points to obtain the immediate fault diagnosis results;

[0153] Select appropriate fault diagnosis means based on actual requirements. Common methods include rule-based expert systems, model-based diagnosis, and data-driven methods. Set the standards and processes for fault diagnosis to ensure the accuracy and effectiveness of instant fault diagnosis. Conduct a detailed analysis of the real-time parameters at the fault traceability points. Specifically, identify abnormal behaviors through time series analysis or anomaly detection algorithms (such as Isolation Forest). To make better judgments, a fault diagnosis standard can be set in advance. For example, when the load exceeds the rated value or the temperature of the robotic arm is abnormal, it is marked as a fault. Generate instant fault diagnosis results based on the analysis results, including the fault type, impact degree, and recommended repair measures.

[0154] Thus, by monitoring the platform operation status and the robotic arm movements in real time, the system can predict potential fault risks and issue early warnings. Through fault prediction, the platform makes adjustments before problems occur, reducing the occurrence of sudden faults and avoiding affecting the overall operation progress.

[0155] Step S54: Based on the operation efficiency value and the instant fault diagnosis results, perform iterative optimization of dynamic loading and unloading management, and construct an intelligent loading and unloading management model.

[0156] Design an intelligent loading and unloading management model. Combining the operation efficiency value of multi-task collaboration and the instant fault diagnosis results, set optimization goals (such as minimizing the loading and unloading time, reducing the failure rate, etc.). Specifically, system dynamics models or optimization algorithms (such as genetic algorithms, particle swarm optimization) can be used to design dynamic management strategies. Verify the intelligent loading and unloading management model through the collected historical data and real-time parameters, and optimize the management strategy by adjusting the model parameters. Further, a feedback mechanism can also be set to ensure that the model can be continuously optimized according to new data. For example, during the optimization process, perform multiple iterations (such as 10 times), evaluate the model performance after each iteration, and ensure continuous improvement of the loading and unloading efficiency and the failure rate. Store the optimized intelligent loading and unloading management model and its parameters in the database. At the same time, generate a model performance evaluation report, including data on the improvement of loading and unloading efficiency and the decrease in the failure rate.

[0157] Thus, through dynamic optimization based on the efficiency value, the system can adjust the operation strategy according to the feedback data of each operation and construct an intelligent loading and unloading management model. This model continuously learns and adjusts, enabling the platform to maintain high-efficiency and flexible operation capabilities in the face of different tasks and environments.

[0158] As another alternative embodiment, the present invention also provides an intelligent self-loading and unloading management method, and its specific execution process is as shown in the following steps S1 to S6:

[0159] Step S1: Obtain the images of goods to be loaded and unloaded and the goods loading and unloading logs; perform edge contour segmentation on the images of goods to be loaded and unloaded, and match the goods information according to the goods loading and unloading logs to obtain the attribute information of each good.

[0160] Specifically, set up a high-definition industrial camera in the area to be loaded and unloaded to ensure that clear images of the goods can be captured. The selected camera should have a resolution of at least 1920x1080 and a frame rate of 30 frames per second to meet the monitoring requirements of the dynamic environment. At the same time, the light source of the camera can be configured to obtain good image quality under different lighting conditions. In addition, an LED (Light Emitting Diode) fill light can be used to provide uniform lighting and avoid the influence of shadows and reflections on the image quality.

[0161] Extract the goods loading and unloading logs from the Warehouse Management System (WMS) or the goods tracking system. Among them, the goods loading and unloading logs at least include information such as loading and unloading time, goods type, quantity, weight, and location. In addition, the data extraction frequency (such as extracting once per hour) can be set to ensure the real-time update and accuracy of the loading and unloading logs.

[0162] Preprocess the collected images of goods to be loaded and unloaded, including noise removal, image enhancement, and grayscale conversion. This process can be achieved through Gaussian filtering and histogram equalization in the OpenCV library. At the same time, in this way, the subsequent segmentation effect can be improved. By converting the image to a grayscale image, the subsequent edge detection process can be simplified.

[0163] When performing edge contour segmentation, select a suitable contour segmentation algorithm, such as the Canny edge detection algorithm. First, apply Gaussian blur to process the image, and set appropriate blur parameters (such as a 5×5 convolution kernel) to reduce the influence of noise. Set the thresholds of the Canny algorithm (such as 100 and 200) to identify the obvious edges in the image, so as to accurately capture the edge contours of the goods. The findContours function in OpenCV can be used to extract the contours of the goods, and the drawContours function can be used to draw the contours on the original image for subsequent verification. Store the extracted contour data (such as contour coordinates and related area information) in the database for subsequent processing.

[0164] Based on the information in the cargo handling log, perform attribute information matching on the images of the cargo to be handled. Set the matching criteria in advance, including key parameters such as cargo type, size, and weight. Use a string matching algorithm (such as the Levenshtein distance) to perform fuzzy matching on the cargo name, so that even if there are minor name inconsistencies, it can be correctly identified. For each identified cargo contour, extract its corresponding information in the cargo handling log, including cargo ID, name, specifications, and quantity, etc. A matching threshold (such as 90% similarity) can be set in advance to ensure the accuracy and reliability of the matching results. Associate the attribute information of each cargo with its corresponding contour data and store it in the database to ensure the integrity of the information. Generate a visualization graph to display the contour of each cargo and its matching attribute information to help operators quickly identify and confirm the cargo.

[0165] Thus, through image edge segmentation technology, ensure that the shape and appearance characteristics of each cargo are accurately extracted, eliminating the risk of misidentification. The precise comparison with the handling log can ensure that the cargo attribute information obtained by the system is consistent with the actual cargo, thus providing accurate basic data for subsequent operations. Specifically, through the dual verification of images and logs, the risk of human errors and information loss can be effectively reduced, ensuring that the system's identification and processing of each cargo are more intelligent and accurate. The accurate cargo attribute information provides accurate data support for the evaluation of handling priorities, path planning, and collaborative control, ensuring the refined management of the entire handling process.

[0166] Step S2: Perform handling risk assessment and dynamic handling priority calculation on the attribute information of each cargo to generate the handling priority of each cargo.

[0167] Determine the key indicators for handling risk assessment, including cargo type, weight, volume, handling environment (such as temperature, humidity), historical damage records, etc. Set a scoring standard (such as 1 to 5 points) for each key indicator, and those skilled in the art can score according to the actual situation. It should be noted that when scoring the key indicators, the particularity of the cargo and environmental factors should be considered. For example, fragile items should get a high score (such as 5 points), while durable items should get a low score (such as 1 point). Fragile cargo (such as glass) gets a high-risk score in terms of weight and type, while large cargo (such as furniture) gets a high score in terms of volume. Extract the relevant data of each cargo from the cargo attribute information database to form a risk assessment data set containing all key indicators.

[0168] To ensure data accuracy, techniques such as data cleaning can be used to handle missing values and outliers. For example, for missing historical damage records, mean filling or most frequent value filling can be adopted. According to the set scoring criteria and weights, calculate the total risk score for each cargo. Preset a scoring threshold. For example, a total risk score greater than 8 is marked as high risk, 5 to 8 is medium risk, and less than 5 is low risk. Store the calculated risk scores in the database, recording the risk level (high, medium, low) and specific scores for each cargo. Extract the risk scores of each cargo from the risk assessment results and combine them with the handling timeliness scores to form the dataset required for priority calculation.

[0169] Thus, by comprehensively evaluating the physical characteristics of the cargo, the system can automatically identify high-risk cargo, ensure special handling measures are taken during loading and unloading, and avoid breakage and accidents. As the environment, platform load, and operation conditions change, the system can adjust the loading and unloading priorities in real time. Some cargo needs to be processed preferentially under high-load conditions, and the system will dynamically adjust the task priorities based on real-time feedback to improve the overall operation efficiency and flexibility of the platform. Through dynamic priority sorting, the system can reasonably allocate platform resources, ensure that high-risk or urgent tasks are carried out first, avoid task backlogs and resource conflicts, and improve the overall operation smoothness.

[0170] Step S3: Obtain the monitoring video of the self-loading and unloading platform; perform visual recognition of the loading and unloading path on the monitoring video of the self-loading and unloading platform, and conduct three-dimensional topological modeling to construct a dynamic loading and unloading model;

[0171] Install high-definition surveillance cameras around the self-loading and unloading platform. Ensure that all loading and unloading areas are covered during installation. Similarly to the previous case, the installed monitoring equipment should have a resolution of at least 1920x1080 to capture clear video details. Configure the shooting angle and height of the camera to ensure that the loading and unloading process can be monitored omnidirectionally and avoid blind spots. To capture fast-moving objects and dynamic changes, the frame rate of video recording can be set to 30 frames per second. The storage format can be selected as H.264 encoding to save storage space and ensure that the video quality is suitable for subsequent processing.

[0172] Process the collected monitoring video through a computer vision library (such as OpenCV). First, decompose the video into single-frame images. Five frames can be extracted per second to reduce the processing burden. Preprocess each frame image, including denoising, enhancing contrast, and grayscale conversion, to improve the effectiveness of subsequent edge detection and feature extraction.

[0173] Use a deep learning-based object detection model (such as YOLOv5 or Faster R-CNN (Faster Region-based Convolutional Neural Network)) to identify the loading and unloading path. The model can be trained with a labeled image dataset first to ensure that the actual model used can accurately identify the path and obstacles. Set the evaluation metrics of the model (such as the mAP value (Mean Average Precision)), with the goal of achieving an accuracy rate of over 80%.

[0174] On the identified loading and unloading path, extract key feature points (such as the starting point, ending point, and turning points), and store this information for subsequent use. At the same time, the recognition results can be saved as structured data (such as JSON format), including the coordinate information and related attributes of the path, for subsequent modeling use.

[0175] Select a suitable 3D modeling software for topological modeling. Ensure that the selected tool supports importing and processing path data from computer vision. Set modeling parameters, such as the resolution and level of detail of the model, to retain necessary details while ensuring performance. Create a 3D model in the modeling software according to the extracted loading and unloading path feature points. Use 3D geometric shapes (such as line segments and faces) to represent the loading and unloading path. Considering the actual environment, the surrounding environment data (such as the positions of shelves and robotic arms) can be imported to ensure the authenticity and accuracy of the model.

[0176] Furthermore, the generated 3D model can be optimized to ensure smooth performance in a dynamic environment. Specifically, the model can be verified through a visualization tool to ensure the coherence and accuracy of the path. Set up a virtual camera for simulation to check the correctness and operability of the model.

[0177] Define the structure of the dynamic loading and unloading model. It includes a physical model, a logical model, and a data model. Set the model update frequency, for example, update once per second, to ensure that the dynamic loading and unloading model reflects the real-time state. Integrate real-time monitoring data (such as sensor data, real-time video streams, etc.) into the dynamic loading and unloading model to achieve real-time data synchronization. In this process, communication protocols such as MQTT (Message Queuing Telemetry Transport) or WebSocket (a protocol that supports full-duplex communication) can be used for data transmission to ensure low latency and high reliability. Test the dynamic loading and unloading model to ensure that it can correctly reflect state changes under different conditions (such as high load, faults, etc.).

[0178] Thus, through video monitoring and visual recognition, the platform can perceive the changes in the operating environment in real time and accurately reflect them in the three-dimensional space. The construction of the twin model enables the state of the virtual platform to be highly synchronized with that of the actual platform, improving the system's response ability to external environmental changes. Through three-dimensional modeling, the system can analyze the operating space in real time, identify the loading and unloading paths and obstacles, provide accurate spatial data for subsequent path planning and task scheduling, reduce manual intervention, and improve the operating safety. Based on the constructed dynamic twin model, the system continuously optimizes and adjusts the operating strategy according to the real-time feedback data to ensure the smooth progress of the entire loading and unloading process. In the subsequent processing flow, the system can analyze the best loading and unloading path for each cargo according to the loading and unloading priority of the cargo.

[0179] Step S4: Analyze the available loading and unloading paths for each cargo in the dynamic loading and unloading model one by one according to the loading and unloading priority of each cargo, and make an optimization decision on the loading and unloading time sequence to construct an optimized path for the loading and unloading time sequence of the cargo.

[0180] Based on the previously constructed dynamic loading and unloading model, it can accurately reflect the real situation of the loading and unloading environment, including cargo, robotic arm and path information. First, determine the path analysis algorithm, such as the A* algorithm or Dijkstra algorithm. These algorithms can effectively find the shortest path and consider obstacles. For each cargo, use the selected path algorithm to analyze the available loading and unloading paths from the current placement position to the target placement position.

[0181] Set algorithm parameters, such as the heuristic function and the path cost calculation formula, to ensure the accuracy of path analysis. For each cargo, calculate the available loading and unloading paths one by one according to the loading and unloading priority and the current state of the cargo. The algorithm will take into account the obstacles on the path and the influence of other cargos to ensure the effectiveness of the calculation. Generate the available loading and unloading paths and path costs for each cargo and record this information for subsequent time sequence optimization.

[0182] Based on the analysis of the available loading and unloading paths, an optimization model for the loading and unloading time sequence of the cargo can be constructed. This model should consider multiple factors, including the loading and unloading priority of the cargo, the available loading and unloading paths, the loading and unloading time, cross-path conflicts, etc. Determine the optimization objective, such as minimizing the overall loading and unloading time or maximizing the operating efficiency, and set appropriate constraint conditions (such as the load capacity of each robotic arm and the operating sequence). Select a suitable optimization algorithm, such as the genetic algorithm, Particle Swarm Optimization (PSO) or Mixed-Integer Linear Programming (MILP), to achieve the optimization decision of the loading and unloading time sequence.

[0183] Select an appropriate algorithm according to the complexity and scale of the problem. And set the corresponding algorithm parameters, such as population size, number of iterations, and fitness function, in order to effectively optimize the loading and unloading time sequence of goods. Input the available loading and unloading paths and loading and unloading priority information into the optimization model, and run the selected algorithm to perform the time sequence optimization calculation. The algorithm will generate the best loading and unloading order and path selection to ensure that high-priority goods are processed first under the lowest risk. During the solution process, by presetting the number of iterations (such as up to 100 iterations), ensure that the algorithm can converge to an effective solution. According to the optimized loading and unloading time sequence decision, construct the optimized loading and unloading time sequence path for each good, and record the key nodes of the path (such as the starting point, ending point, and turning point).

[0184] Generate detailed data of the optimized loading and unloading time sequence path. Include the length of each path, the estimated loading and unloading time, and the information of the loading and unloading equipment passed through. Verify the generated optimized loading and unloading time sequence path to ensure its feasibility in a dynamic environment. Specifically, a simulation tool can be used for path simulation to test the usability and safety of the path. During the simulation, if it is found that there are potential conflicts or problems in the path, the path or the loading and unloading order can be adjusted in a timely manner to ensure the safety and efficiency of the overall loading and unloading process. Store the optimized loading and unloading time sequence path and related information in the database to make the data traceable and complete. Record the ID, optimized path, loading and unloading order, and estimated time of each good. Set the storage format as structured data (such as JSON or CSV) for subsequent analysis and query.

[0185] Thus, through dynamic path planning, avoid conflicts and intersections between paths, and ensure that the robotic arm and other equipment can smoothly execute tasks. Not only path planning, the system can also optimize and adjust the loading and unloading time sequence, avoid conflicts and idle waste of resources, ensure that tasks are executed in the optimal order, and improve the operation efficiency of the entire system.

[0186] Step S5: Perform cooperative operation control of the platform robotic arm according to the optimized loading and unloading time sequence path of goods, and perform quantitative calculation of the cooperative efficiency to generate a multi-task cooperative operation efficiency value;

[0187] First, a multi-robotic arm cooperative control system can be designed on the self-loading and unloading platform to ensure that each robotic arm can cooperate according to the optimized path. Among them, the system should at least include a central control unit responsible for real-time scheduling and coordinating the actions of each robotic arm. Set control parameters, such as control frequency (such as 50Hz), to ensure that the system can respond to changes in the dynamic environment in real time.

[0188] Build a task scheduling mechanism to allocate tasks to each robotic arm according to the optimized path based on the loading and unloading sequence. Adopt a priority-based scheduling algorithm (such as EDF scheduling) to arrange the loading and unloading tasks to ensure that high-priority goods are processed in a timely manner. Set scheduling parameters (such as task response time and maximum waiting time) to optimize the scheduling effect. Input the optimized loading and unloading sequence path into the control unit of each robotic arm through the control system to ensure that the robotic arm performs the loading and unloading tasks according to the preset path.

[0189] Use real-time control algorithms (such as PID control) to adjust the movement of the robotic arm to ensure accuracy and stability during execution. The PID parameters (such as proportional, integral, and derivative coefficients) can be preset to optimize the control effect. At the same time, install sensors (such as position sensors and load sensors) on each robotic arm to monitor the status of the robotic arm in real time. Collect real-time data through the data acquisition system for subsequent analysis. Through real-time monitoring and control optimization, abnormal situations (such as overloading or robotic arm failure) can be quickly responded to and adjusted during the loading and unloading process.

[0190] Determine the evaluation indicators for the efficiency of multi-task collaborative operation. Include loading and unloading execution time, task completion rate, robotic arm utilization rate, etc. In actual calculations, each indicator needs to be quantified for comparison and analysis. During the loading and unloading process, record the loading and unloading execution time and task completion status of each robotic arm in real time, and use data analysis tools for data processing and statistics. For example, count the working time, task completion status, and number of conflicts occurred for each robotic arm to ensure the integrity and accuracy of the data.

[0191] Thus, by comprehensively considering multiple factors such as the loading and unloading priority of goods, path planning, and load status, the system can complete tasks in the shortest time, significantly improving the overall efficiency and accuracy of the loading and unloading operation. The system can coordinate the collaborative work of multiple robotic arms in different loading and unloading tasks, avoiding conflicts, waiting, and ineffective operations between robotic arms. Through optimizing the control strategy, the collaborative ability between devices is enhanced. By calculating the working efficiency, task response time, and collaborative cooperation degree of each robotic arm, the platform can quantify the overall efficiency of multi-task collaboration, helping to optimize the operation sequence, reduce ineffective waiting, and improve resource utilization. The quantification of collaborative efficiency provides a data basis for subsequent optimization, and the system can adjust the operation strategy in real time according to different task loads and efficiency values to further improve the overall collaborative efficiency of the system.

[0192] Step S6: Perform loading and unloading fault prediction based on the real-time parameters of multi-task collaborative control, and conduct dynamic iterative optimization of loading and unloading management based on the multi-task collaborative operation efficiency value to build an intelligent loading and unloading management model.

[0193] On the self-loading and unloading platform, continuously collect real-time control parameters for multi-task collaborative control, such as the motion state, load, temperature, humidity, etc. of the robotic arm. These data can be collected in real time through sensors and monitoring systems to ensure the accuracy and timeliness of the data. Set the data collection frequency, for example, collect once per second, to ensure real-time monitoring of the system state.

[0194] Select a suitable fault prediction model, such as a machine learning-based classification model (e.g., random forest, support vector machine) or a deep learning model (e.g., long short-term memory network) for time series prediction. These models can handle time series data and effectively identify potential fault patterns. In the model training stage, the historical fault data and real-time parameters can be used for training to improve the prediction ability of the model. Set the proportion of the training set and test set according to actual needs (e.g., 80% for training and 20% for testing) to ensure the generalization ability of the model. Input the collected real-time parameters into the trained fault prediction model to evaluate the fault risk of the loading and unloading equipment in real time. A prediction threshold can be set (e.g., mark as high risk when the fault probability is greater than 0.7) to take timely measures. Output the prediction results, including potential fault types, occurrence probabilities, and recommended preventive measures. Store the prediction results in the database for subsequent analysis and decision-making.

[0195] Thus, by real-time monitoring and analyzing the operation data of the platform, the system can predict the faults occurring during the loading and unloading process and take preventive measures in advance to avoid the occurrence of faults. When detecting abnormal robotic arm load or path conflicts, the system can immediately adjust the tasks or notify the operator for intervention. Based on the real-time feedback of the multi-task collaborative efficiency value, the system can continuously optimize the loading and unloading strategy and adjust according to the actual operation situation to improve the overall operation efficiency of the system.

[0196] On the basis of fault prediction, considering the fault prediction results, real-time parameters, and multi-task collaborative operation efficiency value simultaneously, set the optimization objectives (e.g., minimizing the fault occurrence rate and loading and unloading time), and construct a dynamic loading and unloading management iterative optimization model. Select a suitable optimization algorithm according to actual needs, such as genetic algorithm or reinforcement learning, to ensure that the model can make effective decisions in a dynamic environment. At the same time, the key parameters of the optimization model need to be determined, including the weights of fault prediction, the weights of loading and unloading efficiency values, and the system response time, etc. Among them, the above parameters can be obtained through expert evaluation or historical data analysis.

[0197] Set evaluation metrics for the model, such as the percentage increase in optimized loading and unloading efficiency and the percentage reduction in failure rate, for iterative evaluation. During the actual loading and unloading process, run the iterative optimization model in real time and make dynamic adjustments based on real-time data and fault prediction results. The update frequency of the model can be preset (e.g., updated once per minute) to ensure that the model adapts to the rapidly changing environment. By setting up a feedback mechanism, collect the results after each optimization, evaluate the loading and unloading efficiency and the occurrence of faults for subsequent iteration. At the same time, analyze the results of each iteration to determine whether the optimization effect meets the expected goal. If not, the model parameters and optimization strategies need to be dynamically adjusted.

[0198] Thus, through continuous learning and optimization, the system can gradually improve the intelligence and flexibility of decision-making. Through intelligent fault prediction, real-time monitoring, and self-optimization, the system can maintain an efficient, stable, and reliable operating state, reduce downtime, and enhance the continuity of loading and unloading operations and the stability of the system.

[0199] According to the analysis results, adjust the loading and unloading management strategy. For example, adjust the operating sequence of the robotic arm, optimize the loading and unloading path, etc., to improve the overall efficiency. Thus, by integrating the fault prediction and the dynamic loading and unloading management iterative optimization model into an intelligent loading and unloading management model, it is convenient for the system to make better real-time decisions and management for cargo loading and unloading. Ensure that the model can seamlessly connect real-time data and historical data. Finally, deploy the intelligent loading and unloading management model to the central control system of the self-loading and unloading platform to ensure that the system can make dynamic decisions based on real-time parameters and prediction results.

[0200] Refer to Figure 8 , which shows the structural block diagram of an intelligent self-loading and unloading management device provided by an embodiment of the present invention, and specifically may include:

[0201] A data acquisition unit 801, configured to acquire images of goods to be loaded and unloaded, goods loading and unloading logs, and monitoring videos of the self-loading and unloading platform;

[0202] A goods information matching unit 802, configured to perform goods information matching based on edge recognition according to the images of goods to be loaded and unloaded and the goods loading and unloading logs, and obtain the goods attribute information of each good;

[0203] A loading and unloading priority generation unit 803, configured to perform dynamic loading and unloading priority calculation based on loading and unloading risk assessment by combining the goods loading and unloading logs and the goods attribute information of each good, and generate the loading and unloading priority of each good;

[0204] A dynamic loading and unloading model construction unit 804, configured to perform loading and unloading path recognition on the monitoring video and construct a dynamic loading and unloading model;

[0205] The loading and unloading path planning unit 805 is used to analyze available loading and unloading paths according to the dynamic loading and unloading model, optimize the loading and unloading timing decision based on each loading and unloading priority, and construct a timing optimization path for cargo loading and unloading;

[0206] The intelligent loading and unloading management unit 806 is used to perform multi-task collaborative control calculations based on the timing optimization path, obtain real-time control parameters and operation efficiency values for multi-task collaboration, and optimize dynamic loading and unloading management according to the real-time control parameters and the operation efficiency values, and construct an intelligent loading and unloading management model, which is used to make real-time decisions and management for cargo loading and unloading.

[0207] In an alternative embodiment, the cargo information matching unit 802 includes:

[0208] The cargo edge contour segmentation unit is used to segment the cargo edge contour of the to-be-loaded and unloaded cargo image, and extract the edge contour line of each cargo;

[0209] The geometric shape analysis unit is used to perform geometric shape analysis on each of the edge contour lines respectively, and generate geometric shape features of each cargo;

[0210] The cargo surface texture recognition unit is used to recognize the cargo surface texture of the to-be-loaded and unloaded cargo image, and extract cargo surface texture features;

[0211] The depth vision recognition unit is used to perform depth vision recognition respectively based on each of the geometric shape features and the cargo surface texture features, and generate depth vision representations of each cargo;

[0212] The cargo information matching subunit is used to match the cargo information of the cargo loading and unloading log according to each depth vision representation, and obtain the cargo attribute information of each cargo.

[0213] In an alternative embodiment, the loading and unloading priority generation unit 803 includes:

[0214] The loading and unloading risk assessment unit is used to perform loading and unloading risk assessment on each of the cargo attribute information respectively, and generate loading and unloading risk assessment values of each cargo;

[0215] The cargo flow direction mining unit is used to mine the cargo flow direction based on the cargo loading and unloading log, and generate flow direction data of each cargo;

[0216] The final placement position calculation unit is used to calculate the final placement position respectively according to each of the flow direction data, and generate the final placement position of each cargo;

[0217] A real-time platform loading and unloading load analysis unit for performing real-time platform loading and unloading load analysis based on the cargo loading and unloading log to obtain real-time loading and unloading load data;

[0218] A dynamic loading and unloading priority calculation unit for dynamically calculating the loading and unloading priority of the loading and unloading risk assessment value and the final placement position of each cargo based on the real-time loading and unloading load data, and generating the loading and unloading priority of each cargo.

[0219] In an optional embodiment, the dynamic loading and unloading model construction unit 804 includes:

[0220] A loading and unloading path visual recognition unit for performing visual recognition of the loading and unloading path on the monitoring video to obtain multiple loading and unloading paths;

[0221] A platform component analysis unit for analyzing the platform components in the monitoring video to obtain multiple platform components;

[0222] A path obstacle monitoring unit for monitoring path obstacles based on the multiple platform components and marking each path obstacle node;

[0223] A spatial position positioning unit for respectively performing spatial position positioning on each path obstacle node to generate spatial position coordinates;

[0224] A three-dimensional topology modeling unit for performing three-dimensional topology modeling based on the multiple loading and unloading paths and each spatial position coordinate to construct a dynamic loading and unloading model.

[0225] In an optional embodiment, the three-dimensional topology modeling unit includes:

[0226] A current cargo placement position recognition unit for recognizing the current cargo placement position;

[0227] A platform operation space calculation unit for calculating the platform operation space based on the current cargo placement position and each spatial position coordinate to obtain platform operation space characteristics;

[0228] A platform component spatial layout analysis unit for performing platform component spatial layout analysis on the multiple platform components to generate spatial layout data of the platform components;

[0229] A platform three-dimensional topology structure analysis unit for performing platform three-dimensional topology structure analysis based on the spatial layout data to obtain three-dimensional topology structure characteristics;

[0230] A loading and unloading robotic arm state parameter acquisition unit for acquiring the state parameters of the loading and unloading robotic arm of the self-loading and unloading platform;

[0231] An operating state analysis unit for analyzing the operating state of the handling robotic arm state parameters to obtain the operating state characteristics of the robotic arm;

[0232] A robotic arm handling activity range calculation unit for calculating the handling activity range of the robotic arm according to the operating state characteristics of the robotic arm and extracting the handling activity range of the robotic arm;

[0233] A three-dimensional point cloud modeling unit for performing three-dimensional point cloud modeling on the three-dimensional topological structure characteristics according to the platform operation space characteristics to construct a three-dimensional handling platform model;

[0234] A dynamic handling behavior rendering unit for rendering the dynamic handling behavior of the three-dimensional handling platform model according to the handling activity range of the robotic arm and the multiple handling paths to construct a dynamic handling model.

[0235] In an alternative embodiment, the handling path planning unit 805 includes:

[0236] An available handling path analysis unit for analyzing the available handling paths for each cargo one by one according to the dynamic handling model to generate the available handling paths for each cargo;

[0237] A cargo transportation intersection prediction unit for predicting the cargo transportation intersections for each of the available handling paths and marking the handling path intersection points;

[0238] A potential path obstacle detection unit for detecting potential path obstacles for each of the available handling paths and identifying potential path obstacles;

[0239] An available path avoidance analysis unit for performing available path avoidance analysis according to the potential path obstacles and the handling path intersection points to generate the avoidance paths for each cargo;

[0240] A handling timing optimization decision unit for performing handling timing optimization decisions on each of the avoidance paths according to each of the handling priorities to construct an optimized handling timing path for the cargo.

[0241] In an alternative embodiment, the intelligent handling management unit 806 includes:

[0242] A real-time control parameter acquisition unit for controlling the coordinated operation of the platform robotic arm according to the optimized timing path and acquiring the real-time control parameters of multi-task coordination;

[0243] A handling execution time calculation unit for calculating the handling execution time of the real-time control parameters to generate the handling execution time of each cargo;

[0244] A loading and unloading conflict statistics unit, configured to obtain operation simulation data of multi-task collaboration, perform loading and unloading conflict statistics on the operation simulation data, and identify loading and unloading conflict time points;

[0245] A conflict path distribution mining unit, configured to perform conflict path distribution mining on the loading and unloading conflict time points to obtain loading and unloading conflict path distribution data;

[0246] A collaborative efficiency quantification calculation unit, configured to perform collaborative efficiency quantification calculation based on the loading and unloading conflict path distribution data and each loading and unloading execution time, and generate an operation efficiency value of multi-task collaboration.

[0247] In an alternative embodiment, the intelligent loading and unloading management unit 806 includes:

[0248] A loading and unloading fault prediction unit, configured to perform loading and unloading fault prediction according to the real-time control parameters, and generate loading and unloading fault prediction data;

[0249] A fault traceability and location unit, configured to perform fault traceability and location on the loading and unloading fault prediction data to obtain a fault traceability point of the self-loading and unloading platform;

[0250] An instant fault diagnosis unit, configured to perform instant fault diagnosis on the fault traceability point to obtain an instant fault diagnosis result;

[0251] A dynamic loading and unloading management iterative optimization unit, configured to perform dynamic loading and unloading management iterative optimization based on the operation efficiency value and the instant fault diagnosis result, and construct an intelligent loading and unloading management model.

[0252] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For related parts, please refer to the corresponding description in the foregoing method embodiment.

[0253] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory:

[0254] The memory is used to store program codes and transmit the program codes to the processor; the processor is used to execute the intelligent self-loading and unloading management method according to the instructions in the program codes of any embodiment of the present invention.

[0255] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the intelligent self-loading and unloading management method according to any embodiment of the present invention.

[0256] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again.

[0257] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0258] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0259] In addition, in each embodiment of the present invention, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0260] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0261] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims

1. An intelligent self-loading and unloading management method, characterized in that Including: Obtain the images of goods to be loaded and unloaded, the goods loading and unloading logs, and the monitoring videos of the self-loading and unloading platform; Perform edge recognition-based goods information matching according to the images of goods to be loaded and unloaded and the goods loading and unloading logs to obtain the goods attribute information of each good; Combine the goods loading and unloading logs and the goods attribute information of each good to calculate the dynamic loading and unloading priorities based on loading and unloading risk assessment, and generate the loading and unloading priorities of each good, including: performing loading and unloading risk assessment on the goods attribute information of each good respectively to generate the loading and unloading risk assessment values of each good; mining the goods flow directions based on the goods loading and unloading logs to generate the flow direction data of each good; calculating the final placement positions of each good respectively according to the flow direction data of each good to generate the final placement positions of each good; analyzing the real-time platform loading and unloading load according to the goods loading and unloading logs to obtain the real-time loading and unloading load data; performing dynamic loading and unloading priority calculation on the loading and unloading risk assessment values and the final placement positions of each good according to the real-time loading and unloading load data to generate the loading and unloading priorities of each good; wherein, the factors affecting the loading and unloading priorities include the loading and unloading risk assessment values, the goods flow directions, the real-time loading and unloading load data, and the final placement positions. During the generation of the loading and unloading priorities, based on the factors affecting the loading and unloading priorities, generate the loading and unloading priorities of each good according to the factor weighting method; Perform loading and unloading path recognition on the monitoring videos to construct a dynamic loading and unloading model; Perform available loading and unloading path analysis according to the dynamic loading and unloading model, and optimize the loading and unloading timing decision based on each loading and unloading priority to construct the timing optimization path for goods loading and unloading; Perform multi-task collaborative control calculation based on the timing optimization path to obtain the real-time control parameters and operation efficiency values of multi-task collaboration, and perform dynamic loading and unloading management optimization according to the real-time control parameters and the operation efficiency values to construct an intelligent loading and unloading management model, and the intelligent loading and unloading management model is used for real-time decision-making and management of goods loading and unloading.

2. The intelligent self-loading and unloading management method according to claim 1, wherein, The performing edge recognition-based goods information matching according to the images of goods to be loaded and unloaded and the goods loading and unloading logs to obtain the goods attribute information of each good includes: Perform segmentation of the goods edge contours on the images of goods to be loaded and unloaded to extract the edge contour lines of each good; Perform geometric shape analysis on each of the edge contour lines respectively to generate the geometric shape features of each good; Perform recognition of the goods surface texture on the images of goods to be loaded and unloaded to extract the goods surface texture features; Perform depth vision recognition respectively based on each geometric shape feature and the goods surface texture features to generate the depth vision representations of each good; Perform goods information matching on the goods loading and unloading logs according to each depth vision representation to obtain the goods attribute information of each good.

3. The intelligent self-loading and unloading management method according to claim 1, characterized in that, The performing loading and unloading path recognition on the monitoring videos to construct a dynamic loading and unloading model includes: Perform loading and unloading path vision recognition on the monitoring videos to obtain multiple loading and unloading paths; Perform platform component analysis on the monitoring videos to obtain multiple platform components; Perform path obstacle monitoring according to the multiple platform components and mark each path obstacle node; Spatially locate each of the path obstacle nodes to generate spatial position coordinates; Based on the multiple loading and unloading paths and each of the spatial position coordinates, perform three-dimensional topological modeling to construct a dynamic loading and unloading model.

4. The intelligent self-loading and unloading management method according to claim 3, characterized in that, The performing three-dimensional topological modeling based on the multiple loading and unloading paths and each of the spatial position coordinates to construct a dynamic loading and unloading model includes: Identify the current position of the goods placement; Calculate the platform operation space based on the current position of the goods placement and each of the spatial position coordinates to obtain the platform operation space characteristics; Analyze the spatial layout of the multiple platform components to generate the spatial layout data of the platform components; Analyze the three-dimensional topological structure of the platform based on the spatial layout data to obtain the three-dimensional topological structure characteristics; Obtain the state parameters of the loading and unloading robotic arm of the self-loading and unloading platform; Analyze the operating state of the state parameters of the loading and unloading robotic arm to obtain the operating state characteristics of the robotic arm; Calculate the loading and unloading activity range of the robotic arm according to the operating state characteristics of the robotic arm, and extract the loading and unloading activity range of the robotic arm; Perform three-dimensional point cloud modeling on the three-dimensional topological structure characteristics according to the platform operation space characteristics to construct a three-dimensional loading and unloading platform model; Render the dynamic loading and unloading behavior of the three-dimensional loading and unloading platform model according to the loading and unloading activity range of the robotic arm and the multiple loading and unloading paths to construct a dynamic loading and unloading model.

5. The intelligent self-loading and unloading management method according to claim 1, characterized in that The performing available loading and unloading path analysis according to the dynamic loading and unloading model, and making an optimized decision on the loading and unloading time sequence based on each of the loading and unloading priorities to construct an optimized time sequence path for goods loading and unloading includes: Perform available loading and unloading path analysis on each good one by one according to the dynamic loading and unloading model to generate the available loading and unloading paths for each good; Predict the cross of goods transportation for each of the available loading and unloading paths, and mark the cross points of the loading and unloading paths; Detect potential path obstacles for each of the available loading and unloading paths to identify potential path obstacles; Perform available path avoidance analysis according to the potential path obstacles and the cross points of the loading and unloading paths to generate the avoidance paths for each good; Make an optimized decision on the loading and unloading time sequence for each of the avoidance paths according to each of the loading and unloading priorities respectively to construct an optimized time sequence path for goods loading and unloading.

6. The intelligent self-loading and unloading management method according to claim 1, wherein The performing multi-task collaborative control calculation based on the optimized time sequence path to obtain the real-time control parameters and operation efficiency values of multi-task collaboration includes: Perform collaborative operation control of the platform robotic arm according to the optimized time sequence path, and collect the real-time control parameters of multi-task collaboration; Calculate the loading and unloading execution time for each good based on the real-time control parameters to generate the loading and unloading execution time for each good; Obtain the operation simulation data of multi-task collaboration, and perform statistics on the loading and unloading conflicts of the operation simulation data to identify the loading and unloading conflict time points; Mine the distribution of conflict paths for the loading and unloading conflict time points to obtain the distribution data of the loading and unloading conflict paths; Perform collaborative efficiency quantification calculation based on the distribution data of the loading and unloading conflict paths and each of the loading and unloading execution times to generate the operation efficiency value of multi-task collaboration.

7. The intelligent self-loading and unloading management method according to claim 1 or 6, characterized in that The performing dynamic loading and unloading management optimization according to the real-time control parameters and the operation efficiency value to construct an intelligent loading and unloading management model includes: Perform loading and unloading fault prediction based on the real-time control parameters to generate loading and unloading fault prediction data; Locate the fault source of the loading and unloading fault prediction data to obtain the fault source point of the self-loading and unloading platform; Perform immediate fault diagnosis on the fault source point to obtain an immediate fault diagnosis result; Based on the operation efficiency value and the immediate fault diagnosis result, perform iterative optimization of dynamic loading and unloading management to construct an intelligent loading and unloading management model.

8. An intelligent self-loading and unloading management device, characterized in that, Including: A data acquisition unit for acquiring images of goods to be loaded and unloaded, goods loading and unloading logs, and monitoring videos of the self-loading and unloading platform; A goods information matching unit for performing goods information matching based on edge recognition according to the images of goods to be loaded and unloaded and the goods loading and unloading logs to obtain the goods attribute information of each good; A loading and unloading priority generation unit for calculating dynamic loading and unloading priorities based on loading and unloading risk assessment in combination with the goods loading and unloading logs and the goods attribute information of each good to generate the loading and unloading priorities of each good, including: performing loading and unloading risk assessment on the goods attribute information of each good respectively to generate the loading and unloading risk assessment values of each good; excavating the flow direction of goods based on the goods loading and unloading logs to generate the flow direction data of each good; calculating the final placement position of each good respectively according to each flow direction data to generate the final placement position of each good; analyzing the real-time platform loading and unloading load according to the goods loading and unloading logs to obtain real-time loading and unloading load data; calculating the dynamic loading and unloading priorities of the loading and unloading risk assessment values and the final placement positions of each good according to the real-time loading and unloading load data to generate the loading and unloading priorities of each good; wherein, the factors affecting the loading and unloading priorities include the loading and unloading risk assessment values, the flow direction of goods, the real-time loading and unloading load data, and the final placement position. During the generation of the loading and unloading priorities, based on the factors affecting the loading and unloading priorities, the loading and unloading priorities of each good are generated according to the factor weighting method; A dynamic loading and unloading model construction unit for identifying the loading and unloading path of the monitoring video to construct a dynamic loading and unloading model; A loading and unloading path planning unit for analyzing available loading and unloading paths according to the dynamic loading and unloading model and making an optimization decision on the loading and unloading time sequence based on each loading and unloading priority to construct a time sequence optimization path for goods loading and unloading; An intelligent loading and unloading management unit for performing multitask collaborative control calculation based on the time sequence optimization path to obtain real-time control parameters and operation efficiency values of multitask collaboration, and performing dynamic loading and unloading management optimization according to the real-time control parameters and the operation efficiency values to construct an intelligent loading and unloading management model, and the intelligent loading and unloading management model is used for making real-time decisions and management on goods loading and unloading.

9. An electronic device, characterized in that, The device includes a processor and a memory: The memory is used for storing program codes and transmitting the program codes to the processor; The processor is used for executing the intelligent self-loading and unloading management method according to any one of claims 1-7 based on the instructions in the program codes.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used for storing program codes, and the program codes are used for executing the intelligent self-loading and unloading management method according to any one of claims 1-7.

Images