Operation identification method and device for loading, unloading and carrying equipment in warehouse and computer equipment

Abstract

The invention relates to an operation identification method and device for loading and unloading carrying equipment in a warehouse and computer equipment. The method comprises the following steps: acquiring a to-be-strengthened data set in an area with a collision risk and a to-be-optimized data set in an area without the collision risk from a target point cloud data set in a warehouse collected by loading and unloading carrying equipment; performing obstacle detection on the to-be-strengthened data set to obtain an obstacle detection result; carrying out lightweight processing on the to-be-optimized data set to obtain an optimized data set; inputting the obstacle detection result and the optimized data set into an anti-collision model to obtain a predicted running track of the loading and unloading carrying equipment in the target time period; and performing operation identification on the loading, unloading and carrying equipment according to the predicted operation track, and controlling the operation of the loading, unloading and carrying equipment according to an identification result of the operation identification. By adopting the method, the accuracy of the identification result of operation identification of the loading and unloading carrying equipment can be improved.

Description

Technical Field

[0001] This application relates to the technical field of logistics warehousing, and particularly to a method and device for identifying the operations of loading and unloading equipment in a warehouse, as well as a computer device. Background Art

[0002] In the field of logistics warehousing, the efficient operation of a warehouse is crucial for an enterprise's competitiveness. As the core execution unit of warehouse operations, the operation safety of loading and unloading equipment is very important.

[0003] In the related art, a millimeter-wave radar is provided on the loading and unloading equipment. According to the point cloud data collected by the millimeter-wave radar, the operation of the loading and unloading equipment is identified to obtain an identification result, which indicates whether the loading and unloading equipment is about to hit an obstacle. Furthermore, the operation of the loading and unloading equipment is controlled according to the identification result of the operation identification to avoid collisions of the loading and unloading equipment.

[0004] However, in the related art, there is a problem that the identification result of identifying the operation of the loading and unloading equipment is inaccurate. Summary of the Invention

[0005] Based on this, this application provides a method and device for identifying the operations of loading and unloading equipment in a warehouse, as well as a computer device, which can improve the accuracy of the identification result of identifying the operation of the loading and unloading equipment.

[0006] In a first aspect, this application provides a method for identifying the operations of loading and unloading equipment in a warehouse, the method comprising:

[0007] Obtaining a dataset to be enhanced in the area with a collision risk and a dataset to be optimized in the area without a collision risk from the target point cloud dataset in the warehouse collected by the loading and unloading equipment;

[0008] Performing obstacle detection on the dataset to be enhanced to obtain an obstacle detection result;

[0009] Performing lightweight processing on the dataset to be optimized to obtain an optimized dataset;

[0010] Inputting both the obstacle detection result and the optimized dataset into an anti-collision model to obtain a predicted operation trajectory of the loading and unloading equipment in a target time period;

[0011] Identifying the operation of the loading and unloading equipment according to the predicted operation trajectory, and controlling the operation of the loading and unloading equipment according to the identification result of the operation identification.

[0012] In some embodiments, obtaining a dataset to be enhanced in the area with a collision risk and a dataset to be optimized in the area without a collision risk from the target point cloud dataset in the warehouse collected by the loading and unloading equipment includes:

[0013] Obtain the initial point cloud data set in multiple target areas collected by the loading and unloading equipment; wherein, the multiple target areas are areas within multiple different azimuth angle ranges respectively collected by the loading and unloading equipment, and the union of the multiple different azimuth angle ranges is the full azimuth angle;

[0014] Identify the point cloud data in each area with a collision risk in the initial point cloud data set in each target area, and splice the point cloud data in each area with a collision risk to obtain a data set to be enhanced;

[0015] Determine the point cloud data other than the point cloud data in each area with a collision risk in the initial point cloud data set in each target area as the data set to be optimized.

[0016] In some embodiments, perform obstacle detection on the data set to be enhanced to obtain an obstacle detection result, including:

[0017] Use the farthest point sampling algorithm to sample a preset number of point cloud data from the data set to be enhanced to obtain sampled point cloud data;

[0018] Determine an updated obstacle detection model according to the sampled point cloud data;

[0019] Use the updated obstacle detection model to perform obstacle detection on the data set to be enhanced to obtain the obstacle detection result.

[0020] In some embodiments, determining an updated obstacle detection model according to the sampled point cloud data includes:

[0021] Determine the ratio of the number of obstacle point cloud data in the sampled point cloud data to the total number of the sampled point cloud data as the obstacle occupancy ratio of the sampled point cloud data;

[0022] Determine the target weight of each obstacle category under multiple obstacle categories according to the obstacle occupancy ratio of the sampled point cloud data;

[0023] Update the preset obstacle detection model according to the target weight of each obstacle category to obtain an updated obstacle detection model.

[0024] In some embodiments, determining the target weight of each obstacle category under multiple obstacle categories according to the obstacle occupancy ratio of the sampled point cloud data includes:

[0025] Determine a smoothing coefficient for smoothing the label according to the obstacle occupancy ratio of the sampled point cloud data;

[0026] Smooth the true label of each obstacle category with the smoothing coefficient to obtain the target label of each obstacle category;

[0027] Use a preset obstacle detection model to perform obstacle detection on the dataset to be enhanced, and obtain the probabilities of the dataset to be enhanced in each obstacle category;

[0028] According to the target labels of each obstacle category and the probabilities of each obstacle category, determine the loss values of each obstacle category, and according to the loss values of each obstacle category, determine the target weights of each obstacle category.

[0029] In some embodiments, perform lightweight processing on the dataset to be optimized to obtain an optimized dataset, including:

[0030] Segment the point cloud data in the dataset to be optimized to obtain sub-datasets in multiple segmented regions;

[0031] Identify sub-datasets in at least one non-edge region from the sub-datasets in multiple segmented regions;

[0032] Replace the sub-datasets in each non-edge region with the centroid coordinate data of each non-edge region to obtain an optimized dataset.

[0033] In some embodiments, identifying sub-datasets in at least one non-edge region from the sub-datasets in multiple segmented regions includes:

[0034] Determine the normal vectors between each target point in the sub-datasets in each segmented region and multiple neighboring points;

[0035] According to the normal vectors, determine the average angle of the sub-datasets in each segmented region;

[0036] Determine the sub-datasets in each segmented region corresponding to the average angle less than or equal to the preset angle as the sub-datasets in each non-edge region.

[0037] In some embodiments, controlling the operation of the loading and unloading equipment according to the recognition result of the operation recognition includes:

[0038] When the recognition result of the operation recognition indicates that the probability of collision within the target time period is greater than or equal to the first threshold and less than the second threshold, control the loading and unloading equipment to perform a sound alarm at the first alarm frequency;

[0039] When the recognition result of the operation recognition indicates that the probability of collision within the target time period is greater than or equal to the second threshold, control the loading and unloading equipment to perform a sound alarm at the second alarm frequency and increase the control resistance of the loading and unloading equipment; the second alarm frequency is greater than the first alarm frequency.

[0040] In a second aspect, the present application provides an operation recognition device for a loading and unloading equipment in a warehouse, and the device includes:

[0041] A data acquisition module, configured to obtain a dataset to be enhanced in the area with a collision risk and a dataset to be optimized in the area without a collision risk from the target point cloud dataset in the warehouse collected by the loading and unloading equipment;

[0042] An obstacle detection module, configured to perform obstacle detection on the dataset to be enhanced to obtain an obstacle detection result;

[0043] A lightweight processing module, configured to perform lightweight processing on the dataset to be optimized to obtain an optimized dataset;

[0044] A predicted operation trajectory module, configured to input both the obstacle detection result and the optimized dataset into a collision avoidance model to obtain the predicted operation trajectory of the loading and unloading equipment during the target period;

[0045] An identification and control module, configured to perform operation identification on the loading and unloading equipment according to the predicted operation trajectory, and control the operation of the loading and unloading equipment according to the identification result of the operation identification.

[0046] In a third aspect, the present application provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method according to any one of the first aspect are implemented.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method according to any one of the first aspect are implemented.

[0048] In a fifth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method according to any one of the first aspect are implemented.

[0049] In the technical solution provided by the embodiments of the present application, by dividing the target point cloud dataset into a dataset to be enhanced in the area with a collision risk and a dataset to be optimized in the area without a collision risk, and performing obstacle detection on the dataset to be enhanced in the area with a collision risk, focusing on the areas where collisions may occur, it is possible to more intensively detect these areas where collisions may occur, improve the recognition accuracy of obstacle detection, and reduce the influence of information irrelevant to the collision risk on the detection result, thereby improving the accuracy of the determined obstacle detection result, and further improving the accuracy of the recognition result of the operation identification of the loading and unloading equipment; moreover, by performing lightweight processing on the areas where collisions are impossible, the amount of calculation is reduced, the real-time performance and efficiency of the operation identification method are improved, the timeliness of the recognition result of the operation identification of the loading and unloading equipment can be improved, and the problem of reducing the recognition result accuracy due to the delay in determining the collision risk can be avoided, and further the accuracy of the recognition result of the operation identification of the loading and unloading equipment is improved. Brief Description of the Drawings

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0051] Figure 1 Flow schematic diagram of the operation recognition method for loading and unloading equipment in a warehouse provided for some embodiments;

[0052] Figure 2 Flow schematic diagram of the method for obtaining the dataset to be enhanced and the dataset to be optimized provided for some embodiments;

[0053] Figure 3 Flow schematic diagram of the method for obtaining the detection result of an obstacle provided for some embodiments;

[0054] Figure 4 Flow schematic diagram of the method for lightweight processing of the dataset to be optimized to obtain the optimized dataset provided for some embodiments;

[0055] Figure 5 Flow schematic diagram of the method for operation recognition and control of loading and unloading equipment provided for some embodiments;

[0056] Figure 6 Structural schematic diagram of the operation recognition device for loading and unloading equipment in a warehouse provided for some embodiments;

[0057] Figure 7 Structural schematic diagram of a computer device provided for some embodiments. Detailed Embodiments

[0058] The following will describe in detail the embodiments of the technical solutions of the present application with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, so they are only examples and cannot be used to limit the protection scope of the present application.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above description of the drawings are intended to cover non-exclusive inclusion.

[0060] In the description of the embodiments of the present application, technical terms such as "first" and "second" are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality" is more than two, unless otherwise specifically limited. In the description of the embodiments of the present application, "each" means every one or every one of a plurality, unless otherwise specifically limited.

[0061] Referring to "embodiments" herein means that specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0062] In the description of the embodiments of the present application, the term "and / or" is merely an associative relationship describing associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. Additionally, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0063] The computer device in the embodiments of the present application may include one or a combination of at least two of the following: handling equipment, servers, computers, industrial control computers, etc. Exemplarily, the handling equipment may include handling vehicles or robots, etc. Exemplarily, the handling vehicle may include forklifts, transporters, tractors, automated guided vehicles, or trailers, etc.

[0064] Figure 1 A schematic flow diagram of an operation identification method for handling equipment in a warehouse provided for some embodiments is as Figure 1 shown. This method is applied to a computer device, and the method includes:

[0065] S101. Obtain a dataset to be enhanced in the area with a risk of collision and a dataset to be optimized in the area without a risk of collision from the target point cloud dataset in the warehouse collected by the handling equipment.

[0066] Exemplarily, the handling equipment may operate in the warehouse. Exemplarily, a millimeter-wave radar is provided on the handling equipment, and the target point cloud dataset in the warehouse is collected through the millimeter-wave radar. Exemplarily, there are obstacles such as shelves, goods, and buildings (such as columns) in the warehouse.

[0067] Among them, each point cloud data is a three-dimensional coordinate. In some embodiments, the target point cloud data set can be a point cloud data set at all-round angles for the detection of handling equipment. Among them, the all-round angle is an angle within a horizontal range of 360°. In other embodiments, in order to reduce the computational amount, the target point cloud data set can be a point cloud data set in the front and side directions for the detection of handling equipment.

[0068] Exemplarily, the target point cloud data set can be point cloud data collected within a specified period. Again exemplarily, the target point cloud data set can be point cloud data collected at a single time point. Among them, the specified period can be the period between a specified moment and the current moment; the specified moment is a moment before the current moment and at a specified time interval from the current moment.

[0069] Exemplarily, the area with a collision risk may include an area where there is a possibility of colliding with a building, goods, and a shelf within a target period. Exemplarily, the area without a collision risk may include an area where there is no possibility of hitting a building, goods, and a shelf within a target period. Among them, the area without a collision risk can be the area in the detection area of the handling equipment after removing the area with a collision risk.

[0070] Exemplarily, the area where there is no possibility of collision may include an open area. In the area where there is no possibility of collision, the distance between the handling equipment and the fixed structure is relatively far (linear distance > safety threshold). The fixed structure is, for example, a dynamic obstacle (such as a moving good, other working vehicles other than the handling equipment) and a static obstacle (such as a shelf, a fixed good, and a building, etc.). In the area where there is a possibility of collision, the distance between the handling equipment and the fixed structure is relatively close (linear distance ≤ safety threshold), then it is determined that there is a possibility of collision. Here, the actual driving direction of the handling equipment is not considered, only whether it is within the area is considered.

[0071] Among them, the data set to be strengthened and the data set to be optimized can be obtained by dividing the target point cloud data set. In some embodiments, from the target point cloud data set, the point cloud data greater than or equal to the preset density is determined as the data set to be strengthened in the area with a collision risk, and the data other than the data set to be strengthened in the target point cloud data set is used as the data set to be optimized. In other embodiments, classification models such as support vector machines, random forests, neural networks (such as deep learning networks for point cloud data like PointNet, etc.) are used to divide the target point cloud data set into a data set to be strengthened in the area with a collision risk and a data set to be optimized in the area without a collision risk.

[0072] S102. Perform obstacle detection on the data set to be strengthened to obtain an obstacle detection result.

[0073] Exemplarily, the updated obstacle detection model is used to perform obstacle detection on the dataset to be enhanced, and the detection results of the obstacles are obtained.

[0074] In some embodiments, the detection results of the obstacles may include the types of the obstacles (e.g., buildings, shelves, goods, etc.). In other embodiments, the detection results of the obstacles may include the types of the obstacles and the point cloud data in the area where the obstacles are located in the area with a collision risk. Exemplarily, the obstacle detection model is used to identify the obstacle point cloud data in the point cloud data and the categories of the obstacles.

[0075] Exemplarily, the updated obstacle detection model may be a preset obstacle detection model, or the updated obstacle detection model may be a detection model obtained by updating the preset obstacle detection model according to the target weights of each obstacle category. Exemplarily, the obstacle detection model may include a YOLO model. The target weights of each obstacle category are a set of parameters in the YOLO model, which are used to measure the importance of different obstacle types in prediction.

[0076] In one implementation, historical point cloud data, historical obstacle point cloud data, and historical attention weights are obtained, and the historical point cloud data, historical obstacle point cloud data, and historical attention weights are input into a preset model for training; the output of the preset model is used as the target model parameters, and the preset model is updated according to the target model parameters to obtain an obstacle detection model.

[0077] In one implementation, the historical point cloud data is various point cloud data, such as the point cloud data of columns, shelves, goods of various sizes, and other various obstacles in the warehouse, etc., as well as the point cloud data of the surrounding environment outside these objects, and the historical obstacle point cloud data (only including the point cloud data of the objects, not including the point cloud data of the surrounding environment). The historical attention weights are used to force the model to focus on key areas (obstacles that are not easily distinguishable).

[0078] S103. Perform lightweight processing on the dataset to be optimized to obtain an optimized dataset.

[0079] Exemplarily, the lightweight processing may include at least one of uniform sampling, random sampling, filtering processing, feature point extraction, centroid coordinate data replacement, etc.

[0080] In some embodiments, performing lightweight processing on the dataset to be optimized to obtain an optimized dataset may include: performing lightweight processing on the non-edge part of the dataset to be optimized to obtain an optimized dataset. For example, replacing the point cloud data in the non-edge part of the dataset to be optimized with centroid coordinate data to obtain an optimized dataset.

[0081] In some other embodiments, the dataset to be optimized is processed to be lightweight to obtain the optimized dataset, which may include: performing at least one of uniform sampling, random sampling, filtering, feature point extraction, etc. on the dataset to be optimized to obtain the optimized dataset. For example, performing uniform sampling on the dataset to be optimized to obtain the optimized dataset.

[0082] S104. Input both the obstacle detection result and the optimized dataset into the anti-collision model to obtain the predicted operation trajectory of the loading and unloading handling equipment during the target period.

[0083] Among them, the inputs of the anti-collision model are the obstacle detection result and the optimized dataset, and the output is the predicted operation trajectory of the loading and unloading handling equipment during the target period. Exemplarily, the obstacle detection result and the optimized dataset can be respectively: the obstacle detection results at multiple time points corresponding to the point cloud data collected within a specified period and the optimized datasets at multiple time points. Another example, the obstacle detection result and the optimized dataset can be respectively: the obstacle detection result and the optimized dataset corresponding to the point cloud data collected at the current moment.

[0084] Among them, the target period is a time period with a preset duration starting from the current moment.

[0085] Exemplarily, the anti-collision model is trained based on time-series point cloud data and has the ability of trajectory prediction. Exemplarily, the warehouse environment can be scanned and collected at different time points, and the collected point cloud data is arranged in chronological order to form time-series data; the areas with collision risks are identified from the time-series data, and the obstacle detection results of the areas with collision risks are obtained through obstacle detection; the dataset obtained after lightweight processing of the point cloud data in the areas without collision risks in the time-series data; the anti-collision model is generated according to the obstacle detection results of the areas with collision risks in the time-series data and the dataset obtained after lightweight processing in the time-series data.

[0086] In some embodiments, different anti-collision detection models can be selected according to actual requirements and computing resources. For example, if high precision and real-time performance are required, a network based on deep learning (such as RadarMFNet) can be adopted. Another example, if the computing resources are limited, a spatio-temporal convolutional network can be used.

[0087] S105. Identify the operations of the loading and unloading handling equipment according to the predicted operation trajectory, and control the operation of the loading and unloading handling equipment according to the identification result of the operation identification.

[0088] In some embodiments, the operation identification of the handling equipment is performed according to the predicted operation trajectory, including: determining the distance between the handling equipment and the nearest obstacle within the target period according to the predicted operation trajectory, and determining the identification result of the operation identification of the handling equipment according to the distance between the handling equipment and the nearest obstacle within the target period, where the identification result represents the probability of collision within the target period.

[0089] In some embodiments, if it is determined that there is a collision risk according to the identification result of the operation identification, a reminder can be given to the driver or management personnel and a risk alarm can be issued.

[0090] In the technical solution provided by the embodiments of the present application, by dividing the target point cloud data set into a data set to be enhanced in the area with collision risk and a data set to be optimized in the area without collision risk, and performing obstacle detection on the data set to be enhanced in the area with collision risk, focusing on the areas where collisions may occur, it is possible to more intensively detect these areas where collisions may occur, improve the recognition accuracy of obstacle detection, and reduce the influence of information unrelated to collision risk on the detection result, thereby improving the accuracy of the determined obstacle detection result, and further improving the accuracy of the operation identification result of the handling equipment; moreover, by performing lightweight processing on the areas where collisions are not likely to occur, the amount of calculation is reduced, the real-time performance and efficiency of the operation identification method are improved, the timeliness of the operation identification result of the handling equipment can be improved, and the problem of reducing the accuracy of the identification result due to the delay in determining the collision risk can be avoided, thereby further improving the accuracy of the operation identification result of the handling equipment.

[0091] In addition, by controlling the operation of the handling equipment through the more accurate operation identification result, the control reliability of the operation control of the handling equipment can be improved, the anti-collision ability of the handling equipment in a complex warehouse environment can be improved, and the operation safety of the handling equipment can be improved.

[0092] In the technical solution provided by the embodiments of the present application, by dividing the obtained target area point cloud data (dividing it into a data set to be enhanced and a data set to be optimized), performing enhancement processing on the data set to be enhanced, performing lightweight processing on the data set to be optimized, and combining with an anti-collision model for prediction and operation safety identification, the effective monitoring and risk warning of the operation safety of the handling equipment are realized.

[0093] In the technical solution provided by the embodiments of the present application, it is possible to judge in advance the change in the distance between the loading and unloading handling equipment and the obstacle, and issue a warning signal in time, thereby effectively ensuring the safety of the loading and unloading handling equipment during operation. In addition, by strengthening the dataset to be strengthened and lightweighting the dataset to be optimized, it is possible to not only improve the accuracy of the determined obstacle detection result, but also reduce the amount of data input to the anti-collision model, improve the speed of determining the predicted operation trajectory of the loading and unloading handling equipment during the target time period, and save the computing resources for the anti-collision model to make predictions; therefore, by reasonably allocating computing resources, the performance of the computer device is optimized, and the anti-collision ability of the loading and unloading handling equipment in a complex warehouse environment is improved.

[0094] Figure 2 A flowchart showing a method for obtaining a dataset to be strengthened and a dataset to be optimized provided for some embodiments is as Figure 2 shown, Figure 2 The embodiment is a description of step S101. This method is applied to a computer device, and the method includes:

[0095] S1011. Obtain an initial point cloud dataset in multiple target regions collected by the loading and unloading handling equipment; wherein, the multiple target regions are regions within multiple different azimuth angle ranges respectively collected by the loading and unloading handling equipment, and the union of the multiple different azimuth angle ranges is the full azimuth angle.

[0096] In some embodiments, multiple millimeter-wave radars may be provided on the loading and unloading handling equipment, and the detection area of each millimeter-wave radar is a target region, so that the initial point cloud datasets in multiple target regions are respectively detected by the multiple millimeter-wave radars. Exemplarily, there may be overlapping regions among the multiple target regions. Exemplarily, the target region is the data region obtained by the millimeter-wave radar installed on the operation vehicle. For example, if five millimeter-wave radars are installed on the loading and unloading handling equipment, then there are five target regions.

[0097] For example, the multiple target regions may include at least two of the regions directly in front of, to the left of, to the right of, in the front left, in the front right, in the rear left, in the rear right, directly behind the loading and unloading handling equipment, etc.

[0098] Among them, each angle range may be an angle range within the horizontal 360° range. Exemplarily, there is an overlap between two adjacent azimuth angle ranges among the multiple different azimuth angle ranges. For example, taking the front of the loading and unloading handling equipment as 0°, the multiple different azimuth angle ranges are respectively the ranges between 315° and 360° and between 0 and 45°, the range between 30° and 120°, the range between 90° and 180°, the range between 180° and 270°, the range between 240° and 330°, etc.

[0099] The omnidirectional angle is formed by the union of multiple different azimuth angle ranges, so that the point cloud data within the horizontal 360° range can be collected by the loading and unloading equipment.

[0100] S1012. Identify the point cloud data in each area with a collision risk in the initial point cloud dataset in each target area, and splice the point cloud data in each area with a collision risk to obtain a dataset to be enhanced.

[0101] Analyze the areas with a collision risk for the initial point cloud dataset in a target area collected by each millimeter-wave radar to obtain the point cloud data in the areas with a collision risk in the initial point cloud dataset in each target area.

[0102] Since there is an overlapping range between two adjacent azimuth angle ranges among multiple different azimuth angle ranges, therefore, splice (or merge) the point cloud data in the same range in the point cloud data in the area with a collision risk, that is, take the union of the point cloud data in the same area (or the same azimuth angle range) in the area with a collision risk in the point cloud data collected by different millimeter-wave radars to obtain a dataset to be enhanced. For example, taking the multiple different azimuth angle ranges including the range between 30° and 120° and the range between 90° and 180° as an example, there is an overlapping range between 90° and 120° in the range between 30° and 120° and the range between 90° and 180°. Then, the point cloud data in the area with a collision risk within the range between 90° and 120° in the range between 30° and 120° can be taken as the union with the point cloud data in the area with a collision risk within the range between 90° and 120° in the range between 90° and 180°. The point cloud data in the area with a collision risk within the range between 30° and 90°, the point cloud data after taking the union within the range between 90° and 120°, and the point cloud data in the area with a collision risk within the range between 120° and 180° are determined as the dataset to be enhanced.

[0103] The point cloud collected by the millimeter-wave radar is sparse (such as the top of the shelf or the edge of the goods is missing), resulting in incomplete three-dimensional modeling and unable to accurately judge the blind area obstacles. In some possible implementation scenarios, set the X-axis, Y-axis, and Z-axis of the warehouse. The radar respectively collects the local point cloud P1 of the front shelf (the top of the shelf with a height of 3m), the local point cloud P2 of the top of the goods transported by the loading and unloading equipment, and the local point cloud P3 of the side goods (a box with a height of 1m), and splice P1, P2, and P3 to obtain P (P ∈ R 3M ×3), supplement the point clouds from different perspectives to simulate a complete three-dimensional environment (for example, the situation where the handling equipment is carrying goods and there are shelves and goods coexisting around the handling equipment), and P is the dataset to be enhanced. Apply the farthest point sampling algorithm to P for farthest point sampling to retain 1024 points (i.e., the sampled point cloud data), and preferentially select the points at the top of the shelves and the edges of the goods (key contours) to generate new samples. . Exemplarily, the number of point clouds from the shelves is m1 = 600, and the number of points from the goods is m2 = 424.

[0104] In this way, by stitching the local point cloud data from different perspectives (such as the point clouds at the top of the shelves, the top of the goods, and the side of the goods), a more complete three-dimensional environment can be simulated. The above stitching method can effectively supplement the sparse point cloud data collected by the millimeter-wave radar, reduce the blind areas caused by the radar coverage range or angle limitations, and thus provide richer environmental information for subsequent obstacle detection.

[0105] S1013. Determine the point cloud data outside the point cloud data in each area with a collision risk in the initial point cloud dataset in each target area as the dataset to be optimized.

[0106] In some embodiments, the dataset to be optimized can be the point cloud data outside the point cloud data in each area with a collision risk in the initial point cloud dataset in each target area that is not stitched. In other embodiments, the point cloud data outside the point cloud data in each area with a collision risk in the initial point cloud dataset in each target area can be stitched within the same azimuth angle range to obtain the dataset to be optimized.

[0107] In the embodiments of the present application, the datasets in the dataset to be enhanced are aggregated according to the region type (i.e., multiple different azimuth angle ranges. For example, when the handling equipment is traveling in the warehouse, the millimeter-wave radar respectively collects the local point cloud data of the front shelf and the local point cloud data of the goods in the front side. The two shelves are the same, but the acquisition perspectives are different) to obtain an aggregated dataset (i.e., the stitched dataset). By aggregating according to the region type, the point clouds from different perspectives are supplemented to simulate a complete environment, avoiding the situation where the point cloud data collected by the millimeter-wave radar may be incomplete or insufficient when the handling equipment moves in the warehouse. Thus, the point cloud data can be enhanced by supplementing the point clouds from different perspectives, increasing the diversity of the point cloud data, making up for the deficiencies of single-shot acquisition data, and thus providing richer environmental information for three-dimensional modeling.

[0108] Strengthen a part of the point cloud data (i.e., strengthen the dataset to be strengthened by means of point cloud stitching), while perform lightweight processing on the other part, based on the collision risks in different regions and the requirements of computing resource optimization; it can achieve Goal 1: reasonably allocate computing resources, concentrate limited resources on high-risk regions, and at the same time reduce the overall computing burden through lightweight processing, and can achieve Goal 2: through strengthening and lightweight processing, the system can improve real-time performance and efficiency while ensuring the accuracy of collision detection, thereby better ensuring the safety of the loading, unloading and handling equipment operation.

[0109] In the technical solution provided by the embodiment of the present application, by identifying the point cloud data in each area with collision risk in the initial point cloud dataset in each target area, and stitching the point cloud data in each area with collision risk, a dataset to be strengthened is obtained, so as to improve the diversity of the point cloud data in the dataset to be strengthened, make up for the deficiencies of single-shot collected data, provide richer environmental information for 3D modeling, and further improve the accuracy of the obtained obstacle detection results by increasing the richness of the dataset to be strengthened.

[0110] Figure 3 A flowchart of a method for obtaining the detection result of an obstacle provided for some embodiments is as Figure 3 shown Figure 3 The embodiment is a description of step S102. This method is applied to a computer device, and this method includes:

[0111] S1021. Use the farthest point sampling algorithm to sample a preset number of point cloud data from the dataset to be strengthened to obtain sampled point cloud data.

[0112] The farthest point sampling is used to sample M points on a point cloud of N points, so that these points can better represent the overall contour of the point cloud. The idea of the farthest point sampling is to iteratively select M points on the point cloud of N points. Each time, select the point with the maximum minimum distance from all points in the currently selected point set S = {P0, P1,..., P i}, and add it to the set S.

[0113] The preset number is M, and M is an integer greater than or equal to 2. Exemplarily, the value of M can be based on the size of the obstacle. Exemplarily, when the size of the obstacle is greater than the preset size, the value of M can be larger; when the size of the obstacle is less than or equal to the preset size, the value of M can be smaller. Exemplarily, the number of M is in a proportional relationship with the size of the obstacle.

[0114] In some embodiments, the enhanced dataset is clustered to obtain multiple clustering clusters, and the farthest point sampling algorithm is used to sample a preset number of point cloud data for each clustering cluster respectively to obtain the subsampled point cloud data of each clustering cluster, and the subsampled point cloud data of each clustering cluster is determined as the sampled point cloud data.

[0115] S1022. Determine the updated obstacle detection model according to the sampled point cloud data.

[0116] In some embodiments, determining the updated obstacle detection model according to the sampled point cloud data includes: determining the target weight of each obstacle category under multiple obstacle categories according to the sampled point cloud data; updating the preset obstacle detection model according to the target weight of each obstacle category to obtain the updated obstacle detection model. Exemplarily, the target weight may be the weight in the loss function of the obstacle detection model. Among them, updating the preset obstacle detection model according to the target weight of each obstacle category to obtain the updated obstacle detection model may include: replacing the weight in the loss function of the preset obstacle detection model with the target weight to obtain the updated obstacle detection model.

[0117] In other embodiments, determining the updated obstacle detection model according to the sampled point cloud data includes: determining the obstacle occupancy ratio of the sampled point cloud data as the ratio of the number of obstacle point cloud data in the sampled point cloud data to the total number of the sampled point cloud data, and updating the preset obstacle detection model according to the obstacle occupancy ratio to obtain the updated obstacle detection model.

[0118] In still other embodiments, determining the updated obstacle detection model according to the sampled point cloud data includes: determining the obstacle occupancy ratio of the sampled point cloud data as the ratio of the number of obstacle point cloud data in the sampled point cloud data to the total number of the sampled point cloud data; determining the target weight of each obstacle category under multiple obstacle categories according to the obstacle occupancy ratio of the sampled point cloud data; updating the preset obstacle detection model according to the target weight of each obstacle category to obtain the updated obstacle detection model. In this way, considering that in the actual scenario, some obstacle categories may be more likely to be misjudged or missed, through dynamic weight adjustment (i.e., dynamically adjusting the target weight), the updated obstacle detection model can assign higher weights to these difficult-to-classify obstacles, thereby reducing the phenomena of misjudgment and missed detection. The dynamic adjustment mechanism can effectively improve the adaptability of the model to unbalanced data, and when the obstacles are easy to detect, the weights can be appropriately reduced to reduce the consumption of additional computing power.

[0119] Exemplarily, the obstacle proportion can be the sum of the proportions of various types of obstacles. Exemplarily, it can be determined whether each point cloud data in the sampled point cloud data is an obstacle (exemplarily, the obstacle point cloud data in the sampled point cloud data can be determined through the updated obstacle detection model), and then the obstacle proportion can be determined. Also exemplarily, the obstacle type corresponding to the sub-sampled point cloud data of each clustering cluster can be determined, and the ratio between the number of clustering clusters corresponding to each obstacle type and the total number of clustering clusters can be determined as the proportion of each type of obstacle, and the sum of the proportions of each type of obstacle can be determined as the obstacle proportion.

[0120] In some embodiments, according to the obstacle proportion of the sampled point cloud data, determining the target weights of each obstacle category under multiple obstacle categories includes: determining the proportion of each type of obstacle as the target weight of each obstacle category.

[0121] In some other embodiments, according to the obstacle proportion of the sampled point cloud data, determining the target weights of each obstacle category under multiple obstacle categories includes: determining a smoothing coefficient for smoothing the labels according to the obstacle proportion of the sampled point cloud data; using the smoothing coefficient to smooth the true labels of each obstacle category (the true labels include the labels of real obstacles such as buildings, shelves, and goods), to obtain the target labels of each obstacle category; using a preset obstacle detection model to perform obstacle detection on the dataset to be enhanced, to obtain the probabilities of the dataset to be enhanced in each obstacle category; determining the loss value of each obstacle category according to the target label and the probability of each obstacle category, and determining the target weight of each obstacle category according to the loss value of each obstacle category. Through the obstacle proportion, the model recognition tendency can be determined. The model recognition tendency refers to the inherent preference of the model for certain obstacle categories, which is determined by the training data distribution and the loss function design.

[0122] In some embodiments, according to the loss value of each obstacle category, determining the target weight of each obstacle category may include: obtaining the total loss value of the loss values of each obstacle category, and determining the ratio between the loss value of each obstacle category and the total loss value as the target weight of each obstacle category.

[0123] Exemplarily, if the obstacle proportion is relatively high, it indicates that there are more obstacles in the scene and the information about obstacles in the dataset is rich. At this time, the smoothing coefficient can be appropriately reduced to make the model pay more attention to the actual label information and make full use of this rich data to learn the features of obstacles to accurately identify and locate obstacles. If the obstacle proportion is relatively low, it means that the obstacle information in the data is relatively scarce, and there may be noise or incomplete situations. Increasing the smoothing coefficient can smooth the limited label information, reduce the influence of noise, and use the context information of the surrounding data to infer the possible position and attributes of obstacles, thereby improving the accuracy of obstacle recognition.

[0124] The following is an implementation method for calculating the loss value: Regarding the building as immovable like a pillar and the shelf as the same, so the two can be regarded as an individual, that is, both the building and the shelf are treated as shelves. Set the true labels of the shelf and the goods as one-hot encoding (One-Hot encoding) (for example, the shelf is [1, 0] and the goods are [0, 1]). After label smoothing, the true label is adjusted to: , where is the smoothing coefficient (taking 0.4 as an example, exemplarily, the smoothing coefficient can be determined according to the obstacle occupancy ratio of the sampled point cloud data), and K is the number of categories (set to 2).

[0125] The original label of the shelf c1 = [1, 0], after smoothing:

[0126] C1 smooth = 0.6 * [1, 0] + 0.4 * [0.5, 0.5] = [0.6 * 1 + 0.4 * 0.5, 0.6 * 0 + 0.4 * 0.5] = [0.8, 0.2].

[0127] The original label of the goods c2 = [0, 1], after smoothing: c2 smooth = 0.6 * [0, 1] + 0.4[0.5, 0.5] = [0.2, 0.8].

[0128] The prediction result of the obstacle detection model (i.e., the preset obstacle detection model) for obstacles is as follows: The model outputs a probability distribution f( ) = [0.9, 0.1] (that is, the probability of predicting as a shelf is 90%). Using the cross-entropy loss formula: , substitute the values for calculation, , approximately equal to 0.1; The model outputs a probability distribution f( )) = [0.3, 0.7] (that is, the probability of predicting as goods is 70%). Using the cross-entropy loss formula: , substitute the values for calculation, , approximately equal to 0.3; The loss of the shelf : The model predicts the shelf part more accurately (the probability 0.9 is close to the true label 0.8), so the loss is lower; The loss of the goods : The model predicts the goods part poorly (the probability 0.7 deviates from the true label 0.8), so the loss is higher.

[0129] In one implementation method, the calculation formula of the target weight: shelf , goods , it can be seen that the classification of goods is more difficult (with high losses), and a higher weight needs to be assigned in the total loss to force the model to focus on error-prone areas. The point cloud density of the goods at the top and bottom of the shelves is increased, reducing blind spot missed detections, and improving the attention ability of the obstacle detection model to obstacles that are not easily recognized.

[0130] S1023. Use the updated obstacle detection model to perform obstacle detection on the dataset to be enhanced, and obtain the detection results of the obstacles.

[0131] In the technical solution provided by the embodiment of the present application, farthest point sampling is performed on the spliced point cloud data, which can retain key contour points (such as the points at the top of the shelf and the edges of the goods), and generate new sample data (i.e., sampled point cloud data). This method can reduce the amount of data while preferentially retaining the feature points crucial for obstacle recognition; select a preset number of representative points from a large amount of point cloud data, reducing the amount of data for subsequent processing and lowering the computational complexity; preferentially select key contour points (such as the points at the top of the shelf and the edges of the goods), which can better reflect the geometric features and shapes of the obstacles, thereby improving the accuracy of obstacle recognition.

[0132] Figure 4 It is a schematic flowchart of a method for performing lightweight processing on a dataset to be optimized to obtain an optimized dataset provided for some embodiments, as Figure 4 shown, Figure 4 The embodiment is a description of step S103. This method is applied to a computer device, and the method includes:

[0133] S1031. Segment the point cloud data in the dataset to be optimized to obtain sub-datasets in multiple segmented regions.

[0134] Exemplarily, the multiple segmented regions can be multiple different road surface regions in the road surface area of the region without collision risk. Exemplarily, different road surfaces can be road surfaces in different directions and / or different regions.

[0135] In some embodiments, the point cloud data in the dataset to be optimized can be segmented in the way of grid division.

[0136] In some other embodiments, the point cloud data in the dataset to be optimized can be segmented into pavement regions according to different pavement properties. Exemplarily, there may be height differences in different pavement regions within a warehouse, such as loading and unloading platforms and ordinary floors. By calculating the height value of each point in the point cloud and dividing the regions based on a set height threshold, the segmentation of the point cloud data can be achieved. Also exemplarily, different functional pavements may have different slopes. For example, the driving passage for loading and unloading equipment is horizontal, while the drainage area may have a certain slope. The point cloud data is segmented by calculating the slope of the local region of the point cloud. Also exemplarily, the points within the same pavement region are usually connected to each other, and the connectivity between the points is used to segment the point cloud data. Also exemplarily, the point cloud data in the dataset to be optimized is input into a machine learning model (such as a support vector machine or a random forest, etc.) so that the machine learning model outputs sub-datasets in multiple segmented regions. Also exemplarily, the point cloud data in the dataset to be optimized is input into a deep learning-based segmentation model (such as a deep learning model for processing point cloud data like PointNet, etc.) so that the deep learning-based segmentation model outputs sub-datasets in multiple segmented regions.

[0137] S1032. Identify sub-datasets in at least one non-edge region from the sub-datasets in multiple segmented regions.

[0138] For example, identify sub-datasets in P non-edge regions from the sub-datasets in Q segmented regions. P is an integer less than Q.

[0139] In some embodiments, identifying sub-datasets in at least one non-edge region from the sub-datasets in multiple segmented regions includes: determining the normal vectors between each target point in the sub-datasets in each segmented region and multiple neighboring points; determining the average angle of the sub-datasets in each segmented region according to the normal vectors; and determining the sub-datasets in each segmented region corresponding to the average angles less than or equal to the preset angle as the sub-datasets in each non-edge region.

[0140] S1033. Replace the sub-datasets in each non-edge region with the centroid coordinate data of each non-edge region to obtain an optimized dataset.

[0141] In the technical solution provided by the embodiment of the present application, the point cloud data in the dataset to be optimized is segmented to obtain multiple sub-datasets (i.e., the sub-datasets in the above-mentioned multiple segmentation regions), the normal vectors of the target points and each neighborhood point in each sub-dataset are determined, and the average angle in the sub-data is calculated according to the normal vectors. A sub-dataset contains multiple sub-data, and each sub-data contains multiple target points (for example, the point corresponding to each sub-data is a target point, or for example, among the points corresponding to multiple sub-data, a target number of points are randomly or evenly selected as multiple target points), and each target point has multiple neighborhood points. If the average angle ≤ the preset angle, it is determined that the region corresponding to the sub-dataset is a non-edge region, the centroid coordinates of the sub-dataset are calculated, the coordinates in the sub-dataset are replaced according to the centroid coordinates to obtain the target sub-data, and the optimized dataset is determined according to each target sub-data.

[0142] In the technical solution provided by the embodiment of the present application, the data in the dataset to be optimized does not play a major role in anti-collision detection, so lightweight processing is required. However, lightweight processing is prone to overshoot, resulting in geometric size reduction or edge blurring, so a scheme for edge detection of point cloud data is proposed (that is, among the sub-datasets in multiple segmentation regions, the sub-datasets other than the sub-datasets in each non-edge region are determined as the sub-datasets of the edge region).

[0143] Among them, a sub-dataset contains multiple sub-data, each sub-dataset corresponds to a region, the sub-dataset contains multiple target points, and there are multiple neighborhood points around any one target point. Calculate the normal vector of each target point among all target points. . Where Cov represents the covariance matrix. is the coordinate of the neighborhood point, O is the centroid of the neighborhood point (geometric center, that is, the average value of all neighborhood points), k is the number of neighborhood points. represents the offset vector between the neighborhood point and the centroid, T represents the transpose operation of the matrix. represents the matrix multiplication of the offset vector and its transpose (to generate a 3×3 matrix). The covariance matrix is subjected to eigenvalue decomposition to obtain three eigenvalues (λ1, λ2, λ3) and the corresponding eigenvectors (v1, v2, v3). The eigenvector corresponding to the minimum eigenvalue is determined as the normal vector between the target point and each neighborhood point. The normal vectors of each target point among all target points can be obtained through the above method.

[0144] In one implementation, the centroid coordinates of a sub-dataset are calculated in the following way: , where O represents the average coordinate (centroid) of the neighborhood point. represents the target point the coordinate of the j-th neighborhood point of, k is the number of neighborhood points. Calculate the average angle of the target point: , where is the average included angle (i.e., the average included angle of the sub - dataset). The meaning of the above formula is: for each target point , calculate its normal vector and the included angle between it and the normal vectors of all neighborhood points . Finally, take the average of all target points within the sub - dataset to obtain the average included angle of the sub - dataset. Since the normal vectors of edge points are quite different from those of neighborhood points, while non - edge points tend to be consistent, when the average included angle ≤ the preset included angle, it is determined as a non - edge area. By accurately identifying edge points, redundant data interference in flat areas can be avoided.

[0145] In the embodiments of the present application, the point cloud data in the dataset to be optimized is segmented to obtain sub - datasets, the normal vectors of the target points and each neighborhood point in the sub - data are determined, and the average included angle in the sub - data is calculated according to the normal vectors; the sub - dataset contains multiple sub - data, the sub - data contains multiple target points, and each target point has multiple neighborhood points; if the average included angle ≤ the preset included angle, it is determined that the area corresponding to the sub - data is a non - edge area, the centroid coordinates of the sub - data are calculated, the coordinates in the sub - data are replaced according to the centroid coordinates to obtain the target sub - data, and the optimized dataset is determined according to each target sub - data.

[0146] In the technical solution provided by the embodiments of the present application, by determining the sub - datasets in each segmentation area corresponding to the average included angle less than or equal to the preset included angle as the sub - datasets in each non - edge area, edge points can be accurately identified, and redundant data interference in flat areas can be avoided.

[0147] Figure 5 is a schematic flow chart of a method for operation recognition and control of a handling device provided in some embodiments, as Figure 5 shown Figure 5 The embodiment is a description of S105. This method is applied to a computer device and includes:

[0148] S1051. Perform operation recognition on the handling device according to the predicted operation trajectory to obtain the recognition result of the operation recognition.

[0149] In some embodiments, the shortest distance between the predicted operation trajectory of the target time period and the obstacle can be obtained, and the recognition result of the operation recognition representing the probability of collision occurring within the target time period is determined according to the shortest distance.

[0150] S1052. When the recognition result of the operation recognition indicates that the probability of collision occurring within the target time period is greater than or equal to the first threshold and less than the second threshold, control the handling device to give an audible alarm at the first alarm frequency.

[0151] S1053. When the recognition result of the operation recognition indicates that the probability of a collision occurring within the target time period is greater than or equal to the second threshold, control the loading and unloading equipment to give a sound alarm at the second alarm frequency, and increase the control resistance of the loading and unloading equipment; the second alarm frequency is greater than the first alarm frequency.

[0152] In the embodiment of the present application, if there is an operation hazard, the target operation vehicle is controlled. For example, the collision probability is determined through the prediction result. If the first threshold ≤ collision probability < the second threshold, a sound alarm is given; if the second threshold < collision probability, the frequency of the sound alarm is increased, and the control resistance of the target operation vehicle is increased.

[0153] In one implementation, there may be a mapping relationship between the distance and the collision probability. For example, a mapping relationship can be established according to the distance between the vehicle and the obstacle, converting the distance into a collision probability. The closer the distance, the higher the collision probability. By setting different collision probability thresholds, hierarchical early warnings for potential collision risks are carried out. When the collision probability is within the lower threshold range, only a sound alarm is triggered to remind the driver to pay attention; when the collision probability further increases, not only the frequency of the sound alarm is increased, but also the resistance of the throttle and steering wheel of the operation vehicle is increased, forcibly reducing the driving speed and steering flexibility of the vehicle, thereby effectively reducing the possibility of a collision.

[0154] In one implementation, increasing the control resistance of the target operation vehicle may include: increasing the control resistance of the throttle and steering wheel of the target operation vehicle. For example: when the driver turns right, the resistance to turning the steering wheel to the right is increased, making it difficult to turn and move forward (achieved by increasing the control resistance of the throttle), thereby reducing the possibility of a collision. This resistance gradually increases until the target operation vehicle stops.

[0155] Based on the same inventive concept, the embodiment of the present application also provides an operation recognition device for a loading and unloading equipment in a warehouse for implementing the operation recognition method of the loading and unloading equipment in the warehouse involved above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the operation recognition device for a loading and unloading equipment in a warehouse provided below can refer to the limitations on the operation recognition method of the loading and unloading equipment in a warehouse in the above text, and will not be repeated here.

[0156] In an exemplary embodiment, Figure 6 is a schematic structural diagram of an operation recognition device for a loading and unloading equipment in a warehouse provided for some embodiments, as Figure 6 shown. The operation recognition device 600 for a loading and unloading equipment in a warehouse includes:

[0157] The data acquisition module 601 is configured to obtain a dataset to be enhanced in the area with a collision risk and a dataset to be optimized in the area without a collision risk from the target point cloud dataset in the warehouse collected by the handling equipment.

[0158] The obstacle detection module 602 is configured to perform obstacle detection on the dataset to be enhanced to obtain an obstacle detection result.

[0159] The lightweight processing module 603 is configured to perform lightweight processing on the dataset to be optimized to obtain an optimized dataset.

[0160] The predicted operation trajectory module 604 is configured to input both the obstacle detection result and the optimized dataset into the anti-collision model to obtain the predicted operation trajectory of the handling equipment during the target time period.

[0161] The recognition and control module 605 is configured to perform operation recognition on the handling equipment according to the predicted operation trajectory and control the operation of the handling equipment according to the recognition result of the operation recognition.

[0162] In some embodiments, the data acquisition module includes a data acquisition unit, a dataset to be enhanced determination unit, and a dataset to be optimized determination unit. The acquisition unit is configured to obtain an initial point cloud dataset in multiple target areas collected by the handling equipment. Among them, the multiple target areas are areas within multiple different azimuth angle ranges respectively collected by the handling equipment, and the union of the multiple different azimuth angle ranges is the full azimuth angle. The dataset to be enhanced determination unit is configured to identify the point cloud data in each area with a collision risk in the initial point cloud dataset in each target area, splice the point cloud data in each area with a collision risk to obtain the dataset to be enhanced. The dataset to be optimized determination unit is configured to determine the point cloud data other than the point cloud data in each area with a collision risk in the initial point cloud dataset in each target area as the dataset to be optimized.

[0163] In some embodiments, the obstacle detection module includes a sampling unit, a model determination unit, and an obstacle detection unit. The sampling unit is configured to sample a preset number of point cloud data from the dataset to be enhanced using the farthest point sampling algorithm to obtain sampled point cloud data. The model determination unit is configured to determine an updated obstacle detection model according to the sampled point cloud data. The obstacle detection unit is configured to perform obstacle detection on the dataset to be enhanced using the updated obstacle detection model to obtain the detection result of the obstacle.

[0164] In some embodiments, the model determination unit is further configured to determine the ratio of the number of obstacle point cloud data in the sampled point cloud data to the total number of the sampled point cloud data as the obstacle occupancy ratio of the sampled point cloud data; determine the target weights of each obstacle category under multiple obstacle categories according to the obstacle occupancy ratio of the sampled point cloud data; and update the preset obstacle detection model according to the target weights of each obstacle category to obtain an updated obstacle detection model.

[0165] In some embodiments, the model determination unit is further configured to determine a smoothing coefficient for smoothing the labels according to the obstacle occupancy ratio of the sampled point cloud data; perform smoothing processing on the true labels of each obstacle category by using the smoothing coefficient to obtain the target labels of each obstacle category; perform obstacle detection on the dataset to be enhanced by using the preset obstacle detection model to obtain the probabilities of the dataset to be enhanced in each obstacle category; determine the loss values of each obstacle category according to the target labels and the probabilities of each obstacle category, and determine the target weights of each obstacle category according to the loss values of each obstacle category.

[0166] In some embodiments, the lightweight processing module includes a segmentation unit, an identification unit, and a lightweight processing unit. The segmentation unit is configured to segment the point cloud data in the dataset to be optimized to obtain sub-datasets in multiple segmentation regions; the identification unit is configured to identify sub-datasets in at least one non-edge region from the sub-datasets in the multiple segmentation regions; and the lightweight processing unit is configured to replace the sub-datasets in each non-edge region with the centroid coordinate data of each non-edge region to obtain an optimized dataset.

[0167] In some embodiments, the identification unit is further configured to determine the normal vectors between each target point in the sub-dataset in each segmentation region and multiple neighboring points; determine the average angle of the sub-dataset in each segmentation region according to the normal vectors; and determine the sub-datasets in each segmentation region corresponding to the average angles less than or equal to the preset angle as the sub-datasets in each non-edge region.

[0168] In some embodiments, the identification control module includes an identification unit and a control unit. The identification unit is configured to perform operation identification on the loading and unloading equipment according to the predicted operation trajectory to obtain the identification result of the operation identification; the control unit is configured to control the loading and unloading equipment to give an audible alarm at a first alarm frequency when the identification result of the operation identification indicates that the probability of a collision occurring within the target time period is greater than or equal to a first threshold and less than a second threshold; and control the loading and unloading equipment to give an audible alarm at a second alarm frequency and increase the control resistance of the loading and unloading equipment when the identification result of the operation identification indicates that the probability of a collision occurring within the target time period is greater than or equal to the second threshold; the second alarm frequency is greater than the first alarm frequency.

[0169] The description of the above device embodiments is similar to that of the above method embodiments, and has beneficial effects similar to those of the method embodiments. For the technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0170] Each module in the above operation recognition device of the loading, unloading and handling equipment in the warehouse can be implemented in whole or in part by software, hardware and their combination. Each of the above modules can be embedded in the processor in the computer device in hardware form or independent of the processor, or stored in the memory in the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0171] In an exemplary embodiment, Figure 7 A schematic structural diagram of a computer device provided for some embodiments. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through Wireless Fidelity (WIFI), a mobile cellular network, Near Field Communication (NFC) or other technologies. When the computer program is executed by the processor, it realizes an operation recognition method for loading, unloading and handling equipment in a warehouse. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball or a touchpad set on the shell of the computer device, or an external keyboard, a touchpad or a mouse, etc.

[0172] Those skilled in the art can understand that Figure 7 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0173] For example, a computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method according to any one of the above embodiments are implemented.

[0174] For example, in an exemplary embodiment, when the processor is used to execute the computer program, it is implemented as follows: from the target point cloud dataset in the warehouse collected by the handling equipment, obtain the dataset to be enhanced in the area with a risk of collision and the dataset to be optimized in the area without a risk of collision; perform obstacle detection on the dataset to be enhanced to obtain an obstacle detection result; perform lightweight processing on the dataset to be optimized to obtain an optimized dataset; input both the obstacle detection result and the optimized dataset into the anti-collision model to obtain the predicted operation trajectory of the handling equipment during the target period; perform operation recognition on the handling equipment according to the predicted operation trajectory, and control the operation of the handling equipment according to the recognition result of the operation recognition.

[0175] In one embodiment, a computer-readable storage medium is provided. When the computer program is executed by a processor, the steps of the method provided in any one of the above embodiments are implemented.

[0176] For example, in an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: from the target point cloud dataset in the warehouse collected by the handling equipment, obtain the dataset to be enhanced in the area with a risk of collision and the dataset to be optimized in the area without a risk of collision; perform obstacle detection on the dataset to be enhanced to obtain an obstacle detection result; perform lightweight processing on the dataset to be optimized to obtain an optimized dataset; input both the obstacle detection result and the optimized dataset into the anti-collision model to obtain the predicted operation trajectory of the handling equipment during the target period; perform operation recognition on the handling equipment according to the predicted operation trajectory, and control the operation of the handling equipment according to the recognition result of the operation recognition.

[0177] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the method provided in any one of the above embodiments are implemented.

[0178] For example, in an exemplary embodiment, a computer program product is provided, including a computer program, which when executed by a processor, implements the following steps: obtaining a dataset to be enhanced in the area with a collision risk and a dataset to be optimized in the area without a collision risk from the target point cloud dataset in the warehouse collected by the handling equipment; performing obstacle detection on the dataset to be enhanced to obtain an obstacle detection result; performing lightweight processing on the dataset to be optimized to obtain an optimized dataset; inputting both the obstacle detection result and the optimized dataset into a collision prevention model to obtain a predicted operation trajectory of the handling equipment during the target period; identifying the operation of the handling equipment according to the predicted operation trajectory, and controlling the operation of the handling equipment according to the identification result of the operation identification.

[0179] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0180] The processor, each functional module or each functional unit in any embodiment of the present application may include any one or more of the following integrations: general-purpose processor, application specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), central processing unit (CPU), graphics processing unit (GPU), embedded neural network processor (NPU), controller, microcontroller, microprocessor, programmable logic device, discrete gate or transistor logic device, discrete hardware component, quantum computing-based data processing logic unit, artificial intelligence (AI) processor, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0181] The memory or computer-readable storage medium in any embodiment of the present application may include at least one of non-volatile memory and volatile memory. The non-volatile memory includes the integration of one or more of the following: Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Ferromagnetic Random Access Memory (FRAM), Flash Memory, magnetic surface memory, optical disc, Compact Disc Read-Only Memory (CD-ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, Resistive Random Access Memory (ReRAM), Magnetoresistive Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene memory, volatile memory, etc. The volatile memory includes the integration of one or more of the following: Random Access Memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), etc.

[0182] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope recorded in the present application.

[0183] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for identifying the operation of loading and unloading equipment in a warehouse, characterized in that, The method includes: Obtaining a dataset to be enhanced in the area with a collision risk and a dataset to be optimized in the area without a collision risk from the target point cloud dataset in the warehouse collected by the handling equipment; Performing obstacle detection on the dataset to be enhanced to obtain an obstacle detection result; Performing lightweight processing on the dataset to be optimized to obtain an optimized dataset; Inputting both the obstacle detection result and the optimized dataset into a collision prevention model to obtain a predicted operation trajectory of the handling equipment during a target period; Performing operation recognition on the handling equipment according to the predicted operation trajectory, and controlling the operation of the handling equipment according to the recognition result of the operation recognition.

2. The method according to claim 1, characterized in that, The step of obtaining a dataset to be enhanced in the area with a collision risk and a dataset to be optimized in the area without a collision risk from the target point cloud dataset in the warehouse collected by the handling equipment includes: Obtaining an initial point cloud dataset in a plurality of target areas collected by the handling equipment; wherein, the plurality of target areas are areas within a plurality of different azimuth angle ranges respectively collected by the handling equipment, and the union of the plurality of different azimuth angle ranges is the full azimuth angle; Identifying the point cloud data in each area with a collision risk in the initial point cloud dataset in each target area, and splicing the point cloud data in each area with a collision risk to obtain the dataset to be enhanced; Determining the point cloud data other than the point cloud data in each area with a collision risk in the initial point cloud dataset in each target area as the dataset to be optimized.

3. The method according to claim 1 or 2, characterized in that, The step of performing obstacle detection on the dataset to be enhanced to obtain an obstacle detection result includes: Sampling a preset number of point cloud data from the dataset to be enhanced by using the farthest point sampling algorithm to obtain sampled point cloud data; Determining an updated obstacle detection model according to the sampled point cloud data; Performing obstacle detection on the dataset to be enhanced by using the updated obstacle detection model to obtain the detection result of the obstacle.

4. The method according to claim 3, wherein The step of determining an updated obstacle detection model according to the sampled point cloud data includes: Determining the ratio of the number of obstacle point cloud data in the sampled point cloud data to the total number of the sampled point cloud data as the obstacle occupancy ratio of the sampled point cloud data; Determining the target weight of each obstacle category under a plurality of obstacle categories according to the obstacle occupancy ratio of the sampled point cloud data; Updating a preset obstacle detection model according to the target weight of each obstacle category to obtain the updated obstacle detection model.

5. The method according to claim 4, wherein The step of determining the target weight of each obstacle category under a plurality of obstacle categories according to the obstacle occupancy ratio of the sampled point cloud data includes: Determining a smoothing coefficient for smoothing the label according to the obstacle occupancy ratio of the sampled point cloud data; Smoothing the true label of each obstacle category by using the smoothing coefficient to obtain the target label of each obstacle category. Using the preset obstacle detection model to perform obstacle detection on the to-be-strengthened dataset, and obtaining the probabilities of the to-be-strengthened dataset in each of the obstacle categories; According to the target labels of each of the obstacle categories and the probabilities of each of the obstacle categories, determining the loss values of each of the obstacle categories, and determining the target weights of each of the obstacle categories according to the loss values of each of the obstacle categories.

6. The method according to claim 1 or 2, characterized in that, The lightweight processing of the to-be-optimized dataset to obtain the optimized dataset includes: Segmenting the point cloud data in the to-be-optimized dataset to obtain sub-datasets in a plurality of segmented regions; Identifying sub-datasets in at least one non-edge region from the sub-datasets in the plurality of segmented regions; Replacing the sub-datasets in each of the non-edge regions with the centroid coordinate data of each of the non-edge regions to obtain the optimized dataset.

7. The method according to claim 6, wherein The identifying sub-datasets in at least one non-edge region from the sub-datasets in the plurality of segmented regions includes: Determining the normal vectors between each target point in the sub-datasets in each of the segmented regions and a plurality of neighboring points; Determining the average angle of the sub-datasets in each of the segmented regions according to the normal vectors; Determining the sub-datasets in each of the segmented regions corresponding to the average angles less than or equal to a preset angle as the sub-datasets in each of the non-edge regions.

8. The method according to claim 1 or 2, characterized in that, The controlling the operation of the loading and unloading handling device according to the recognition result of the operation recognition includes: When the recognition result of the operation recognition indicates that the probability of a collision occurring within the target time period is greater than or equal to a first threshold and less than a second threshold, controlling the loading and unloading handling device to perform a sound alarm at a first alarm frequency; When the recognition result of the operation recognition indicates that the probability of a collision occurring within the target time period is greater than or equal to the second threshold, controlling the loading and unloading handling device to perform a sound alarm at a second alarm frequency and increasing the control resistance of the loading and unloading handling device; the second alarm frequency is greater than the first alarm frequency.

9. An operation recognition device for loading and unloading equipment in a warehouse, characterized in that, The device includes: A data acquisition module, configured to obtain a to-be-strengthened dataset in a region with a collision risk and a to-be-optimized dataset in a region without a collision risk from the target point cloud dataset in the warehouse collected by the loading and unloading handling device; An obstacle detection module, configured to perform obstacle detection on the to-be-strengthened dataset to obtain an obstacle detection result; A lightweight processing module, configured to perform lightweight processing on the to-be-optimized dataset to obtain an optimized dataset; A predicted operation trajectory module, configured to input both the obstacle detection result and the optimized dataset into an anti-collision model to obtain a predicted operation trajectory of the loading and unloading handling device in a target time period; An identification control module, configured to perform operation identification on the loading and unloading handling device according to the predicted operation trajectory, and control the operation of the loading and unloading handling device according to the recognition result of the operation identification.

10. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images