A Smart Scheduling Method Based on 3D Intelligent Detection

Abstract

This invention discloses an intelligent scheduling method based on 3D intelligent detection, comprising the following steps: camera frames are fed into a target detection algorithm for target detection; the detection results are compared and added to a target tracking list; then, the tracked objects are fed into a target classification network for classification, while simultaneously reconstructing 3D position information using depth maps; the classified position information and prior information are fed into a decision module for intelligent decision-making, and then displayed on a client and controlled by a lower-level machine. This invention improves the intelligent scheduling platform by utilizing deep neural networks and LiDAR, enabling precise, efficient, and real-time scheduling of controlled objects within the monitored range without adding additional environmental equipment, by effectively utilizing the position, category, and environmental information of the monitored area.

Description

Technical Field

[0001] This invention belongs to the field of intelligent scheduling and relates to an intelligent scheduling method based on three-dimensional intelligent detection. Background Technology

[0002] With the continuous advancement of intelligent and unmanned operations in the logistics, transportation, and service industries, effectively controlling and scheduling multiple unmanned devices within a scenario, such as logistics vehicles and service robots, and establishing an intelligent scheduling platform to ensure efficient operation while effectively handling various emergencies, has become an urgent problem to solve. While some solutions have emerged in the field of intelligent scheduling, these solutions generally require large-scale additions or modifications to the site or controlled objects, resulting in high costs.

[0003] In existing warehouse navigation intelligent vehicle scheduling methods based on global vision (CN201911046869.9), the OpenCV open-source vision library is used to extract feature patterns from AGV vehicles, including binarization of HSV images and contour detection using the Canny edge detection algorithm. This approach is easily affected by the environment, resulting in poor system robustness. This method uses a target detection network for target identification and classification, offering strong adaptability, applicability to various environments, and good transfer capabilities. Simultaneously, LiDAR is used for localization, resulting in better accuracy and adaptability to various target detection and localization tasks. In contrast, the target fusion method and system based on multi-camera and LiDAR (CN202111490323.X) utilizes multi-camera data for target detection, performs 3D clustering using LiDAR point clouds, and then fuses and matches the information from both. This method uses 3D clustering, which is time-consuming and involves processing ground point cloud information, significantly impacting the clustering results. Summary of the Invention

[0004] To address the aforementioned problems, this invention integrates depth sensors, lightweight deep learning target detection, target classification algorithms, and intelligent decision-making methods to modularly design the scheduling system. This enables state perception, intelligent decision-making, and real-time scheduling without requiring large-scale modifications to the site or controlled objects. The invention feeds camera frames into a target detection algorithm for target detection; then, the detection results are compared and added to a target tracking list; next, the tracked objects are fed into a target classification network for classification, while simultaneously reconstructing 3D position information using depth maps; finally, the classified position information, along with prior information, is fed into a decision-making module for intelligent decision-making, which is then displayed on a client and used to control lower-level machines. This invention improves the intelligent scheduling platform using deep neural networks and LiDAR, effectively utilizing the position, category, and environmental information of the monitored area to perform precise, efficient, and real-time scheduling of controlled objects within the area without adding additional environmental equipment.

[0005] The present invention is achieved by at least one of the following technical solutions.

[0006] An intelligent scheduling method based on 3D intelligent detection includes the following steps:

[0007] 1) The dataset automatic annotation module acquires the current frame and sends it to the object detection network for detection;

[0008] 2) Use the target tracking network to track the detection results of the target detection network and output the tracking results;

[0009] 3) Use a target classification network to classify the tracking results, and input the target bounding box and depth map into the 3D mapping module to reconstruct the 3D position, and output the 3D coordinates in the world coordinate system;

[0010] 4) Input the three-dimensional coordinates of the tracked object, the classification results, and the environmental information into the intelligent decision-making module, use behavior tree to make intelligent decisions, display the decision results on the client, and control and schedule the lower-level machine through serial communication.

[0011] Furthermore, the controlled object positions detected by the target detection network are filtered and then fed into the target tracking network. The target tracking network performs feature extraction and modeling of the tracked object in each frame, obtains the inter-frame correlation metric, and determines the next frame position of the tracked object based on the inter-frame correlation, thereby obtaining the output of the target tracking module.

[0012] Furthermore, the results detected by the target detection network are input into the target tracking network. If it is the first time entering the target tracking network, N trackers are initialized with these results. The number of trackers N is the same as the number of detection boxes detected by the target detection network. If it is not the first time entering the target tracking module, the remaining trackers use Kalman filters to predict using the tracking boxes and calculate the overlap with each of the input detection boxes. Three cases are obtained: some detection boxes and tracking boxes match successfully, some detection boxes do not match, and some tracking boxes do not match. If a detection box and a tracking box match successfully, the Kalman filter of the corresponding tracker is updated using the detection box. If a tracking box does not match a detection box, its own Kalman filter is updated using the tracking box, and the self-update count of the tracker is incremented by one. If a detection box cannot match successfully, a tracker is initialized using this detection box.

[0013] Mix all trackers together and check if the number of frames present after mixing exceeds a threshold. If it does, delete the tracker; otherwise, proceed as follows:

[0014] First, it checks if the overlap of tracking boxes exceeds a threshold. If it does, the reset flags of both trackers are set to true. Then, it checks if the consecutive hit count of all trackers exceeds a threshold. If it does not exceed the threshold, the reset flags of the trackers are set to true. Next, it checks if the number of resets of the trackers exceeds a threshold. If it does, the number of resets is set to zero, and the reset flags of the trackers are set to true. Finally, it checks if the target confidence exceeds a threshold. If it does not exceed the threshold, the reset flags of the trackers are set to true. Trackers with true reset flags input the image information within the tracking boxes into the deep neural network for classification.

[0015] Furthermore, the target classification network uses the EfficientNet feature extraction network as the feature extraction backbone network. The image output by the target tracking network is input into the feature extraction backbone network, and the feature extraction results of the feature extraction backbone network are then input into the fully connected layer and the regression layer to obtain the class probability corresponding to the image. The class with the highest probability is selected as the image class attribute for target classification. The ID obtained in this classification is compared with the ID obtained in the previous classification. If the IDs are the same, the consecutive classification count is incremented by one; otherwise, the consecutive classification count is set to zero. At the same time, the target confidence is reset to the confidence of the classification result, and the reset timer is set to zero. Then, the task of the target tracking module is completed, and all trackers that have not been deleted are retained for the next tracking task.

[0016] Furthermore, the 3D position reconstruction of the 3D mapping module includes the following steps:

[0017] 0) Define the following coordinates:

[0018] The lidar coordinate system is as follows: with the midpoint of the bottom of the lidar as the origin, the x-axis points in front of the lidar, the z-axis is vertically upward, and the y-axis faces forward and is horizontal to the left.

[0019] Camera coordinate system: with the camera optical center as the origin, facing forward, the x-axis is horizontal to the right, the y-axis is vertical to the ground, and the z-axis is parallel to the optical axis and points forward;

[0020] Pixel coordinate system: with the top left corner of the image as the origin, the x-axis is horizontal to the right and the y-axis is vertical downward;

[0021] 1) Start a thread to handle receiving data from the LiDAR;

[0022] 2) Convert the lidar data to a Cartesian coordinate system format;

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] in:

[0030] The measurement ID is the label ID of the data packet;

[0031] scan width This represents the horizontal resolution value.

[0032] beam_altitude_angles is the elevation angle of each laser beam;

[0033] beam_azimuth_angles is the azimuth angle of each laser beam;

[0034] θ encoder The rotation angle of the encoder built into the lidar;

[0035] θ azimuth The azimuth angle of the lidar beam;

[0036] The elevation angle of the lidar beam;

[0037] r is range_mm, which is the sum of the magnitude of the distance vector from the center of the lidar origin coordinate system to the lidar front-end optical device and the magnitude of the distance vector from the lidar front-end optical device to the object being detected;

[0038] n is lidar_origin_to_beam_origin_mm, which is the magnitude of the distance vector from the center of the lidar origin coordinate system to the front-end optical device of the lidar.

[0039] x, y, z are the Cartesian coordinates of the point cloud;

[0040] 3) Project the point cloud data obtained from the LiDAR onto the image to obtain a depth map:

[0041]

[0042] The depth map is a single-channel image with the same resolution as the camera image, where each pixel is filled with the corresponding Zc value; (X l X l Z l ) is the coordinate of the point cloud in the lidar coordinate system, u and v are the pixel coordinates of the corresponding pixel in the image, Zc is the Z-axis value of the corresponding point cloud in the camera coordinate system, M1 is the intrinsic parameter matrix of the camera, M2 is the extrinsic parameter matrix of the lidar to the camera, and M1 and M2 are obtained by joint calibration.

[0043] 4) Based on the target bounding boxes obtained from target recognition, the depth information within the bounding boxes is processed using the k-clustering method:

[0044] First, iterate through all depth values ​​(range) within the target bounding box and calculate the average value (range_mean) of all depth information. Then, initialize three clusters and use [range_mean - area_thresh, range_mean, range_mean + area_thresh] as the representative values ​​for these three clusters.

[0045] Then, iterate through all depth values ​​(range) within the target bounding box again. Calculate the absolute value of each depth value with the representative values ​​of the three clusters, compare the three absolute values, and assign the depth value of this point to the cluster with the smallest absolute value. After one iteration, calculate the average depth values ​​of the three clusters and use this depth value as the new representative value for the three clusters, and start a new iteration. Iteration stops when the absolute value between the new and old representative values ​​is less than `stop_num`, or when the number of iterations exceeds `iter_num`. At this point, re-sort the representative values ​​of the three clusters, and the smallest value is taken as the foreground depth representative value (range). frontThe number of depth values ​​belonging to this cluster is used as the front. num The largest value is considered to be the range representing the background depth. background The number of depth values ​​belonging to this cluster is used as the background. num The values ​​of medium are considered to be the range of medium depth of field. middle The number of depth values ​​belonging to this cluster is used as the middle. num Next, the depth representative value z of the target is calculated. represent .

[0046] Furthermore, the depth representative value z of the target is calculated. represent There are two calculation methods:

[0047] 1. If the device is viewing the target at eye level or from below, then use the range. middle For z represent ;

[0048] II. If the equipment is viewing the target from above, then z represent =range front *front num +range middle *middle num +range background *background num ;

[0049] 5) Obtain the target's coordinates in the world coordinate system:

[0050]

[0051] Where (X) w Y w Z w Z represents the coordinates of the point cloud in the world coordinate system. represent The depth representation of the target is shown; a and b are the midpoints of the target detection box; M1 is the intrinsic parameter matrix of the camera; M3 is the extrinsic parameter matrix from the world coordinate system to the camera coordinate system, obtained through the PNP algorithm and manual calibration.

[0052] Furthermore, the object detection network inputs the collected image frames into the feature extraction backbone network, and then inputs the feature maps of different scales extracted from the backbone network into the detection neck and the decoupled detection head. Finally, the detection head performs regression to obtain the rectangular box orientation of the object and the corresponding class probability.

[0053] Furthermore, the target classification network adopts a local deployment method of deep learning neural networks. The image segmented by the joint detection of target detection and target tracking is input into the feature extraction backbone network. The feature extraction results of the feature extraction backbone network are then input into the fully connected layer and the regression layer to obtain the category probability corresponding to the image. The category with the highest probability is selected as the image category attribute.

[0054] Furthermore, the intelligent decision-making module customizes the intelligent scheduling logic scheme before the system is activated, based on the specific intelligent scheduling tasks and performance requirements, and implements it using a behavior tree scheme. By adopting the behavior tree algorithm, the location and category information of the controlled target and the messages from other auxiliary sensors in the receiving site are input into the behavior tree. The site situation perception and strategy selection are performed according to the pre-built behavior tree flow logic. Finally, the intelligent decision-making results are output to the system through the action nodes in the behavior tree for front-end display or lower-level machine scheduling.

[0055] Furthermore, the automatic dataset annotation module includes annotating and initializing the image sequence of the controlled object to be detected for the specific task, feeding the initial dataset into the object detection network for training to obtain preliminary weights, further acquiring images to obtain an expanded image sequence, loading the preliminary weights into the object detection network, using the weights to infer the expanded image sequence to obtain the pre-selected positions of the target objects in the expanded image sequence, and then manually confirming and fine-tuning the pre-selected positions of the target objects to obtain an expanded dataset, and then using the expanded dataset to train the target detection network, and the process is repeated cyclically.

[0056] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0057] The control system of this invention abstracts business functions into seven modules based on the characteristics of the application scenario. The modules are loosely coupled and can be flexibly customized according to different needs of the scheduling platform usage scenario. The intelligent decision-making module can be customized according to different tasks and requirements, and can maximize the response to specific scheduling needs. This invention combines a full-field depth and distance detection sensor with an image sensor, making the controlled object detection and positioning process efficient and stable. It can achieve the positioning progress required for scheduling tasks without installing additional position sensors or other devices on the controlled object and the site itself, reducing deployment costs and improving performance compared to current intelligent scheduling platforms. The intelligent dataset annotation process allows the scheduling platform to continuously self-optimize during actual deployment and scheduling processes, continuously improving accuracy and possessing self-learning capabilities. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the control process of an intelligent scheduling platform based on three-dimensional intelligent detection according to an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the target detection architecture according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the target classification architecture in an embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram of the target tracking architecture according to an embodiment of the present invention;

[0062] Figure 5 This is a schematic diagram of the lidar detection depth map according to an embodiment of the present invention;

[0063] Figure 6 This is a schematic diagram of the intelligent decision-making logic in an embodiment of the present invention;

[0064] Figure 7 This is a schematic diagram of the intelligent dataset annotation platform according to an embodiment of the present invention;

[0065] Figure 8 This is a schematic diagram of the front-end display of an intelligent scheduling platform based on three-dimensional intelligent detection according to an embodiment of the present invention;

[0066] Figure 9 This is a schematic diagram of the front-end display of an intelligent scheduling platform based on three-dimensional intelligent detection, according to an embodiment of the present invention. Detailed Implementation

[0067] The present invention will be further described below with reference to the embodiments and the accompanying drawings.

[0068] Example 1

[0069] An intelligent scheduling platform based on 3D intelligent detection includes modules such as a target detection module, a target tracking module, a target classification module, a 3D mapping and intelligent scheduling module, a dataset automatic annotation module, and a user interface display module. The target detection module uses a target detection network for detection and inputs the detection results into the target tracking module. The target tracking module uses a target tracking network to track the detection results of the target detection network and inputs the output tracking results into the target classification module. The target classification network classifies the tracking results into targets. Simultaneously, the target bounding box and depth map are input into the 3D mapping module for 3D position reconstruction, outputting 3D coordinates in the world coordinate system. The 3D coordinates of the tracked object, the classification results, and environmental information are jointly input into the intelligent decision-making module, which uses a behavior tree for intelligent decision-making. The decision results are displayed on the client and controlled and scheduled by the lower-level machine via serial communication.

[0070] This invention provides an intelligent scheduling method based on three-dimensional intelligent detection, such as... Figure 1 As shown, the specific steps include:

[0071] 1) The current frame is acquired from the camera and sent to the object detection module for detection. The object detection module employs an object detection algorithm. In this embodiment, the object detection algorithm used is YOLOX, and the object detection network used is the YOLOX-s network.

[0072] The YOLOX-s architecture consists of three parts: a feature extraction backbone network, a detection neck, and decoupled detection heads. The feature extraction backbone network is identical to that of YOLOv5-s, using residual stacking of convolutional layers, batch normalization layers, and activation function layers. However, YOLOX-s replaces the activation function with SiLU. The detection neck uses a feature pyramid structure for feature fusion. Finally, three decoupled heads output the target box's category prediction, location information, and foreground / background information, respectively.

[0073] The collected image frames are input into the feature extraction backbone network, and the feature maps of different scales extracted from the backbone network are then input into the detection neck and the decoupled detection head. Finally, the detection head performs regression to obtain the possible rectangular box orientation of the object and the corresponding class probability.

[0074] 2) Input the target detection results from the target detection module into the target tracking module. If this is the first time entering the target tracking module, initialize N trackers with these results. The number of trackers N is the same as the number of inputs (i.e., the number of detection boxes detected by the target detection network). Each tracker includes a Kalman filter (with the pixel coordinates of the center point of the detection box and the area and aspect ratio of the detection box as state variables), an ID number (default is -1 during initialization), the number of frames present (age), a reset timer (reset_count), the number of consecutive hits (hit_count), the number of self-updates (renew_count), a reset flag (reset_flag), the target confidence score (prob), and the number of consecutive classifications (classic_count).

[0075] If this is not the first time the tracker has entered the target tracking module, the remaining trackers use a Kalman filter to predict the tracking boxes and calculate the overlap ratio with each pair of input detection boxes. The Hungarian algorithm is used to calculate the overlap ratio, based on the Intersection over Union (IOU) of the two boxes (i.e., selecting pairs with high overlap ratios for matching). This results in three scenarios: some detection boxes successfully match the tracking boxes, some detection boxes fail to match, and some tracking boxes fail to match. If a detection box successfully matches a tracking box, the Kalman filter of the corresponding tracker is updated using the detection box; if a tracking box does not match a detection box, its own Kalman filter is updated using the tracking box, and the tracker's self-update count is incremented; if a detection box fails to match, a new tracker is initialized using this detection box. Then, all trackers are mixed together, and their existence frame count is checked to see if it exceeds a threshold. If it does, the tracker is deleted; otherwise, the following process is performed: First, it checks if any tracking boxes have an overlap ratio higher than the threshold (measured using IOU). If it is too high, the reset flags of both trackers are set to true. Next, it checks whether the consecutive hit count of all trackers exceeds a threshold. If it does not exceed the threshold, the tracker's reset flag is set to true. Then, it checks whether the tracker's reset count exceeds a threshold. If it does, the reset count is set to zero, and the tracker's reset flag is set to true. Finally, it checks whether the target confidence level exceeds a threshold. If it does not, the tracker's reset flag is set to true. In this series of checks, if a tracker needs to set its reset flag to true even once, no further checks are required. Trackers that survive all checks (without the reset flag being set to true) have their consecutive classification count incremented by one by the reset timer. Trackers with a true reset flag input the image information within the tracking box into the target classification module.

[0076] 3) The target classification network uses a deep neural network to classify the tracking results, such as... Figure 3 As shown, the image segmented by the joint detection and tracking of the target is input into the feature extraction backbone network (EfficientNet feature extraction network). The feature extraction results from the backbone network are then input into the fully connected layer and the regression layer to obtain the class probability corresponding to the image. The class with the highest probability is selected as the image class attribute for target classification, with classes 1-5 for vehicles. After classification, the ID obtained in this classification is compared with the ID obtained in the previous classification. If they are the same, the consecutive classification count is incremented by one; otherwise, the consecutive classification count is reset to zero. Simultaneously, the target confidence is reset to the confidence of the classification result, and the reset timer is set to zero. The target tracking module then completes its task, and all trackers that are not yet deleted are retained for the next tracking task.

[0077] The target tracking results and depth map are input into the 3D mapping module for 3D position reconstruction. The steps are as follows:

[0078] S0) Define the coordinate system:

[0079] The lidar coordinate system is as follows: with the midpoint of the bottom of the lidar as the origin, the x-axis points in front of the lidar, the z-axis is vertically upward, and the y-axis faces forward and is horizontal to the left.

[0080] Camera coordinate system: with the camera optical center as the origin, facing forward, the x-axis is horizontal to the right, the y-axis is vertical to the ground, and the z-axis is parallel to the optical axis and points forward;

[0081] Pixel coordinate system: with the top left corner of the image as the origin, the x-axis points horizontally to the right, and the y-axis points vertically downwards.

[0082] S1) Start a thread specifically responsible for receiving data from the LiDAR;

[0083] S2) Convert the lidar data to a Cartesian coordinate system format according to the following formula.

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] Where measurementID is the label ID of the data packet.

[0091] scan width This represents the horizontal resolution value.

[0092] beam_altitude_angles represents the elevation angle of each laser beam.

[0093] beam_azimuth_angles is the azimuth angle of each laser beam.

[0094] θ encoder The rotation angle of the encoder built into the lidar.

[0095] θ azimuth The azimuth angle of the lidar beam.

[0096] The elevation angle of the lidar beam.

[0097] r is range_mm, which is the sum of the magnitude of the distance vector from the center of the lidar origin coordinate system to the lidar front-end optical device and the magnitude of the distance vector from the lidar front-end optical device to the object being detected.

[0098] n is lidar_origin_to_beam_origin_mm, which is the magnitude of the distance vector from the center of the lidar origin coordinate system to the front-end optical device of the lidar.

[0099] x, y, z are the Cartesian coordinates of the point cloud.

[0100] S3) According to the formula:

[0101]

[0102] The point cloud data obtained from the LiDAR is projected onto the image to obtain a depth map. The result is as follows: Figure 5 As shown. Each pixel is filled with the corresponding Zc value; in the above formula, the subscript l indicates the coordinates of the point cloud in the LiDAR coordinate system, u and v are the pixel coordinates of the corresponding pixel in the image, Zc is the Z-axis value of the corresponding point cloud in the camera coordinate system, M1 is the intrinsic parameter matrix of the camera, and M2 is the extrinsic parameter matrix of the LiDAR to the camera. M1 and M2 are obtained using joint calibration;

[0103] S4) Based on the target bounding boxes obtained from target recognition, the depth information within the target bounding boxes is processed using k-clustering: First, iterate through all depth values ​​(range) within the target bounding box and calculate the average value (range_mean). Initialize three clusters, using [range_mean - area_thresh, range_mean, range_mean + area_thresh] as the representative values ​​for these three clusters. Then, iterate through all depth values ​​(range) within the target bounding box again, calculating the absolute value of each depth value with the representative value of the three clusters, and comparing the magnitudes of the three absolute values. Assign the depth value of the point to the cluster with the smaller absolute value. After one iteration, calculate the average depth value of the three clusters and use this depth value as the new representative value for the three clusters, and start a new iteration. Iteration stops when the absolute value between the new and old representative values ​​is less than stop_num, or when the number of iterations exceeds iter_num. At this point, re-sort the representative values ​​of the three clusters, and the smallest value is taken as the foreground depth representative value (range). front The number of depth values ​​belonging to this cluster is used as the front. num The largest value is considered to represent the background depth range. background The number of depth values ​​belonging to this cluster is used as the background.num The values ​​representing medium depth of field are considered to be within the range. middle The number of depth values ​​belonging to this cluster is used as the middle. num Next, the depth representation value z of the target is calculated. represent There are two calculation methods:

[0104] If the device is viewing the target at eye level or from below, then take the range. middle For z represent .

[0105] If the device is looking down at the target, then z represent =(range front *front num +range middle *middle num +range background *background num ) / (front num +middle num +background num The above values ​​are area_thresh (1.5), stop_num (0.01), and iter_num (30). These three values ​​can be adjusted according to the actual situation.

[0106] S5) Obtain the coordinates of the target in the world coordinate system:

[0107]

[0108] The values ​​with the subscript 'w' represent the coordinates of the point cloud in the world coordinate system. Z represent The depth is obtained through the above steps; u and v are the midpoints of the target detection boxes; M1 is the intrinsic parameter matrix of the camera; M3 is the extrinsic parameter matrix from the world coordinate system to the camera coordinate system, obtained through the PNP algorithm and manual calibration.

[0109] After the above steps, the three-dimensional coordinates in the world coordinate system are output;

[0110] 4) The 3D coordinates of the tracked object, along with its classification, battery level, site facility occupancy, and task duration, are input into the intelligent decision-making module. Using a customized intelligent scheduling logic scheme based on specific intelligent scheduling tasks and performance requirements before system activation, the location and category information of the controlled target, along with messages from other auxiliary sensors in the site, are input into the behavior tree. The module then performs site situational awareness and strategy selection according to the pre-built behavior tree flow logic. An example of the intelligent decision-making logic is... Figure 6 As shown, the intelligent decision-making results are finally output to the system through the action nodes in the behavior tree for front-end display or lower-level machine scheduling.

[0111] The intelligent decision-making logic is as follows: a main decision-making process loop is constructed under the root node; under the main decision-making loop, a series of decision conditions are set, such as "whether the base has sufficient power", "whether there are visitors", and "whether there are vehicles in key areas". If the decision conditions are met, the corresponding action is executed and the process continues; if the decision conditions are not met, the process continues; within the task execution cycle, the process jumps to the sub-decision loop; in this loop, decision conditions such as whether the vehicle has sufficient power and whether there is a pickup command are set, are set. If the conditions are met, the corresponding decision is executed; if not, the condition loop continues.

[0112] 5) After filtering information from the system such as object power, site facility occupancy, and task time with the intelligent decision-making results, the information is displayed on the client side. Control and scheduling are then performed on the lower-level machine via serial communication, with the following effect: Figure 8 , Figure 9 As shown.

[0113] For the automatic dataset annotation module, image sequences of controlled objects to be detected for the specific task are collected and annotated to initialize the dataset. This initial dataset is then fed into the object detection network for training, yielding preliminary weights. Further image acquisition results in an augmented image sequence. These preliminary weights are then loaded into the object detection network, and inference is performed on the augmented image sequence using these weights to obtain pre-selected locations of target objects within the sequence. These pre-selected locations are then manually confirmed and fine-tuned to obtain the augmented dataset. The augmented dataset is then used to train the object detection network, and this process is repeated cyclically. The user interface is shown below. Figure 7 As shown in Table 1, the performance of the target detection module, based on experiments, is as follows.

[0114] Table 1 Performance of the target detection module

[0115] Class Ave.IOU mAP@0.50 Recall Car 84.6％ 98.99％ 99.85％

[0116] The average inference time per frame is 3ms.

[0117] The performance of the target classification module is shown in Table 2 below:

[0118] Table 2 Performance of the target classification module

[0119]

[0120]

[0121] The average inference time per frame is 1.758ms.

[0122] This invention balances algorithm complexity, detection accuracy, and functional completeness, and has achieved good results in implementation.

[0123] Example 2

[0124] Regarding the 3D mapping module, due to site and cost limitations, it is not suitable to install LiDAR in all situations. To address this, the 3D coordinate reconstruction using LiDAR in Example 1 is changed to using a binocular camera. The characteristic of the 3D coordinate reconstruction process using a binocular camera is that two parallel cameras simultaneously acquire images, feature points are extracted from the two images, and matched. The matched features are then processed using a formula... Where f is the focal length of the camera, b is the epipolar distance of the binocular camera, and x l x is the focal distance between the left optical axis and the camera, the line connecting the feature points, and the optical center plane. r The focal distance between the right optical axis and the camera, the line connecting the feature points, and the optical center plane is used to calculate the depth Z. The resulting depth map can then be integrated into the lidar depth map processing workflow. This improvement enhances the invention's applicability in various environments and reduces installation costs.

[0125] Example 3

[0126] In terms of intelligent decision-making, traditional behavior tree algorithms may struggle to describe the diverse decision-making needs of users. To address this, the intelligent scheduling step in Example 1 using behavior trees is replaced with a posture determination step using behavior trees. The posture determination step using behavior trees is characterized by the user pre-labeling potential scheduling behaviors or suggestions based on the controlled target's possible position and classification information, and external sensor information. This data-label pair is encoded, used to construct a random forest model, and fed into the training process. The model automatically searches for potential scheduling suggestions based on unlabeled data and outputs these suggestions to the scheduling system for subsequent operations. This improvement enhances the platform's ability to describe complex user scheduling task requirements and improves its generalization capabilities.

[0127] Example 4

[0128] Due to varying installation budgets and environments, deep learning object detection schemes are unsuitable for detecting controlled objects in some situations. To address this, the deep learning object detection module in Example 1 is replaced with a foreground separation module. The foreground separation module employs adaptive Gaussian background modeling. It uses 3 to 5 Gaussian models to characterize the features of each pixel in the image to model the background. Then, it matches the pixels of the current frame with the model; if a match is successful, the pixel is considered background; otherwise, it is considered foreground. Simultaneously, during foreground separation, it continuously updates its Gaussian model parameters, aiming to make the modeled background more closely resemble the background of the current video frame, exhibiting adaptive characteristics. This improvement enhances the generalization ability of the invention, reduces its implementation cost, and broadens its application scope.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention in detail to help understand the present invention. Any improvements and substitutions made by those skilled in the art without departing from the principles of the present invention shall fall within the protection scope of the present invention.

Claims

1. An intelligent scheduling method based on three-dimensional intelligent detection, characterized in that, Includes the following steps: 1) The dataset automatic annotation module acquires the current frame and sends it to the object detection network for detection; The target detection network includes a feature extraction backbone network, a detection neck, and a decoupled detection head. The collected image frames are input into the feature extraction backbone network, and the feature maps of different scales extracted by the backbone network are input into the detection neck and the decoupled detection head. Finally, the detection head performs regression to obtain the rectangular box orientation of the object and the corresponding class probability. The controlled object position detected by the target detection network is then fed into the target tracking network after a filtering algorithm. 2) Use the target tracking network to track the detection results of the target detection network and output the tracking results; The results detected by the target detection network are input into the target tracking network. If it is the first time entering the target tracking network, N trackers are initialized with these results. The number of trackers N is the same as the number of detection boxes detected by the target detection network. If it is not the first time entering the target tracking module, the remaining trackers use Kalman filters to predict using the tracking boxes and calculate the overlap with each of the input detection boxes. Three cases are obtained: some detection boxes and tracking boxes match successfully, some detection boxes do not match, and some tracking boxes do not match. If a detection box and a tracking box match successfully, the Kalman filter of the corresponding tracker is updated using the detection box. If a tracking box does not match a detection box, its own Kalman filter is updated using the tracking box, and the self-update count of the tracker is incremented by one. If a detection box cannot match successfully, a tracker is initialized using this detection box. Mix all trackers together and check if the number of frames present after mixing exceeds a threshold. If it does, delete the tracker; otherwise, proceed as follows: First, it checks if the overlap of tracking boxes exceeds a threshold. If it does, the reset flags of both trackers are set to true. Next, it checks if the consecutive hit count of all trackers exceeds a threshold. If it does not exceed the threshold, the reset flags of the trackers are set to true. Then, it checks if the number of resets of the trackers exceeds a threshold. If it does, the number of resets is set to zero, and the reset flags of the trackers are set to true. Finally, it checks if the target confidence exceeds a threshold. If it does not exceed the threshold, the reset flags of the trackers are set to true. Trackers with true reset flags input the image information within their tracking boxes into a deep neural network for classification. 3) Use a target classification network to classify the tracking results, and input the target bounding box and depth map into the 3D mapping module to reconstruct the 3D position, and output the 3D coordinates in the world coordinate system; 4) Input the three-dimensional coordinates of the tracked object, the classification results, and the environmental information into the intelligent decision-making module, use behavior tree to make intelligent decisions, display the decision results on the client, and control and schedule the lower-level machine through serial communication.

2. The intelligent scheduling method based on three-dimensional intelligent detection according to claim 1, characterized in that, The target classification network uses EfficientNet feature extraction network as the feature extraction backbone. The image output from the target tracking network is input into the feature extraction backbone, and the feature extraction results of the feature extraction backbone are then input into the fully connected layer and the regression layer to obtain the class probability corresponding to the image. The class with the highest probability is selected as the image class attribute for target classification. The ID obtained from the current classification is compared with the ID obtained from the previous classification. If the IDs are the same, the consecutive classification count is incremented by one; otherwise, the consecutive classification count is set to zero. At the same time, the target confidence is reset to the confidence of the classification result, and the reset timer is set to zero. Then, the task of the target tracking module is completed. All trackers that have not been deleted are retained for the next tracking task.

3. The intelligent scheduling method based on three-dimensional intelligent detection according to claim 1, characterized in that, The 3D position reconstruction of the 3D mapping module includes the following steps: 0) Define the following coordinates: The lidar coordinate system is as follows: with the midpoint of the bottom of the lidar as the origin, the x-axis points in front of the lidar, the z-axis is vertically upward, and the y-axis faces forward and is horizontal to the left. Camera coordinate system: with the camera optical center as the origin, facing forward, the x-axis is horizontal to the right, the y-axis is vertical to the ground, and the z-axis is parallel to the optical axis and points forward; Pixel coordinate system: with the top left corner of the image as the origin, the x-axis is horizontal to the right and the y-axis is vertical downward; 1) Start a thread to receive data from the LiDAR; 2) Convert the lidar data to a Cartesian coordinate system format; in: measurementID is the label ID of the data packet; scan width This represents the horizontal resolution value. beam_altitude_angles is the elevation angle of each laser beam; beam_azimuth_angles is the azimuth angle of each laser beam; θ encoder The rotation angle of the encoder built into the lidar; θ azimuth The azimuth angle of the lidar beam; The elevation angle of the lidar beam; r is range_mm, which is the sum of the magnitude of the distance vector from the center of the lidar origin coordinate system to the lidar front-end optical device and the magnitude of the distance vector from the lidar front-end optical device to the object being detected; n is lidar_origin_to_beam_origin_mm, which is the magnitude of the distance vector from the center of the lidar origin coordinate system to the front-end optical device of the lidar. x, y, z are the Cartesian coordinates of the point cloud; 3) Project the point cloud data obtained from the LiDAR onto the image to obtain a depth map: The depth map is a single-channel image with the same resolution as the camera image, where each pixel is filled with the corresponding Z-axis. c Value; (X) l Y l Z l Z represents the coordinates of the point cloud in the lidar coordinate system, u and v are the pixel coordinates of the corresponding pixel in the image, and Z is the coordinates of the point cloud in the lidar coordinate system. c M1 is the Z-axis value of the corresponding point cloud in the camera coordinate system, M2 is the intrinsic parameter matrix of the camera, and M2 is the extrinsic parameter matrix of the LiDAR to the camera. M1 and M2 are obtained by joint calibration. 4) Based on the target bounding boxes obtained from target recognition, the depth information within the bounding boxes is processed using the k-clustering method: First, iterate through all depth values ​​(range) within the target bounding box and calculate the average value (range_mean) of all depth information. Then, initialize three clusters and use [range_mean - area_thresh, range_mean, range_mean + area_thresh] as the representative values ​​for these three clusters. Then, iterate through all depth values ​​(range) within the target bounding box again. Calculate the absolute value of each depth value with the representative values ​​of the three clusters, compare the three absolute values, and assign the depth value of this point to the cluster with the smallest absolute value. After one iteration, calculate the average depth values ​​of the three clusters and use this depth value as the new representative value for the three clusters, and start a new iteration. Iteration stops when the absolute value between the new and old representative values ​​is less than `stop_num`, or when the number of iterations exceeds `iter_num`. At this point, re-sort the representative values ​​of the three clusters, and the smallest value is taken as the foreground depth representative value (range). front The number of depth values ​​belonging to this cluster is used as the front. num The largest value is considered to be the range representing the background depth. background The number of depth values ​​belonging to this cluster is used as the background. num The value of a medium range is considered to be the representative value of medium depth of field. middle The number of depth values ​​belonging to this cluster is used as the middle. num Next, the depth representative value z of the target is calculated. represent .

4. The intelligent scheduling method based on three-dimensional intelligent detection according to claim 3, characterized in that, Calculate the depth representation value z of the target represent There are two calculation methods:

1. If the device is viewing the target at eye level or from below, then use the range. middle For z represent ; II. If the equipment is viewing the target from above, then z represent =(range front *front num +range middle *middle num +range background *background num ) / (front num +middle num +background num ); 5) Obtain the target's coordinates in the world coordinate system: Where (X) w Y w Z w Z represents the coordinates of the point cloud in the world coordinate system. represent The depth representation of the target is shown; a and b are the midpoints of the target detection box; M1 is the intrinsic parameter matrix of the camera; M3 is the extrinsic parameter matrix from the world coordinate system to the camera coordinate system, obtained through the PNP algorithm and manual calibration.

5. The intelligent scheduling method based on three-dimensional intelligent detection according to claim 1, characterized in that, The target classification network adopts a local deployment method of deep learning neural networks. The image segmented by joint detection of target detection and target tracking is input into the feature extraction backbone network. The feature extraction results of the feature extraction backbone network are then input into the fully connected layer and the regression layer to obtain the category probability of the image. The category with the highest probability is selected as the image category attribute.

6. The intelligent scheduling method based on three-dimensional intelligent detection according to claim 1, characterized in that, The intelligent decision-making module customizes the intelligent scheduling logic scheme before the system is activated, based on the specific intelligent scheduling tasks and performance requirements, and implements it using a behavior tree scheme. The behavior tree algorithm is used to input the location and category information of the controlled target and receive messages from other auxiliary sensors in the site into the behavior tree. The site situation perception and strategy selection are performed according to the pre-built behavior tree flow logic. Finally, the intelligent decision-making results are output to the system through the action nodes in the behavior tree for front-end display or lower-level machine scheduling.

7. The intelligent scheduling method based on three-dimensional intelligent detection according to any one of claims 1 to 6, characterized in that, The automatic dataset annotation module includes annotating and initializing the image sequence of the controlled object to be detected for the specific task, feeding the initial dataset into the object detection network for training to obtain preliminary weights, further acquiring images to obtain an expanded image sequence, loading the preliminary weights into the object detection network, using the weights to infer the pre-selected positions of the target objects in the expanded image sequence, and then manually confirming and fine-tuning the pre-selected positions of the target objects to obtain an expanded dataset, which is then used to train the object detection network. This process is repeated cyclically.

Images