AGV cargo state detection method, device, equipment and medium

Abstract

The invention discloses an AGV cargo state detection method, device and equipment and a medium, and relates to the field of automatic guided vehicle freight, and the method comprises the steps: collecting image data and point cloud data of a target cargo on an automatic guided vehicle, processing the image data of the target cargo by using a preset training model to obtain boundary coordinate data related to the boundary feature points of the target cargo; generating a point cloud cluster corresponding to the boundary feature point based on the boundary coordinate data and the point cloud data, and determining the current pose of the target cargo through the position information of the point cloud cluster; and based on the cargo type corresponding to the target cargo and the current pose, determining whether there is a to-be-processed cargo having a freight risk in the target cargo, and intervening the cargo having the freight risk based on a preset cargo adjustment scheme. According to the method, the cargo state is detected in real time, the cargos of the automatic guided vehicle are correspondingly adjusted, and normal transportation of the cargos is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of automatic guided vehicle freight transportation, and in particular to an AGV cargo status detection method, device, equipment and medium. Background Art

[0002] At present, in the field of automated warehousing and logistics, automated guided vehicle forklifts have been widely used in cargo handling and warehouse operations. In actual use, during the handling tasks of the automated guided vehicle, the goods are often tilted due to uneven ground, steering, etc.; the automated guided vehicle inserts the goods at the wrong height and overturns the goods; when stacking soft packages, when the fork of the automated guided vehicle is pulled out, the stacked soft packages are subjected to unbalanced force, causing the goods to collapse. Traditional automated guided vehicles usually do not have the function of real-time detection of the status of the goods. Therefore, how to detect the status of the goods in real time and adjust the goods according to the different status of the goods to ensure the normal transportation of the goods by the automated guided vehicle is a problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide an AGV cargo status detection method, device, equipment and medium, which can detect the cargo status of the automatic guided vehicle in real time and make corresponding adjustments to the cargo of the automatic guided vehicle to ensure the normal transportation of the cargo of the automatic guided vehicle. The specific scheme is as follows:

[0004] In a first aspect, the present application provides an AGV cargo status detection method, comprising:

[0005] The preset deep imaging camera is used to collect image data and point cloud data of the target cargo on the automatic guided vehicle, and the image data of the target cargo is processed using a preset training model to obtain boundary coordinate data related to boundary feature points of the target cargo; the preset training model is a deep learning model determined based on the yolov5 target detection algorithm;

[0006] Generate a point cloud cluster corresponding to the boundary feature point based on the boundary coordinate data, the preset depth imaging camera's own equipment parameters and the point cloud data, and determine the current posture of the target cargo through the position information of the point cloud cluster;

[0007] Based on the cargo type corresponding to the target cargo and the current position, it is determined whether there is cargo to be processed with freight risks among the target cargo, and based on the cargo adjustment plan corresponding to the cargo type, cargo transportation intervention is performed on the cargo to be processed with freight risks.

[0008] Optionally, the AGV cargo status detection method further includes:

[0009] The preset deep imaging camera is used to collect cargo image data to obtain historical cargo image data, and the yolov5 target detection algorithm is used to perform deep learning training on the target feature points in the historical cargo image data to obtain the preset training model.

[0010] Optionally, the processing the image data of the target goods by using a preset training model to obtain boundary coordinate data related to boundary feature points of the target goods includes:

[0011] Using the preset training model and the cargo type corresponding to the target cargo to identify the image data to extract boundary feature points of the target cargo;

[0012] Corresponding boundary coordinate data are determined based on the matrix coordinate data of the minimum circumscribed matrix corresponding to the boundary feature points.

[0013] Optionally, generating a point cloud cluster corresponding to the boundary feature point based on the boundary coordinate data, the preset device parameters of the depth imaging camera, and the point cloud data, and determining the current position of the target cargo through the position information of the point cloud cluster includes:

[0014] Aligning the boundary coordinate data, the device parameters of the preset depth imaging camera and the point cloud data to obtain point cloud data corresponding to the boundary feature points, and screening the point cloud data to obtain corresponding point cloud clusters;

[0015] The characteristic posture of the boundary feature point is determined by the position information of the point cloud cluster, and the current posture of the target cargo is determined according to the characteristic posture.

[0016] Optionally, if the cargo type corresponding to the target cargo is multi-layer cargo, then judging whether there is cargo to be processed with freight risks in the target cargo based on the cargo type corresponding to the target cargo and the current position, and performing cargo transportation intervention on the cargo to be processed with freight risks based on the cargo adjustment plan corresponding to the cargo type, includes:

[0017] When the automatic guided vehicle performs a cargo handling operation for the target cargo, judging, based on the current posture, whether a relative displacement between two cargoes in adjacent cargo layers in the target cargo is greater than a first preset displacement threshold;

[0018] If the relative displacement between the two goods is greater than the first preset displacement threshold, the fork on the automatic guided vehicle is controlled to adjust the overall pitch angle of the target goods to cause relative sliding between the two goods so that the relative displacement between the two goods is reduced to a target displacement value; the target displacement value is less than a second preset displacement threshold; the second preset displacement threshold is less than the first preset displacement threshold.

[0019] Optionally, if the cargo type corresponding to the target cargo is inserted cargo, then judging whether there is cargo to be processed with freight risks in the target cargo based on the cargo type corresponding to the target cargo and the current position, and performing cargo transportation intervention on the cargo to be processed with freight risks based on the cargo adjustment plan corresponding to the cargo type, includes:

[0020] When the automatic guided vehicle performs a cargo insertion operation for the target cargo, acquiring a current tilt angle of the target cargo and a distance change value of a relative distance between the target cargo and the automatic guided vehicle based on the current posture;

[0021] If the current inclination angle is greater than a preset cargo inclination angle threshold, and / or the data difference ratio between the distance change value of the relative distance between the target cargo and the automatic guided vehicle and the vehicle travel distance when the automatic guided vehicle performs the cargo insertion operation is greater than a preset percentage, the automatic guided vehicle is controlled to terminate the cargo insertion operation through a preset task interruption instruction.

[0022] Optionally, if the cargo type corresponding to the target cargo is soft-package stacked cargo, then judging whether there is cargo to be processed with freight risks in the target cargo based on the cargo type corresponding to the target cargo and the current position, and performing cargo transportation intervention on the cargo to be processed with freight risks based on the cargo adjustment plan corresponding to the cargo type, includes:

[0023] When the automatic guided vehicle performs a stacking pallet placement operation for the target cargo, determining a maximum stacking sinkage value of the target cargo by using a preset neural network model and the current posture;

[0024] Determine a target placement position of the stacking pallet based on the maximum stacking sinkage value, and control the stacking pallet to perform a stacking pallet pressing operation on the target cargo through the fork of the automatic guided vehicle and the target placement position;

[0025] During the stacking tray pressing operation, detecting the height change and the tilt angle of the target cargo to obtain a current height change value and a current tilt angle change value of the target cargo;

[0026] When the current height change value is zero and equal to the previous height change value, the fork is controlled to be withdrawn from under the stacking pallet through a preset fork withdrawal instruction to complete the stacking pallet placement operation;

[0027] When the real-time tilt angle change value is greater than a preset tilt angle change value, the fork is controlled to lift the stacking pallet through a preset task interruption instruction to terminate the stacking pallet placement operation.

[0028] In a second aspect, the present application provides an AGV cargo status detection device, comprising:

[0029] A coordinate data acquisition module, used to collect image data and point cloud data of target goods on the automatic guided vehicle using a preset deep imaging camera, and process the image data of the target goods using a preset training model to obtain boundary coordinate data related to boundary feature points of the target goods; the preset training model is a deep learning model determined based on the yolov5 target detection algorithm;

[0030] A current posture determination module, used to generate a point cloud cluster corresponding to the boundary feature point based on the boundary coordinate data, the device parameters of the preset depth imaging camera and the point cloud data, and determine the current posture of the target cargo through the position information of the point cloud cluster;

[0031] A cargo transportation intervention module is used to determine whether there is any cargo to be processed with freight risks among the target cargo based on the cargo type corresponding to the target cargo and the current position, and to perform cargo transportation intervention on the cargo to be processed with freight risks based on the cargo adjustment plan corresponding to the cargo type.

[0032] In a third aspect, the present application provides an electronic device, including:

[0033] Memory, used to store computer programs;

[0034] The processor is used to execute the computer program to implement the aforementioned AGV cargo status detection method.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned AGV cargo status detection method is implemented.

[0036] In the present application, a preset deep imaging camera is first used to collect image data and point cloud data of target cargo on an automatic guided vehicle, and the image data of the target cargo is processed using a preset training model to obtain boundary coordinate data related to boundary feature points of the target cargo; the preset training model is a deep learning model trained based on the yolov5 target detection algorithm; a point cloud cluster corresponding to the boundary feature point is generated based on the boundary coordinate data, the device parameters of the preset deep imaging camera itself, and the point cloud data, and the current posture of the target cargo is determined through the position information of the point cloud cluster; based on the cargo type corresponding to the target cargo and the current posture, it is determined whether there is any cargo to be processed with freight risks in the target cargo, and based on the cargo adjustment plan corresponding to the cargo type, cargo transportation intervention is performed on the cargo to be processed with freight risks. The image data and point cloud data of the target cargo are collected by a preset depth imaging camera, and the feature data obtained after the image data is extracted by using the point cloud data and the preset training model generates the posture, that is, by considering the two-dimensional image features and three-dimensional image features of the target cargo to generate the current posture of the target cargo, the freight status of the target cargo can be detected more accurately; by real-time monitoring of the target cargo status, the potential freight risk of the target cargo can be detected in time; according to the target cargo corresponding to different cargo types, adjustments can be made, and potential freight risks can be handled in time, reducing the possibility of freight risks, thereby improving the reliability of the automatic guided vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 This is a flow chart of an AGV cargo status detection method disclosed in this application;

[0039] Figure 2 This is a system framework diagram of an AGV cargo status detection method disclosed in this application;

[0040] Figure 3 This is a schematic diagram of the structure of an AGV cargo status detection device disclosed in this application;

[0041] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] At present, in the field of automated warehousing and logistics, automated guided vehicle forklifts have been widely used in cargo handling and warehouse operations. In actual use, during the automated guided vehicle handling task, the goods are often tilted due to uneven ground, steering and other reasons; the automated guided vehicle inserts the goods at the wrong height and overturns the goods; during the soft package stacking operation, when the automated guided vehicle fork is pulled out, the stacked soft packages are subjected to unbalanced force, causing the goods to collapse. Traditional automated guided vehicles usually do not have the function of real-time detection of the status of the goods. Therefore, this application will specifically introduce an AGV cargo status detection method that can solve the above problems.

[0044] See also Figure 1 As shown, an embodiment of the present invention discloses an AGV cargo status detection method, which may include:

[0045] Step S11, using a preset deep imaging camera to collect image data and point cloud data of the target cargo on the automatic guided vehicle, and using a preset training model to process the image data of the target cargo to obtain boundary coordinate data related to the boundary feature points of the target cargo; the preset training model is a deep learning model determined based on the yolov5 target detection algorithm.

[0046] In this embodiment, the preset deep imaging camera is used to collect cargo image data to obtain historical cargo image data, and the yolov5 (you only look once version5, i.e., target detection algorithm) target detection algorithm is used to perform deep learning training on the target feature points in the historical cargo image data to obtain the preset training model. Specifically, yolov5 is used to perform deep learning and training on the key target feature points in the pre-collected historical cargo image data. The pre-trained model is obtained by extracting and learning the key target feature points in the historical cargo image data, such as edges, corners, specific identifiers, etc., so that the image data of the target cargo can be processed by the pre-trained model.

[0047] In this embodiment, the image data of the target cargo is processed by using a preset training model to obtain boundary coordinate data related to the boundary feature points of the target cargo, including: using the preset training model and the cargo type corresponding to the target cargo to identify the image data to extract the boundary feature points of the target cargo; determining the corresponding boundary coordinate data based on the matrix coordinate data of the minimum circumscribed matrix corresponding to the boundary feature points. It should be pointed out that when the automatic guided vehicle is running, it is first necessary to determine whether to start the cargo status detection system according to the current operation task, whether there are cargo on the vehicle, etc., and send an enable command or a stop command to the edge computing module through the communication module. When the cargo status detection system is running, the preset depth imaging camera is used to collect the image data and point cloud data of the target cargo in real time, and the collected cargo image data and point cloud data are transmitted to the edge computing module through the communication module. The edge computing module recognizes the collected cargo image data through the pre-trained model, and extracts the coordinate data of the minimum circumscribed matrix of the cargo image feature points after the recognition is completed. It should be noted that different types of goods correspond to different extraction methods. For example, box-type goods will identify the feature points of each corner of the box and the top view surface; soft-package type goods will identify the feature points of each corner of the soft package and the curved surface; cylindrical goods will identify the front edges of the cylinder and the circular surface when viewed from above, etc.

[0048] Step S12: generating a point cloud cluster corresponding to the boundary feature point based on the boundary coordinate data, the preset depth imaging camera's own equipment parameters and the point cloud data, and determining the current posture of the target cargo through the position information of the point cloud cluster.

[0049] In this embodiment, the boundary coordinate data, the device parameters of the preset depth imaging camera and the point cloud data are aligned to obtain the point cloud data corresponding to the boundary feature point, and the point cloud data is screened to obtain the corresponding point cloud cluster; the characteristic pose of the boundary feature point is determined by the position information of the point cloud cluster, and the current pose of the target cargo is determined according to the characteristic pose. That is to say, after the cargo image data recognition and extraction is completed, the point cloud algorithm built into the edge computing module aligns the coordinate data, the device parameters of the preset depth imaging camera and the point cloud data collected by the preset depth imaging camera, calculates the point cloud data of the boundary feature point, and screens out the corresponding point cloud cluster, and then calculates the characteristic pose of the boundary feature point according to the position information of the point cloud cluster, thereby restoring the current pose of the cargo; wherein the device parameters of the preset depth imaging camera include but are not limited to camera internal parameters, distortion coefficient and other data; the characteristic pose may include distance, left and right offset, rotation angle and other data. It can be understood that the boundary feature points of box-type cargo and soft-pack type cargo are the feature points of each corner point of the box and soft-pack.

[0050] Step S13: determine whether there is any cargo to be processed with freight risk among the target cargo based on the cargo type corresponding to the target cargo and the current position, and perform cargo transportation intervention on the cargo to be processed with freight risk based on the cargo adjustment plan corresponding to the cargo type.

[0051] In this embodiment, different intervention measures are taken for the pending goods with freight risks based on the goods type in the order message issued by the WMS (Warehouse Management System, i.e., logistics management system).

[0052] In a specific embodiment, when the cargo type corresponding to the target cargo is multi-layer cargo, when the automatic guided vehicle performs cargo handling operations for the target cargo, it is determined based on the current posture whether the relative displacement between two cargoes in adjacent cargo layers in the target cargo is greater than a first preset displacement threshold; if the relative displacement between the two cargoes is greater than the first preset displacement threshold, the fork on the automatic guided vehicle is controlled to adjust the overall pitch angle of the target cargo, so that relative sliding occurs between the two cargoes, so that the relative displacement between the two cargoes is reduced to a target displacement value; the target displacement value is less than a second preset displacement threshold; the second preset displacement threshold is less than the first preset displacement threshold. It can be understood that for multi-layer cargo, during the handling of the target cargo, relative sliding may occur between target cargoes on different layers due to turning, deceleration, etc., causing the cargo to slip. Therefore, when the edge computing module calculates that the relative displacement between two cargoes in adjacent cargo layers in the target cargo is greater than the first preset displacement threshold based on the current posture, the fork on the automatic guided vehicle is controlled to adjust the overall pitch angle of the target cargo, so that the relative displacement between the two cargoes is reduced to the target displacement value. For small cargo, the first preset displacement threshold and the second preset displacement threshold can be set to smaller values, such as setting the first preset displacement threshold to 50mm and the second preset displacement threshold to 20mm. That is to say, when the edge computing module calculates that the upper target cargo exceeds the lower target cargo by 50mm based on the current posture, the fork pitch command is sent to the vehicle control module through the communication module to adjust the overall pitch angle of the target cargo, so that the upper target cargo stops sliding outward and corrects the previous sliding distance. When the sliding distance is corrected to within 20mm, the edge computing module sends a horizontal command to adjust the fork to a horizontal state. On the contrary, if the lower target cargo exceeds the upper target cargo by 50mm, a downward tilt command is sent to make the fork and the target cargo tilt downward as a whole to correct the deviation. For large cargo, the first preset displacement threshold and the second preset displacement threshold can be set to larger values, such as setting the first preset displacement threshold to 300mm and the second preset displacement threshold to 100mm.

[0053] In a second specific implementation, when the cargo type corresponding to the target cargo is inserted cargo, when the automatic guided vehicle performs a cargo insertion operation for the target cargo, the current tilt angle of the target cargo and the distance change value of the relative distance between the target cargo and the automatic guided vehicle are obtained based on the current posture; if the current tilt angle is greater than a preset cargo tilt angle threshold, and / or the data difference ratio between the distance change value of the relative distance between the target cargo and the automatic guided vehicle and the vehicle travel distance when the automatic guided vehicle performs the cargo insertion operation is greater than a preset percentage, the automatic guided vehicle is controlled to terminate the cargo insertion operation through a preset task interrupt instruction. That is to say, this embodiment is aimed at the situation where the automatic guided vehicle hits the cargo because the fork is not aligned with the pallet insertion hole when the automatic guided vehicle inserts the cargo. The current inclination angle of the target cargo and the distance change value of the relative distance between the target cargo and the automatic guided vehicle are calculated based on the current posture by the edge computing module. When the inclination angle of the target cargo increases continuously and exceeds the preset cargo inclination angle threshold, and / or the data difference ratio between the distance change value of the relative distance between the target cargo and the automatic guided vehicle and the vehicle walking distance when the automatic guided vehicle performs the cargo insertion operation is greater than a preset percentage, a task interruption instruction is issued to the automatic guided vehicle through the communication module. For example, when it is detected that the inclination angle of the target cargo increases continuously and exceeds 2°, it is determined that during the insertion and extraction of the cargo, the fork is not aligned with the pallet insertion hole, causing the automatic guided vehicle to hit the cargo. At this time, a task interruption instruction is issued to the automatic guided vehicle through the communication module, and the cargo pickup task is re-performed. When it is detected that the data difference ratio between the distance change value of the relative distance between the target cargo and the automatic guided vehicle and the vehicle walking distance when the automatic guided vehicle performs the cargo insertion operation is greater than 15%, it is determined that the automatic guided vehicle continues to drive without successfully inserting the cargo or the cargo falls. At this time, a task interruption instruction is issued to the automatic guided vehicle through the communication module, and the cargo pickup task is re-performed.

[0054] In a third specific implementation, when the cargo type corresponding to the target cargo is soft-package stacked cargo, when the automatic guided vehicle performs a stacking pallet placement operation for the target cargo, the maximum stacking sinkage value of the target cargo is determined by a preset neural network model and the current posture; the target placement position of the stacking pallet is determined based on the maximum stacking sinkage value, and the stacking pallet is controlled to perform a stacking pallet pressing operation on the target cargo through the fork of the automatic guided vehicle and the target placement position; during the stacking pallet pressing operation, the height change and inclination angle of the target cargo are detected to obtain the current height change value and the current inclination angle change value of the target cargo; when the current height change value is zero and equal to the previous height change value, the fork is controlled to be pulled out from under the stacking pallet through a preset fork withdrawal instruction to complete the stacking pallet placement operation; when the real-time inclination angle change value is greater than the preset inclination change value, the fork is controlled to lift the stacking pallet through a preset task interruption instruction to terminate the stacking pallet placement operation.

[0055] It is understandable that, for soft package stacking, when the stacking pallet presses down the stacked target soft package, the soft package sinks unbalancedly, resulting in unstable stacking of the stacking pallet, and when the fork is pulled out, the goods fall over. In this embodiment, the yolov5 model is first used to segment all the soft packages on the top layer of the stacked soft package, and the positions of the four corner points of the soft package and the surface features are extracted to calculate the length, width and surface curvature of each soft package. It is understandable that after the soft package is pressed down by the stacking pallet, the length or width will be reduced and the surface curvature will be increased. The edge computing module calculates the sinkable height of all soft packages on the top layer, and calculates the optimal solution of the stacking center of the stacking pallet according to the sinkable height to obtain the optimal coordinates of the stacking center of the stacking pallet, and then sends the optimal coordinates to the vehicle control system, which converts the optimal coordinates into map coordinates to obtain the target placement position of the stacking pallet. After the automatic guided vehicle walks to the target placement position, it lowers the fork and stacks the stacking pallet on the stacked soft package. During the stacking process, the cargo status detection system detects the target cargo height of the stacked soft package in real time. When it is detected that the height of the target cargo has not changed for three consecutive frames, it is determined that the target cargo is stacked steadily. At this time, the vehicle control module controls the fork to be pulled out based on the fork withdrawal instruction issued by the edge computing module. If the cargo status detection system detects that the tilt angle change of the stacked soft package is greater than 2°, it is determined that the target cargo has a risk of overturning. At this time, the vehicle control module controls the fork to extend to the previous position based on the task interruption instruction issued by the edge computing module, and lifts the stacking pallet to terminate the stacking pallet placement operation.

[0056] As can be seen from the above, this embodiment first uses a preset deep imaging camera to collect image data and point cloud data of the target cargo on the automatic guided vehicle, and uses a preset training model to process the image data of the target cargo to obtain boundary coordinate data related to the boundary feature points of the target cargo; the preset training model is a deep learning model trained based on the yolov5 target detection algorithm; based on the boundary coordinate data, the preset deep imaging camera's own equipment parameters and the point cloud data, a point cloud cluster corresponding to the boundary feature point is generated, and the current posture of the target cargo is determined through the position information of the point cloud cluster; based on the cargo type corresponding to the target cargo and the current posture, it is determined whether there is any cargo to be processed with freight risks in the target cargo, and based on the cargo adjustment plan corresponding to the cargo type, cargo transportation intervention is performed on the cargo to be processed with freight risks. The image data and point cloud data of the target cargo are collected by a preset depth imaging camera, and the feature data obtained after the image data is extracted by using the point cloud data and the preset training model generates the posture, that is, by considering the two-dimensional image features and three-dimensional image features of the target cargo to generate the current posture of the target cargo, the freight status of the target cargo can be detected more accurately; by real-time monitoring of the target cargo status, the potential freight risk of the target cargo can be detected in time; according to the target cargo corresponding to different cargo types, adjustments can be made, and potential freight risks can be handled in time, reducing the possibility of freight risks, thereby improving the reliability of the automatic guided vehicle.

[0057] It should be noted that the system framework diagram of the AGV cargo status detection method of the present application is as follows: Figure 2 As shown, it includes: a communication module, a depth imaging camera, an edge computing module, and a vehicle control module. The communication module is used for data transmission; the depth imaging camera is used to collect cargo image data, and transmit the collected cargo image data to the edge computing module through the communication module; the edge computing module is used to analyze the received data information to restore the current position of the cargo; the vehicle control module is used to control the operation of the automatic guided vehicle.

[0058] Accordingly, see Figure 3 As shown, the embodiment of the present application also provides an AGV cargo status detection device, which may include:

[0059] A coordinate data acquisition module 11 is used to collect image data and point cloud data of target goods on the automatic guided vehicle using a preset deep imaging camera, and process the image data of the target goods using a preset training model to obtain boundary coordinate data related to boundary feature points of the target goods; the preset training model is a deep learning model determined based on the yolov5 target detection algorithm;

[0060] A current posture determination module 12 is used to generate a point cloud cluster corresponding to the boundary feature point based on the boundary coordinate data, the device parameters of the preset depth imaging camera and the point cloud data, and determine the current posture of the target cargo through the position information of the point cloud cluster;

[0061] The cargo transportation intervention module 13 is used to determine whether there is any cargo to be processed with freight risks among the target cargo based on the cargo type corresponding to the target cargo and the current position, and to perform cargo transportation intervention on the cargo to be processed with freight risks based on the cargo adjustment plan corresponding to the cargo type.

[0062] As can be seen from the above, the present application uses a preset depth imaging camera to collect image data and point cloud data of the target cargo, and uses the point cloud data and the preset training model to extract the feature data obtained after extracting the image data to generate the posture, that is, by considering the two-dimensional image features and three-dimensional image features of the target cargo to generate the current posture of the target cargo, the freight status of the target cargo can be detected more accurately; by real-time monitoring of the target cargo status, the potential freight risks of the target cargo can be detected in time; by adjusting the target cargo corresponding to different cargo types, the potential freight risks can be handled in time, reducing the possibility of freight risks, and thereby improving the reliability of the automatic guided vehicle.

[0063] In some specific embodiments, it also includes:

[0064] The model training unit is used to collect cargo image data using the preset deep imaging camera to obtain historical cargo image data, and to perform deep learning training on target feature points in the historical cargo image data using the yolov5 target detection algorithm to obtain the preset training model.

[0065] In some specific implementations, the coordinate data acquisition module 11 includes:

[0066] A feature point extraction unit, used to identify the image data using the preset training model and the type of goods corresponding to the target goods to extract boundary feature points of the target goods;

[0067] The coordinate data determining unit is used to determine the corresponding boundary coordinate data based on the matrix coordinate data of the minimum circumscribed matrix corresponding to the boundary feature point.

[0068] In some specific implementations, the current posture determination module 12 includes:

[0069] A point cloud cluster obtaining unit, configured to register the boundary coordinate data, the device parameters of the preset depth imaging camera and the point cloud data to obtain point cloud data corresponding to the boundary feature points, and to filter the point cloud data to obtain corresponding point cloud clusters;

[0070] The current posture determination unit is used to determine the characteristic posture of the boundary feature point through the position information of the point cloud cluster, and determine the current posture of the target cargo according to the characteristic posture.

[0071] In some specific embodiments, the cargo transportation intervention module 13 includes:

[0072] a relative displacement judgment unit, configured to judge, when the automatic guided vehicle performs a cargo handling operation on the target cargo, whether a relative displacement between two cargoes in adjacent cargo layers in the target cargo is greater than a first preset displacement threshold based on the current posture;

[0073] The relative displacement reduction unit is used to control the fork on the automatic guided vehicle to adjust the overall pitch angle of the target cargo if the relative displacement between the two cargoes is greater than the first preset displacement threshold, so as to cause relative sliding between the two cargoes so that the relative displacement between the two cargoes is reduced to a target displacement value; the target displacement value is less than a second preset displacement threshold; and the second preset displacement threshold is less than the first preset displacement threshold.

[0074] In some specific embodiments, the cargo transportation intervention module 13 includes:

[0075] a cargo information determination unit, configured to obtain, based on the current posture, a current tilt angle of the target cargo and a distance change value of a relative distance between the target cargo and the automatic guided vehicle when the automatic guided vehicle performs a cargo insertion operation for the target cargo;

[0076] The cargo insertion termination unit is used to control the automatic guided vehicle to terminate the cargo insertion operation through a preset task interruption instruction if the current inclination angle is greater than a preset cargo inclination angle threshold, and / or the data difference ratio between the distance change value of the relative distance between the target cargo and the automatic guided vehicle and the vehicle travel distance when the automatic guided vehicle performs the cargo insertion operation is greater than a preset percentage.

[0077] In some specific embodiments, the cargo transportation intervention module 13 includes:

[0078] a stacking sinkage determination unit, configured to determine a maximum stacking sinkage value of the target cargo by using a preset neural network model and the current posture when the automatic guided vehicle performs a stacking pallet placement operation for the target cargo;

[0079] A stacking pallet pressing unit, used to determine a target placement position of the stacking pallet based on the maximum stacking sinkage value, and control the stacking pallet to perform a stacking pallet pressing operation on the target cargo through the fork of the automatic guided vehicle and the target placement position;

[0080] A cargo information detection unit, used to detect the height change and tilt angle of the target cargo during the stacking tray pressing operation to obtain a current height change value and a current tilt angle change value of the target cargo;

[0081] A stacking pallet extraction unit, used for controlling the fork to be extracted from under the stacking pallet through a preset fork extraction instruction to complete the stacking pallet placement operation when the current height change value is zero and equal to the previous height change value;

[0082] The stacking pallet lifting unit is used to control the fork to lift the stacking pallet to terminate the stacking pallet placement operation through a preset task interruption instruction when the real-time tilt angle change value is greater than a preset tilt angle change value.

[0083] Furthermore, the present application also discloses an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the AGV cargo status detection method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0084] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0085] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0086] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the AGV cargo status detection method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0087] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed AGV cargo status detection method is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0088] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0089] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0090] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0091] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0092] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An AGV cargo status detection method, characterized in that: include: Using a preset depth imaging camera to collect image data and point cloud data of target goods on the automatic guided vehicle, and using a preset training model to process the image data of the target goods to obtain boundary coordinate data related to boundary feature points of the target goods; The preset training model is a deep learning model determined based on the yolov5 target detection algorithm; Generate a point cloud cluster corresponding to the boundary feature point based on the boundary coordinate data, the preset depth imaging camera's own equipment parameters and the point cloud data, and determine the current posture of the target cargo through the position information of the point cloud cluster; Based on the cargo type corresponding to the target cargo and the current position, it is determined whether there is cargo to be processed with freight risks among the target cargo, and based on the cargo adjustment plan corresponding to the cargo type, cargo transportation intervention is performed on the cargo to be processed with freight risks.

2. The AGV cargo status detection method according to claim 1, characterized in that: Also includes: The preset deep imaging camera is used to collect cargo image data to obtain historical cargo image data, and the yolov5 target detection algorithm is used to perform deep learning training on the target feature points in the historical cargo image data to obtain the preset training model.

3. The AGV cargo status detection method according to claim 1, characterized in that: The method of processing the image data of the target goods by using a preset training model to obtain boundary coordinate data related to boundary feature points of the target goods includes: Using the preset training model and the cargo type corresponding to the target cargo to identify the image data to extract boundary feature points of the target cargo; Corresponding boundary coordinate data are determined based on the matrix coordinate data of the minimum circumscribed matrix corresponding to the boundary feature points.

4. The AGV cargo status detection method according to claim 1, characterized in that: The step of generating a point cloud cluster corresponding to the boundary feature point based on the boundary coordinate data, the preset device parameters of the depth imaging camera, and the point cloud data, and determining the current position of the target cargo through the position information of the point cloud cluster includes: Aligning the boundary coordinate data, the device parameters of the preset depth imaging camera and the point cloud data to obtain point cloud data corresponding to the boundary feature points, and screening the point cloud data to obtain corresponding point cloud clusters; The characteristic posture of the boundary feature point is determined by the position information of the point cloud cluster, and the current posture of the target cargo is determined according to the characteristic posture.

5. The AGV cargo status detection method according to claim 1, characterized in that: If the cargo type corresponding to the target cargo is multi-layer cargo, then judging whether there is cargo to be processed with freight risks in the target cargo based on the cargo type corresponding to the target cargo and the current position, and performing cargo transportation intervention on the cargo to be processed with freight risks based on the cargo adjustment plan corresponding to the cargo type, includes: When the automatic guided vehicle performs a cargo handling operation for the target cargo, judging, based on the current posture, whether a relative displacement between two cargoes in adjacent cargo layers in the target cargo is greater than a first preset displacement threshold; If the relative displacement between the two goods is greater than the first preset displacement threshold, the fork on the automatic guided vehicle is controlled to adjust the overall pitch angle of the target goods to cause relative sliding between the two goods so that the relative displacement between the two goods is reduced to a target displacement value; the target displacement value is less than a second preset displacement threshold; the second preset displacement threshold is less than the first preset displacement threshold.

6. The AGV cargo status detection method according to claim 1, characterized in that: If the cargo type corresponding to the target cargo is insertion cargo, then judging whether there is cargo to be processed with freight risk in the target cargo based on the cargo type corresponding to the target cargo and the current position, and performing cargo transportation intervention on the cargo to be processed with freight risk based on the cargo adjustment plan corresponding to the cargo type, includes: When the automatic guided vehicle performs a cargo insertion operation for the target cargo, acquiring a current tilt angle of the target cargo and a distance change value of a relative distance between the target cargo and the automatic guided vehicle based on the current posture; If the current inclination angle is greater than a preset cargo inclination angle threshold, and / or the data difference ratio between the distance change value of the relative distance between the target cargo and the automatic guided vehicle and the vehicle travel distance when the automatic guided vehicle performs the cargo insertion operation is greater than a preset percentage, the automatic guided vehicle is controlled to terminate the cargo insertion operation through a preset task interruption instruction.

7. The AGV cargo status detection method according to claim 1, characterized in that: If the cargo type corresponding to the target cargo is soft-package stacked cargo, then judging whether there is cargo to be processed with freight risks in the target cargo based on the cargo type corresponding to the target cargo and the current position, and performing cargo transportation intervention on the cargo to be processed with freight risks based on the cargo adjustment plan corresponding to the cargo type, includes: When the automatic guided vehicle performs a stacking pallet placement operation for the target cargo, determining a maximum stacking sinkage value of the target cargo by using a preset neural network model and the current posture; Determine a target placement position of the stacking pallet based on the maximum stacking sinkage value, and control the stacking pallet to perform a stacking pallet pressing operation on the target cargo through the fork of the automatic guided vehicle and the target placement position; During the stacking tray pressing operation, detecting the height change and the tilt angle of the target cargo to obtain a current height change value and a current tilt angle change value of the target cargo; When the current height change value is zero and equal to the previous height change value, the fork is controlled to be withdrawn from under the stacking pallet through a preset fork withdrawal instruction to complete the stacking pallet placement operation; When the real-time tilt angle change value is greater than a preset tilt angle change value, the fork is controlled to lift the stacking pallet through a preset task interruption instruction to terminate the stacking pallet placement operation.

8. An AGV cargo status detection device, characterized in that: include: A coordinate data acquisition module, used to collect image data and point cloud data of target goods on the automatic guided vehicle using a preset depth imaging camera, and process the image data of the target goods using a preset training model to obtain boundary coordinate data related to boundary feature points of the target goods; The preset training model is a deep learning model determined based on the yolov5 target detection algorithm; A current posture determination module, used to generate a point cloud cluster corresponding to the boundary feature point based on the boundary coordinate data, the device parameters of the preset depth imaging camera and the point cloud data, and determine the current posture of the target cargo through the position information of the point cloud cluster; A cargo transportation intervention module is used to determine whether there is any cargo to be processed with freight risks among the target cargo based on the cargo type corresponding to the target cargo and the current position, and to perform cargo transportation intervention on the cargo to be processed with freight risks based on the cargo adjustment plan corresponding to the cargo type.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the AGV cargo status detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the AGV cargo status detection method according to any one of claims 1 to 7 is implemented.

Images