Target recognition method and device

Abstract

An embodiment of the present invention provides a method, device, storage medium and electronic device for target recognition, the method comprising: obtaining calibration parameters of a depth camera and characteristic parameters of a target forklift, calibrating a first position of the depth camera relative to a target forklift arm according to the calibration parameters of the depth camera and the characteristic parameters of the target forklift, obtaining a target depth image taken by the depth camera, stratifying the target depth image according to depth to obtain a first group of stratified depth images, inputting the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model, and determining a target contour of the target object and a second position of the target object relative to the target forklift based on the calibration parameters of the depth camera, the first position and the target recognition result, thereby solving the technical problem of low efficiency of forklift operation existing in the related art.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a method, device, storage medium and electronic device for target identification. Background Art

[0002] At present, the demand for automated forklifts in logistics and warehousing systems is relatively high. However, the control methods of automated forklifts in related technologies make it difficult to achieve precise automated control of forklifts in complex scenarios, and it is also difficult to efficiently identify objects that need to be controlled by forklifts.

[0003] Currently, no effective solution has been proposed to the technical problem that it is difficult to effectively identify the object that the forklift needs to control, resulting in low efficiency in forklift operations. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, storage medium and electronic device for target recognition, so as to at least solve the technical problem in the related art that it is difficult to effectively identify the object to be controlled by a forklift, thereby causing low efficiency in forklift operation.

[0005] According to one embodiment of the present invention, there is provided a target recognition method, comprising: obtaining calibration parameters of a depth camera and characteristic parameters of a target forklift; calibrating a first position of the depth camera relative to a target forklift arm according to the calibration parameters of the depth camera and the characteristic parameters of the target forklift, wherein the target forklift is equipped with the target forklift arm; obtaining a target depth image captured by the depth camera; stratifying the target depth image according to depth to obtain a first group of stratified depth images; inputting the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model; and determining a target contour of the target object and a second position of the target object relative to the target forklift based on the calibration parameters of the depth camera, the first position and the target recognition result.

[0006] According to another embodiment of the present invention, a target recognition device is provided, including: a first acquisition module, used to acquire calibration parameters of a depth camera and characteristic parameters of a target forklift; a calibration module, used to calibrate a first position of the depth camera relative to a target forklift arm according to the calibration parameters of the depth camera and the characteristic parameters of the target forklift, wherein the target forklift is equipped with the target forklift arm; a second acquisition module, used to acquire a target depth image captured by the depth camera; a stratification module, used to stratify the target depth image according to depth to obtain a first group of stratified depth images; a processing module, used to input the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model; a determination module, used to determine a target contour of the target object and a second position of the target object relative to the target forklift based on the calibration parameters of the depth camera, the first position and the target recognition result.

[0007] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0008] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0009] Through the present invention, the position of the depth camera relative to the target forklift arm is calibrated by obtaining the calibration parameters of the depth camera and the characteristic parameters of the target forklift. After obtaining the target depth image captured by the depth camera, the target depth image is layered according to the depth to obtain a first group of depth images after layering. The first group of depth images is input into the target recognition model to obtain the target recognition result output by the target recognition model. Finally, based on the calibration parameters of the depth camera, the position of the depth camera relative to the target forklift arm and the target recognition result, the target contour of the target object and the position of the target object relative to the target forklift are determined. By performing subsequent operations according to the contour of the object and the position of the object relative to the forklift, the technical solution of controlling the forklift to operate by simply calculating the forklift coordinates in the related art is replaced. Therefore, the technical problem of difficulty in effectively identifying the object to be controlled by the forklift, which leads to low efficiency of the forklift operation, can be solved in the related art, so as to achieve the technical effect of improving the accuracy of the forklift in identifying the object to be controlled and improving the forklift operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0011] Figure 1 is a hardware structure block diagram of a mobile terminal of a target recognition method according to an embodiment of the present invention;

[0012] Figure 2 is a flow chart of an optional target recognition method according to an embodiment of the present invention;

[0013] Figure 3 is a flow chart of an optional target recognition method according to an embodiment of the present invention;

[0014] Figure 4 is a flow chart of an optional target recognition method according to an embodiment of the present invention;

[0015] Figure 5 is a schematic diagram of an optional target recognition system according to an embodiment of the present invention;

[0016] Figure 6 It is a schematic structural diagram of an optional target recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a target recognition method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1More or fewer components as shown, or with Figure 1 Different configurations shown.

[0020] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the target recognition method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0021] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0022] In this embodiment, a target recognition method running on a mobile terminal, a computer terminal or a similar computing device is provided. Figure 2 is a flow chart of an optional target recognition method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0023] S202, obtaining calibration parameters of the depth camera and characteristic parameters of the target forklift;

[0024] S204, calibrating a first position of the depth camera relative to a target forklift arm according to calibration parameters of the depth camera and characteristic parameters of the target forklift, wherein the target forklift is equipped with a target forklift arm;

[0025] S206, obtaining a target depth image captured by the depth camera;

[0026] S208, stratifying the target depth image according to depth to obtain a first group of stratified depth images;

[0027] S210, inputting the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model;

[0028] S212: Determine a target contour of the target object and a second position of the target object relative to the target forklift based on the target recognition result, the first spatial position, and the calibration parameters of the depth camera.

[0029] Optionally, in this embodiment, the first position and the second position may include but are not limited to two-dimensional position information, three-dimensional position information, and position information that can be calibrated currently or in the future.

[0030] Optionally, in this embodiment, the depth camera can be installed in the middle of a forklift arm, including but not limited to the target forklift, and the forklift arm can include a left forklift arm and a right forklift arm. By installing the depth camera in the middle of the left and right forklift arms of the forklift, recording the calibration parameters of the depth camera and the position parameters of the forklift, and automatically calibrating the spatial position of the depth camera relative to the forklift arm, the depth image of the target can be acquired through the depth camera.

[0031] Specifically, the depth camera can be installed between the left and right forklift arms of the forklift by including but not limited to recording the calibration parameters such as the horizontal field of view angle, vertical field of view angle and oblique field of view angle of the depth camera, recording the width of the forklift, the minimum height of the forklift arm from the ground and the maximum height of the forklift arm from the ground, and the depth camera can rise and fall with the forklift arm. The depth camera collects the depth image in front as the target depth image, and detects the position of the front end of the forklift arm in the depth image. The distance between the depth camera and the left and right forklift arms and the distance between the depth camera and the front end of the forklift arm are calculated in combination with the calibration parameters of the depth camera. When the distance between the depth camera and the front end of the forklift arm is equal to the minimum depth recognized by the depth camera, the above parameters can be manually calibrated.

[0032] Optionally, in this embodiment, the target recognition model may include but is not limited to a model to be trained constructed using the L-BFGS maximum entropy classification algorithm, and the initial recognition model is trained into the target recognition model through the TensorFlow machine learning framework.

[0033] Optionally, in this embodiment, the target recognition result may include but is not limited to identifying which layer of depth image in the target depth image contains the target contour of the target object.

[0034] Optionally, in this embodiment, each label in the above-mentioned set of labels is used to indicate the layer where the contour contained in the corresponding sample depth image in the first set of sample depth images is located. For example, if the target contour of the target object is recognized in the i-th layer, the i-th layer target depth image is determined as the target recognition result to be output. The i-th layer target depth image may also record the parameters of the target forklift, the contour parameters of the target object, etc., and the target forklift can be automatically controlled to perform the target operation through the target recognition result. The target operation may include but is not limited to moving forward, moving backward, stopping, turning left, turning right, raising the forklift arm, lowering the forklift arm, etc.

[0035] Optionally, in this embodiment, the depth camera may include but is not limited to a depth camera based on binocular stereo vision or time-of-flight.

[0036] The above is only an example, and this embodiment does not make any specific limitation.

[0037] Through this embodiment, the position of the depth camera relative to the target forklift arm is calibrated by obtaining the calibration parameters of the depth camera and the characteristic parameters of the target forklift. After obtaining the target depth image captured by the depth camera, the target depth image is layered according to the depth to obtain a first group of layered depth images. The first group of depth images is input into the target recognition model to obtain the target recognition result output by the target recognition model. Finally, based on the calibration parameters of the depth camera, the position of the depth camera relative to the target forklift arm and the target recognition result, the target contour of the target object and the position of the target object relative to the target forklift are determined. By performing subsequent operations according to the contour of the object and the position of the object relative to the forklift, the technical solution of controlling the forklift to operate by simply calculating the forklift coordinates in the related art is replaced. Therefore, the technical problem that it is difficult to effectively identify the object to be controlled by the forklift in the related art, which causes low efficiency in forklift operation, can be solved, so as to achieve the technical effect of improving the accuracy of the forklift in identifying the object to be controlled and improving the forklift operation efficiency.

[0038] In an optional embodiment, the target depth image is layered according to depth to obtain a first group of layered depth images, including: obtaining a depth range of the target depth image; discretizing the target depth image within the depth range according to a preset resolution to obtain a group of the first group of depth images divided according to depth.

[0039] Optionally, in this embodiment, the depth camera starts to collect depth images of the front environment objects and preprocesses the depth images: if the depth range collected by the depth camera is d min to d max, remove all points that are not within this depth range, and then compare them with the feature data set at a certain resolution in d min to d max If the depth map is discretized within the range, then the entire depth map is Where i is the depth, X i Represents all points in the depth data set within this depth range, and then the adjacent X with data i Merge into one layer L i ( L i The depth range is X n Depth to X m The depth of the depth map can be expressed as Where L i It may include but is not limited to arranging in ascending order of depth, corresponding to the first group of depth images mentioned above.

[0040] In an optional embodiment, inputting the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model includes:

[0041] Determine a second group of depth images layer by layer from the first group of depth images in a predetermined order, wherein the second group of depth images is a group of depth images containing the target contour of the target object in the first group of depth images;

[0042] The target recognition result is determined from the second group of depth images, wherein the target recognition result includes the second group of depth images and a layer corresponding to each depth image in the second group of depth images.

[0043] In an optional embodiment, a second group of depth images is determined layer by layer from the first group of depth images in a predetermined order, including: removing depth images containing a number of discretized points less than a first preset threshold from the first group of depth images to obtain a third group of depth images; binarizing each depth image in the third group of depth images to obtain a fourth group of depth images; performing a first target processing on the fourth group of depth images to obtain a fifth group of depth images, wherein the first target processing is used for image denoising and / or image smoothing; performing a second target processing on the fifth group of depth images to obtain the second group of depth images, wherein the second target processing is used to search for the target contour layer by layer in the order of the depth range from small to large, determine the second group of depth images, and determine a sixth group of depth images in the fifth group of depth images except the second group of depth images, and the sixth group of depth images is used to dynamically adjust the minimum value of the depth range.

[0044] Optionally, in this embodiment, after the depth image is layered, a noise layer with few depth points is filtered out before recognition, and then recognition is started layer by layer in order from near to far.

[0045] For example, for L i The layer is first binarized, and then the corrosion and expansion algorithms are used multiple times to remove noise and smooth the image. Then the contour algorithm is used to find the target contour layer by layer. When the object contour is detected in this layer, the required target contour is obtained after eliminating interference items according to parameters such as aspect ratio, major and minor axis and threshold. When the target contour is recognized, the L algorithm stops. j (j>i) layer identification. Among them, when L m After the target contour is identified in the layer, d can be dynamically adjusted according to the layer depth. min is d min '(d m -Δδ), so that subsequent target recognition can quickly locate the layer where the target contour is located. When the target contour is not recognized in all layers, d min 'Resize to d min .

[0046] It should be noted that, in this embodiment, the process of identifying all the filtered layers can be but is not limited to a parallel recognition process, and the recognition results are arranged from near to far according to the depth of the layer, and the nearest layer depth in the recognition result is recorded. In the next recognition, the nearest depth in the previous recognition result is preferentially selected to start recognition. If the target contour is recognized, the recognition of other layers will no longer be performed.

[0047] In an optional embodiment, before inputting the first group of depth images into the target recognition model, the method further includes: repeatedly performing the following steps until the difference between the first recognition probability and the second recognition probability output by the initial recognition model is less than or equal to the predetermined threshold: acquiring a first group of sample depth images and a second group of sample depth images captured by a depth camera, and a group of labels corresponding one-to-one to the first group of sample depth images; inputting the first group of sample depth images and the second group of sample depth images into the initial recognition model to generate the first sample recognition result, wherein the first sample recognition result includes the first recognition probability and the second recognition probability output by the initial recognition model, and the first recognition probability The rate is used to represent the probability that the first sample recognition result recognizes that the contour of the sample object is included in the sample depth image, and the second recognition probability is used to represent the probability that the first sample recognition result recognizes that the contour of the sample object is not included in the sample depth image; when the difference between the first recognition probability and the second recognition probability is greater than the predetermined threshold, the parameters in the initial recognition model are adjusted; when the difference between the first recognition probability and the second recognition probability is less than the predetermined threshold, the first sample recognition result is compared with the set of labels to obtain the first set of sample depth images and the layer corresponding to each sample depth image in the first set of sample depth images.

[0048] Optionally, in this embodiment, the first recognition probability is that the recognition result of the initial recognition model is a contour containing the sample object, and the second recognition probability is that the recognition result of the initial recognition model is a contour not containing the sample object.

[0049] Optionally, in this embodiment, the depth image collected by the depth camera is used as a training data set after recognition. In order to ensure the reliability of the training data set, the depth images collected for multiple consecutive recognitions of the target contour may be considered to contain the target, and the depth images collected for multiple consecutive failures to recognize the target contour may be considered not to contain the target. For example, in F i-1 、F i 、F i+1 These three depth images all contain targets, so extract F i As a training data set containing the target, the same treatment is applied to the case where the target is not contained. The mathematical model is constructed using the L-BFGS maximum entropy classification algorithm, and the above training data set is trained using the TensorFlow machine learning framework. When the number of training data sets increases gradually and the trained model becomes increasingly accurate (corresponding to the difference between the first recognition probability and the second recognition probability being less than the predetermined threshold), the trained model is used in the layered depth map recognition. The process is as follows: According to Li The depth of L is gradually increased, and the mathematical model is used to determine whether there is a target contour. i After the target is identified in the image processing algorithm, the characteristic parameters of the target contour are identified by using the above-mentioned contour recognition, and L k The data of (k<i) is used for subsequent forklift motion control.

[0050] In an optional embodiment, a first group of sample depth images, a second group of sample depth images and a group of labels are input into an initial recognition model to generate a first sample recognition result, including: arranging the first group of sample depth images according to depth size based on a group of labels to obtain a first group of sample depth images arranged in a target order, wherein the target order is an order arranged from small to large depth; inputting the first group of sample depth images and the second group of sample depth images arranged in the target order into the initial recognition model to generate a first sample recognition result; comparing the first sample recognition result with a group of labels to obtain the first group of sample depth images and the layer corresponding to each sample depth image in the first group of sample depth images.

[0051] Optionally, in this embodiment, the above-mentioned target order includes but is not limited to the order from small to large depth values. The above-mentioned first sample recognition result can determine whether the output result of the above-mentioned initial recognition model contains contour information of the sample object or does not contain contour information of the sample object, and in the case of containing contour information of the sample object, the layer corresponding to the depth of the sample depth image.

[0052] In an optional embodiment, the first group of sample depth images, the second group of sample depth images and the group of labels are input into the initial recognition model to generate the first sample recognition result, including: removing the depth images containing the number of discretized points less than the second preset threshold from the first group of sample depth images and the second group of sample depth images to obtain a third group of sample depth images; performing a third target processing on each sample depth image in the third group of sample depth images to obtain a fourth group of sample depth images, wherein the third target processing is used to determine the sample depth image including the contour of the sample object from the sample depth images; and comparing the fourth group of sample depth images with the group of labels to generate the first sample recognition result. .

[0053] Optionally, in this embodiment, several consecutive frames of depth images in which the target contour is recognized and the depth images in which the target contour is not recognized can be used as training data sets, but are not limited to, and the two types of data are labeled as "contains the target" and "does not contain the target", and then the L-BFGS maximum entropy classification algorithm is used for the training data set. The obtained mathematical model will be used to determine whether the target contour is contained in the layered depth image, so as to quickly locate the L i layer, accelerates the recognition of target contours, and the positioning process is as follows: According to L i The depth of L is gradually increased, and the mathematical model is used to determine whether there is a target contour. i After the target is identified in the image processing algorithm, the characteristic parameters of the target contour are identified, and the identification of subsequent layers is stopped, and L is retained. k The data of (k<i) is used for subsequent forklift motion control.

[0054] In an optional embodiment, after determining the target contour of the target object and the second position of the target object relative to the target forklift based on the calibration parameters of the depth camera, the first position and the target recognition result, the method further includes: controlling the target forklift to perform a target operation corresponding to the second position.

[0055] Optionally, in this embodiment, the target recognition results may include but are not limited to the recognized target contour combined with the collected depth information and the calibration parameters of the depth camera to calculate the characteristic parameters of the target object and the position information relative to the forklift, the characteristic parameters include: the position and depth of the upper left, lower left, upper right, lower right and center, and the width and height of the target object, and the spatial position information includes: the distance of the target from the forklift, the angle between the plane where the target contour is located and the plane where the forklift is located, and the inclination of the target contour relative to the plane where the left and right forklift arms are located.

[0056] Specifically, for example, a two-dimensional coordinate system can be established for the depth map, and the value of each point represents the depth information of the point. This coordinate system can be expressed as the position and depth information of the four vertices and the center point of the upper left, lower left, upper right, and lower right corresponding to the target contour. The width and height of the target can be calculated using trigonometric functions in combination with the camera calibration parameters and the position coordinates and depth information of the four vertices of the target contour; the angle between the plane where the target contour is located and the plane where the forklift is located can be calculated based on the depth difference between the left and right ends of the target contour and the calibration parameters of the camera; the inclination of the target contour relative to the plane where the left and right forklift arms are located can be calculated based on the height difference between the left and right ends of the target contour and the calibration parameters of the camera.

[0057] In an optional embodiment, controlling the target forklift to perform a target operation corresponding to the second spatial position includes: when the layer where the target contour of the target object is located is not the layer closest to the target forklift in the target depth image, determining whether there is an obstacle that hinders the movement of the target forklift on the layer between the layer where the target contour is located and the target forklift; and when the obstacle exists on the layer between the layer where the target contour is located and the target forklift, controlling the target forklift to stop moving.

[0058] Optionally, in this embodiment, when it is detected that the layer where the target contour of the target object is located is the layer closest to the target forklift in the target depth image, a movement instruction is sent to the target forklift to control the target forklift to move toward the target object so that the forklift arm faces the target object and inserts the target object to move the target object. That is to say, when the layer L where the target object is located is i When it is the layer closest to the forklift, the forklift can be controlled to move toward the target according to the contour information of the target object and its spatial position information relative to the forklift.

[0059] Optionally, in this embodiment, when judging L k When there is an obstacle at the (k<i) layer that affects the movement of the forklift, the forklift may be controlled to stop moving, or the forklift may be controlled to bypass the obstacle and continue to move toward the target object.

[0060] In an optional embodiment, controlling the target forklift to move toward the target object includes: controlling the target forklift to move toward the target object according to the target contour and spatial position information of the target object relative to the target forklift.

[0061] Optionally, in this embodiment, the characteristic parameters of the target object are determined according to the above-mentioned target contour and the calibration parameters of the depth camera installation to control the forklift to move toward the target, the forklift is controlled to move forward or backward according to the minimum depth of the upper left, lower left, upper right, lower right and center points of the target object and the distance between the depth camera and the front end of the forklift arm, the forklift is controlled to move left and right and the forklift arm is controlled to move up and down according to the position of the center point of the target object and the distance between the depth camera and the left and right forklift arms, and the forklift rotation angle is controlled according to the depth difference between the upper left, upper right and lower left and lower right of the target to ensure that the forklift arm is facing the target.

[0062] In an optional embodiment, when the target recognition result also indicates the target position of the target object, the target forklift is controlled to perform a target operation corresponding to the target position, including: when the layer where the target contour of the target object is located is not the layer closest to the target forklift in the target depth image, determining whether there is an obstacle that hinders the movement of the target forklift on the layer between the layer where the target contour is located and the target forklift; when there is no obstacle on the layer between the layer where the target contour is located and the target forklift, controlling the target forklift to move toward the target object.

[0063] Optionally, in this embodiment, when the target contour of the target object is located in the layer L i There are other levels L away from the forklift k When (k<i) exists, L will be judged first k Whether there are obstacles on the (k<i) layer that affect the movement of the forklift, the forklift is controlled to move toward the target object.

[0064] In an optional embodiment, controlling the target forklift to perform the target operation corresponding to the second position includes: determining a preset motion trajectory of the target forklift based on the second position, wherein the preset motion trajectory is the motion trajectory of the target forklift arm aimed at the target object; when the actual operation trajectory of controlling the target forklift to move toward the target object does not match the preset motion trajectory, controlling the target forklift to update the movement direction to control the target forklift to move according to the preset motion trajectory.

[0065] Optionally, in this embodiment, whether the forklift's motion trajectory is conducive to the forklift arm aligning with the target can be determined in real time, including but not limited to, when the forklift is moving in the correct direction, the movement of the forklift under the current instruction is not interfered with, otherwise the control instruction is updated to intervene in the forklift movement in time.

[0066] In an optional embodiment, determining a preset motion trajectory of the target forklift based on the second spatial position includes: when the target depth detected by the depth camera is greater than the sum of the distance between the depth camera and the front end of the target forklift arm and the preset safety distance, adding parameters for controlling the forward motion of the target forklift to the preset motion trajectory; and / or when the target depth detected by the depth camera is less than the preset safety distance, adding parameters for controlling the backward motion of the target forklift to the preset motion trajectory; and / or when the horizontal distance difference between the center of the target object and the center of the depth image is greater than a third threshold, adding parameters for controlling the left and right translational motion of the target forklift to the preset motion trajectory; and / or when the vertical distance difference between the center of the target object and the center of the depth image is greater than a fourth threshold, adding parameters for controlling the up and down translational motion of the target forklift to the preset motion trajectory. The method comprises the steps of: adding the parameters for controlling the target forklift to perform a first alarm operation and stopping the movement of the forklift arm to the preset motion trajectory when the height to be raised of the forklift arm of the target forklift is greater than the maximum ground clearance height of the forklift arm or the height to be lowered is less than the minimum ground clearance height of the forklift arm, wherein the first alarm operation is used to indicate that the forklift arm cannot be aligned with the target object; and / or when the plane where the target object is located is not parallel to the plane of the target forklift, adding the parameters for controlling the target forklift to perform a rotation operation to the preset motion trajectory; and / or when the angle formed by the target object and the plane where the forklift arm is located is greater than a fifth threshold, adding the parameters for controlling the target forklift to perform a second alarm operation to the preset motion trajectory, wherein the second alarm operation is used to indicate that the target object is tilted.

[0067] Optionally, in this embodiment, when the target depth detected by the depth camera is greater than the distance d between the depth camera and the front end of the forklift arm min When the safety distance is added, the forklift is controlled to move forward; when the forklift is less than the safety distance from the target, the forklift is controlled to move backward; when the horizontal distance difference between the center of the target object and the center of the depth image is greater than the threshold, the forklift is controlled to translate left and right, and when the vertical distance difference between the center of the target object and the center of the depth image is greater than the threshold, the forklift arm is controlled to rise and fall. If the height to which the forklift arm needs to rise or fall is greater than or less than the maximum height above the ground and the minimum height above the ground of the forklift arm, an alarm is issued and the forklift arm movement is stopped; when the plane where the target contour of the target object is located is not parallel to the plane where the forklift is located (that is, when the depth difference between the left and right ends exceeds the threshold): 1) the depth on the left is greater than the depth on the right, the forklift is controlled to rotate left; 2) the depth on the right is greater than the depth on the left, the forklift is controlled to rotate right; when the angle formed by the target contour of the target object and the plane where the left and right forklift arms are located exceeds a certain threshold, it is considered that the target object is tilted, which is not conducive to forklift handling, and an alarm is issued.

[0068] Optionally, in this embodiment, Figure 3 Yes, if Figure 3 As shown, Figure 3 The meanings of the symbols in the middle are as follows:

[0069] G: ground;

[0070] V1: The field of view angle plane of the depth camera. V1 moves with the up and down movement of the forklift arm and is perpendicular to G;

[0071] L1: The plane where the forklift arm is located and is perpendicular to G;

[0072] L2: The plane where the target contour is located and is perpendicular to G;

[0073] d1: The distance from the upper left corner of the target contour to L1;

[0074] d1': the position of the upper left corner of the target contour on V1;

[0075] d2: The distance from the upper right corner of the target contour to L1;

[0076] d2': the position of the upper right corner of the target contour on V1;

[0077] d3: The distance from the lower right corner of the target contour to L1;

[0078] d3': the position of the lower right corner of the target contour on V1;

[0079] d4: The distance from the lower left corner of the target contour to L1;

[0080] d4': the position of the lower left corner of the target contour on V1;

[0081] c1: the center position of V1;

[0082] c2: target contour center position;

[0083] c2': the position of C2 on V1;

[0084] The above-mentioned controlling the target forklift to perform the target operation corresponding to the second spatial position may include but is not limited to the following examples:

[0085] S1. When the horizontal distance between C2' and C1 is greater than a certain threshold, control the forklift to move left and right;

[0086] S2. When the vertical distance between C2' and C1 is greater than a certain threshold, control the forklift arm to rise and fall;

[0087] S3. When L1 and L2 form a certain angle, that is, when the distance difference between d1 and d2 or the distance difference between d3 and d4 is greater than a certain threshold, rotate the forklift (control L1 and L2 to be parallel) to reduce the distance difference between d1 and d2 or the distance difference between d3 and d4;

[0088] S4. When the vertical distance between d1' and d2' or the vertical distance between d3' and d4' is greater than a certain threshold, it means that the target is tilted, and the forklift stops moving and issues an alarm;

[0089] S5. When the horizontal distance between d1' and d2' or the horizontal distance between d3' and d4' is smaller than the horizontal distance between the two forklift arms, the forklift stops moving and gives an alarm.

[0090] The present invention will be further explained below in conjunction with specific embodiments:

[0091] Figure 4 1 is a flow chart of an optional target recognition method according to an embodiment of the present invention, the flow chart includes the following steps:

[0092] Step 402: Install the depth camera between the left and right forklift arms of the forklift, record the calibration parameters of the depth camera and the characteristic parameters of the forklift, and automatically calibrate the spatial position of the depth camera relative to the forklift arms.

[0093] Step 404: collect the depth image in front of the forklift, and implement stratification from small to large depth.

[0094] Step 406: Process the layered depth map layer by layer, and use an image processing algorithm to identify and extract the contour of the target object.

[0095] Step 408: Use the identified depth map as a training data set, build a mathematical model, and use machine learning self-training to accelerate the determination of whether the depth map contains the contour of the target object.

[0096] Step 410: Combine the identified target object contour with the depth data and the calibration parameters of the depth camera to calculate the contour information of the target object and the spatial position information relative to the forklift.

[0097] Specifically, in step 402, the calibration parameters of the depth camera such as the horizontal field of view angle, the vertical field of view angle, and the oblique field of view angle are recorded, the width of the forklift, the minimum height of the forklift arm from the ground, and the maximum height of the forklift arm from the ground are recorded, and the depth camera is installed between the left and right forklift arms of the forklift, and the depth camera can rise and fall with the forklift arm. The depth camera collects the depth map in front and detects the position of the front end of the forklift arm in the depth map. Combined with the calibration parameters of the depth camera, the distance between the depth camera and the left and right forklift arms and the distance between the depth camera and the front end of the forklift arm are calculated. In particular, when the distance between the depth camera and the front end of the forklift arm is less than the minimum depth recognized by the cell depth camera, the above parameters need to be manually calibrated.

[0098] In step 404, the depth camera starts to collect depth images of the front environment objects and preprocesses the depth images: if the depth range collected by the depth camera is d min to d max , remove all points that are not within this depth range, and then process the feature data set at a certain resolution in d min to d max If the depth map is discretized within the range, then the entire depth map is Where i is the depth, X i Represents all points in the depth data set within this depth range, and then the adjacent X with data i Merge into one layer L i ( L i The depth range is X n Depth to X m The depth of the depth map can be expressed as Where L i Arrange them in order of depth from smallest to largest.

[0099] After the depth image is layered in step 406, the noise layer with few depth points is filtered out before recognition, and then recognition is started layer by layer in the order from near to far. i The layer is first binarized, and then the erosion and dilation algorithms are used multiple times to remove noise and smooth the image. The contour algorithm is then used to find the contour of the target object layer by layer. When an object contour is detected in this layer, the required target object contour is obtained after eliminating interference items based on parameters such as aspect ratio, major and minor axis, and threshold. When the target object contour is recognized, the L algorithm stops. j (j>i) layer identification. Among them, when L m After the target object contour is identified in the layer, d can be dynamically adjusted according to the layer depth. min is d min '(d m-Δδ), so that the subsequent target object recognition can quickly locate the layer where the target object contour is located. When the target object contour is not recognized in all layers, d min 'Resize to d min .

[0100] In particular, the algorithm execution order can be improved:

[0101] 1) Start parallel recognition of all filtered layers, arrange the recognition results from near to far according to the depth of the layer, and record the depth of the nearest layer in the recognition results;

[0102] 2) In the next recognition, the nearest depth in the last recognition result is selected as the starting point. If the contour of the target object is recognized, no other layers are recognized;

[0103] 3) When the target object contour is not recognized in 2), repeat step 1).

[0104] In step 408, the depth image captured by the depth camera is used as a training data set after recognition. In order to ensure the reliability of the training data set, the depth images that have been continuously recognized to have the contour of the target object for many times can be considered to contain the target object, and the depth images that have not been continuously recognized to have the contour of the target object can be considered to not contain the target object. For example, in F i-1 、F i 、F i+1 These three depth images all contain the target object, so extract F i As a training data set containing the target object, the same treatment is applied to the case where the target object is not contained. The mathematical model is constructed using the L-BFGS maximum entropy classification algorithm, and the above training data set is trained using the TensorFlow machine learning framework. When the number of training data sets increases gradually and the trained model becomes increasingly accurate (the loss function is small enough), the trained model is used in the layered depth map recognition. The process is as follows: According to L i The depth of L is gradually increased, and the mathematical model is used layer by layer to determine whether there is a target object contour. i After the target object is identified in the image processing algorithm, the characteristic parameters of the target object contour are identified by using the above-mentioned contour recognition, and L k The data of (k<i) is used for subsequent forklift motion control.

[0105] In step 410, the identified target object contour is combined with the collected depth information and the calibration parameters of the camera to calculate the characteristic parameters of the target object and the spatial position information relative to the forklift. The characteristic parameters include: the position and depth of the upper left, lower left, upper right, lower right and center, and the width and height of the target object. The spatial position information includes: the distance of the target object from the forklift, the angle between the plane where the target object contour is located and the plane where the forklift is located, and the inclination of the target object contour relative to the plane where the left and right forklift arms are located.

[0106] Specifically, a two-dimensional coordinate system can be established for the depth map, and the value of each point represents the depth information of the point. This coordinate system can concisely and clearly represent the position and depth information of the four vertices and the center point of the upper left, lower left, upper right, and lower right corresponding to the target object outline. The width and height of the target object can be calculated using trigonometric functions in combination with the camera calibration parameters and the position coordinates and depth information of the four vertices of the target object outline; the angle between the plane where the target object outline is located and the plane where the forklift is located can be calculated based on the depth difference between the left and right ends of the target object outline and the calibration parameters of the camera; the inclination of the target object outline relative to the plane where the left and right forklift arms are located can be calculated based on the height difference between the left and right ends of the target object outline and the calibration parameters of the camera.

[0107] Figure 5 is a schematic diagram of an optional target recognition system according to an embodiment of the present invention. Figure 5 As shown, the system includes:

[0108] Data acquisition unit, data processing and control unit, wherein the data processing and control unit further comprises: an identification module 502 , a learning module 504 , a control module 506 and an abnormality module 508 .

[0109] The data acquisition unit uses a structured light depth camera, and optionally a depth camera based on binocular stereo vision or time-of-flight can be used; the data acquisition unit is connected to the data processing and control unit using an Internet port or a universal serial bus;

[0110] The recognition module 502 is used to recognize the contour of the target object in the depth image and calculate the characteristic parameters of the contour of the target object and the spatial position information of the target object relative to the forklift. The recognition method and process have been described above and will not be repeated here;

[0111] Learning module 504, the recognition module uses the recognition results of "with target object" and "without target object" as labels to label the depth map after layering, and uses this as a training data set to train the mathematical model. When the mathematical model is accurate enough and the loss function is small enough, the mathematical model is used to accelerate the recognition module to L iThe judgment of whether there is a target object in the layer; in particular, in order to ensure the reliability of the training data set, the depth image frames with consistent and continuous recognition results can be used as the training data set. How to ensure the reliability of the data set has been described above and will not be repeated here.

[0112] The control module 506 is used to control the forklift to move toward the target object according to the characteristic parameters of the target object calculated in the recognition module 502 and the calibration parameters of the camera installation, control the forklift to move forward or backward according to the minimum depth of the upper left, lower left, upper right, lower right and center point of the target object and the distance between the depth camera and the front end of the forklift arm, control the left and right translation of the forklift and the up and down lifting of the forklift arm according to the position of the center point of the target object and the distance between the depth camera and the left and right forklift arms, and control the rotation angle of the forklift according to the depth difference between the upper left, upper right and lower left and lower right of the target object to ensure that the forklift arm is facing the target object;

[0113] When the recognition module detects the target object, and when the target object is located in layer L i When the depth of the target object detected by the depth camera is greater than the distance d between the depth camera and the front end of the forklift arm, a movement instruction is sent to the forklift to move toward the target object so that the forklift arm faces the target object and inserts into the target object. min When the safety distance is added, the forklift is controlled to move forward; when the forklift is less than the safety distance from the target object, the forklift is controlled to move backward; when the horizontal distance difference between the center of the target object and the center of the depth map is greater than the threshold, the forklift is controlled to move left and right, and when the vertical distance difference between the center of the target object and the center of the depth map is greater than the threshold, the forklift arm is controlled to rise and fall. If the height to which the forklift arm needs to rise or fall is greater than or less than the maximum height above the ground and the minimum height above the ground of the forklift arm, an alarm is issued and the forklift arm movement is stopped; when the plane where the target object contour is located is not parallel to the plane where the forklift is located (that is, when the depth difference between the left and right ends exceeds the threshold): 1) the depth on the left is greater than the depth on the right, the forklift is controlled to rotate left; 2) the depth on the right is greater than the depth on the left, the forklift is controlled to rotate right; when the angle formed by the target object contour and the plane where the left and right forklift arms are located exceeds a certain threshold, it is considered that the target object is tilted, which is not conducive to forklift handling, and an alarm is issued; when the target object is located at layer L i There are other layers L near the forklift k When (k<i) exists, L is determined based on the width and height of the forklift. kWhether there is an obstacle in the (k<i) layer that hinders the movement of the forklift; the control module 506 will also judge in real time whether the movement trajectory of the forklift is conducive to the forklift arm of the forklift to align with the target object. When the forklift moves in the correct direction, the control module will not interfere with the movement of the forklift under the current instruction, otherwise it will update the control instruction to interfere with the movement of the forklift; the control module 506 also detects whether there is a sudden intrusion of foreign objects in the current movement direction of the forklift, and stops the forklift in time in case of emergency;

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0115] In the present embodiment, a target recognition device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0116] Figure 6 is a schematic diagram of the structure of an optional target recognition device according to an embodiment of the present invention. Figure 6 As shown, the device comprises:

[0117] A first acquisition module 602 is used to acquire calibration parameters of the depth camera and characteristic parameters of the target forklift;

[0118] A calibration module 604, configured to calibrate a first position of the depth camera relative to a target forklift arm according to a calibration parameter of the depth camera and a characteristic parameter of a target forklift, wherein the target forklift arm is installed on the target forklift;

[0119] A second acquisition module 606 is used to acquire a target depth image captured by the depth camera;

[0120] A layering module 608, configured to layer the target depth image according to depth to obtain a first group of layered depth images;

[0121] A processing module 610 is used to input the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model;

[0122] The determination module 612 is configured to determine a target contour of the target object and a second position of the target object relative to the target forklift based on the calibration parameters of the depth camera, the first position, and the target recognition result.

[0123] In an optional embodiment, the stratification module 608 is used to stratify the target depth image according to depth in the following manner to obtain a first group of stratified depth images: obtain the depth range of the target depth image; discretize the target depth image within the depth range according to a preset resolution to obtain a set of the first group of depth images divided according to depth.

[0124] In an optional embodiment, the processing module 610 is used to input the first group of depth images into a target recognition model in the following manner to obtain a target recognition result output by the target recognition model: determining a second group of depth images layer by layer in order from small to large depth range from the first group of depth images, wherein the second group of depth images is a group of depth images that contain the target contour of the target object in the first group of depth images; determining the target recognition result from the second group of depth images, wherein the target recognition result includes the second group of depth images and the layer corresponding to each depth image in the second group of depth images.

[0125] In an optional embodiment, the determination module 612 is used to determine the second group of depth images from the first group of depth images layer by layer in the order of depth range from small to large in the following manner: remove the depth images containing the number of discretized points less than the first preset threshold from the first group of depth images to obtain a third group of depth images; binarize each depth image in the third group of depth images to obtain a fourth group of depth images; perform first target processing on the fourth group of depth images to obtain a fifth group of depth images, wherein the first target processing is used for image denoising and / or image smoothing; perform second target processing on the fifth group of depth images to obtain the second group of depth images, wherein the second target processing is used to search for the target contour layer by layer in the order of the depth range from small to large, determine the second group of depth images, and determine a sixth group of depth images in the fifth group of depth images except the second group of depth images, and the sixth group of depth images is used to dynamically adjust the minimum value of the depth range.

[0126] In an optional embodiment, the device is also used to: before inputting the first group of depth images into the target recognition model, repeatedly perform the following steps until the difference between the first recognition probability and the second recognition probability output by the initial recognition model is less than or equal to the predetermined threshold: obtain a first group of sample depth images and a second group of sample depth images captured by a depth camera, and a group of labels corresponding to the first group of sample depth images one by one; input the first group of sample depth images and the second group of sample depth images into the initial recognition model to generate the first sample recognition result, wherein the first sample recognition result includes the first recognition probability and the second recognition probability output by the initial recognition model, and the first recognition probability The rate is used to represent the probability that the first sample recognition result recognizes that the contour of the sample object is included in the sample depth image, and the second recognition probability is used to represent the probability that the first sample recognition result recognizes that the contour of the sample object is not included in the sample depth image; when the difference between the first recognition probability and the second recognition probability is greater than the predetermined threshold, the parameters in the initial recognition model are adjusted; when the difference between the first recognition probability and the second recognition probability is less than the predetermined threshold, the first sample recognition result is compared with the set of labels to obtain the first set of sample depth images and the layer corresponding to each sample depth image in the first set of sample depth images.

[0127] In an optional embodiment, the device is also used to input the first group of sample depth images, the second group of sample depth images and the group of labels into the initial recognition model to generate the first sample recognition result in the following manner: remove the depth images containing the number of discretized points less than a second preset threshold from the first group of sample depth images and the second group of sample depth images to obtain a third group of sample depth images; perform third target processing on each sample depth image in the third group of sample depth images to obtain a fourth group of sample depth images, wherein the third target processing is used to determine a sample depth image including the outline of the sample object from the sample depth images; compare the fourth group of sample depth images with the group of labels to generate the first sample recognition result.

[0128] In an optional embodiment, the device is also used to: after determining the target contour of the target object and the second position of the target object relative to the target forklift based on the calibration parameters of the depth camera, the first position and the target recognition result, control the target forklift to perform a target operation corresponding to the second position.

[0129] In an optional embodiment, the device is also used to control the target forklift to perform the target operation corresponding to the second spatial position in the following manner: when the layer where the target contour of the target object is located is not the layer closest to the target forklift in the target depth image, determine whether there is an obstacle that hinders the movement of the target forklift on the layer between the layer where the target contour is located and the target forklift; when the obstacle exists on the layer between the layer where the target contour is located and the target forklift, control the target forklift to stop moving.

[0130] In an optional embodiment, the device is also used to control the target forklift to perform the target operation corresponding to the second position in the following manner: determine a preset motion trajectory of the target forklift based on the second position, wherein the preset motion trajectory is the motion trajectory of the target forklift arm aimed at the target object; when the actual operation trajectory of controlling the target forklift to move toward the target object does not match the preset motion trajectory, control the target forklift to update the movement direction so as to control the target forklift to move according to the preset motion trajectory.

[0131] In an optional embodiment, the device is also used to determine the preset motion trajectory of the target forklift based on the second spatial position in the following manner: when the target depth detected by the depth camera is greater than the sum of the distance between the depth camera and the front end of the target forklift arm and the preset safety distance, the parameters for controlling the forward movement of the target forklift are added to the preset motion trajectory; and / or when the target depth detected by the depth camera is less than the preset safety distance, the parameters for controlling the backward movement of the target forklift are added to the preset motion trajectory; and / or when the horizontal distance difference between the center of the target object and the center of the depth image is greater than a third threshold, the parameters for controlling the left and right translation movement of the target forklift are added to the preset motion trajectory; and / or when the vertical distance difference between the center of the target object and the center of the depth image is greater than a fourth threshold, the target forklift is controlled to translate up and down The parameters of the motion are added to the preset motion trajectory; and / or when the height to which the forklift arm of the target forklift needs to rise is greater than the maximum ground clearance of the forklift arm or the height to be lowered is less than the minimum ground clearance of the forklift arm, the parameters for controlling the target forklift to perform a first alarm operation and stopping the movement of the forklift arm are added to the preset motion trajectory, wherein the first alarm operation is used to indicate that the forklift arm cannot be aligned with the target object; and / or when the plane where the target object is located is not parallel to the plane of the target forklift, the parameters for controlling the target forklift to perform a rotation operation are added to the preset motion trajectory; and / or when the angle formed by the target object and the plane where the forklift arm is located is greater than a fifth threshold, the parameters for controlling the target forklift to perform a second alarm operation are added to the preset motion trajectory, wherein the second alarm operation is used to indicate that the target object is tilted.

[0132] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0133] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0134] In this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0135] S1, obtaining the calibration parameters of the depth camera and the characteristic parameters of the target forklift;

[0136] S2, calibrating a first position of the depth camera relative to a target forklift arm according to calibration parameters of the depth camera and characteristic parameters of the target forklift, wherein the target forklift is equipped with a target forklift arm;

[0137] S3, obtaining the target depth image captured by the depth camera;

[0138] S4, stratifying the target depth image according to depth to obtain a first group of stratified depth images;

[0139] S5, inputting the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model;

[0140] S6, determining a target contour of the target object and a second position of the target object relative to the target forklift based on the target recognition result, the first spatial position, and the calibration parameters of the depth camera.

[0141] The computer readable storage medium is further configured to store a computer program for performing the following steps:

[0142] S1, obtaining the calibration parameters of the depth camera and the characteristic parameters of the target forklift;

[0143] S2, calibrating a first position of the depth camera relative to a target forklift arm according to calibration parameters of the depth camera and characteristic parameters of the target forklift, wherein the target forklift is equipped with a target forklift arm;

[0144] S3, obtaining the target depth image captured by the depth camera;

[0145] S4, stratifying the target depth image according to depth to obtain a first group of stratified depth images;

[0146] S5, inputting the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model;

[0147] S6, determining a target contour of the target object and a second position of the target object relative to the target forklift based on the target recognition result, the first spatial position, and the calibration parameters of the depth camera.

[0148] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0149] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0150] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0151] In an exemplary embodiment, the processor may be configured to perform the following steps via a computer program:

[0152] S1, obtaining the calibration parameters of the depth camera and the characteristic parameters of the target forklift;

[0153] S2, calibrating a first position of the depth camera relative to a target forklift arm according to calibration parameters of the depth camera and characteristic parameters of the target forklift, wherein the target forklift is equipped with a target forklift arm;

[0154] S3, obtaining the target depth image captured by the depth camera;

[0155] S4, stratifying the target depth image according to depth to obtain a first group of stratified depth images;

[0156] S5, inputting the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model;

[0157] S6, determining a target contour of the target object and a second position of the target object relative to the target forklift based on the target recognition result, the first spatial position, and the calibration parameters of the depth camera.

[0158] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0159] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A target recognition method, characterized in that: include: Obtain the calibration parameters of the depth camera and the characteristic parameters of the target forklift; Calibrate a first position of the depth camera relative to a target forklift arm according to a calibration parameter of the depth camera and a characteristic parameter of a target forklift, wherein the target forklift arm is installed on the target forklift; Acquire a target depth image captured by the depth camera; Layering the target depth image according to depth to obtain a first group of layered depth images; Inputting the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model; Determining a target contour of a target object and a second position of the target object relative to the target forklift based on a calibration parameter of the depth camera, the first position, and the target recognition result; The step of inputting the first group of depth images into a target recognition model to obtain a target recognition result output by the target recognition model includes: determining a second group of depth images layer by layer in a predetermined order from the first group of depth images, wherein the second group of depth images is a group of depth images containing the target contour of the target object in the first group of depth images; determining the target recognition result from the second group of depth images, wherein the target recognition result includes the second group of depth images and the layer corresponding to each depth image in the second group of depth images; Determine a second group of depth images layer by layer from the first group of depth images in a predetermined order, including: removing depth images containing a number of discretized points less than a first preset threshold from the first group of depth images to obtain a third group of depth images; binarizing each depth image in the third group of depth images to obtain a fourth group of depth images; performing a first target processing on the fourth group of depth images to obtain a fifth group of depth images, wherein the first target processing is used for image denoising and / or image smoothing; performing a second target processing on the fifth group of depth images to obtain the second group of depth images, wherein the second target processing is used to search for the target contour layer by layer in the order of the depth range of the target depth image from small to large, determine the second group of depth images, and determine a sixth group of depth images other than the second group of depth images in the fifth group of depth images, and the sixth group of depth images is used to dynamically adjust the minimum value of the depth range.

2. The method according to claim 1, characterized in that The target depth image is layered according to the depth to obtain a first group of layered depth images, including: Acquire the depth range of the target depth image; The target depth image is discretized within the depth range according to a preset resolution to obtain a group of the first depth images divided according to the depth.

3. The method according to claim 1, characterized in that: Before inputting the first set of depth images into the target recognition model, the method further includes: Repeat the following steps until the difference between the first recognition probability and the second recognition probability output by the initial recognition model is less than or equal to a predetermined threshold: Acquire a first group of sample depth images and a second group of sample depth images captured by a depth camera, and a group of labels corresponding one-to-one to the first group of sample depth images; Inputting the first group of sample depth images and the second group of sample depth images into the initial recognition model to generate a first sample recognition result, wherein the first sample recognition result includes the first recognition probability and the second recognition probability output by the initial recognition model, the first recognition probability is used to represent the probability that the first sample recognition result recognizes that the contour of the sample object is included in the sample depth image, and the second recognition probability is used to represent the probability that the first sample recognition result recognizes that the contour of the sample object is not included in the sample depth image; When the difference between the first recognition probability and the second recognition probability is greater than the predetermined threshold, adjusting the parameters in the initial recognition model; When the difference between the first recognition probability and the second recognition probability is less than the predetermined threshold, the first sample recognition result is compared with the set of labels to obtain the first set of sample depth images and the layer corresponding to each sample depth image in the first set of sample depth images.

4. The method according to claim 3, characterized in that Inputting the first group of sample depth images, the second group of sample depth images, and the group of labels into the initial recognition model to generate the first sample recognition result, including: Removing depth images including the number of discretized points less than a second preset threshold from the first group of sample depth images and the second group of sample depth images to obtain a third group of sample depth images; Performing a third target processing on each sample depth image in the third group of sample depth images to obtain a fourth group of sample depth images, wherein the third target processing is used to determine a sample depth image including a contour of the sample object from the sample depth images; The fourth group of sample depth images is compared with the group of labels to generate the first sample recognition result.

5. The method according to claim 1, characterized in that After determining a target contour of the target object and a second position of the target object relative to the target forklift based on the calibration parameters of the depth camera, the first position, and the target recognition result, the method further includes: The target forklift is controlled to perform a target operation corresponding to the second position.

6. The method according to claim 5, characterized in that Controlling the target forklift to perform a target operation corresponding to the second spatial position includes: In a case where the layer where the target contour of the target object is located is not the layer closest to the target forklift in the target depth image, determining whether there is an obstacle that hinders the movement of the target forklift on a layer between the layer where the target contour is located and the target forklift; When the obstacle exists on the layer where the target contour is located and the layer between the target forklift, the target forklift is controlled to stop moving.

7. The method according to claim 5, characterized in that The controlling the target forklift to perform a target operation corresponding to the second position includes: Determining a preset motion trajectory of the target forklift based on the second position; When the actual running trajectory of controlling the target forklift to move toward the target object does not match the preset motion trajectory, the target forklift is controlled to update the motion direction so as to control the target forklift to move according to the preset motion trajectory.

8. The method according to claim 7, characterized in that Determining a preset motion trajectory of the target forklift based on the second spatial position includes: When the target depth detected by the depth camera is greater than the sum of the distance between the depth camera and the front end of the target forklift arm and the preset safety distance, adding a parameter for controlling the forward motion of the target forklift to the preset motion trajectory; and / or When the target depth detected by the depth camera is less than the preset safety distance, adding parameters for controlling the backward movement of the target forklift to the preset motion trajectory; and / or When the horizontal distance difference between the center of the target object and the center of the depth image is greater than a third threshold, adding parameters for controlling the target forklift to perform left and right translational motion to the preset motion trajectory; and / or When the vertical distance difference between the center of the target object and the center of the depth image is greater than a fourth threshold, adding parameters for controlling the target forklift to perform an up and down translation motion to the preset motion trajectory; and / or When the height to be raised by the forklift arm of the target forklift is greater than the maximum ground clearance of the forklift arm or the height to be lowered is less than the minimum ground clearance of the forklift arm, controlling the target forklift to perform a first alarm operation and adding a parameter for stopping the movement of the forklift arm to the preset motion trajectory, wherein the first alarm operation is used to indicate that the forklift arm cannot be aligned with the target object; and / or When the plane where the target object is located is not parallel to the plane of the target forklift, adding parameters for controlling the target forklift to perform a rotation operation to the preset motion trajectory; and / or When the angle formed by the target object and the plane where the forklift arm is located is greater than a fifth threshold, a parameter for controlling the target forklift to perform a second alarm operation is added to the preset motion trajectory, wherein the second alarm operation is used to indicate that the target object is tilted.

Images