Flatbed carriage detection method and device based on unmanned guided vehicle and unmanned guided vehicle

Abstract

The present invention provides a flatbed carriage detection method, device and unmanned guided vehicle based on an unmanned guided vehicle. The method uses an image retrieval network to extract the front reference image and the rear reference image that are closest to the real-time monitored RGB data stream from the flatbed graphics library that is updated and maintained in real time, and then matches the feature point pairs with the front image and the rear image in the RGB data stream to locate the pixel coordinates, and then converts them into the front space coordinates and the rear space coordinates in the absolute space coordinate system, and fits the linear equation that matches the first length side to guide the loading operation position according to each position of the first length side. In complex scenarios such as the unfixed parking position of the truck and the various types and specifications of the goods, the position of the flatbed carriage can be accurately identified, and the cargo specifications are combined to provide a basis for the cargo position division, which is convenient for accurate navigation to the designated cargo position and realizes flexible loading.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a flatbed carriage detection method and device based on an unmanned guided vehicle, and an unmanned guided vehicle. Background Art

[0002] With the rapid development of computer and robot technology, the widespread application of Automated Guided Vehicle (AGV) in the logistics industry marks the development of warehouse logistics technology towards automation and intelligence. Research on loading has remained at the model loading under specific devices, such as installing laser radar at the entrance and exit of the garage, loading by mechanical arms, and loading by telescopic conveyors. All of these require modification of the logistics site and rely on special devices, which are suitable for container loading.

[0003] However, in complex warehousing and logistics scenarios where flatbed trucks are not parked in a standardized manner in the garage, the positions are not fixed, and the types and specifications of goods are diverse, it is easy for the goods to be misaligned and tipped over. This model of loading is not applicable, making it impossible for AGV to successfully complete loading in complex warehousing and logistics scenarios with flatbed trucks. Summary of the invention

[0004] The present invention provides a flatbed carriage detection method and device based on an unmanned guided vehicle and an unmanned guided vehicle, which are used to solve the defect that the cargo space cannot be accurately located due to the irregular parking of the flatbed truck during the loading process.

[0005] The present invention provides a flatbed carriage detection method based on an unmanned guided vehicle, comprising:

[0006] When it is determined that the interval distance between the unmanned guided vehicle and the first end point in the first length side of the flatbed carriage to be detected is greater than or equal to a first preset threshold, the camera module on the unmanned guided vehicle is started to continuously receive the RGB data stream collected by the camera module; wherein the RGB data stream has the front image and the rear image of the flatbed carriage to be detected as the start frame and the end frame respectively;

[0007] Matching the front and rear reference images of the vehicle corresponding to the RGB data stream according to the flatbed vehicle graphics library; wherein the flatbed vehicle graphics library stores a partial image retaining the front and rear endpoints of the flatbed vehicle, and the pixel coordinates of the endpoints marked in the partial image;

[0008] Matching the front image and the rear image of the vehicle in the RGB data stream with the front reference image and the rear reference image corresponding to the RGB data stream respectively to determine the pixel coordinates of the front and the rear of the vehicle;

[0009] Based on the pixel coordinates of the front of the vehicle, the pixel coordinates of the rear of the vehicle and the point cloud data of the front and rear of the vehicle, the space coordinates of the front of the vehicle and the space coordinates of the rear of the vehicle of the flatbed vehicle to be detected in the absolute space coordinate system are obtained; wherein the point cloud data of the front and rear of the vehicle are determined according to the depth data stream collected by the camera module while collecting the RGB data stream and the camera internal parameters of the camera module;

[0010] The linear equation is constructed using the spatial coordinates of the front of the vehicle and the spatial coordinates of the rear of the vehicle to determine the position information of the first length side of the flatbed car to be detected in the absolute spatial coordinate system; wherein the position information of the first length side of the flatbed car to be detected in the absolute spatial coordinate system is used to control the unmanned guided vehicle to carry out loading operations from the first length side.

[0011] According to a flatbed carriage detection method based on an unmanned guided vehicle provided by the present invention, the front image and the rear image of the vehicle in the RGB data stream are matched with the front reference image and the rear reference image corresponding to the RGB data stream for feature points, respectively, to determine the front pixel coordinates and the rear pixel coordinates, including:

[0012] Extracting feature points and descriptors from the vehicle front image and the vehicle front reference image to obtain a first feature point set and a second feature point set;

[0013] Determine a feature point in the second feature point set whose pixel coordinates are less than a second preset threshold from the pixel coordinates of the front end point carried in the vehicle front reference image as a first detection key point;

[0014] Filtering out a point pair including the first detection key point from a first point pair set obtained by matching the first feature point set with the second feature point set, and setting the pixel coordinates of the first detection key point to the pixel coordinates of the front of the vehicle corresponding to the front endpoint in the front of the vehicle image;

[0015] Accordingly, feature points and descriptors are extracted from the vehicle rear image and the vehicle rear reference image to obtain a third feature point set and a fourth feature point set;

[0016] Determine a feature point in the third feature point set whose pixel coordinates are less than the pixel coordinates of the rear end point carried in the vehicle rear reference image by a distance less than the second preset threshold as a second detection key point;

[0017] A point pair including the second detection key point is screened out from a second point pair set obtained by matching the third feature point set with the fourth feature point set, and the pixel coordinates of the second detection key point are set to the pixel coordinates of the rear end of the vehicle corresponding to the tail endpoint in the rear end image.

[0018] According to a flatbed carriage detection method based on an unmanned guided vehicle provided by the present invention, the front space coordinates and the rear space coordinates of the flatbed carriage to be detected in an absolute space coordinate system are obtained based on the front pixel coordinates, the rear pixel coordinates and the front and rear point cloud data of the carriage, including:

[0019] The pixel coordinates of the front of the vehicle and the pixel coordinates of the rear of the vehicle are used to index the front and rear point cloud data of the vehicle compartment to obtain the front coordinates and the rear coordinates of the flatbed vehicle compartment to be detected in the camera coordinate system;

[0020] The front coordinates and the rear coordinates of the vehicle are converted into coordinate systems to obtain the front space coordinates and the rear space coordinates of the flatbed vehicle to be detected in the absolute space coordinate system;

[0021] The absolute space coordinate system is a geodetic coordinate system or an absolute coordinate system whose origin is fixed at any corner point in the storage space.

[0022] According to a flatbed carriage detection method based on an unmanned guided vehicle provided by the present invention, before acquiring the front space coordinates and the rear space coordinates of the flatbed carriage to be detected in the absolute space coordinate system based on the front pixel coordinates, the rear pixel coordinates and the front and rear point cloud data of the carriage, the method further includes:

[0023] Based on the depth data stream and the camera internal and external parameters of the camera module, the depth data stream and the RGB data stream are respectively aligning after distortion correction;

[0024] The distortion-corrected depth data stream is filtered and mapped to a point cloud image to obtain the head and tail point cloud data of the carriage aligned with the RGB data stream.

[0025] According to a flatbed carriage detection method based on an unmanned guided vehicle provided by the present invention, after matching the front image and the rear image of the vehicle in the RGB data stream with the front reference image and the rear reference image corresponding to the RGB data stream for feature points, and determining the front pixel coordinates and the rear pixel coordinates, the method further includes:

[0026] Constructing a straight line equation based on the width of the flatbed carriage to be detected, the front space coordinates and the rear space coordinates, and determining the position information of the second length side of the flatbed carriage to be detected in the absolute space coordinate system;

[0027] The position information of the first length side and / or the second length side of the flatbed carriage to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to perform loading operations on the corresponding side.

[0028] According to a flatbed carriage detection method based on an unmanned guided vehicle provided by the present invention, the position information of the first length side and / or the second length side of the flatbed carriage to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to perform loading operations on the corresponding side, including:

[0029] Determine the cargo operation position based on the position information of the first length side and / or the second length side of the flatbed carriage to be detected in the absolute space coordinate system and the cargo attribute information;

[0030] The cargo operation position is sent to the unmanned guided vehicle, so that the unmanned guided vehicle controls the fork arm to transport the cargo carried on the fork arm from the corresponding length side to the cargo operation position to complete the loading operation.

[0031] The present invention also provides a flatbed carriage detection device based on an unmanned guided vehicle, comprising:

[0032] An image acquisition module is used to start the camera module on the unmanned guided vehicle to continuously receive the RGB data stream acquired by the camera module when it is determined that the interval distance between the unmanned guided vehicle and the first end point of the first length side of the flatbed carriage to be detected is greater than or equal to a first preset threshold; wherein the RGB data stream has the front image and the rear image of the flatbed carriage to be detected as the starting frame and the ending frame respectively;

[0033] An image retrieval module is used to match the front reference image and the rear reference image corresponding to the RGB data stream according to the flatbed vehicle graphics library; wherein the flatbed vehicle graphics library stores a partial image retaining the front and rear endpoints of the flatbed vehicle, and the pixel coordinates of the endpoints marked in the partial image;

[0034] An image matching module is used to match the front image and the rear image of the vehicle in the RGB data stream with the front reference image and the rear reference image corresponding to the RGB data stream, respectively, to determine the pixel coordinates of the front and the rear of the vehicle;

[0035] A head and tail positioning module, used to obtain the head space coordinates and the tail space coordinates of the flatbed car to be detected in the absolute space coordinate system based on the head pixel coordinates, the tail pixel coordinates and the head and tail point cloud data of the car; wherein the head and tail point cloud data of the car are determined based on the depth data stream collected by the camera module while collecting the RGB data stream and the camera internal parameters of the camera module;

[0036] The car body fitting module is used to construct a straight line equation using the space coordinates of the front of the car and the space coordinates of the rear of the car to determine the position information of the first length side of the flatbed car body to be detected in the absolute space coordinate system; wherein the position information of the first length side of the flatbed car body to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to carry out loading operations from the first length side.

[0037] The present invention also provides an unmanned guided vehicle, comprising an unmanned guided vehicle body, and a camera module installed in the longitudinal axis direction of the unmanned guided vehicle body, and also comprising a processor arranged in the unmanned guided vehicle body, and when the processor executes the program, the flatbed carriage detection method based on the unmanned guided vehicle as described in any of the above items is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the flatbed carriage detection method based on an unmanned guided vehicle as described in any one of the above is implemented.

[0039] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for detecting a flatbed compartment based on an unmanned guided vehicle.

[0040] The flatbed carriage detection method, device and unmanned guided vehicle provided by the present invention use an image retrieval network to extract the front reference image and the rear reference image that are closest to the real-time monitored RGB data stream from the flatbed graphics library that is updated and maintained in real time, and then match the feature point pairs with the front image and the rear image in the RGB data stream to locate the pixel coordinates, and use the located front pixel coordinates and rear pixel coordinates to obtain the front space coordinates and rear space coordinates in the absolute space coordinate system by indexing the front and rear point cloud data of the carriage, and fit the linear equation that matches the first length side to guide the loading operation position according to each position of the first length side. In complex scenarios such as the unfixed parking position of the truck and the various types and specifications of the goods, the position of the flatbed carriage can be accurately identified, and the basis for the division of cargo spaces is provided in combination with the cargo specifications, so as to facilitate accurate navigation to the designated cargo space and realize flexible loading. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1It is one of the flow diagrams of the flatbed carriage detection method based on the unmanned guided vehicle provided by the present invention;

[0043] Figure 2 It is one of the image matching visualization schematic diagrams provided by the present invention;

[0044] Figure 3 This is the second image matching visualization schematic diagram provided by the present invention;

[0045] Figure 4 This is the second flow chart of the flatbed carriage detection method based on the unmanned guided vehicle provided by the present invention;

[0046] Figure 5 It is a structural schematic diagram of a flatbed carriage detection device based on an unmanned guided vehicle provided by the present invention;

[0047] Figure 6 It is a structural schematic diagram of the unmanned guided vehicle provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0050] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0051] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0052] Figure 1FIG. 1 is one of the flow charts of the flatbed compartment detection method based on an unmanned guided vehicle provided by the present invention. Figure 1 As shown, the flatbed car detection method based on an unmanned guided vehicle provided in an embodiment of the present invention includes: step 101, when it is determined that the interval distance between the unmanned guided vehicle and the first endpoint of the first length side of the flatbed car to be detected is greater than or equal to a first preset threshold, starting the camera module on the unmanned guided vehicle to continuously receive the RGB data stream collected by the camera module.

[0053] The RGB data stream uses the front image and the rear image of the flatbed carriage to be detected as the start frame and the end frame respectively.

[0054] It should be noted that the executor of the unmanned vehicle-based flatbed carriage detection method provided in the embodiment of the present invention is a flatbed carriage detection device based on an unmanned vehicle. The flatbed carriage detection device based on an unmanned vehicle can be a central processing unit (CPU) built into the unmanned vehicle, or a development board based on CPU integration to perform information processing and program operation.

[0055] The application scenario of the flatbed carriage detection method based on an unmanned guided vehicle provided in an embodiment of the present invention is that when facing a flatbed carriage used to load goods, the unmanned guided vehicle controls the camera module to start the visual task at a certain safe distance from the flatbed carriage to identify and locate the front and rear of the flatbed carriage.

[0056] The camera module includes both a color camera and a depth camera. The types of depth cameras include but are not limited to binocular cameras, TOF cameras or structured light cameras. The different types of depth cameras only have different ranging principles. Within the effective ranging range, the ranging accuracy of the depth camera can reach the centimeter level or even higher. The smaller the ranging range, the higher the accuracy. The range of variation of its effective ranging results is still at the centimeter level.

[0057] It should be noted that the first preset threshold is a threshold set for the vertical distance between the unmanned guided vehicle and the first endpoint of the first length side of the flatbed carriage to be detected. This value is often limited by the perception distance and perception accuracy of the depth camera, and additionally considers factors such as the camera's field of view, forklift arm length, and forklift path planning. The setting of this distance is often not more than 3 meters.

[0058] Exemplarily, the present invention sets the first preset threshold for triggering the start of the visual task to be 2 meters, denoted as D.

[0059] It should be noted that a flatbed car is composed of a frame and a chassis made of different numbers of flat wooden boards horizontally assembled together. The four sides of the flatbed car can be opened for cargo loading operations. The side parallel to the longitudinal center axis of the flatbed car body is the length side, and the side parallel to the longitudinal center axis of the body is the width side.

[0060] Among them, the first length side refers to the length side that the lens of the camera module is facing. In the first length side, the one close to the front of the vehicle is the leading end point, and the one far from the front of the vehicle is the trailing end point.

[0061] Specifically, in step 101, the flatbed car detection device based on the unmanned guided vehicle can monitor the distance between the unmanned guided vehicle and the surrounding objects in real time through the sensing element during the movement of the unmanned guided vehicle. When it is monitored that the interval distance between the unmanned guided vehicle and the first end point of the first length side of the flatbed car to be detected is greater than or equal to a first preset threshold, the visual task on the unmanned guided vehicle is driven to start, and while controlling the unmanned guided vehicle to move from the first end point to the rear end point of the car body while maintaining a fixed distance from the vehicle body, the camera module is started, the trigger mode of the camera module is set to continuous trigger, and the RGB data stream is obtained.

[0062] The RGB data stream contains all the local RGB images of the carriage continuously collected by the camera module during the movement. The starting image frame of the RGB data stream is the front image of the carriage with the front end of the carriage at the center of the camera module's field of view, and the ending image frame of the RGB data stream is the rear image of the carriage with the rear end of the carriage at the center of the camera module's field of view.

[0063] It is understandable that the above-mentioned visual tasks can also be judged by the user through the visual images on the front end and triggered manually.

[0064] Step 102: Match the front and rear reference images corresponding to the RGB data stream according to the flatbed vehicle graphics library, wherein the flatbed vehicle graphics library stores a partial image retaining the front and rear endpoints of the flatbed vehicle, and the pixel coordinates of the endpoints marked in the partial image.

[0065] It should be noted that the flatbed truck graphics library includes several side images of flatbed trucks collected at different angles, heights, and light conditions. To reduce the amount of calculation, the images are cropped to retain the local images of the front and rear endpoints of the flatbed truck. When the endpoints of the flatbed truck are manually annotated using the labelme software, the pixel coordinates of the annotated endpoints are saved in a *.json file.

[0066] Specifically, in step 102, the flatbed car detection device based on the unmanned guided vehicle can perform feature extraction on the RGB data stream, and then select an endpoint local image whose similarity with the extracted feature vector exceeds a certain threshold from the flatbed car graphics library, and output it as the corresponding front vehicle reference image and rear vehicle reference image according to the endpoint type marked in the endpoint local image.

[0067] Among them, the method of screening based on vector similarity in the flatbed truck graphics library includes but is not limited to Euclidean distance, Mahalanobis distance, Manhattan distance, Chebyshev distance, Minkowski distance, Hamming distance, cosine similarity, Pearson correlation coefficient, etc.

[0068] Preferably, the embodiment of the present invention can use an image retrieval network such as MobileNetVlad for processing, and its specific implementation process includes a training phase and an application phase, wherein:

[0069] In order to reduce the amount of calculation during the training phase, the images in the flatbed car graphics library are resampled to half the size of the original images and input into the MobileNetVlad network model, which extracts the features of the images and saves them in a vector. Then, the similarity is expressed by calculating the difference between the two vectors, and the gradient back propagation is used to repeatedly update the model parameters until convergence, thereby achieving the purpose of optimizing the model.

[0070] In the application stage, the RGB data stream is directly input into the MobileNetVlad network, which will output the images in the flatbed car graphics library that are most similar to the front and rear images in the RGB data stream and whose similarity exceeds a certain threshold as the front and rear reference images.

[0071] Step 103: perform feature point matching on the front image and the rear image of the vehicle in the RGB data stream with the front reference image and the rear reference image corresponding to the RGB data stream, respectively, to determine the pixel coordinates of the front pixel and the rear pixel.

[0072] Specifically, in step 103, the flatbed carriage detection device based on the unmanned guided vehicle respectively compares the vehicle front image and the vehicle front reference image closest to it in the image library, takes the pixel point in the vehicle front image that matches the endpoint marked corresponding to the vehicle front reference image as the front endpoint, assigns the pixel coordinates corresponding to the marked point in the vehicle front reference image to the front endpoint, and outputs the pixel coordinates of the vehicle front.

[0073] Similarly, for the rear image and the closest rear reference image in the library, the pixel point in the rear image that matches the corresponding marked endpoint of the rear reference image is used as the rear endpoint, and the pixel coordinates corresponding to the marked point in the rear reference image are assigned to the rear endpoint, and the pixel coordinates of the rear are output.

[0074] Step 104: Based on the pixel coordinates of the front of the vehicle, the pixel coordinates of the rear of the vehicle and the point cloud data of the front and rear of the vehicle, obtain the space coordinates of the front of the vehicle and the space coordinates of the rear of the vehicle of the flatbed vehicle to be detected in the absolute space coordinate system.

[0075] Among them, the point cloud data of the front and rear of the carriage are determined based on the depth data stream collected by the camera module while collecting the RGB data stream and the camera internal parameters of the camera module.

[0076] It should be noted that the point cloud data at the front and rear of the carriage refers to the use of the internal and external parameters of the camera module to convert the depth map in the depth data stream obtained when collecting the RGB data stream from the image coordinate system to the world coordinate system to obtain a set of three-dimensional point cloud coordinates.

[0077] Specifically, in step 104, the flatbed car detection device based on the unmanned guided vehicle uses the pixel coordinates of the front and rear of the car to index the point cloud data of the front and rear of the car, respectively, and converts the indexed three-dimensional point cloud coordinates from the world coordinate system to the absolute space coordinate system, and obtains the space coordinates of the front (denoted as (X1, Y1, Z1)) and the space coordinates of the rear (denoted as (X2, Y2, Z2)) of the first length side of the flatbed car to be detected in the absolute space coordinate system.

[0078] Step 105: construct a straight line equation using the front space coordinates and the rear space coordinates to determine the position information of the first length side of the flatbed carriage to be detected in the absolute space coordinate system. The position information of the first length side of the flatbed carriage to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to carry out loading operations from the first length side.

[0079] Specifically, in step 105, the flatbed carriage detection device based on the unmanned guided vehicle uses the space coordinates of the front of the vehicle and the space coordinates of the rear of the vehicle in the absolute space coordinate system to construct the straight line equation as follows:

[0080]

[0081] Thus, the position information of the first length side of the flatbed car to be detected in the absolute space coordinate system can be determined, and the length and height of the flatbed car can also be fitted. Furthermore, according to the size of the pallet cargo, the corresponding cargo position is divided in the position information of the first length side in the absolute space coordinate system to guide the forklift to load and navigate to the working position of the first length side.

[0082] The embodiment of the present invention uses an image retrieval network to extract the front reference image and the rear reference image that are closest to the real-time monitored RGB data stream from the flatbed graphics library that is updated and maintained in real time, and then matches the feature point pairs with the front image and the rear image in the RGB data stream to locate the pixel coordinates, and uses the located front pixel coordinates and rear pixel coordinates to obtain the front space coordinates and rear space coordinates in the absolute space coordinate system in the front and rear point cloud data index of the carriage, and fits the straight line equation that matches the first length side to guide the loading operation position according to the various positions of the first length side. In complex scenarios such as the unfixed parking position of the truck and the various types and specifications of the goods, the position of the flatbed carriage can be accurately identified, and the cargo specifications are combined to provide a basis for the division of cargo spaces, which is convenient for accurate navigation to the designated cargo space and realizes flexible loading.

[0083] On the basis of any of the above embodiments, the front image and the rear image of the vehicle in the RGB data stream are respectively matched with the front reference image and the rear reference image corresponding to the RGB data stream for feature points to determine the pixel coordinates of the front and the rear of the vehicle, including: extracting feature points and descriptors from the front image and the front reference image to obtain a first feature point set and a second feature point set.

[0084] Specifically, in step 103, the flatbed carriage detection device based on the unmanned guided vehicle uses the SuperPoint network to extract feature points and descriptors from the vehicle front image and the vehicle front reference image to obtain a first feature point set and a second feature point set, respectively.

[0085] The embodiment of the present invention does not specifically limit the parameter setting of the SuperPoint network.

[0086] Exemplarily, the confidence threshold is set to 0.25, the weight is set to outdoor detection, and the non-maximum suppression radius is 4. Here, the feature point is a corner point that does not change with factors such as light and shooting angle.

[0087] A feature point in the second feature point set whose distance from the pixel coordinates of the first endpoint carried by the vehicle front reference image is less than a second preset threshold is determined as a first detection key point.

[0088] It should be noted that the second preset threshold is a threshold set for the distance between the feature point and the marked point in the vehicle head reference image. The smaller the value, the higher the credibility of the corresponding feature point in the vehicle head reference image as the marked point.

[0089] Exemplarily, the present invention sets the second preset threshold to 5 pixels.

[0090] Specifically, the flatbed car detection device based on the unmanned guided vehicle reads the *.json annotation file of the vehicle head reference image, and selects the feature point closest to the pixel coordinates of the annotation point of the vehicle head reference image in the second feature point set in combination with the second preset threshold. If the point is within five pixels around the annotation point, the point is used as the first detection key point of the front end of the flatbed car in the reference image.

[0091] If the first detection key point is not found, the confidence threshold of the feature point extracted by the SuperPoint network is lowered and feature extraction is performed again.

[0092] A point pair including the first detection key point is screened out from a first point pair set obtained by matching the first feature point set with the second feature point set, and the pixel coordinates of the first detection key point are set to the vehicle front pixel coordinates corresponding to the first endpoint in the vehicle front image.

[0093] Specifically, the flatbed carriage detection device based on the unmanned guided vehicle can use the SuperGlue network to match the first feature point set and the second feature point set to obtain a first point pair set containing several matching point pairs, and take the point pair where the first detection key point is located from the first point pair set, locate the other feature point in the point pair as the head end point in the vehicle head image, and assign the pixel coordinates of the first detection key point to the feature point to output the pixel coordinates of the vehicle head.

[0094] Figure 2 is one of the image matching visualization diagrams provided by the present invention. The matching result of the vehicle head image and the vehicle head reference image is as follows: Figure 2 As shown, the bold line is the matching result of the point pair where the first detection key point is located.

[0095] If the point pair containing the first detection key point cannot be screened out in the first point pair set, the matching threshold parameter of the SuperGlue network structure is correspondingly lowered to expand the matching point pairs as much as possible.

[0096] Correspondingly, feature points and descriptors are extracted from the vehicle rear image and the vehicle rear reference image to obtain a third feature point set and a fourth feature point set.

[0097] A feature point whose pixel coordinates in the third feature point set are at a distance less than the second preset threshold from the pixel coordinates of the rear end point carried in the vehicle rear reference image is determined as a second detection key point.

[0098] A point pair including the second detection key point is screened out from a second point pair set obtained by matching the third feature point set with the fourth feature point set, and the pixel coordinates of the second detection key point are set to the pixel coordinates of the rear end of the vehicle corresponding to the tail endpoint in the rear end image.

[0099] Specifically, the flatbed carriage detection device based on the unmanned guided vehicle extracts feature points from the vehicle rear image and the vehicle rear reference image using the processing logic described in the above process to obtain the third feature point set and the fourth feature point set, and also selects the feature points within five pixels around the marked points of the vehicle rear reference image from the fourth feature point as the second detection key point. From the second point pair set obtained by matching feature points from the third feature point set and the fourth feature point set, the point pair where the second detection key point is located is taken, and the other feature point in the point pair is positioned as the tail end point in the vehicle rear image, and the pixel coordinates of the second detection key point are assigned to the feature point to output the pixel coordinates of the vehicle rear.

[0100] Figure 3 This is the second visualization diagram of image matching provided by the present invention. The matching result of the rear image and the rear reference image is as follows: Figure 3 As shown, the bold line is the matching result of the point pair where the second detection key point is located.

[0101] If the point pair containing the second detection key point cannot be screened out in the second point pair set, the matching threshold parameter of the SuperGlue network structure is correspondingly lowered to expand the matching point pairs as much as possible.

[0102] The embodiment of the present invention uses a feature extraction network to extract feature points from the front image and the front reference image respectively, and uses the feature point closest to the marked point in the front reference image as the detection key point of the front end of the flatbed truck in the reference image. In the feature point matching process, the matching point pixel coordinates of the detection key point at the front end of the flatbed truck are taken as the output of the front pixel coordinates. The same processing flow is then used to locate the pixel coordinates of the rear end of the vehicle from the rear image and the rear reference image. In complex scenarios where the parking position of the truck is not fixed, the key positions of the front and rear ends of the flatbed car can be accurately identified.

[0103] On the basis of any of the above embodiments, based on the front pixel coordinates, the rear pixel coordinates and the front and rear point cloud data of the carriage, the front spatial coordinates and the rear spatial coordinates of the flatbed carriage to be detected in the absolute space coordinate system are obtained, including: using the front pixel coordinates and the rear pixel coordinates to index in the front and rear point cloud data of the carriage, to obtain the front coordinates and the rear coordinates of the flatbed carriage to be detected in the camera coordinate system.

[0104] Specifically, the flatbed car detection device based on the unmanned guided vehicle indexes the front and rear point cloud data of the car according to the front pixel coordinates and the rear pixel coordinates of the car, and obtains the front coordinates (denoted as (x1, y1, z1)) and rear coordinates (denoted as (x2, y2, z2)) of the flatbed car to be detected in the camera coordinate system respectively.

[0105] It can be understood that if the front coordinate or the rear coordinate is empty, the pixel coordinate of the adjacent point is obtained and output as the front coordinate or the rear coordinate.

[0106] The front coordinates and the rear coordinates of the vehicle are converted into coordinate systems to obtain the front space coordinates and the rear space coordinates of the flatbed vehicle to be detected in the absolute space coordinate system.

[0107] The absolute space coordinate system is a geodetic coordinate system or an absolute coordinate system whose origin is fixed at any corner point in the storage space.

[0108] It should be noted that since AGV navigation relies on antenna receiving signals, the coordinates of the forklift itself in the absolute space coordinate system are actually the position of the antenna, which is set as (X h ,Z h ). Ignore the height information on the Y axis, perform dimensionality reduction on the camera coordinate system, simplify the vehicle head coordinate to (x1, z1), and obtain the antenna coordinate system by translating the camera plane coordinate system. The actual distance difference between the origins of the two coordinate systems in the X-axis direction is measured, recorded as Δx, and the distance difference in the Z-axis direction is recorded as Δz. Then, the antenna coordinate system is rotated around the Y axis by θ to obtain the absolute space coordinate system.

[0109] The absolute space coordinate system may be a custom coordinate system in the logistics site, which is not specifically limited in the embodiment of the present invention. For example, the absolute space coordinate system may be customizable to set the origin to a corner point in the logistics site.

[0110] The absolute space coordinate system can also be a geocentric coordinate system that makes the origin coincide with the center of mass of the earth and is established with the ellipsoid and the normal as the reference.

[0111] Specifically, in step 104, the flatbed carriage detection device based on the unmanned guided vehicle first sets the coordinates of the vehicle head (x1, z1) in the antenna coordinate system to (x1+Δx, z1+Δz), and sets the antenna coordinates to (X h ,Z h ) into the following formula to calculate the spatial coordinates of the front of the vehicle (X1, Y1, Z1):

[0112] X1=(x1+Δx)cosθ+(z1+Δz)sinθ+X h

[0113] Y1=y1

[0114] Z1=-(x1+Δx)sinθ+(z1+Δz)cosθ+Z h

[0115] Similarly, the position of the forklift antenna in the absolute space coordinate system is set to (X t ,Z t), the coordinates of the rear of the vehicle (x2, z2) in the antenna coordinate system are (x2+Δx, z2+Δz). t ,Z t ) into the following formula to calculate the rear space coordinates (X2, Y2, Z2):

[0116] X2=(x2+Δx)cosθ+(z2+Δz)sinθ+X t

[0117] Y2=y2

[0118] Z2=-(x2+Δx)sinθ+(z2+Δz)cosθ+Z t

[0119] The embodiment of the present invention first uses the pixel coordinates of the front and rear of the vehicle to index in the point cloud data, and converts the obtained coordinates of the front and rear of the vehicle in the camera coordinate system to the absolute space coordinate system of AGV navigation to obtain the spatial coordinates of the front and rear of the vehicle. In complex scenarios where the parking position of the truck is not fixed, the key positions of the front and rear ends of the flatbed carriage in the storage space can be accurately identified.

[0120] On the basis of any of the above embodiments, before obtaining the front space coordinates and the rear space coordinates of the flatbed car to be detected in the absolute space coordinate system based on the front pixel coordinates, the rear pixel coordinates and the front and rear point cloud data of the car, it also includes: based on the depth data stream and the camera internal and external parameters of the camera module, the depth data stream and the RGB data stream are respectively aligning after distortion correction.

[0121] Specifically, before step 104, the flatbed carriage detection device based on the unmanned guided vehicle performs corresponding distortion correction on the depth data stream and the RGB data stream, and then uses the camera internal and external parameters of the camera module to convert the coordinate system of the depth data stream to align it with the RGB data stream after the distortion correction.

[0122] The distortion-corrected depth data stream is filtered and mapped to a point cloud image to obtain the head and tail point cloud data of the carriage aligned with the RGB data stream.

[0123] Specifically, the flatbed carriage detection device based on the unmanned guided vehicle filters the depth data stream to remove noise, and then maps it to a point cloud image to obtain the head and tail point cloud data of the carriage aligned with the RGB data.

[0124] The embodiment of the present invention sets corresponding distortion correction processes for both the depth data stream and the RGB data stream, which can greatly improve the accuracy in the subsequent image processing process.

[0125] On the basis of any of the above embodiments, after matching the front image and the rear image of the vehicle in the RGB data stream with the front reference image and the rear reference image corresponding to the RGB data stream for feature points and determining the front pixel coordinates and the rear pixel coordinates, it also includes: constructing a straight line equation based on the width of the flatbed vehicle to be detected, the front spatial coordinates and the rear spatial coordinates, and determining the position information of the second length side of the flatbed vehicle to be detected in the absolute spatial coordinate system.

[0126] Specifically, after step 104, the flatbed car detection device based on the unmanned guided vehicle can also combine the car width of the flatbed car to be detected, convert the front space coordinates and the rear space coordinates of the first length side into the first and last endpoint coordinates of the second length side, and fit the position information of the second length side in the absolute space coordinate system using the straight line equation constructed by the first and last endpoint coordinates of the second length side.

[0127] The position information of the first length side and / or the second length side of the flatbed carriage to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to perform loading operations on the corresponding side.

[0128] Specifically, the flatbed carriage detection device based on the unmanned guided vehicle can plan the AGV operation position based on the position information of the first length side or the second length side in the absolute space coordinate system, and control the unmanned guided vehicle to perform loading operations in sequence according to the planned operation points on the first length side or the second length side.

[0129] Among them, the second length side refers to a length side that is parallel to the first length side and is not in the framing content of the camera module. Among the second length sides, the one close to the front of the vehicle is the leading endpoint, and the one far away from the front of the vehicle is the trailing endpoint.

[0130] Preferably, the position information of the first length side and the second length side in the absolute space coordinate system can also be used to plan the operating positions of both sides of the AGV, and control the unmanned guided vehicle to perform loading operations at the first length side and the second length side according to the planned operating points at the same time.

[0131] It is understandable that an unmanned guided vehicle equipped with an unmanned guided vehicle-based flatbed carriage detection device may be placed on each of the first length side and the second length side, respectively, to carry out visual tasks on both sides simultaneously and parse the straight line equations of the corresponding sides.

[0132] The embodiment of the present invention can determine the front and rear endpoints and the straight line equation of the edge of the carriage on the other side according to the width of the carriage of the flatbed truck and the front and rear spaces on either side, thereby facilitating loading operations on both sides of the carriage at the same time and improving the work efficiency of the unmanned guided vehicle.

[0133] On the basis of any of the above embodiments, the position information of the first length side and / or the second length side of the flatbed car to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to perform loading operations on the corresponding sides, including: determining the cargo operation position based on the position information of the first length side and / or the second length side of the flatbed car to be detected in the absolute space coordinate system and the cargo attribute information.

[0134] Specifically, the flatbed carriage detection device based on the unmanned guided vehicle can predict the storage scale of the cargo according to the cargo attribute information (such as basic attribute information such as cargo type, cargo name, cargo size, cargo weight, cargo quantity, etc.), and use this as the basis for dividing the area matching its storage scale based on the position information of the first length side and / or the second length side in the absolute space coordinate system, and use the positions contained in the area to plan the cargo operation position.

[0135] The cargo operation position is sent to the unmanned guided vehicle, so that the unmanned guided vehicle controls the fork arm to transport the cargo carried on the fork arm from the corresponding length side to the cargo operation position to complete the loading operation.

[0136] Specifically, the flatbed carriage detection device based on the unmanned guided vehicle sends the planned cargo operation position to the unmanned guided vehicle.

[0137] The unmanned guided vehicle receives and responds to the control instruction carrying the cargo operation position, drives the unmanned guided vehicle to move to the position of the corresponding length side, controls the fork arm to move to the cargo operation position, and performs loading operations on the cargo of corresponding specifications.

[0138] Figure 4 This is the second flow chart of the method for detecting a flatbed carriage based on an unmanned guided vehicle provided by the present invention. Figure 4 As shown, the embodiment of the present invention provides a complete implementation process of a flatbed carriage detection method based on an unmanned guided vehicle:

[0139] Step 1: Build the flatbed truck graphics library:

[0140] Several side images of flatbed trucks were collected at different angles, heights, and lighting conditions. To reduce the amount of calculation, the images were cropped and the local images of the flatbed truck endpoints were retained. The endpoints of the flatbed truck were manually annotated using labelme software.

[0141] Step 2: Use the TOF camera to continuously acquire RGBD images:

[0142] After the RGB data stream in the RGBD image is distorted, the depth data stream is aligned with the RGB distortion-corrected image using the camera's internal and external parameters. Then, a filter is used to remove noise and map it to a point cloud image to obtain the head and tail point cloud data of the carriage aligned with the RGB data stream.

[0143] Step 3: Image retrieval module:

[0144] The RGB data stream of the front end of the flatbed truck obtained by the camera is input into the MobileNetVlad network, which will output a picture in the flatbed truck graphics library that is most similar to the picture and whose similarity exceeds a certain threshold. This picture is called the front end reference picture.

[0145] Step 4: Feature extraction:

[0146] The SuperPoint network is used to extract feature points and descriptors from the RGB data of the front end of the flatbed truck and the front reference image, and the *.json annotation file of the front reference image is read to obtain the nearest feature point near the annotation point. If the point is within five pixels around the annotation point, the point is used as the first detection key point of the front end of the flatbed truck in the reference image.

[0147] Step 5: Feature matching module:

[0148] The SuperGlue network is used to match the feature points of the reference image and the matching image. Among several matching point pairs, the matching point pixel coordinates of the first detection key point at the front end of the flatbed truck are taken.

[0149] Step 6: Distance measurement module:

[0150] According to the first detection key point index to the point cloud data obtained in step 2, the coordinates of the front of the flatbed car are obtained. When the coordinates of the spatial point are empty, the pixel coordinates of the adjacent points are obtained as the coordinates of the front of the car.

[0151] If the camera RGB image resolution is set to 1280*720, steps 1 to 6 take about 170ms.

[0152] Step 7: Coordinate transformation:

[0153] Convert the vehicle head coordinates in the camera coordinate system to the vehicle head space coordinates in the absolute space coordinate system of AGV navigation.

[0154] Step 8: Get the space coordinates of the rear of the vehicle

[0155] Repeat steps 2 to 6 using the rear image in the RGB data stream to obtain the rear spatial coordinates of the flatbed car's rear end in the absolute spatial coordinate system.

[0156] Step 9: Construct the equation of a straight line

[0157] In the absolute space coordinate system, the linear equation is constructed using the space coordinates of the front and rear of the vehicle, thereby determining the length of the flatbed truck and the height of the vehicle compartment.

[0158] The cargo space on the flatbed truck is divided according to the size of the pallet cargo, and the precise location of the forklift loading navigation is determined. In addition, according to the width of the flatbed truck's compartment, the equation of the first and last endpoints on the other side of the flatbed truck and the edge of the compartment can be determined, making it convenient to carry out loading operations on both sides of the compartment at the same time.

[0159] The embodiment of the present invention divides corresponding cargo spaces in the first length side and / or the second length side according to cargo attribute information of the pallet cargo, and plans cargo operation positions to guide the unmanned guided vehicle to operate according to the cargo operation positions. The precise position of the forklift loading navigation can be determined, providing an effective solution for cargo space division and flexible loading.

[0160] Figure 5 FIG. 1 is a schematic diagram of the structure of a flatbed carriage detection device based on an unmanned guided vehicle provided by the present invention. Figure 5 As shown, the device includes an image acquisition module 510, an image retrieval module 520, an image matching module 530, a head and tail positioning module 540 and a car body fitting module 550, wherein:

[0161] The image acquisition module 510 is used to start the camera module on the unmanned guided vehicle to continuously receive the RGB data stream acquired by the camera module when it is determined that the interval distance between the unmanned guided vehicle and the first end point of the first length side of the flatbed carriage to be detected is greater than or equal to a first preset threshold. The RGB data stream has the front image and the rear image of the flatbed carriage to be detected as the starting frame and the ending frame respectively.

[0162] The image retrieval module 520 is used to match the front reference image and the rear reference image corresponding to the RGB data stream according to the flatbed vehicle graphics library, wherein the flatbed vehicle graphics library stores a partial image retaining the front and rear endpoints of the flatbed vehicle and the pixel coordinates of the endpoints marked in the partial image.

[0163] The image matching module 530 is used to perform feature point matching on the front image and the rear image of the vehicle in the RGB data stream with the front reference image and the rear reference image corresponding to the RGB data stream, respectively, to determine the pixel coordinates of the front pixel and the rear pixel.

[0164] The head and tail positioning module 540 is used to obtain the head space coordinates and the tail space coordinates of the flatbed car to be detected in the absolute space coordinate system based on the head pixel coordinates, the tail pixel coordinates and the head and tail point cloud data of the car. The head and tail point cloud data of the car are determined based on the depth data stream collected by the camera module while collecting the RGB data stream and the camera internal parameters of the camera module.

[0165] The car body fitting module 550 is used to construct a straight line equation using the front space coordinates and the rear space coordinates to determine the position information of the first length side of the flatbed car body to be detected in the absolute space coordinate system. The position information of the first length side of the flatbed car body to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to carry out loading operations from the first length side.

[0166] Specifically, the image acquisition module 510, the image retrieval module 520, the image matching module 530, the head and tail positioning module 540 and the vehicle compartment fitting module 550 are electrically connected in sequence.

[0167] The image acquisition module 510 can monitor the distance between the unmanned guided vehicle and surrounding objects in real time through the sensing element during its movement. When it is monitored that the interval between the unmanned guided vehicle and the first end point of the first length side of the flatbed carriage to be detected is greater than or equal to a first preset threshold, the visual task on the unmanned guided vehicle is driven to start, and while controlling the unmanned guided vehicle to move from the first end point to the rear end point of the carriage while maintaining a fixed distance from the vehicle body, the camera module is started, the trigger mode of the camera module is set to continuous trigger, and the RGB data stream is obtained.

[0168] The image retrieval module 520 can perform feature extraction on the RGB data stream, and then select an endpoint local image whose similarity with the extracted feature vector exceeds a certain threshold from the flatbed vehicle graphics library, and output it as the corresponding front vehicle reference image and rear vehicle reference image according to the endpoint type marked in the endpoint local image.

[0169] The image matching module 530 respectively compares the vehicle front image and the vehicle front reference image closest to it in the gallery, takes the pixel point in the vehicle front image that matches the corresponding marked endpoint in the vehicle front reference image as the front endpoint, assigns the pixel coordinates corresponding to the marked point in the vehicle front reference image to the front endpoint, and outputs the pixel coordinates of the vehicle front.

[0170] Similarly, for the rear image and the closest rear reference image in the library, the pixel point in the rear image that matches the corresponding marked endpoint of the rear reference image is used as the rear endpoint, and the pixel coordinates corresponding to the marked point in the rear reference image are assigned to the rear endpoint, and the pixel coordinates of the rear are output.

[0171] The head and tail positioning module 540 uses the pixel coordinates of the front and rear of the vehicle to index the head and tail point cloud data of the vehicle, and converts the indexed three-dimensional point cloud coordinates from the world coordinate system to the absolute space coordinate system to obtain the head space coordinates and the tail space coordinates of the first length side of the flatbed vehicle to be detected in the absolute space coordinate system.

[0172] The car fitting module 550 constructs a straight line equation using the space coordinates of the front and rear of the vehicle in the absolute space coordinate system, thereby determining the position information of the first length side of the flatbed car to be detected in the absolute space coordinate system, and can also fit the length and height of the flatbed car. Further, according to the size of the pallet cargo, the corresponding cargo position is divided in the position information of the first length side in the absolute space coordinate system to guide the forklift to load and navigate to the operating position of the first length side.

[0173] Optionally, the image matching module 530 includes a first feature extraction unit, a first detection key point screening unit, a first matching unit, a second feature extraction unit, a second detection key point screening unit and a second matching unit, wherein:

[0174] The first feature extraction unit is used to extract feature points and descriptors from the vehicle front image and the vehicle front reference image to obtain a first feature point set and a second feature point set.

[0175] The first detection key point screening unit is used to determine as the first detection key point a feature point in the second feature point set whose distance between the pixel coordinates and the pixel coordinates of the head end point carried by the vehicle head reference image is less than a second preset threshold.

[0176] The first matching unit is used to filter out point pairs including the first detection key points from a first point pair set obtained by matching the first feature point set with the second feature point set, and set the pixel coordinates of the first detection key points to the pixel coordinates of the front of the vehicle corresponding to the front endpoint in the front of the vehicle image.

[0177] The second feature extraction unit is used to extract feature points and descriptors from the vehicle rear image and the vehicle rear reference image to obtain a third feature point set and a fourth feature point set.

[0178] The second detection key point screening unit is used to determine as a second detection key point a feature point whose pixel coordinates in the third feature point set are at a distance less than the second preset threshold from the pixel coordinates of the rear end point carried by the rear vehicle reference image.

[0179] The second matching unit is used to filter out point pairs including the second detection key points from the second point pair set obtained by matching the third feature point set with the fourth feature point set, and set the pixel coordinates of the second detection key points to the pixel coordinates of the rear end of the vehicle corresponding to the rear end point in the rear end image.

[0180] Optionally, the head and tail positioning module 540 includes a head and tail positioning unit and a coordinate system conversion unit, wherein:

[0181] The head and tail positioning unit is used to use the head pixel coordinates and the tail pixel coordinates to index the head and tail point cloud data of the carriage to obtain the head coordinates and the tail coordinates of the flatbed carriage to be detected in the camera coordinate system.

[0182] The coordinate system conversion unit is used to perform coordinate system conversion on the front coordinates and the rear coordinates to obtain the front space coordinates and the rear space coordinates of the flatbed carriage to be detected in the absolute space coordinate system.

[0183] The absolute space coordinate system is a geodetic coordinate system or an absolute coordinate system whose origin is fixed at any corner point in the storage space.

[0184] Optionally, the device further comprises a distortion correction module and a point cloud mapping module, wherein:

[0185] The distortion correction module is used to align the depth data stream and the RGB data stream after distortion correction based on the depth data stream and the camera internal and external parameters of the camera module.

[0186] The point cloud mapping module is used to filter the depth data stream after distortion correction and map it to a point cloud image to obtain the head and tail point cloud data of the carriage aligned with the RGB data stream.

[0187] Optionally, the device further comprises a side fitting module and a loading and unloading planning module, wherein:

[0188] The side fitting module is used to construct a straight line equation based on the width of the flatbed car to be detected, the front space coordinates and the rear space coordinates, and determine the position information of the second length side of the flatbed car to be detected in the absolute space coordinate system.

[0189] The loading and unloading planning module is used to control the unmanned guided vehicle to perform loading operations on the corresponding sides by using the position information of the first length side and / or the second length side of the flatbed carriage to be detected in the absolute space coordinate system.

[0190] Optionally, the loading and unloading planning module includes an operation planning unit and a control unit, wherein:

[0191] The operation planning unit is used to determine the cargo operation position based on the position information of the first length side and / or the second length side of the flatbed carriage to be detected in the absolute space coordinate system and the cargo attribute information.

[0192] The control unit is used to send the cargo operation position to the unmanned guided vehicle, so that the unmanned guided vehicle controls the fork arm to move the cargo carried on the fork arm from the corresponding length side to the cargo operation position to complete the loading operation.

[0193] The flatbed car detection device based on an unmanned guided vehicle provided in an embodiment of the present invention is used to execute the above-mentioned flatbed car detection method based on an unmanned guided vehicle of the present invention. Its implementation method is consistent with the implementation method of the flatbed car detection method based on an unmanned guided vehicle provided by the present invention, and can achieve the same beneficial effects, which will not be repeated here.

[0194] The embodiment of the present invention uses an image retrieval network to extract the front reference image and the rear reference image that are closest to the real-time monitored RGB data stream from the flatbed graphics library that is updated and maintained in real time, and then matches the feature point pairs with the front image and the rear image in the RGB data stream to locate the pixel coordinates, and uses the located front pixel coordinates and rear pixel coordinates to obtain the front space coordinates and rear space coordinates in the absolute space coordinate system in the front and rear point cloud data index of the carriage, and fits the straight line equation that matches the first length side to guide the loading operation position according to the various positions of the first length side. In complex scenarios such as the unfixed parking position of the truck and the various types and specifications of the goods, the position of the flatbed carriage can be accurately identified, and the cargo specifications are combined to provide a basis for the division of cargo spaces, which is convenient for accurate navigation to the designated cargo space and realizes flexible loading.

[0195] Figure 6 Schematic diagram of the structure of the unmanned guided vehicle provided by the present invention. Figure 6 As shown, the unmanned guided vehicle includes an unmanned guided vehicle body 610, and a camera module 620 installed in the longitudinal axis direction of the unmanned guided vehicle body 610, and also includes a processor 630 arranged in the unmanned guided vehicle body 610. When the processor 630 executes the program, the above-mentioned flatbed carriage detection method based on the unmanned guided vehicle is implemented.

[0196] Specifically, the camera module 620 is installed in the longitudinal axis direction of the unmanned guided vehicle body 610, and its optical center is marked as O, which is the origin of the camera coordinate system. The main antenna is installed 1 meter above the camera module 620 along the longitudinal axis direction of the unmanned guided vehicle body 610. The center of the main antenna is marked as O1, which is the origin of the antenna coordinate system. In the actual geographical space of the warehouse, the Beidou base station is used as the origin, marked as Ow, and the north direction is the Z axis and the east direction is the X axis. In which:

[0197] The camera module 620 is used to continuously shoot the flatbed truck compartment from the front end to the rear end, and simultaneously obtain the RGB data stream and the depth data stream.

[0198] Processor 630 is used to call the logic instructions in the memory to execute the flatbed car detection method based on the unmanned guided vehicle, and sequentially goes through the steps of flatbed car warehouse building, TOF camera image acquisition, image retrieval, feature extraction, feature matching, ranging, coordinate transformation, straight line construction, etc. to identify the position of the flatbed car, providing a solution for cargo space division and flexible loading.

[0199] Among them, the unmanned transport vehicle can be an unmanned intelligent forklift or a manned traditional forklift.

[0200] The embodiment of the present invention uses an image retrieval network to extract the front reference image and the rear reference image that are closest to the real-time monitored RGB data stream from the flatbed graphics library that is updated and maintained in real time, and then matches the feature point pairs with the front image and the rear image in the RGB data stream to locate the pixel coordinates, and uses the located front pixel coordinates and rear pixel coordinates to obtain the front space coordinates and rear space coordinates in the absolute space coordinate system in the front and rear point cloud data index of the carriage, and fits the straight line equation that matches the first length side to guide the loading operation position according to the various positions of the first length side. In complex scenarios such as the unfixed parking position of the truck and the various types and specifications of the goods, the position of the flatbed carriage can be accurately identified, and the cargo specifications are combined to provide a basis for the division of cargo spaces, which is convenient for accurate navigation to the designated cargo space and realizes flexible loading.

[0201] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0202] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the flatbed car detection method based on an unmanned guided vehicle provided by the above-mentioned methods, the method including: when it is determined that the interval distance between the unmanned guided vehicle and the first endpoint of the first length side of the flatbed car to be detected is greater than or equal to a first preset threshold, starting the camera module on the unmanned guided vehicle to continuously receive the RGB data stream captured by the camera module; wherein the RGB data stream uses the front image and the rear image of the flatbed car to be detected as the starting frame and the ending frame respectively; matching the front reference image and the rear reference image corresponding to the RGB data stream according to the flatbed car graphics library; wherein the flatbed car graphics library stores a partial image that retains the front and rear endpoints of the flatbed car, and in the partial image The pixel coordinates of the marked endpoints; the front image and the rear image of the vehicle in the RGB data stream are matched with the front reference image and the rear reference image corresponding to the RGB data stream for feature points, respectively, to determine the front pixel coordinates and the rear pixel coordinates; based on the front pixel coordinates, the rear pixel coordinates and the front and rear point cloud data of the carriage, the front space coordinates and the rear space coordinates of the flatbed carriage to be detected in the absolute space coordinate system are obtained; wherein the front and rear point cloud data of the carriage are determined according to the depth data stream collected by the camera module while collecting the RGB data stream and the camera intrinsic parameters of the camera module; the straight line equation is constructed using the front space coordinates and the rear space coordinates to determine the position information of the first length side of the flatbed carriage to be detected in the absolute space coordinate system; wherein the position information of the first length side of the flatbed carriage to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to carry out loading operations from the first length side.

[0203] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the flatbed car detection method based on an unmanned guided vehicle provided by the above-mentioned methods, the method comprising: when it is determined that the interval distance between the unmanned guided vehicle and the first endpoint in the first length side of the flatbed car to be detected is greater than or equal to a first preset threshold, starting a camera module on the unmanned guided vehicle to continuously receive an RGB data stream captured by the camera module; wherein the RGB data stream has a front image and a rear image of the flatbed car to be detected as a start frame and an end frame respectively; matching a front reference image and a rear reference image corresponding to the RGB data stream according to a flatbed car graphics library; wherein the flatbed car graphics library stores a local image retaining the front and rear endpoints of the flatbed car, and the pixel coordinates of the endpoints annotated in the local image; converting the RGB data stream into a reference image; wherein the RGB data stream is a reference image of the front and rear of the flatbed car ... The front image and the rear image of the vehicle in the GB data stream are matched with the front reference image and the rear reference image corresponding to the RGB data stream for feature points, respectively, to determine the front pixel coordinates and the rear pixel coordinates; based on the front pixel coordinates, the rear pixel coordinates and the front and rear point cloud data of the car, the front spatial coordinates and the rear spatial coordinates of the flatbed car to be detected in the absolute spatial coordinate system are obtained; wherein, the front and rear point cloud data of the car are determined based on the depth data stream collected by the camera module while collecting the RGB data stream and the camera intrinsic parameters of the camera module; the straight line equation is constructed using the front spatial coordinates and the rear spatial coordinates to determine the position information of the first length side of the flatbed car to be detected in the absolute spatial coordinate system; wherein, the position information of the first length side of the flatbed car to be detected in the absolute spatial coordinate system is used to control the unmanned guided vehicle to carry out loading operations from the first length side.

[0204] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0205] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A flatbed carriage detection method based on an unmanned guided vehicle, characterized in that: include: When it is determined that the interval distance between the unmanned guided vehicle and the first end point in the first length side of the flatbed carriage to be detected is greater than or equal to a first preset threshold, the camera module on the unmanned guided vehicle is started to continuously receive the RGB data stream collected by the camera module; wherein the RGB data stream has the front image and the rear image of the flatbed carriage to be detected as the start frame and the end frame respectively; Matching the front and rear reference images of the vehicle corresponding to the RGB data stream according to the flatbed vehicle graphics library; wherein the flatbed vehicle graphics library stores a partial image retaining the front and rear endpoints of the flatbed vehicle, and the pixel coordinates of the endpoints marked in the partial image; Extract feature points and descriptors from the front image and the front reference image to obtain a first feature point set and a second feature point set; determine a feature point in the second feature point set whose distance between the pixel coordinates and the pixel coordinates of the leading end point carried by the front reference image is less than a second preset threshold as a first detection key point; filter out point pairs containing the first detection key point from a first point pair set obtained by feature point matching between the first feature point set and the second feature point set, and set the pixel coordinates of the first detection key point to the front pixel coordinates of the vehicle corresponding to the leading end point in the front image; correspondingly, extract feature points and descriptors from the rear image and the rear reference image to obtain a third feature point set and a fourth feature point set; determine a feature point in the third feature point set whose distance between the pixel coordinates and the pixel coordinates of the trailing end point carried by the trailing reference image is less than the second preset threshold as a second detection key point; filter out point pairs containing the second detection key point from a second point pair set obtained by feature point matching between the third feature point set and the fourth feature point set, and set the pixel coordinates of the second detection key point to the rear pixel coordinates of the vehicle corresponding to the trailing end point in the rear image; Based on the pixel coordinates of the front of the vehicle, the pixel coordinates of the rear of the vehicle and the point cloud data of the front and rear of the vehicle, the space coordinates of the front of the vehicle and the space coordinates of the rear of the vehicle of the flatbed vehicle to be detected in the absolute space coordinate system are obtained; wherein the point cloud data of the front and rear of the vehicle are determined according to the depth data stream collected by the camera module while collecting the RGB data stream and the camera internal parameters of the camera module; The linear equation is constructed using the spatial coordinates of the front of the vehicle and the spatial coordinates of the rear of the vehicle to determine the position information of the first length side of the flatbed car to be detected in the absolute spatial coordinate system; wherein the position information of the first length side of the flatbed car to be detected in the absolute spatial coordinate system is used to control the unmanned guided vehicle to carry out loading operations from the first length side.

2. The flatbed carriage detection method based on an unmanned guided vehicle according to claim 1 is characterized in that: The method of obtaining the space coordinates of the front and rear of the flatbed carriage to be detected in the absolute space coordinate system based on the front pixel coordinates, the rear pixel coordinates and the front and rear point cloud data of the carriage includes: The pixel coordinates of the front of the vehicle and the pixel coordinates of the rear of the vehicle are used to index the front and rear point cloud data of the vehicle compartment to obtain the front coordinates and the rear coordinates of the flatbed vehicle compartment to be detected in the camera coordinate system; The front coordinates and the rear coordinates of the vehicle are converted into coordinate systems to obtain the front space coordinates and the rear space coordinates of the flatbed vehicle to be detected in the absolute space coordinate system; The absolute space coordinate system is a geodetic coordinate system or an absolute coordinate system whose origin is fixed at any corner point in the storage space.

3. The flatbed carriage detection method based on an unmanned guided vehicle according to claim 1, characterized in that: Before acquiring the front space coordinates and the rear space coordinates of the flatbed carriage to be detected in the absolute space coordinate system based on the front pixel coordinates, the rear pixel coordinates and the front and rear point cloud data of the carriage, the method further includes: Based on the depth data stream and the camera internal and external parameters of the camera module, the depth data stream and the RGB data stream are respectively aligning after distortion correction; The distortion-corrected depth data stream is filtered and mapped to a point cloud image to obtain the head and tail point cloud data of the carriage aligned with the RGB data stream.

4. The flatbed carriage detection method based on an unmanned guided vehicle according to claim 1, characterized in that: After matching the front image and the rear image of the vehicle in the RGB data stream with the front reference image and the rear reference image corresponding to the RGB data stream for feature points to determine the front pixel coordinates and the rear pixel coordinates, the method further includes: Constructing a straight line equation based on the width of the flatbed carriage to be detected, the front space coordinates and the rear space coordinates, and determining the position information of the second length side of the flatbed carriage to be detected in the absolute space coordinate system; The position information of the first length side and / or the second length side of the flatbed carriage to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to perform loading operations on the corresponding side.

5. The method for detecting a flatbed carriage based on an unmanned guided vehicle according to claim 4, characterized in that: The method of controlling the unmanned guided vehicle to perform loading operation on the corresponding side by using the position information of the first length side and / or the second length side of the flatbed carriage to be detected in the absolute space coordinate system comprises: Determine the cargo operation position based on the position information of the first length side and / or the second length side of the flatbed carriage to be detected in the absolute space coordinate system and the cargo attribute information; The cargo operation position is sent to the unmanned guided vehicle, so that the unmanned guided vehicle controls the fork arm to transport the cargo carried on the fork arm from the corresponding length side to the cargo operation position to complete the loading operation.

6. A flatbed carriage detection device based on an unmanned guided vehicle, characterized in that: include: An image acquisition module is used to start the camera module on the unmanned guided vehicle to continuously receive the RGB data stream acquired by the camera module when it is determined that the interval distance between the unmanned guided vehicle and the first end point of the first length side of the flatbed carriage to be detected is greater than or equal to a first preset threshold; wherein the RGB data stream has the front image and the rear image of the flatbed carriage to be detected as the starting frame and the ending frame respectively; An image retrieval module is used to match the front reference image and the rear reference image corresponding to the RGB data stream according to the flatbed vehicle graphics library; wherein the flatbed vehicle graphics library stores a partial image retaining the front and rear endpoints of the flatbed vehicle, and the pixel coordinates of the endpoints marked in the partial image; an image matching module, for extracting feature points and descriptors from the front image of the vehicle and the front reference image to obtain a first feature point set and a second feature point set; determining a feature point whose distance between pixel coordinates in the second feature point set and the pixel coordinates of the front endpoint carried by the front reference image is less than a second preset threshold as a first detection key point; filtering out point pairs containing the first detection key point from a first point pair set obtained by matching the first feature point set with the second feature point set, and setting the pixel coordinates of the first detection key point to the front pixel coordinates of the vehicle corresponding to the front endpoint in the front image of the vehicle; correspondingly, extracting feature points and descriptors from the rear image of the vehicle and the rear reference image to obtain a third feature point set and a fourth feature point set; determining a feature point whose distance between pixel coordinates in the third feature point set and the pixel coordinates of the rear endpoint carried by the rear reference image is less than the second preset threshold as a second detection key point; filtering out point pairs containing the second detection key point from a second point pair set obtained by matching the third feature point set with the fourth feature point set, and setting the pixel coordinates of the second detection key point to the rear pixel coordinates of the vehicle corresponding to the rear endpoint in the rear image of the vehicle; A head and tail positioning module, used to obtain the head space coordinates and the tail space coordinates of the flatbed car to be detected in the absolute space coordinate system based on the head pixel coordinates, the tail pixel coordinates and the head and tail point cloud data of the car; wherein the head and tail point cloud data of the car are determined according to the depth data stream collected by the camera module while collecting the RGB data stream; The car body fitting module is used to construct a straight line equation using the space coordinates of the front of the car and the space coordinates of the rear of the car to determine the position information of the first length side of the flatbed car body to be detected in the absolute space coordinate system; wherein the position information of the first length side of the flatbed car body to be detected in the absolute space coordinate system is used to control the unmanned guided vehicle to carry out loading operations from the first length side.

7. An unmanned guided vehicle, comprising an unmanned guided vehicle body and a camera module installed in the longitudinal axis direction of the unmanned guided vehicle body, characterized in that: It also includes a processor arranged in the unmanned guided vehicle body, and when the processor executes a program, it implements the flatbed carriage detection method based on the unmanned guided vehicle as claimed in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the flatbed carriage detection method based on an unmanned guided vehicle as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the flatbed carriage detection method based on an unmanned guided vehicle as described in any one of claims 1 to 5 is implemented.

Images