Forklift pallet segmentation and positioning method and device and intelligent forklift

Abstract

The application provides a forklift pallet segmentation and positioning method and device and an intelligent forklift, and relates to the technical field of computer vision. The method comprises the following steps: inputting a warehouse color image into a segmentation model to obtain a first pallet image output by the segmentation model; obtaining a pallet pixel area, a pallet corner point coordinate and a pallet center point coordinate corresponding to each pallet by using a pallet contour coordinate set parsed from the first pallet image; performing threshold condition constraint on the pallet pixel area, the pallet center point coordinate and the pallet center point depth distance of all pallets, and performing condition constraint on the pallet center point coordinate in the Y-axis, i.e., the condition that the pallet center point coordinate is first upward and then downward, to screen a target pallet from the pallet contour coordinate set and determine the spatial coordinate of the center of the target pallet in a camera coordinate system; and performing coordinate system conversion on the spatial coordinate of the center of the target pallet in the camera coordinate system, and issuing the position coordinate of the target pallet in a warehouse plane coordinate system obtained by conversion to a forklift.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a forklift pallet segmentation and positioning method and device and an intelligent forklift. BACKGROUND

[0002] In recent years, the demand for automated handling equipment in the warehouse logistics industry has grown rapidly, and automated handling equipment represented by intelligent forklifts has developed rapidly. As one of the key technologies for realizing unmanned operation of intelligent forklifts, pallet recognition and positioning has attracted increasing attention.

[0003] There are two kinds of mainstream pallet recognition and positioning methods. In terms of point cloud recognition of pallets, a pallet template is preset for different types of pallets, and the position of the pallet is located by template matching. This method consumes a lot of computing resources, has a large amount of calculation and a long calculation time, and is difficult to realize real-time recognition and positioning of pallets, which seriously restricts the improvement of forklift operation efficiency. In terms of image recognition of pallets, the target detection method is used to locate the position of the pallet. This method cannot classify each pixel point, and often cannot accurately obtain the pixel coordinates of the target pallet. In addition, with the update of computer vision technology, although researchers have combined point cloud recognition and image recognition to solve the problem of single pallet recognition, they have not considered the situation of multi-type pallets stacked on top of each other and arranged in parallel in front and back in actual application scenarios. Therefore, the existing pallet positioning and recognition method cannot balance the recognition accuracy and speed of multi-type pallets placed irregularly, which reduces the actual working efficiency of the forklift. SUMMARY

[0004] The present application provides a forklift pallet segmentation and positioning method, device and intelligent forklift to solve the defects of low recognition accuracy and speed of multi-type pallets placed irregularly in the prior art.

[0005] The present application provides a forklift pallet segmentation and positioning method, device and intelligent forklift to solve the defects of low recognition accuracy and speed of multi-type pallets placed irregularly in the prior art.

[0006] The warehouse color image is input into the segmentation model to obtain a first pallet image output by the segmentation model; wherein the pallet pixel area of the target pallet type in the first pallet image is set to a color and pixel value matching the target pallet type, the target pallet type is any one of the pallet types, and the warehouse color image is color image information collected by a camera module on the forklift at a safe distance from the photographed warehouse pallet.

[0007] The pallet contour coordinate set parsed from the first pallet image is used to obtain the pallet pixel area, pallet corner point coordinates and pallet center point coordinates corresponding to each pallet.

[0008] The tray pixel area, tray center point coordinate and tray center point depth distance of all trays are subjected to threshold condition constraints, and the tray center point coordinate is subjected to a condition constraint that it is first upward and then downward in the Y axis, so as to screen out a target tray from the tray contour coordinate set and determine the spatial coordinate of the center of the target tray in the camera coordinate system; wherein the tray center point depth distance is determined by indexing the tray center point coordinate in the warehouse depth image, and the warehouse depth image is depth image information collected by the camera module at the same time of shooting the warehouse color image;

[0009] The spatial coordinate of the center of the target tray in the camera coordinate system is subjected to coordinate system conversion, and the position coordinate of the target tray in the warehouse plane coordinate system obtained by conversion is sent to a forklift, so that the forklift implements work on the target tray according to the position coordinate of the target tray in the warehouse plane coordinate system.

[0010] The segmentation model is trained according to a sample warehouse color image, a corresponding labeled tray type label in the sample warehouse color image, and a region color and a region pixel value corresponding to the tray type label.

[0011] According to the forklift tray segmentation and positioning method provided by the application, the segmentation model includes a category segmentation layer and a tray separation layer.

[0012] The warehouse color image is input into the category segmentation layer for semantic segmentation, and a second tray image output by the category segmentation layer is obtained; wherein a tray pixel area composed of pixel values corresponding to each tray type in the second tray image is colored as a color matching the tray type.

[0013] The second tray image is input into the tray separation layer for comparison of the upper and lower limits of the color threshold, and the first tray image output by the tray separation layer is obtained.

[0014] According to the forklift tray segmentation and positioning method provided by the application, the tray contour coordinate set parsed from the first tray image is used to obtain the tray pixel area, tray corner point coordinate and tray center point coordinate corresponding to each tray, including:

[0015] The contour edge of the first tray image is extracted, and the contour coordinates of the tray pixel area of each tray under the target tray type are integrated into the tray contour coordinate set;

[0016] The contour coordinates of each tray pixel area in the tray contour coordinate set are enveloped to obtain the first tray pixel area, first tray corner point coordinate and first tray center point coordinate of each tray.

[0017] In a case where the left upper corner point coordinate and the left lower corner point coordinate carried in the first tray corner point coordinate of the tray with the first tray center point coordinate in the middle position of the frame are updated to the left boundary of the frame, the second tray pixel area and the second tray center point coordinate of all complete trays in the left boundary field of view are obtained;

[0018] In a case where the right upper corner point coordinate and the right lower corner point coordinate carried in the first tray corner point coordinate of the tray with the first tray center point coordinate in the middle position of the frame are updated to the right boundary of the frame, the third tray pixel area and the third tray center point coordinate of all complete trays in the right boundary field of view are obtained.

[0019] According to the fork truck tray segmentation and positioning method provided by the application, the tray pixel area, the tray center point coordinate and the tray center point depth distance of all trays are subjected to threshold condition constraints, and the tray center point coordinate in the Y axis is subjected to a condition constraint of being first upward and then downward, and target tray contour coordinates are selected from a tray contour coordinate set, which comprises:

[0020] Based on the tray with the first tray center point coordinate in the first threshold range and the first tray pixel area in the second threshold range, the corresponding tray contour coordinate of the tray in the tray contour coordinate set is determined as a candidate tray center point coordinate in the same vertical warehouse range;

[0021] After a tray with a tray center point depth distance obtained by using the candidate tray center point coordinate in the third threshold range of the warehouse depth image is taken as a candidate tray in the front row in the same vertical warehouse range, the center spatial coordinate of each candidate tray in the camera coordinate system is obtained;

[0022] The candidate tray with the minimum Y coordinate value in the center spatial coordinate is taken as the target tray in the front row and the uppermost layer in the same vertical warehouse range, and the center spatial coordinate of the target tray is output.

[0023] The first threshold range is determined based on the second tray center point coordinate and the third tray center point coordinate; the second threshold range is determined based on the second tray pixel area and the third tray pixel area; and the third threshold range is determined based on a safe distance maintained by the camera module and the warehouse tray being photographed.

[0024] According to the fork truck tray segmentation and positioning method provided by the application, the spatial coordinate of the center of the target tray in the camera coordinate system is subjected to coordinate system conversion, and the position coordinate of the target tray in the warehouse plane coordinate system obtained by conversion is fed to a fork truck, so that the fork truck implements work on the target tray according to the position coordinate of the target tray in the warehouse plane coordinate system, which comprises:

[0025] projecting the spatial coordinates of the center of the target pallet in the camera coordinate system to the XOZ plane to obtain two-dimensional coordinates of the target pallet in the main antenna coordinate system;

[0026] converting the two-dimensional coordinates of the target pallet in the main antenna coordinate system to the warehouse plane coordinate system to obtain the position coordinates of the target pallet in the warehouse plane coordinate system.

[0027] The origin of the camera coordinate system is located at the optical center of the camera module, the camera optical axis Z is parallel to the ground and points to the front of the vehicle, and the X and Y axes point to the left side of the forklift and the ground, respectively. The origin of the main antenna coordinate system is the center point of the main antenna installed along the longitudinal axis of the vehicle and behind the camera module, with the forward direction of the forklift as the Z axis and the left side of the vehicle as the X axis. The warehouse plane coordinate system is a Gauss plane rectangular coordinate system with the Beidou base station as the origin and the north direction as the Z axis and the east direction as the X axis.

[0028] According to the forklift pallet segmentation and positioning method provided by the application, after the candidate pallet with the minimum Y coordinate value in the center spatial coordinates is selected as the target pallet in the front row and the uppermost layer in the same vertical warehouse range, the method further comprises:

[0029] Based on the pallet corner point coordinates of the target pallet, the left short side midpoint coordinates and the right short side midpoint coordinates are determined.

[0030] Based on the left short side midpoint coordinates, the right short side midpoint coordinates and the warehouse depth image, the inclination angle of the target pallet relative to the camera module is determined.

[0031] The inclination angle of the target pallet relative to the camera module is used to convert the working position of the forklift.

[0032] The application also provides a forklift pallet segmentation and positioning device, comprising:

[0033] The pallet segmentation module is used to input the warehouse color image into the segmentation model to obtain the first pallet image output by the segmentation model. In the first pallet image, the pixel area of the target pallet type is set to a color and pixel value matching the target pallet type. The target pallet type is any type of pallet, and the warehouse color image is color image information collected by the camera module on the forklift at a safe distance from the photographed warehouse pallet.

[0034] The pallet recognition module is used to obtain the pallet pixel area, pallet corner point coordinates and pallet center point coordinates corresponding to each pallet by analyzing the pallet contour coordinates set from the first pallet image.

[0035] A tray positioning module is configured to perform threshold condition constraint on tray pixel area, tray center point coordinate and tray center point depth distance of all trays, and constraint on the condition that the tray center point coordinate is first upward and then downward in the Y axis, to filter out a target tray from the tray contour coordinate set, and determine the spatial coordinate of the center of the target tray in the camera coordinate system; wherein the tray center point depth distance is determined by indexing the tray center point coordinate in the warehouse depth image, and the warehouse depth image is the depth image information collected by the camera module when the warehouse color image is captured;

[0036] A coordinate conversion module is configured to perform coordinate system conversion on the spatial coordinate of the center of the target tray in the camera coordinate system, and send the position coordinate of the target tray in the warehouse plane coordinate system obtained by conversion to a forklift, so that the forklift implements work on the target tray according to the position coordinate of the target tray in the warehouse plane coordinate system.

[0037] Wherein, the segmentation model is trained according to the sample warehouse color image, the corresponding labeled tray type label in the sample warehouse color image, and the region color and region pixel value corresponding to the tray type label.

[0038] The application further provides an intelligent forklift, which comprises a vehicle body, a camera module arranged on the vehicle body in the longitudinal direction of the vehicle body, and a main antenna arranged behind the camera module along the longitudinal direction of the vehicle body, and a processor arranged on the vehicle body, wherein the processor implements the segmentation and positioning method of the forklift tray according to any one of the above when executing the program.

[0039] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program implements the segmentation and positioning method of the forklift tray according to any one of the above when executed by a processor.

[0040] The application further provides a computer program product, which comprises a computer program, and the computer program implements the segmentation and positioning method of the forklift tray according to any one of the above when executed by a processor.

[0041] The application provides a forklift pallet segmentation and positioning method and device and an intelligent forklift, which utilizes a semantic segmentation network to segment different types of pallets from a warehouse color image, separates a first pallet image belonging to the same type of pallet, extracts the contour of the first pallet image through image processing, obtains the pallet pixel area, pallet corner point coordinates and pallet center point coordinates of each pallet of the same type, and combines the corresponding threshold value and logic processing based on the principle of first upper and then lower to locate the spatial coordinates of the center of a target pallet in the front row and the uppermost layer in the camera coordinate system, and convert the spatial coordinates to the warehouse plane coordinate system, so as to guide the forklift to implement work by using the position coordinates of the target pallet in the warehouse plane coordinate system, realize the complex scene of multi-layer stacking and front and rear parallel arrangement of different types of pallets, accurately locate the target pallet among numerous pallets, balance the accuracy and speed of pallet positioning, and further improve the work efficiency of the forklift. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0043] Figure 1 is a flowchart of the forklift pallet segmentation and positioning method provided by the application;

[0044] Figure 2 is a schematic view of pallet multi-layer stacking and front and rear arrangement;

[0045] Figure 3 is a schematic view of pallet multi-layer stacking and front and rear arrangement in the intermediate field of view of the camera;

[0046] Figure 4 is a schematic view of pallet multi-layer stacking and front and rear arrangement in the left limit field of view of the camera;

[0047] Figure 5 is a schematic view of pallet multi-layer stacking and front and rear arrangement in the right limit field of view of the camera;

[0048] Figure 6 is a structural schematic view of the forklift pallet segmentation and positioning device provided by the application;

[0049] Figure 7 is one of the structural schematic views of the forklift provided by the application;

[0050] Figure 8 is one of the structural schematic views of the forklift provided by the application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0052] The terms "first", "second", and the like in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class and do not limit the number of objects, for example, the first object can be one or more.

[0053] It should be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of the present application, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0054] The terms "include" and "contain" indicate the presence of the described features, whole, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0055] Figure 1 is a flowchart of the method for splitting and positioning the forklift pallet provided by the present application. As shown in Figure 1 The method for splitting and positioning the forklift pallet provided by the embodiment of the present application comprises: step 101, inputting a warehouse color image into a segmentation model to obtain a first pallet image output by the segmentation model.

[0056] Wherein, the pallet pixel area of the target pallet type in the first pallet image is set to a color and pixel value matched with the target pallet type, the target pallet type is any one of the pallet types, and the warehouse color image is color image information collected by a camera module on a forklift under the condition that the forklift maintains a safe distance from the warehouse pallet being photographed. The segmentation model is trained according to a sample warehouse color image, a corresponding pallet type label annotated in the sample warehouse color image, and a region color and a region pixel value corresponding to the pallet type label.

[0057] It should be noted that the execution subject of the fork truck pallet segmentation and positioning method provided by the embodiment of the present application is a fork truck pallet segmentation and positioning device. The fork truck pallet segmentation and positioning device can be a central processing unit (CPU) built in a fork truck, or a development board integrated based on the CPU, to perform information processing and program running.

[0058] The application scenario of the fork truck pallet segmentation and positioning method provided by the embodiment of the present application is that, during the travel of the fork truck, a vision task is started at a position at a certain safety distance from the pallet, the storage state of different goods placed on different categories of pallets is scanned, each category of pallet contained in the RGBD image shot is identified, and a specified pallet in a certain category is positioned.

[0059] The camera module can be a depth camera. The depth camera types include but are not limited to binocular cameras, TOF cameras or structured light cameras, etc., and different types of depth cameras only differ in ranging principles. Within the effective ranging range, the ranging accuracy of the depth camera can reach centimeter level or even higher, and the smaller the ranging range, the higher the accuracy, and the change range of the effective ranging result is still in centimeter level.

[0060] The safety distance between the camera module and the goods is often limited by the depth camera sensing distance and sensing accuracy, segmentation model accuracy, and additional factors such as camera field of view, fork truck arm length, fork truck path planning, etc. The distance is often set to no more than 3 meters.

[0061] For example, the safety distance of the vision task set by the present application is 2 meters, denoted as D.

[0062] It should be noted that the segmentation model can be a neural network model, and the structure and parameters of the neural network include but are not limited to the number of layers of the input layer, the hidden layer and the output layer of the neural network, and the weight parameters of each layer, etc. The type and structure of the neural network are not specifically limited in the embodiment of the present application.

[0063] For example, the segmentation model can be a semantic segmentation network, which is composed of an input layer, a hidden layer and an output layer, wherein:

[0064] The input layer is at the most front end of the entire network, and directly receives the warehouse color image shot by the camera module arranged on the fork truck for the multi-layer stacked and front-to-back parallel arranged goods pallets.

[0065] The hidden layer can have one or more layers, and the warehouse color image input is calculated by the neurons of the hidden layer in a weighted summation manner, and the calculation formula can be expressed as:

[0066] z=b+w1*x1+w2*x2+…+wm*xm

[0067] Wherein, z is the weight summation value of the hidden layer output, x1, x2, x3…xm are m feature vectors of each sample, b is the bias, w1, w2…wm are the weights corresponding to each feature vector.

[0068] The output layer is the last layer, which is used to decode the vector obtained after weighted summation, and the pixel area of the tray containing different tray types is assigned to the corresponding pixel value and recolored to the corresponding color for visualization in the front end.

[0069] It should be noted that the sample data includes sample warehouse color images of multiple types and quantities of trays under different light, different angles, different heights, and different distances corresponding to the sample data, and a manual annotation method, which annotates the class label of the complete tray area appearing in the sample warehouse color image. After the sample data is divided into a training set and a test set according to a certain proportion.

[0070] Exemplarily, the proportion of the training set and the test set in the sample data includes but is not limited to 9:1, 8:2, etc., and the embodiments of the present application do not make specific limitations thereto.

[0071] Wherein, the class label can be denoted as tuopan[i], and the variable i represents different tray types. In addition, the matching label color (such as red [128, 0, 0], green [0, 128, 0]) is set under the tuopan[i] parameter, and each pixel point in the tray pixel area labeled as tuopan[i] is also reset to the matching label pixel value (such as 1, 2, 3, etc., and 0 is set as the pixel value corresponding to the background area). The correspondence of different types of tray labels, region colors, and region pixel values is completed, and the segmentation base map mask of the tray is generated.

[0072] It should be noted that the label name, color type, pixel value size, etc. during tray labeling in the present application only represent one example of the present application, which is convenient for explaining the processing steps.

[0073] Specifically, in step 101, the segmentation and positioning device of the forklift tray initializes the weight coefficients between the layers of the constructed segmentation model, and then inputs a group of sample warehouse color images in the training set and the annotation content of the sample warehouse color images to the neural network under the current weight coefficient, and sequentially calculates the output of each node of the input layer, the hidden layer and the output layer. The accumulated error between the final output result of the output layer and its actual connection position state type is corrected according to the gradient descent method. According to the above process, the weight coefficients of the input layer and the hidden layer can be obtained by traversing all samples in the training set.

[0074] The forklift pallet segmentation and positioning device restores the segmentation model in step 101 according to the weight coefficients of the input layer and the hidden layer of the neural network, inputs each warehouse color image to be identified in the test set into the trained segmentation model, and can obtain the area where different pixel values are located, that is, a specified type of pallet, and different areas correspond to different RGB color first pallet images.

[0075] In step 102, the forklift pallet segmentation and positioning device obtains the pallet pixel area, the pallet corner point coordinates, and the pallet center point coordinates corresponding to each pallet by using the pallet contour coordinate set parsed from the first pallet image.

[0076] Specifically, in step 102, the forklift pallet segmentation and positioning device performs inflation and corrosion operations on the first pallet image to generate a binary image, and then extracts the pallet contour edge. After integrating the pallet contour coordinates of each pallet pixel area of the same type into a pallet contour coordinate set, the set is further processed to obtain the pallet pixel area, the pallet corner point coordinates, and the pallet center point coordinates corresponding to each pallet of the same type.

[0077] In step 103, the forklift pallet segmentation and positioning device filters out the target pallet from the pallet contour coordinate set by performing threshold condition constraints on the pallet pixel area, the pallet center point coordinates, and the pallet center point depth distance of all pallets, and performing a condition constraint on the Y-axis of the pallet center point coordinates from top to bottom. The spatial coordinates of the center of the target pallet in the camera coordinate system are determined.

[0078] The pallet center point depth distance is determined by indexing the pallet center point coordinates in the warehouse depth image. The warehouse depth image is the depth image information collected by the camera module when the warehouse color image is captured.

[0079] It should be noted that the pallet center point depth distance is obtained by searching the two-dimensional pallet center point coordinates in the warehouse depth image collected when the warehouse color image is captured to obtain the three-dimensional spatial coordinates of the pallet center corresponding to the pallet center point coordinates in the camera coordinate system.

[0080] Specifically, in step 103, the forklift pallet segmentation and positioning device performs threshold comparison related constraints on the pallet pixel area, the pallet center point coordinates, and the pallet center point depth distance of each pallet in the set, and can filter out multiple pallets whose pallet center point coordinates are in the same vertical range and whose pallet pixel area and depth distance meet the field of view distance requirements. Then, according to the constraint condition that the Y-axis coordinate value of the pallet center point is the largest, the uppermost pallet is positioned from the multiple front-row and vertically distributed pallets as the target pallet for priority operation, and the converted three-dimensional spatial coordinates of the center point of the target pallet in the camera coordinate system are taken as the center spatial coordinates.

[0081] Step 104, coordinate system conversion is performed on the spatial coordinates of the center of the target pallet in the camera coordinate system, and the position coordinates of the target pallet in the warehouse plane coordinate system obtained through the conversion are sent to the forklift, so that the forklift implements work on the target pallet according to the position coordinates of the target pallet in the warehouse plane coordinate system.

[0082] Specifically, in step 104, the segmentation and positioning device of the forklift pallet projects the spatial coordinates of the center of the target pallet in the camera coordinate system to the warehouse plane coordinate system, obtains the position coordinates of the target pallet in the actual geographical space, and sends the position coordinates to the forklift.

[0083] The forklift receives the control instruction carrying the position coordinates of the target pallet in the warehouse plane coordinate system sent by the segmentation and positioning device of the forklift pallet, and drives the relevant mechanical components to move the forks of the forklift to the position coordinates in the control instruction, thereby completing the loading and unloading work on the target pallet.

[0084] The embodiment of the application utilizes the semantic segmentation network to segment different types of pallets in the warehouse color image, separates the first pallet image belonging to the same type of pallet, extracts the contour of the first pallet image through image processing, obtains the pallet pixel area, pallet corner point coordinates and pallet center point coordinates corresponding to each pallet of the same type, and combines the corresponding threshold value and the logic processing based on the principle of first upper and then lower, to locate the spatial coordinates of the center of the target pallet in the front row and the uppermost layer in the camera coordinate system, and convert it to the warehouse plane coordinate system, so as to guide the forklift to implement work by using the position coordinates of the target pallet in the warehouse plane coordinate system, realize accurate positioning of the target pallet in the complex scene of different types of pallets stacked in multiple layers and arranged in front and back, and balance the accuracy and speed of pallet positioning, thereby further improving the work efficiency of the forklift.

[0085] On the basis of any of the above embodiments, the segmentation model comprises a category segmentation layer and a pallet separation layer.

[0086] The warehouse color image is input into the category segmentation layer for semantic segmentation, and a second pallet image output by the category segmentation layer is obtained; wherein the pallet pixel area composed of the pixel value corresponding to each pallet type in the second pallet image is colored as a color matching the pallet type.

[0087] The second pallet image is input into the pallet separation layer for comparison of the upper and lower limits of the color threshold value, and the first pallet image output by the pallet separation layer is obtained.

[0088] Specifically, the segmentation model is composed of an input layer, a hidden layer and an output layer. The hidden layer is used to extract features of the input warehouse color image through its own neurons, and extract feature information belonging to a certain type of pallet.

[0089] Exemplarily, Figure 2 is a schematic diagram of pallet multi-layer stacking and front-back arrangement space provided by the present application. As Figure 2 shown, a warehouse scene of pallet multi-layer stacking and front-back arrangement is given:

[0090] Preferably, the segmentation model is a DDRnets network, and the hidden layer thereof contains at least two layers, i.e., a category segmentation layer and a pallet separation layer, wherein:

[0091] The category segmentation layer processes the video frames of the camera in real time through a semantic segmentation network to obtain a second pallet image containing multiple types of pallets.

[0092] Among them, the area where the different pixel values in the second pallet image are located represents different pallets, and different areas correspond to different RGB colors.

[0093] The pallet separation layer filters according to different pixel values to obtain a specific type of pallet, and generates a visual first pallet image P1 of a certain specified pallet type. Conversely, without filtering, a visual first pallet image P2 of all pallet types can be generated.

[0094] Exemplarily, Figure 3 is a schematic diagram of pallet multi-layer stacking and front-back arrangement in the intermediate field of view of the camera provided by the present application. As Figure 3 shown, a filtering condition of limiting the color threshold to red is given, and only the pixel area colored red is retained in the first pallet image to realize the specification of the pallet type.

[0095] There are two specific processing methods for different situations. The first method is to set the upper and lower limits of the color threshold for the visual image P2 using the color traversal method, extract the pallet area of a certain specified color, then use the contour detection method to perform inflation and corrosion operations on the image to generate a binary image, and then extract the contour edge of the pallet to obtain the contour coordinates of each pallet pixel area of the pallet type corresponding to the color threshold.

[0096] The second method is to directly use the contour detection method to perform inflation and corrosion operations on P1 to generate a binary image, and then extract the contour edge of the pallet to obtain the contour coordinates of each pallet pixel area of the specified pallet type.

[0097] The embodiment of the present application performs semantic segmentation on the warehouse color image through the category segmentation layer to obtain a second pallet image visually distinguishing different pallet types, and then uses the pallet separation layer to compare the upper and lower limits of the color threshold of the second pallet image to obtain a first pallet image visually distinguishing only the specified pallet type. The different types of pallets can be separated through the semantic segmentation network first, and then the image processing method is used to separate different pallets of the same type, so that the accuracy and speed of pallet recognition and positioning are improved in the warehouse scene of multiple categories of pallets.

[0098] On the basis of any of the above embodiments, the pallet contour coordinate set parsed from the first pallet image is used to obtain the pallet pixel area, pallet corner point coordinates and pallet center point coordinates corresponding to each pallet, including: performing contour edge extraction on the first pallet image, and integrating the contour coordinates of each pallet pixel area of the target pallet type into the pallet contour coordinate set.

[0099] Specifically, the forklift pallet segmentation and positioning device performs inflation and corrosion operations on the first pallet image to generate a binary image, and then performs contour edge extraction on the pallet to integrate the contour coordinates of each pallet pixel area of the specified pallet type into the pallet contour coordinate set.

[0100] The contour coordinates of each pallet pixel area in the pallet contour coordinate set are enveloped to obtain the first pallet pixel area, first pallet corner point coordinates and first pallet center point coordinates of each pallet.

[0101] Specifically, since the side of the pallet is a regular rectangle, the forklift pallet segmentation and positioning device uses the minimum area rectangle envelope method to envelope the pallet contour coordinates of each pallet in the pallet contour coordinate set, and the length and width of the minimum area envelope rectangle of each pallet are calculated according to the length and width to calculate the rectangular area S i , which is the first pallet pixel area corresponding to the pallet, the first pallet center point coordinates are marked as (u i ,v i ), and the four corner point coordinates in the first pallet corner point coordinates are marked as the left upper corner point , the left lower corner point , the right lower corner point and the right upper corner point

[0102] In the case where the left upper corner point coordinates and the left lower corner point coordinates carried in the first pallet corner point coordinates of the pallet with the first pallet center point coordinates in the middle position of the frame are updated as the left boundary of the frame, the second pallet pixel area and the second pallet center point coordinates of all complete pallets in the left boundary field are obtained.

[0103] Specifically, the fork truck pallet segmentation and positioning device moves the pallet originally in the middle position of the camera field of view to the left boundary position of the camera field of view, and the minimum envelope rectangular area S L As the second pallet pixel area corresponding to the pallet, the second pallet center point coordinates are marked as (U L ,V L ).

[0104] Exemplarily, Figure 4 is a schematic diagram of the multi-layer stacking and front-rear arrangement of the pallets in the left limit field of view of the camera provided by the present application. As Figure 4 shown, the embodiment of the present application can determine whether the field of view is completed to the left by whether the upper left corner point coordinates and the lower left corner point coordinates carried in the first pallet corner point coordinates of the pallet originally in the middle position of the camera field of view reach the left boundary.

[0105] The embodiment of the present application can visually display the first pallet image at the front end, and the user monitors in real time whether the left side of the pallet originally in the middle position of the camera field of view reaches the left boundary of the viewfinder during the left translation of the fork truck, to determine whether the field of view is completed to the left.

[0106] In the case where the upper right corner point coordinates and the lower right corner point coordinates carried in the first pallet corner point coordinates of the pallet with the center point coordinates in the middle of the frame are updated to the right boundary of the frame, the third pallet pixel area and the third pallet center point coordinates of all complete pallets in the right boundary field of view are obtained.

[0107] Specifically, the fork truck pallet segmentation and positioning device moves the pallet originally in the middle position of the camera field of view to the left boundary position of the camera field of view, and the minimum envelope rectangular area S R As the third pallet pixel area corresponding to the pallet, the third pallet center point coordinates are marked as (U R ,V R ).

[0108] Exemplarily, Figure 5 is a schematic diagram of the multi-layer stacking and front-rear arrangement of the pallets in the right limit field of view of the camera provided by the present application. As Figure 5 shown, the embodiment of the present application can determine whether the field of view is completed to the right by whether the upper right corner point coordinates and the lower right corner point coordinates carried in the first pallet corner point coordinates of the pallet originally in the middle position of the camera field of view reach the right boundary.

[0109] In this embodiment of the invention, the first pallet image can also be visualized at the front end. During the process of the forklift moving to the right, the user can monitor in real time whether the right side of the pallet, which was originally in the middle of the camera's field of view, reaches the right boundary of the viewfinder to determine whether the field of view has been moved to the right.

[0110] In this embodiment of the invention, contour extraction is performed on the first tray image to obtain a set of contour coordinates. Coordinate conversion is then performed on the contour information in the central field of view, the left limiting field of view, and the right limiting field of view to obtain at least the tray pixel area and the coordinates of the tray center point in the corresponding field of view. This allows for correction using relevant positional information in the left and right limiting fields of view during subsequent separation of different trays of the same type, avoiding errors caused by relying solely on the image in the central field of view for tray semantic segmentation.

[0111] Based on any of the above embodiments, threshold constraints are applied to the pallet pixel area, pallet center point coordinates, and pallet center point depth distance of all pallets, as well as the condition constraint that the pallet center point coordinates are first up and then down in the Y-axis. The target pallet outline coordinates are then selected from the set of pallet outline coordinates, including: determining the pallet center point coordinates of candidate pallets within the same vertical storage range based on the pallet outline coordinates of pallets whose first pallet center point coordinates are within a first threshold range and whose first pallet pixel area is within a second threshold range.

[0112] The first threshold range is determined based on the coordinates of the center point of the second tray and the center point of the third tray. The second threshold range is determined based on the pixel area of ​​the second tray and the pixel area of ​​the third tray.

[0113] Specifically, the forklift pallet segmentation and positioning device takes into account the factor of pallet semantic segmentation error, using the width pixel coordinate value U in the coordinates of the second pallet center point and the third pallet center point at the left and right extreme positions. L and U R Based on this, the width pixel adjustment amount is set to ΔU. L and ΔU R Center point width pixel coordinates u i In the first threshold range [U L +ΔU L U R +ΔU R Only those trays within the specified vertical range are likely to be within the correct range.

[0114] Among them, U L +ΔU L U R +ΔU R These are the left and right extreme pixel distance thresholds, ΔU L and ΔUR The optimal value can be determined based on actual testing.

[0115] Next, the pixel area S of the second tray at the left and right extreme positions is... L and the third tray pixel area S R arithmetic mean Based on this, the pixel area adjustment amount is set to ΔS. L and ΔS R Tray pixel area S i Second threshold range Only those within the specified range are likely to be pallets of relatively accurate pallet type within the same vertical range.

[0116] in, and These are the upper and lower thresholds for pixel area, ΔS. L and ΔS R The optimal value can be determined based on actual testing.

[0117] Finally, the tray pixel coordinates after processing the center point pixel distance threshold and pixel area threshold are defined as (u i ,v i According to (u) i ,v i In the warehouse depth image, index the three-dimensional coordinates (x, y) of the candidate pallets that meet the criteria in the camera coordinate system. i ,y i ,z i ), which is used as the center spatial coordinate of each candidate pallet in the front row within the same vertical storage range in the camera coordinate system.

[0118] It should be noted that pixel area thresholding can, to some extent, remove the influence of background noise caused by inaccurate semantic segmentation.

[0119] After selecting pallets whose center point depth distance from the warehouse depth image obtained using the candidate pallet center point coordinates is within the third threshold range as candidate pallets within the same vertical warehouse range, the center spatial coordinates of each candidate pallet in the camera coordinate system are obtained.

[0120] The third threshold range is determined based on the safe distance maintained by the camera module and the warehouse pallet being photographed.

[0121] Specifically, the forklift pallet segmentation and positioning device indexes the center point coordinates (x, y) of each candidate pallet for each pallet. i ,y i ,z iValues ​​with infinity (±INF) and null values ​​(NAN) are removed. Multiple distance measurements are then performed, and outliers that might cause sudden changes are avoided by removing the maximum and minimum values. Finally, the average coordinates over multiple valid distance measurements are calculated, and the coordinates of the candidate pallet center point are updated.

[0122] Considering the impact of depth camera ranging errors, the depth distance adjustment ΔD is set based on the safe distance D between the camera module and the warehouse pallet being photographed when the vision task is enabled. F and ΔD B z in the coordinates of the center point of the candidate tray i Within the third threshold range [D+ΔD] F ,D+ΔD B The pallets inside are considered to be in the front row, closer to the forklift.

[0123] Where D+ΔD F and D+ΔD B These are the front and rear depth distance thresholds, ΔD. F and ΔD B The optimal value can be determined based on actual testing. The center space coordinates after processing the pallet center point depth-distance threshold are as follows:

[0124] The candidate pallet with the smallest Y-coordinate value in the central spatial coordinates is selected as the target pallet that is in the front row and on the top layer within the same vertical storage range, and the central spatial coordinates of the target pallet are output.

[0125] Specifically, the forklift pallet segmentation and positioning device processes multi-layered pallets stacked on the same side according to a top-to-bottom logic. The center spatial coordinates of the multiple pallets after multi-dimensional threshold constraints are defined as follows: According to height coordinates Arranged in ascending order of size, the tray with the smallest height coordinate is the tray at the top, i.e., the target tray. The center spatial coordinates of the target tray are then (x...). m ,y m ,z m ).

[0126] This invention imposes constraints on the pixel area, center point coordinates, and depth distance of all pallets from three dimensions: pixel area, center point coordinates, and depth distance. This yields candidate pallets within the same vertical range and in the front row. Then, based on a top-down processing logic, the topmost candidate pallet is selected as the target pallet, and its center spatial coordinates are calculated. This fully considers the complex scenarios of multi-layered stacking and parallel arrangement of different types of pallets, enabling accurate location of the target pallet among numerous pallets.

[0127] On the basis of any of the above embodiments, the spatial coordinates of the center of the target pallet in the camera coordinate system are subjected to coordinate system conversion, and the position coordinates of the target pallet in the warehouse plane coordinate system obtained by the conversion are fed to the forklift, so that the forklift implements work on the target pallet according to the position coordinates of the target pallet in the warehouse plane coordinate system, including: projecting the spatial coordinates of the center of the target pallet in the camera coordinate system onto the XOZ plane to obtain two-dimensional coordinates of the target pallet in the main antenna coordinate system.

[0128] The origin of the camera coordinate system is located at the optical center of the camera module, the camera optical axis Z is parallel to the ground and points to the front of the vehicle, and the X and Y axes point to the left side of the forklift and the ground, respectively. The origin of the main antenna coordinate system is the center point of the main antenna installed behind the camera module along the longitudinal axis direction of the vehicle, and the forward direction of the forklift is the Z axis and the left side of the vehicle is the X axis.

[0129] It should be noted that the camera module is installed along the longitudinal axis direction of the vehicle body, and in the camera coordinate system with the optical center of the camera module as the origin, the camera optical axis Z is parallel to the ground and points to the front of the vehicle, and the X and Y axes point to the left side of the vehicle and the ground, respectively.

[0130] For the forklift, two antennas are installed on the roof of the forklift, one in front and one behind along the longitudinal axis direction of each, and the antenna farther away from the camera module is the main antenna, while the relatively closer one is the slave antenna. The distance between the main antenna and the slave antenna is at least 1 meter.

[0131] The main antenna coordinate system is established with the center of the main antenna as the origin, the forward direction of the vehicle as the Z axis, and the left side of the vehicle as the X axis.

[0132] Specifically, the forklift pallet segmentation and positioning device performs dimensionality reduction processing on the camera coordinate system, ignores the height information, projects the coordinates onto the XOZ plane to obtain the camera plane coordinate system, and the main antenna coordinate system is obtained by translation from the camera plane coordinate system. The calculation formula is as follows:

[0133]

[0134] Wherein, x m and z m are the width and depth coordinates of the target pallet in the camera plane coordinate system. x g and z g are the coordinates of the target pallet in the main antenna coordinate system. Δx and Δz are the distance differences of the origins of the two coordinate systems in the X and Z axes directions, respectively, which can be obtained by actual measurement of the installation positions of the main antenna and the camera module and assigned values.

[0135] Convert the two-dimensional coordinates of the target pallet in the main antenna coordinate system to the storage plane coordinate system to obtain the position coordinates of the target pallet in the storage plane coordinate system.

[0136] The storage plane coordinate system is a Gauss plane rectangular coordinate system with the Beidou base station as the origin and the north direction as the Z axis and the east direction as the X axis.

[0137] It should be noted that the storage plane coordinate system is a Gauss plane rectangular coordinate system with the Beidou base station as the origin and the north direction as the Z axis and the east direction as the X axis.

[0138] Specifically, the forklift pallet segmentation and positioning device converts the two-dimensional coordinates of the target pallet in the main antenna coordinate system through translation matrix and rotation matrix to obtain the position coordinates of the target pallet in the storage plane coordinate system, and the calculation formula is as follows.

[0139]

[0140] Wherein, X and Z are the coordinates of the target pallet in the storage plane coordinate system. Theta is the azimuth angle, that is, the angle between the main antenna and the north direction. Delta x w and delta z w are the coordinates of the center of the main antenna, that is, the origin of the main antenna coordinate system in the storage plane coordinate system, wherein delta x w is the east coordinate, and delta z w is the north coordinate. Delta x w , delta z w and theta can be directly calculated and output by the Beidou terminal installed on the unmanned forklift.

[0141] Through formulas (1) and (2), the coordinates of the target pallet in the storage plane coordinate system are obtained, as shown in the following formula:

[0142]

[0143] Wherein, the definitions of the variables in formula (3) are the same as those in formulas (1) and (2).

[0144] The embodiment of the application converts the center space of the target pallet in the camera coordinate system to the main antenna coordinate system and then to the storage plane coordinate system, and finally obtains the position coordinates of the target pallet in the storage plane coordinate system, which can guide the navigation coordinates of the intelligent forklift and improve the forklift operation efficiency.

[0145] On the basis of any of the above embodiments, after the candidate pallet with the minimum Y coordinate value in the central space coordinates is taken as the target pallet in the front row and the uppermost layer in the same vertical storage range, the method further comprises: determining a left short side midpoint coordinate and a right short side midpoint coordinate based on pallet corner point coordinates of the target pallet.

[0146] Specifically, after the forklift pallet segmentation and positioning device determines the target pallet, the left upper corner point and the left lower corner point are calculated based on the left upper corner point and the left lower corner point carried by the target pallet's pallet corner point coordinates. The right short side midpoint coordinate is calculated based on the right upper corner point and the right lower corner point

[0147] . Based on the left short side midpoint coordinate, the right short side midpoint coordinate and the storage depth image, the tilt angle of the target pallet relative to the camera module is determined.

[0148] The tilt angle of the target pallet relative to the camera module is used for conversion of the working position of the forklift.

[0149] Specifically, the forklift pallet segmentation and positioning device indexes the three-dimensional coordinates of the left short side midpoint (Xl, Yl, Zl) and the three-dimensional coordinates of the right short side midpoint (Xr, Yr, Zr) in the storage depth image based on the left short side midpoint coordinate and the right short side midpoint coordinate, respectively.

[0150] Then, based on the trigonometric relationship between the left short side midpoint and the right short side midpoint, the calculation formula of the tilt angle is obtained:

[0151]

[0152] wherein, alpha is the tilt angle of the target pallet relative to the camera module, and the unit is radian system; alpha is used for conversion of the working position of the target pallet in the forklift navigation system. Delta Z is the difference between the depth coordinate values of the left short side midpoint and the right short side midpoint, i.e. Zr-Zl. Delta X is the difference between the horizontal coordinate values of the left short side midpoint and the right short side midpoint, i.e. Xr-Xl.

[0153] The embodiment of the application calculates the depth difference of the two short side midpoints based on the left short side midpoint coordinate and the right short side midpoint coordinate of the minimum area envelope rectangle of the target pallet, and calculates the tilt angle of the pallet using the trigonometric relationship, which can provide a basis for the navigation of the intelligent forklift and improve the forklift working efficiency.

[0154] Figure 6is a structural schematic view of a forklift pallet segmentation and positioning device provided by the present application. On the basis of any of the above embodiments, as shown in the figure, the device includes a pallet segmentation module 610, a pallet identification module 620, a pallet positioning module 630, and a coordinate conversion module 640, wherein: Figure 6

[0155] The pallet segmentation module 610 is configured to input a warehouse color image into a segmentation model and obtain a first pallet image output by the segmentation model. In the first pallet image, the pixel area of a target pallet type is set to a color and pixel value matching the target pallet type. The target pallet type is any one of the pallet types, and the warehouse color image is color image information collected by a camera module on a forklift at a safe distance from the warehouse pallet being photographed.

[0156] The pallet identification module 620 is configured to obtain the pixel area, corner point coordinates, and center point coordinates of each pallet using a set of pallet contour coordinates parsed from the first pallet image.

[0157] The pallet positioning module 630 is configured to perform threshold condition constraints on the pixel area, center point coordinates, and depth distance of the center point of all pallets, and perform a condition constraint on the center point coordinates of the pallets in the Y-axis from top to bottom, filter out a target pallet from the set of pallet contour coordinates, and determine the spatial coordinates of the center of the target pallet in the camera coordinate system. The depth distance of the center point of the pallet is determined by indexing the center point coordinates of the pallet in a warehouse depth image. The warehouse depth image is depth image information collected by the camera module at the same time as the warehouse color image is photographed.

[0158] The coordinate conversion module 640 is configured to perform coordinate system conversion on the spatial coordinates of the center of the target pallet in the camera coordinate system, and send the position coordinates of the target pallet in the warehouse plane coordinate system obtained by the conversion to a forklift, so that the forklift performs work on the target pallet according to the position coordinates of the target pallet in the warehouse plane coordinate system.

[0159] The segmentation model is trained based on a sample warehouse color image, a corresponding labeled pallet type label in the sample warehouse color image, and the region color and region pixel value corresponding to the pallet type label.

[0160] Specifically, the pallet segmentation module 610, the pallet identification module 620, the pallet positioning module 630, and the coordinate conversion module 640 are sequentially electrically connected.

[0161] ​The tray segmentation module 610 initializes the weight coefficient between the layers of the built segmentation model, and then inputs a set of sample warehouse color images in the training set and the annotation content of the sample warehouse color images into the neural network under the current weight coefficient, and sequentially calculates the output of each node of the input layer, the hidden layer and the output layer. The cumulative error between the final output result of the output layer and the actual connection position state type is corrected according to the gradient descent method. The weight coefficients between the input layer and the hidden layer are obtained by traversing all the samples in the training set according to the above process.

[0162] According to the weight coefficients of the input layer and the hidden layer of the neural network, the segmentation model is restored, and each warehouse color image to be recognized in the test set is input into the trained segmentation model, so that the area of different pixel values is obtained, which represents a specified type of tray, and different areas correspond to different RGB color first tray images.

[0163] The tray recognition module 620 performs dilation and corrosion operations on the first tray image to generate a binary image, and then extracts the tray contour edge. After integrating the tray contour coordinates corresponding to each tray pixel area of the same type into a tray contour coordinate set, each tray contour in the set is further processed to obtain the tray pixel area, tray corner point coordinates and tray center point coordinates corresponding to each tray of the same type.

[0164] The tray positioning module 630 performs threshold comparison related constraints on the tray pixel area, tray center point coordinates and depth distance of the tray center point of each tray in the set, so as to filter out multiple trays whose tray center point coordinates are in the same vertical range, and whose tray pixel area and depth distance meet the field distance requirement. Then, according to the constraint condition that the Y-axis coordinate value of the tray center point is the largest, the uppermost tray is positioned from the multiple front-row and vertically distributed trays as the target tray of the priority operation, and the three-dimensional space coordinates of the center point of the target tray in the camera coordinate system are converted into the center space coordinates.

[0165] The coordinate conversion module 640 projects the space coordinates of the center of the target tray in the camera coordinate system to the warehouse plane coordinate system to obtain the position coordinates of the target tray in the actual geographical space, and sends the position coordinates to the forklift.

[0166] The forklift receives and responds to the control instruction carrying the position coordinates of the target tray in the warehouse plane coordinate system sent by the segmentation and positioning device of the forklift tray, drives the related mechanical parts, so that the forks of the forklift move to the position coordinates in the control instruction, and completes the loading and unloading operation of the target tray.

[0167] Optionally, the segmentation model includes a category segmentation layer and a tray separation layer.

[0168] Correspondingly, the tray segmentation module 610 comprises a category segmentation unit and a tray separation unit, wherein:

[0169] The category segmentation unit is configured to input the warehouse color image into the category segmentation layer for semantic segmentation, and obtain a second tray image output by the category segmentation layer. Each tray pixel region composed of pixel values corresponding to a tray type in the second tray image is colored as a color matching the tray type.

[0170] The tray separation unit is configured to input the second tray image into the tray separation layer for comparison of upper and lower thresholds of color threshold values, and obtain the first tray image output by the tray separation layer.

[0171] Optionally, the tray recognition module 620 comprises a contour extraction unit, a first recognition unit, a second recognition unit, and a third recognition unit, wherein:

[0172] The contour extraction unit is configured to extract a contour edge of the first tray image, and integrate contour coordinates of a tray pixel region of each tray under a target tray type into a set of tray contour coordinates.

[0173] The first recognition unit is configured to envelope contour coordinates of each tray pixel region in the set of tray contour coordinates to obtain a first tray pixel area, a first tray corner point coordinate, and a first tray center point coordinate of each tray.

[0174] The second recognition unit is configured to, in a case where the first tray center point coordinate of a tray located at a middle position of a frame is updated to a left boundary of the frame, the left upper corner point coordinate and the left lower corner point coordinate carried in the first tray corner point coordinate, obtain a second tray pixel area and a second tray center point coordinate of all complete trays in a left boundary field of view.

[0175] The third recognition unit is configured to, in a case where the first tray center point coordinate of a tray located at a middle position of a frame is updated to a right boundary of the frame, the right upper corner point coordinate and the right lower corner point coordinate carried in the first tray corner point coordinate, obtain a third tray pixel area and a third tray center point coordinate of all complete trays in a right boundary field of view.

[0176] Optionally, the tray positioning module 630 comprises a first constraint unit, a second constraint unit, and a target tray screening unit, wherein:

[0177] The first constraint unit is configured to determine candidate pallet center point coordinates in the same vertical warehouse range based on the first pallet center point coordinates being in a first threshold range and the first pallet pixel area being in a second threshold range of the pallet contour coordinates corresponding to the pallet in the set of pallet contour coordinates.

[0178] The second constraint unit is configured to obtain a center spatial coordinate of each candidate pallet in a camera coordinate system after a pallet with a pallet center point depth distance obtained from the warehouse depth image using the candidate pallet center point coordinates being in a third threshold range is determined as a candidate pallet in the front row in the same vertical warehouse range.

[0179] The target pallet screening unit is configured to determine a candidate pallet with a minimum Y coordinate value in the center spatial coordinates as a target pallet in the front row and the uppermost layer in the same vertical warehouse range, and output the center spatial coordinate of the target pallet.

[0180] The first threshold range is determined based on the second pallet center point coordinates and the third pallet center point coordinates. The second threshold range is determined based on the second pallet pixel area and the third pallet pixel area. The third threshold range is determined based on a safe distance maintained by the camera module from the warehouse pallet being photographed.

[0181] Optionally, the coordinate conversion module 640 includes a first conversion unit and a second conversion unit, wherein:

[0182] The first conversion unit is configured to project the spatial coordinate of the center of the target pallet in the camera coordinate system onto the XOZ plane to obtain a two-dimensional coordinate of the target pallet in the main antenna coordinate system.

[0183] The second conversion unit is configured to convert the two-dimensional coordinate of the target pallet in the main antenna coordinate system to a warehouse plane coordinate system to obtain a position coordinate of the target pallet in the warehouse plane coordinate system.

[0184] The origin of the camera coordinate system is located at the optical center of the camera module, the camera optical axis Z is parallel to the ground and points to the front of the vehicle, and the X and Y axes point to the left side of the forklift and the ground, respectively. The origin of the main antenna coordinate system is the center point of the main antenna installed behind the camera module along the longitudinal axis of the vehicle. The Z axis points to the forward direction of the forklift, and the X axis points to the left side of the vehicle. The warehouse plane coordinate system is a Gauss plane rectangular coordinate system with the Beidou base station as the origin and the north direction as the Z axis and the east direction as the X axis.

[0185] Optionally, the device further includes a short side midpoint conversion module and an inclination angle acquisition module, wherein:

[0186] A short side midpoint conversion module is configured to determine a left short side midpoint coordinate and a right short side midpoint coordinate based on the tray corner point coordinates of the target tray.

[0187] An inclination angle acquisition module is configured to determine an inclination angle of the target tray relative to the camera module based on the left short side midpoint coordinate, the right short side midpoint coordinate, and the warehouse depth image.

[0188] The inclination angle of the target tray relative to the camera module is used to convert a working position of a forklift.

[0189] The forklift tray segmentation and positioning device provided by the embodiments of the present application is used to implement the forklift tray segmentation and positioning method provided by the present application, and has the same beneficial effects as the forklift tray segmentation and positioning method provided by the present application. Therefore, details are not described herein.

[0190] The embodiments of the present application segment different types of trays from warehouse color images by using a semantic segmentation network, separate a first tray image belonging to the same type of tray, extract the contour of the first tray image through image processing, obtain the tray pixel area, tray corner point coordinates, and tray center point coordinates of each tray of the same type, and combine the corresponding threshold value and the logic processing based on the principle of first upper and then lower to locate the spatial coordinates of the center of the target tray in the front row and the uppermost layer in the camera coordinate system, and convert the spatial coordinates to the warehouse plane coordinate system. The position coordinates of the target tray in the warehouse plane coordinate system are used to guide the forklift to implement work, so that the target tray can be accurately positioned in a complex scene of different types of tray multi-layer stacking and front and rear parallel arrangement, the accuracy and speed of tray positioning can be considered, and the working efficiency of the forklift is further improved.

[0191] Figure 7 is one of the structural schematic diagrams of the forklift provided by the present application. Figure 8 is one of the structural schematic diagrams of the forklift provided by the present application. Based on any of the above embodiments, as shown in Figure 7 and Figure 8 The intelligent forklift includes a vehicle body 1, a camera module 2 installed in the longitudinal axis direction of the vehicle body, a main antenna 3 installed in the longitudinal axis direction of the vehicle body 1 and behind the camera module 2, and a processor 4 installed in the vehicle body 1. The processor 4 implements the forklift tray segmentation and positioning method as described above when executing the program.

[0192] Specifically, the camera module 2 is installed in the longitudinal axis direction of the vehicle body 1, and the optical center is marked as O, which is the origin of the camera coordinate system. The main antenna 3 is installed 1 meter above the camera module 2 along the longitudinal axis direction of the vehicle body 1, and the center of the main antenna is marked as O1, which is the origin of the main antenna coordinate system. In the actual geographic space of the warehouse, the warehouse plane coordinate system is established with the Beidou base station as the origin, marked as Ow, and the north direction as the Z axis and the east direction as the X axis. Wherein:

[0193] The camera module 2 is used to shoot the tray stacking scene in the cargo hold, and obtain the warehouse color image and the warehouse depth image.

[0194] The processor 4 is used to perform semantic segmentation, separation processing of the same type of tray, definition of tray pixel coordinates, positioning of the target tray, coordinate conversion and the like on the warehouse color image and the warehouse depth image.

[0195] The forklift can be an unmanned intelligent forklift or a manned traditional forklift.

[0196] The embodiment of the application separates the first tray image belonging to the same type of tray by using the semantic segmentation network to segment the different types of trays in the warehouse color image, extracts the contour of the first tray image through image processing, obtains the tray pixel area, tray corner point coordinates and tray center point coordinates corresponding to each tray of the same type, and combines the corresponding threshold value and the logic processing based on the principle of first upper and then lower, to position the spatial coordinates of the center of the target tray in the front row and the uppermost layer in the camera coordinate system, and convert it to the warehouse plane coordinate system, so as to guide the forklift to implement the work by using the position coordinates of the target tray in the warehouse plane coordinate system, realize accurate positioning of the target tray in the complex scene of different types of tray multi-layer stacking and front and rear parallel arrangement, and balance the accuracy and speed of tray positioning, to further improve the work efficiency of the forklift.

[0197] In addition, the logic instructions in the memory can be realized in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0198] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the forklift pallet segmentation and positioning method provided by the above-mentioned methods, the method comprising: inputting a warehouse color image into a segmentation model to obtain a first pallet image output by the segmentation model; wherein the pixel area of a target pallet type in the first pallet image is set to a color and pixel value matching the target pallet type, the target pallet type is any one of the pallet types, and the warehouse color image is color image information collected by a camera module on a forklift at a safe distance from the warehouse pallet being photographed; obtaining the pixel area of each pallet, the corner point coordinates of the pallet and the center point coordinates of the pallet from the pallet contour coordinate set parsed from the first pallet image; performing threshold condition constraints on the pixel area of all pallets, the center point coordinates of the pallet and the depth distance of the center point of the pallet, and performing condition constraints on the center point coordinates of the pallet in the Y axis from top to bottom, to screen out the target pallet from the pallet contour coordinate set and determine the spatial coordinates of the center of the target pallet in the camera coordinate system; wherein the depth distance of the center point of the pallet is determined by indexing the center point coordinates of the pallet in the warehouse depth image, and the warehouse depth image is depth image information collected by the camera module at the same time as the warehouse color image is photographed; performing coordinate system conversion on the spatial coordinates of the center of the target pallet in the camera coordinate system, and issuing the position coordinates of the target pallet in the warehouse plane coordinate system obtained by conversion to the forklift, so that the forklift can perform work on the target pallet according to the position coordinates of the target pallet in the warehouse plane coordinate system; wherein the segmentation model is trained according to a sample warehouse color image, a corresponding labeled pallet type label in the sample warehouse color image, and the area color and area pixel value corresponding to the pallet type label.

[0199] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the forklift pallet segmentation and positioning method provided by the above method, the method comprising: inputting a warehouse color image into a segmentation model to obtain a first pallet image output by the segmentation model; wherein the pallet pixel regions of a target pallet type in the first pallet image are all set to a color and pixel value matching the target pallet type, the target pallet type being any one of pallet types, and the warehouse color image being color image information collected by a camera module on a forklift at a safe distance from the warehouse pallet being photographed; obtaining a pallet pixel area, pallet corner point coordinates and pallet center point coordinates corresponding to each pallet using pallet contour coordinate sets parsed from the first pallet image; filtering out a target pallet from the pallet contour coordinate sets and determining the spatial coordinates of the center of the target pallet in a camera coordinate system by performing threshold condition constraints on the pallet pixel area, pallet center point coordinates and pallet center point depth distance of all pallets, and performing a condition constraint on the center point coordinates of the target pallet in the Y-axis being first upwards and then downwards; wherein the pallet center point depth distance is determined using the pallet center point coordinates indexed in a warehouse depth image, and the warehouse depth image is depth image information collected by the camera module at the same time as the warehouse color image is photographed; and performing coordinate system conversion on the spatial coordinates of the center of the target pallet in the camera coordinate system, and issuing the position coordinates of the target pallet in a warehouse plane coordinate system obtained by the conversion to a forklift, so that the forklift performs work on the target pallet according to the position coordinates of the target pallet in the warehouse plane coordinate system; wherein the segmentation model is trained according to a sample warehouse color image, a corresponding pallet type label annotated in the sample warehouse color image, and region colors and region pixel values corresponding to the pallet type label.

[0200] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0201] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0202] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of splitting and positioning a forklift pallet, characterized by, The warehouse color image is input into a segmentation model to obtain a first pallet image output by the segmentation model; wherein the pallet pixel area of a target pallet type in the first pallet image is set to a color and pixel value matching the target pallet type, the target pallet type is any one of the pallet types, and the warehouse color image is color image information collected by a camera module on a forklift at a safe distance from a warehouse pallet being photographed; A set of pallet contour coordinates is parsed from the first pallet image to obtain a pallet pixel area, a pallet corner point coordinate, and a pallet center point coordinate corresponding to each pallet; The pallet pixel area, the pallet center point coordinate, and the pallet center point depth distance of all pallets are subjected to threshold condition constraints, and the target pallets in the front row and the uppermost layer are screened out from the set of pallet contour coordinates according to the condition constraint that the pallet center point coordinates are in the Y-axis from top to bottom, and the spatial coordinates of the center of the target pallet in the camera coordinate system are determined; wherein the pallet center point depth distance is determined by indexing the pallet center point coordinates in the warehouse depth image, and the warehouse depth image is depth image information collected by the camera module at the same time of photographing the warehouse color image; The spatial coordinates of the center of the target pallet in the camera coordinate system are subjected to coordinate system conversion, and the position coordinates of the target pallet in the warehouse plane coordinate system obtained by the conversion are sent to the forklift, so that the forklift performs work on the target pallet according to the position coordinates of the target pallet in the warehouse plane coordinate system. The segmentation model is trained according to a sample warehouse color image, a corresponding pallet type label annotated in the sample warehouse color image, and a region color and a region pixel value corresponding to the pallet type label. The segmentation model includes a category segmentation layer and a pallet separation layer.

2. The method of splitting and positioning a forklift pallet of claim 1, wherein, The warehouse color image is input into the category segmentation layer for semantic segmentation to obtain a second pallet image output by the category segmentation layer; wherein each pallet pixel area composed of pixel values corresponding to each pallet type in the second pallet image is colored as a color matching the pallet type. The second pallet image is input into the pallet separation layer for comparison of the upper and lower limits of the color threshold to obtain the first pallet image output by the pallet separation layer. The set of pallet contour coordinates parsed from the first pallet image is used to obtain a pallet pixel area, a pallet corner point coordinate, and a pallet center point coordinate corresponding to each pallet, including:

3. The method of splitting and positioning a forklift pallet of claim 1, wherein, The first pallet image is subjected to contour edge extraction, and the contour coordinates of the pallet pixel area of each pallet of the target pallet type are integrated into the set of pallet contour coordinates; The contour coordinates of each pallet pixel area in the set of pallet contour coordinates are enveloped to obtain a first pallet pixel area, a first pallet corner point coordinate, and a first pallet center point coordinate of each pallet. ​ In a case where the upper left corner point coordinate and the lower left corner point coordinate carried in the first tray corner point coordinate of the tray with the first tray center point coordinate in the middle of the frame are updated to the left boundary of the frame, second tray pixel areas and second tray center point coordinates of all complete trays in the left boundary field of view are obtained; In a case where the upper right corner point coordinate and the lower right corner point coordinate carried in the first tray corner point coordinate of the tray with the first tray center point coordinate in the middle of the frame are updated to the right boundary of the frame, third tray pixel areas and third tray center point coordinates of all complete trays in the right boundary field of view are obtained.

4. The method of splitting and positioning a forklift pallet of claim 3, wherein, The threshold condition constraint on the tray pixel area, the tray center point coordinate and the tray center point depth distance of all trays, and the condition constraint that the tray center point coordinate is first upward and then downward in the Y axis, filter out the target tray contour coordinate from the tray contour coordinate set, including: Based on the tray with the first tray center point coordinate in the first threshold range and the first tray pixel area in the second threshold range, the corresponding tray contour coordinate of the tray in the tray contour coordinate set is determined as the candidate tray center point coordinate in the same vertical storage range; After the tray with the tray center point depth distance obtained by the candidate tray center point coordinate in the storage depth image is in the third threshold range is taken as the candidate tray in the front row in the same vertical storage range, the center spatial coordinate of each candidate tray in the camera coordinate system is obtained; The candidate tray with the minimum Y coordinate value in the center spatial coordinate is taken as the target tray in the front row and the uppermost layer in the same vertical storage range, and the center spatial coordinate of the target tray is output; The first threshold range is determined based on the second tray center point coordinate and the third tray center point coordinate; the second threshold range is determined based on the second tray pixel area and the third tray pixel area; and the third threshold range is determined based on the safe distance maintained by the camera module and the storage tray being shot.

5. The method of splitting and positioning a forklift pallet of claim 4, wherein, The space coordinate of the center of the target tray in the camera coordinate system is converted, the position coordinate of the target tray in the storage plane coordinate system is obtained, and the position coordinate of the target tray in the storage plane coordinate system is fed back to the forklift, so that the forklift can implement work on the target tray according to the position coordinate of the target tray in the storage plane coordinate system, including: The space coordinate of the center of the target tray in the camera coordinate system is projected onto the XOZ plane to obtain the two-dimensional coordinate of the target tray in the main antenna coordinate system; The two-dimensional coordinate of the target tray in the main antenna coordinate system is converted to the storage plane coordinate system to obtain the position coordinate of the target tray in the storage plane coordinate system; The space coordinate of the center of the target tray in the camera coordinate system is projected onto the XOZ plane to obtain the two-dimensional coordinate of the target tray in the main antenna coordinate system; The two-dimensional coordinate of the target tray in the main antenna coordinate system is converted to the storage plane coordinate system to obtain the position coordinate of the target tray in the storage plane coordinate system; The origin of the camera coordinate system is located at the optical center of the camera module, the camera optical axis Z is parallel to the ground and points to the front of the vehicle, and the X and Y axes point to the left side of the forklift and the ground, respectively. The origin of the main antenna coordinate system is the center point of the main antenna installed along the longitudinal axis of the vehicle and behind the camera module, with the forward direction of the forklift as the Z axis and the left side of the vehicle as the X axis. The warehouse plane coordinate system is a Gauss plane rectangular coordinate system with the Beidou base station as the origin and the north direction as the Z axis and the east direction as the X axis.

6. The method of splitting and positioning a forklift pallet of claim 4, wherein, After the candidate pallet with the minimum Y coordinate value in the central space coordinate is selected as the target pallet in the front row and the uppermost layer in the same vertical warehouse range, the method further includes: determining left and right short side midpoint coordinates based on the target pallet corner point coordinates; determining the inclination angle of the target pallet relative to the camera module based on the left and right short side midpoint coordinates and the warehouse depth image; wherein the inclination angle of the target pallet relative to the camera module is used to convert the working position of the forklift.

7. A fork truck pallet splitting and positioning apparatus, characterized by, It includes: a pallet segmentation module for inputting a warehouse color image into a segmentation model to obtain a first pallet image output by the segmentation model; wherein the pallet pixel area of a target pallet type in the first pallet image is set to a color and pixel value matching the target pallet type, the target pallet type being any one of the pallet types, and the warehouse color image being color image information collected by a camera module on a forklift at a safe distance from the photographed warehouse pallet; a pallet recognition module for obtaining the pixel area, corner point coordinates and center point coordinates of each pallet corresponding to the pallet contour coordinates set parsed from the first pallet image; a pallet positioning module for threshold condition constraints on the pixel area, center point coordinates and center point depth distance of all pallets, and for condition constraints on the Y-axis of the center point coordinates of the pallets from front to back, screening out a target pallet in the front row and the uppermost layer from the pallet contour coordinate set, and determining the space coordinates of the center of the target pallet in the camera coordinate system; wherein the center point depth distance of the pallet is determined by indexing the center point coordinates of the pallet in the warehouse depth image, and the warehouse depth image is depth image information collected by the camera module while photographing the warehouse color image; a coordinate conversion module for coordinate system conversion of the space coordinates of the center of the target pallet in the camera coordinate system, and issuing the position coordinates of the target pallet in the warehouse plane coordinate system to the forklift for the forklift to implement work on the target pallet according to the position coordinates of the target pallet in the warehouse plane coordinate system. The segmentation model is trained based on sample warehouse color images, corresponding pallet type labels in the sample warehouse color images, and region colors and region pixel values corresponding to the pallet type labels.

8. An intelligent forklift truck comprising a vehicle body, a camera module arranged to be mounted in the direction of the longitudinal axis of the vehicle body, and a main antenna arranged to be mounted in the direction of the longitudinal axis of the vehicle body and behind the camera module. Also included is a processor disposed in the vehicle body, the processor implementing the method of segmenting and positioning a forklift pallet as claimed in any one of claims 1 to 6 when executing a program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method of segmenting and positioning a forklift pallet as claimed in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the method of segmenting and positioning a forklift pallet as claimed in any one of claims 1 to 6.

Images