Method and apparatus for detecting boarding and alighting of an article

Abstract

The application provides a method and device for detecting loading and unloading of goods, wherein the method for detecting loading and unloading of goods comprises the following steps: detecting the weight of goods on a warehouse forklift; reading the labels and the number of labels of the goods on the warehouse forklift; identifying the type and the visual number of the goods on the warehouse forklift; calculating the type and the number of the goods on the warehouse forklift based on the weight of the goods, the labels of the goods, the number of the labels of the goods, the type of the goods and the visual number of the goods; and confirming the current warehouse-in and warehouse-out state of the warehouse forklift, and synchronizing the current warehouse-in and warehouse-out state, the type of the goods and the number of the goods to a warehouse management system, wherein the current warehouse-in and warehouse-out state comprises a warehouse-out state and a warehouse-in state. The application realizes the detection of loading and unloading of goods on the warehouse forklift, and improves the management efficiency of the reference warehouse-in and warehouse-out.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of warehouse management, and particularly relates to a method and device for detecting loading and unloading of goods. BACKGROUND

[0002] In a warehouse management system, warehouse management is set at the loading and unloading, and the goods are registered and managed by a warehouse manager;

[0003] With the development of radio frequency identification technology, the warehouse manager can read the goods tags at the loading and unloading through a reader and directly synchronize to the system to confirm the loading and unloading of the goods, thereby improving the efficiency of warehouse management;

[0004] However, the single loading and unloading management is too heavy for the warehouse manager, and the warehouse manager needs to scan each loading and unloading goods one by one. If the warehouse loading and unloading RFID detection is completely relied on, there is a certain error rate. SUMMARY

[0005] The present application aims at the problem of goods loading and unloading management, and proposes a method and device for detecting loading and unloading of goods.

[0006] The present application proposes a method for detecting loading and unloading of goods, which is set in a warehouse forklift, and the method comprises the following steps:

[0007] detecting the weight of goods on the warehouse forklift;

[0008] reading the goods tags and the number of goods tags on the warehouse forklift;

[0009] identifying the type of goods and the visual quantity of goods on the warehouse forklift;

[0010] calculating the type of goods and the quantity of goods on the warehouse forklift based on the weight of goods, the goods tags, the number of goods tags, the type of goods and the visual quantity of goods on the warehouse forklift; and

[0011] confirming the current loading and unloading state of the warehouse forklift, and synchronizing the current loading and unloading state, the type of goods and the quantity of goods to a warehouse management system, wherein the current loading and unloading state comprises a loading state and an unloading state.

[0012] In some embodiments, reading the goods tags and the number of goods tags on the warehouse forklift comprises:

[0013] respectively using two readers arranged at the diagonal positions of the base of the warehouse forklift to read the goods tags and the number of goods tags.

[0014] In some embodiments, the type and quantity of the items on the warehouse forklift are calculated based on the weight of the items on the warehouse forklift, the item label, the number of item labels, the type of the items, and the visual quantity of the items, including:

[0015] When the reading results of the two readers are consistent, the item label and the number of item labels are used as the type and quantity of the items on the warehouse forklift.

[0016] In some embodiments, the type and quantity of the items on the warehouse forklift are calculated based on the weight of the items on the warehouse forklift, the item label, the number of item labels, the type of the items, and the visual quantity of the items, including:

[0017] When the reading results of the two readers are inconsistent, the structure data is constructed based on the output results of the readers, the weight of the items on the warehouse forklift, the type of the items, and the visual quantity of the items;

[0018] The structure data is input into a pre-trained warehouse identification model, and the pre-trained warehouse identification model outputs the type and quantity of the items on the warehouse forklift;

[0019] The type and visual quantity of the items on the warehouse forklift are identified, including starting to collect images when the weight sensor detects that there is weight applied to the warehouse forklift, and stopping collecting images when the weight sensor detects that the weight applied to the warehouse forklift has not changed;

[0020] The type and visual quantity of the items on the warehouse forklift are identified based on the collected multiple images.

[0021] In some embodiments, the warehouse identification model includes:

[0022] A first feature extraction network for obtaining a difference information feature vector of the reading results of the readers;

[0023] A second feature extraction network for obtaining an item image feature vector;

[0024] An output network inputting the difference information feature vector, the item image feature vector, and the weight of the items, and outputting the type and quantity of the items on the warehouse forklift.

[0025] In some embodiments, the first feature extraction network includes:

[0026] An imaging module for encoding the output information of the reader label into a difference information feature image;

[0027] An image feature extraction network for extracting a difference information feature vector of the difference information feature image.

[0028] In some embodiments, the output network comprises:

[0029] a flattening layer configured to flatten the difference information feature vector to the same length or width as the item image feature vector;

[0030] a concatenation layer configured to concatenate the difference information feature vector and the item image feature vector in the length direction or the width direction into a concatenated feature vector;

[0031] a multi-layer fully connected layer configured to fuse the concatenated feature vector;

[0032] an output layer configured to output the type and quantity of the items on the warehouse forklift according to the concatenated feature vector and the item weight.

[0033] In some embodiments, a feature secondary extraction layer is further arranged between the concatenation layer and the multi-layer fully connected layer, and is configured to abstract the concatenated feature vector.

[0034] In some embodiments, the input of the first feature extraction network comprises the item weight, so that the first feature extraction network calculates the difference information feature vector with reference to the item weight; and / or,

[0035] the input of the second feature extraction network comprises the item weight, so that the second feature extraction network calculates the item image feature vector with reference to the item weight.

[0036] The present application also proposes a device for detecting the loading and unloading of items, which comprises:

[0037] a weight detection device configured to detect the weight of the items on the warehouse forklift;

[0038] an RFID reader configured to read the item labels and the quantity of the item labels on the warehouse forklift

[0039] an image recognition device configured to collect the images of the items on the warehouse forklift;

[0040] one or more processors configured to calculate the type and quantity of the items on the warehouse forklift based on the weight of the items on the warehouse forklift, the item labels, the quantity of the item labels, the type of the items and the visual quantity of the items; and confirm the current in-out warehouse state of the warehouse forklift, and synchronize the current in-out warehouse state, the type of the items and the quantity of the items to the warehouse management system, wherein the current in-out warehouse state comprises an out-of-warehouse state and an in-warehouse state.

[0041] The application embodiment improves the reliability of the RFID reading result by using the weight of the goods on the warehouse forklift to assist verification when the reader reads the goods label and the goods quantity on the forklift, and when the RFID reading result is inconsistent, uses the image collected by the image sensor to identify the type of goods on the warehouse forklift and the visual quantity of the goods to provide a reference information, and then uses the RFID reading result and the identification result of the image collected by the image sensor to comprehensively confirm the type of goods on the warehouse forklift and the quantity of the goods; finally, the type of goods on the warehouse forklift and the quantity of the goods, and the current in-out warehouse state of the warehouse forklift, are sent to the warehouse management system for secondary confirmation, thereby improving the reliability of the warehouse management system. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A flowchart of a method for detecting the loading and unloading of goods according to an embodiment of the application;

[0043] Figure 2 A warehouse in-out model block diagram of a method for detecting the loading and unloading of goods according to an embodiment of the application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0045] In the description of the application, the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0046] In the description of the present application, the term "for example" is used to mean "serving as an example, instance, or illustration." Any embodiment described as "for example" in the present application is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the present application. In the following description, for purposes of explanation, specific details are set forth. It is apparent to those skilled in the art that the present application can be practiced without the specific details presented. In other instances, well-known structures and processes are not shown in detail to avoid obscuring the present application. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features presented.

[0047] Example 1

[0048] The present application provides a method for detecting the loading and unloading of goods, which is arranged on a warehouse forklift. In the present application, the warehouse forklift is provided with a weighing sensor for detecting the weight of goods, a reader for reading RFID tags, a camera for collecting images of goods on the warehouse forklift, a communication module in communication connection with a warehouse management system, and a processor.

[0049] The method can confirm the number and type of goods on the forklift, and repeatedly confirm the number and type of goods in combination with the warehouse management system. It should be noted that in the warehouse management system of the present application, there is a detection of warehouse entry and exit, and in combination with the detection of goods on the warehouse forklift, repeated confirmation is achieved.

[0050] Reference Figure 1 In some embodiments, the method comprises:

[0051] S100, detecting the weight of goods on the warehouse forklift;

[0052] Exemplarily, a weighing sensor can be arranged at the bottom of the warehouse forklift to detect the weight of goods stacked on the warehouse forklift. It should be noted that the warehouse management system has pre-recorded the types and weights of various goods, so after the weighing sensor collects the weight of the goods, the processor can calculate the number of goods according to the weight of the goods in combination with the type of the goods, or can infer the type of the goods according to the weight of the goods in combination with the number of the goods, so the weight is an important reference information for the loading and unloading detection of goods in the present application.

[0053] S200, reading the labels of goods on the warehouse forklift and the number of labels;

[0054] Exemplarily, one or more readers can be arranged on the warehouse forklift to read the RFID tags of the articles on the forklift, and then the quantity and type of the articles are obtained. It should be noted that during the reading of the tags by the readers, problems such as tag overlapping, metal shielding, and mutual crosstalk can occur, resulting in inaccurate reading results of the readers.

[0055] In the present application, a plurality of readers can be arranged, and the reading results of the plurality of readers are combined to confirm the quantity and type of the articles. When the plurality of detection results are consistent, it is considered that the reading results of the readers are correct. Further, when the plurality of detection results are consistent, the weight of each type of article is multiplied by the quantity based on the detection results and the type and quantity information to obtain the article weight of the forklift, and the article weight collected by the weighing sensor is combined to further verify the correctness of the detection results.

[0056] Exemplarily, if the reading results of the plurality of readers are inconsistent, the article type and the article quantity on the warehouse forklift need to be confirmed in combination with the article weight and the image information collected in step S300, and the confirmation result is output to the warehouse management system and the warehouse management personnel are reminded to compare and confirm the detection results. In this way, the warehouse management personnel do not need to manually input the detection results, but only need to detect the results on the forklift, and when it is confirmed that there is no problem, they can directly click to confirm, thereby improving the warehouse management efficiency.

[0057] S300, identifying the article type and the article visual quantity on the warehouse forklift;

[0058] Exemplarily, an image sensor can be arranged at the push handle position of the forklift. When the weight sensor detects that a weight is applied to the warehouse forklift, the image collection is started, and the image collection is stopped when the weight sensor detects that the weight applied to the warehouse forklift has not changed. In this way, the collected images are all images during the stacking process, reducing the redundancy of the images and greatly reducing the computational power consumption of the processor.

[0059] In the present application, if the collection results of the RFID reader and / or the weighing sensor are consistent, the detection result can be directly output without using the image data collected by the image sensor. However, if the collection results of the RFID reader and / or the weighing sensor are inconsistent, the image data collected by the image sensor is used to assist in confirming the article quantity and type on the forklift. It should be noted that during the loading and unloading of the articles, the RFID reads the RFID tags in real time, and the error rate is very low. Therefore, the frequency of using the image sensor to collect data is low. However, the present application is configured to use the image data collected by the image sensor to assist in confirming the article quantity and type on the forklift only when the collection results of the RFID reader and / or the weighing sensor are inconsistent. This reduces the amount of image processing and greatly reduces the computational power consumption of the processor.

[0060] Further, a processor with poor computing power can be used to record the timestamp when the collection results of the RFID reader and / or the weighing sensor are inconsistent during the stacking process, then extract the corresponding video data collected by the image sensor, identify the number and type of the articles, and perform auxiliary verification. This process can continue until the end of the stacking, prompting the stacker to compare and confirm the detection results, and confirming that there is no problem, directly click to confirm, which improves the warehouse management efficiency.

[0061] The demand for embedding image processing algorithms with low computing power into warehouse forklifts is met.

[0062] S400, based on the weight of the articles on the warehouse forklift, the article label, the number of article labels, the type of the articles and the visual quantity of the articles, the type of the articles and the number of the articles on the warehouse forklift are calculated; and

[0063] Exemplarily, this step distinguishes between the case where the collection results of the RFID reader and / or the weighing sensor are consistent, and the case where the collection results of the RFID reader and / or the weighing sensor are inconsistent.

[0064] When the collection results are consistent, a large amount of computing power is not needed, and the reading results of the RFID can be directly used as the final results. When the collection results are inconsistent, the timestamp when the collection results of the RFID reader and / or the weighing sensor are inconsistent is recorded, then the corresponding video data collected by the image sensor is extracted, the number and type of the articles are identified, and auxiliary verification is performed.

[0065] S500, confirming the current warehouse in-out state of the warehouse forklift, and synchronizing the current warehouse in-out state, the type of the articles and the number of the articles to the warehouse management system, the current warehouse in-out state including: the out-of-warehouse state and the in-warehouse state.

[0066] Exemplarily, the warehouse forklift and the host of the warehouse management system establish a communication connection, and synchronize the type of the articles and the number of the articles to the warehouse management system. The warehouse management system itself already knows the in-out state of the batch of articles. The current in-out state, the type of the articles and the number of the articles are synchronized to the warehouse management system, which is repeated to ensure the reliability of the in-out detection of the articles.

[0067] The embodiments of the present application improve the reliability of the RFID reading result by using the weight of the goods on the warehouse forklift to assist verification when the reader reads the goods label and the goods quantity on the forklift, and when the RFID reading result is inconsistent, using the image collected by the image sensor to identify the type of goods on the warehouse forklift and the visual quantity of the goods to provide a reference information, and then using the reading result of the RFID and the identification result of the image collected by the image sensor to comprehensively confirm the type of goods on the warehouse forklift and the quantity of the goods; finally, the type of goods on the warehouse forklift and the quantity of the goods, and the current in-out warehouse state of the warehouse forklift, are sent to the warehouse management system for secondary confirmation, thereby improving the reliability of the warehouse management system.

[0068] In some embodiments, reading the goods label and the goods label quantity on the warehouse forklift comprises: respectively using two readers arranged at the diagonal positions of the base of the warehouse forklift to read the goods label and the goods label quantity. The two readers arranged at the diagonal positions of the base of the warehouse forklift are far apart in position and can well cover the position of the goods on the forklift, reducing the shielding and improving the reading accuracy of the readers.

[0069] In some embodiments, calculating the type of goods and the quantity of goods on the warehouse forklift based on the weight of the goods, the goods label, the goods label quantity, the type of goods and the visual quantity of the goods on the warehouse forklift comprises:

[0070] When the reading results of the two readers are consistent, the goods label and the goods label quantity are used as the type of goods and the quantity of goods on the warehouse forklift.

[0071] When the reading results of the two readers are consistent, it can be considered that the reading result is correct.

[0072] In other embodiments, when the reading results of the two readers are consistent, the weight of the goods can also be combined to assist verification of whether the reading result of the two readers is reliable, that is, the product of the total amount and the quantity of the goods using the reading result of the reader is used to obtain the theoretical weight of the goods on the forklift, and when the theoretical weight of the goods is consistent with the weight collected by the weight sensor, it is considered that the reading result of the reader is reliable.

[0073] In some embodiments, calculating the type of goods and the quantity of goods on the warehouse forklift based on the weight of the goods, the goods label, the goods label quantity, the type of goods and the visual quantity of the goods on the warehouse forklift comprises:

[0074] When the reading results of the two readers are inconsistent, the output results of the readers, the weight of the goods on the warehouse forklift, the type of goods and the visual quantity of the goods are used to construct structure data;

[0075] The structural data is input into a pre-trained warehouse-in and warehouse-out identification model, and the pre-trained warehouse-in and warehouse-out identification model outputs the type and quantity of the goods on the warehouse forklift;

[0076] In the present application, the pre-trained warehouse-in and warehouse-out identification model can use a regression model, obtain a warehouse-in and warehouse-out identification model based on training data and data labeling, and then transplant the warehouse-in and warehouse-out model to the warehouse forklift after lightening the warehouse-in and warehouse-out model.

[0077] In the present application, image recognition can use a single image, that is, an image after the stacking is completed, but the error is large. Image recognition can also use full-process video data, but at this time, the input of the warehouse-in and warehouse-out model has many redundant images, resulting in large calculation overhead and interference with the recognition result. In order to control the cost, the processor of the warehouse forklift has low computing power and is difficult to bear a large amount of calculation.

[0078] Further, the type and visual quantity of the goods on the warehouse forklift are identified, including starting to collect images when the weight sensor detects that a weight is applied to the warehouse forklift, and stopping collecting images when the weight sensor detects that the weight applied to the warehouse forklift does not change.

[0079] The type and visual quantity of the goods on the warehouse forklift are identified based on the collected multiple images.

[0080] Reference Figure 2 In some embodiments, the warehouse-in and warehouse-out identification model includes:

[0081] A first feature extraction network is configured to obtain a difference information feature vector of the reading result of the reader;

[0082] A second feature extraction network is configured to obtain an image feature vector of the goods;

[0083] An output network is configured to input the difference information feature vector, the image feature vector of the goods, and the weight of the goods, and the output of the output network is the type and quantity of the goods on the warehouse forklift.

[0084] In the embodiments of the present application, the first feature extraction network can be a recurrent neural network, such as an RNN, an LSTM, or a GRU network. In order to improve the recognition efficiency, the first feature extraction network of the present application needs to encode the output information of the reader label into a difference information feature image, then extract the image feature vector, and finally splice the feature vectors corresponding to the image collected by the image sensor to realize the same dimension or type of the feature vectors, thereby improving the recognition efficiency. Specifically, in some embodiments, the first feature extraction network includes:

[0085] an imaging module configured to encode the output information of the reader tag into a difference information feature image;

[0086] an image feature extraction network configured to extract a difference information feature vector from the difference information feature image.

[0087] The imaging module can use Gram angle field transformation or Markov transformation for image encoding, or can obtain a time-frequency transformation of the output information of the reader tag, such as wavelet transformation or short-time Fourier transformation, and then convert it into a heat map.

[0088] The image feature extraction network is preferably a convolutional neural network, which can efficiently extract image feature information.

[0089] In some embodiments, the output network comprises:

[0090] a flattening layer configured to flatten the difference information feature vector to the same length or width as the item image feature vector;

[0091] a concatenation layer configured to concatenate the difference information feature vector and the item image feature vector into a concatenated feature vector in the length direction or the width direction;

[0092] a feature secondary extraction layer configured to abstract the concatenated feature vector;

[0093] a multi-layer fully connected layer configured to fuse the concatenated feature vector;

[0094] an output layer configured to output the type and quantity of the item on the warehouse forklift according to the concatenated feature vector and the item weight.

[0095] In a convolutional neural network (CNN) architecture, the flattening layer (Flatten Layer) serves as a connection between the convolution / pooling layer and the fully connected layer. In this application, the flattening layer can serve the purpose of adjusting the feature vector, flattening the difference information feature vector to the same length or width as the item image feature vector, so as to perform image splicing. Therefore, the flattening here is a modification of the dimension of the feature vector to meet the splicing requirement.

[0096] In the subsequent process, the concatenated feature vector is flattened and sent to the multi-layer fully connected layer, and finally the output layer is used to output the result.

[0097] In some embodiments, the input of the first feature extraction network comprises the item weight, so that the first feature extraction network calculates the difference information feature vector with reference to the item weight.

[0098] It should be noted that the feature extraction network is only needed to be trained according to the labeled information and the input information, that is, the high-value input information can improve the effect of the feature extraction network. In the present application, the weight of the article is taken as the reference information, as one of the inputs of the feature extraction network, so as to improve the effect of feature extraction.

[0099] Similarly, the input of the second feature extraction network includes the weight of the article, so that the second feature extraction network can refer to the weight of the article to calculate the article image feature vector.

[0100] Specifically, the feature network can have two input ends to input the reading result of the reader or the image collected by the image sensor, and the weight of the article.

[0101] In other embodiments, the weight of the article can be input into one of the intermediate layers of the feature extraction network.

[0102] Example 2

[0103] The present application also provides a device for detecting the loading and unloading of articles, which is exemplarily a warehouse forklift or an upgrade assembly for upgrading the warehouse forklift. The device comprises:

[0104] a weight detection device for detecting the weight of the article on the warehouse forklift;

[0105] an RFID reader for reading the article label and the number of article labels on the warehouse forklift

[0106] an image recognition device for collecting the image of the article on the warehouse forklift;

[0107] one or more processors for calculating the type and number of articles on the warehouse forklift based on the weight, label, number of labels, type and visual quantity of the articles on the warehouse forklift; and confirming the current warehouse-in and warehouse-out state of the warehouse forklift, and synchronizing the current warehouse-in and warehouse-out state, the type and the number of articles to the warehouse management system, wherein the current warehouse-in and warehouse-out state includes the warehouse-out state and the warehouse-in state.

[0108] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0109] Those skilled in the art will appreciate that embodiments of the present application can be devised for a variety of applications. It is intended that the present application be limited only by the scope of the appended claims, and it is intended that various modifications and alterations made by those skilled in the art be considered as within the scope of the present application. The embodiments of the present application will be described with reference to the attached drawings identified below.

[0110] The present application is described in reference to the drawings using a flowchart and / or a block diagram of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0111] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0113] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such modifications and variations as fall within the scope of the present application.

[0114] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for detecting the loading and unloading of an article, provided in a warehouse forklift, characterized by, The method comprises: detecting the weight of the goods on the warehouse forklift; reading the goods label and the number of goods labels on the warehouse forklift; identifying the type of goods and the visual quantity of goods on the warehouse forklift; calculating the type and quantity of goods on the warehouse forklift based on the weight of the goods, the goods label, the number of goods labels, the type of goods and the visual quantity of goods on the warehouse forklift; and confirming the current warehouse in-out state of the warehouse forklift, and synchronizing the current warehouse in-out state, the type of goods and the quantity of goods to the warehouse management system, wherein the current warehouse in-out state includes the out-of-warehouse state and the in-warehouse state. The method comprises: respectively using two readers arranged at the diagonal positions of the base of the warehouse forklift to read the goods label and the number of goods labels; based on the weight of the goods, the goods label, the number of goods labels, the type of goods and the visual quantity of goods on the warehouse forklift, constructing structure data based on the output results of the two readers, the weight of the goods, the type of goods and the visual quantity of goods on the warehouse forklift when the reading results of the two readers are inconsistent; feeding the structure data into a pre-trained warehouse in-out identification model, wherein the pre-trained warehouse in-out identification model outputs the type and quantity of goods on the warehouse forklift; the warehouse in-out identification model comprises a first feature extraction network for obtaining a difference information feature vector of the reading results of the readers, a second feature extraction network for obtaining a goods image feature vector, and an output network inputting the difference information feature vector, the goods image feature vector and the weight of the goods, wherein the output of the output network is the type and quantity of goods on the warehouse forklift; the output network comprises a flattening layer for flattening the difference information feature vector to the same length or width as the goods image feature vector, a concatenation layer for concatenating the difference information feature vector and the goods image feature vector into a concatenated feature vector from the length direction or the width direction, a multi-layer fully connected layer for fusing the concatenated feature vector, and an output layer for outputting the type and quantity of goods on the warehouse forklift according to the concatenated feature vector and the weight of the goods; identifying the type of goods and the visual quantity of goods on the warehouse forklift comprises starting to collect images when the weight sensor detects that the weight applied to the warehouse forklift, and stopping collecting images when the weight sensor detects that the weight applied to the warehouse forklift has not changed; identifying the type of goods and the visual quantity of goods on the warehouse forklift based on the collected multiple images. based on the weight of the goods, the goods label, the number of goods labels, the type of goods and the visual quantity of goods on the warehouse forklift, calculating the type and quantity of goods on the warehouse forklift comprises: when the reading results of the two readers are consistent, using the goods label and the number of goods labels as the type and quantity of goods on the warehouse forklift.

2. The method of claim 1, wherein The first feature extraction network comprises: an imaging module for encoding the output information of the reader label into a difference information feature image; 3. The method of claim 1, wherein the detecting the boarding and the alighting of the article comprises: detecting the boarding and the alighting of the article based on the information of the article. ​ ​ The image feature extraction network is configured to extract a difference information feature vector of a difference information feature image.

4. The method of claim 1, wherein the method further comprises: determining the weight of the object. Between the concatenation layer and the multi-layer full connection layer, a feature secondary extraction layer is further arranged, configured to abstract the concatenation feature vector.

5. The method of claim 1, wherein the method further comprises: determining the weight of the object. The input of the first feature extraction network comprises the weight of the object, so that the first feature extraction network calculates the difference information feature vector by referring to the weight of the object; and / or the input of the second feature extraction network comprises the weight of the object, so that the second feature extraction network calculates the object image feature vector by referring to the weight of the object.

Images