A multi-modal telescopic forklift attachment identification method and system

Abstract

The present invention relates to a multi-modal telescopic forklift attachment identification method and system, wherein an electronic tag is provided on the attachment, a first identification result is obtained by identifying the electronic tag, an attachment image is acquired, a second identification result is obtained by visually identifying the attachment image, and the attachment of the telescopic forklift is identified according to the first identification result and the second identification result. The present invention not only enables the telescopic forklift to cope with more complex and changeable working environments, but also ensures the reliability of attachment identification.

Description

Technical Field

[0001] The present invention relates to the technical field of telescopic arm forklifts, and more particularly to a multi-modal telescopic arm forklift attachment identification method and system. Background Art

[0002] In the field of industrial automation, telescopic forklifts have become an ideal choice for many complex work scenarios due to their wide arm reach and strong adaptability. However, different attachments have significant differences in performance parameters such as maximum load and arm reach when handling specific scenarios. For telescopic forklifts or intelligent forklifts (such as unmanned forklifts) that require remote control, it is particularly important to accurately and automatically identify the attachments in order to match the appropriate load map.

[0003] At present, the identification of accessories is based on RFID identification technology, which uses radio waves to communicate with the RFID tag on the object through the RFID reader, so as to realize the automatic identification of accessories, such as patent CN110298424A. RFID technology has the advantages of non-contact, fast reading, and strong anti-interference ability, but it has the following limitations when used for accessory identification, including: the effective identification distance is usually short, which may cause inaccurate or failed identification in long distances or complex spatial structures, and the RFID tag is easily damaged during operation, which may cause identification failure, thereby affecting the reliability of the entire system.

[0004] In view of this, in order to overcome the limitations of a single identification method, the present application provides a multi-modal telescopic arm forklift attachment identification method. Summary of the invention

[0005] In view of this, in order to overcome the defects of RFID identification technology, the present invention provides a multimodal telescopic arm forklift attachment identification method and system, which aims to improve the robustness and fault tolerance of attachment identification based on RFID identification technology and combined with the attachment visual recognition method.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] On the one hand, the present application discloses a multi-modal telescopic forklift attachment identification method, comprising:

[0008] The attachment is provided with an electronic tag, and a first identification result is obtained by identifying the electronic tag;

[0009] Acquire the attachment image, and obtain a second recognition result by visually recognizing the attachment image;

[0010] The attachment of the telescopic forklift is identified according to the first identification result and the second identification result.

[0011] Preferably, different accessories are matched with different codes.

[0012] When writing the code into the electronic tag,

[0013] Convert the binary code into hexadecimal, and the converted hexadecimal data is written into the EPC area of ​​the tag;

[0014] When identifying an electronic tag,

[0015] Extract key information starting from the fifth byte of the hexadecimal data.

[0016] And according to the key information, in combination with the preset mapping relationship table, a first recognition result is obtained;

[0017] The mapping relationship table represents the corresponding relationship between the electronic tags and the accessory categories.

[0018] Preferably, the step of visually identifying the attachment image includes:

[0019] The tool image is converted to the target scale through the scale calculation function.

[0020] The feature of the attachment image is extracted through a convolutional neural network to obtain a feature map;

[0021] Extract conditional information related to the target scale in the feature map;

[0022] Based on feature graph and condition information Generate adaptive attention weights;

[0023] After performing element-wise multiplication of the adaptive attention weight and the feature map, the adaptive attention weight and the feature map are concatenated to obtain the feature to be detected;

[0024] The attachment is identified according to the feature to be detected to obtain a second identification result.

[0025] Preferably, the attachment image is converted to the target scale by the following formula:

[0026]

[0027] In the formula, Indicates the attachment image. represents the scale calculation function, and Respectively represent the height and width of the target image, and Respectively represent the height and width of the attachment image.

[0028] Preferably, the method for extracting conditional information related to the target scale in the feature map is:

[0029]

[0030] In the formula, Represents the conditional information corresponding to the feature map of the i-th layer, represents the feature map of the i-th layer, Represents the feature map Perform global average pooling, Represents a splicing operation, and Represent the weight matrices of the two fully connected layers, and Respectively represent the bias vectors of the corresponding fully connected layers, Represents the Sigmoid activation function.

[0031] Preferably, the method for generating adaptive attention weights based on feature maps and conditional information is:

[0032]

[0033] In the formula, represents the adaptive attention weight, Indicates conditional information The weight matrix of Representation feature map The weight matrix of and Respectively represent condition information and feature map The bias vector of Represents the weight matrix corresponding to the fully connected layer.

[0034] Preferably, identifying the attachment of the telescopic forklift according to the first recognition result and the second recognition result includes:

[0035] When the attachment is in the retracted state, the first recognition result is used as the final recognition result;

[0036] When the extension distance of the attachment exceeds the identification distance of the electronic tag, the second identification result is used as the final identification result;

[0037] When the attachment is in the process of extension and retraction and the distance does not exceed the electronic tag recognition distance, if the first recognition result is the same as the second recognition result, the current recognition result is used as the final recognition result; if the first recognition result is different from the second recognition result, the sliding window smoothing mechanism is triggered to determine the final recognition result.

[0038] Preferably, the sliding window smoothing mechanism is triggered to determine the final recognition result, and the steps include:

[0039] The first recognition result and the second recognition result are stored in frames,

[0040] Triggering a sliding window, wherein the sliding window starts from the first frame and ends at the stored current frame;

[0041] The recognition results in the sliding window are counted, and the recognition result with the largest number is taken as the final recognition result.

[0042] Preferably, the sliding window smoothing mechanism is triggered to determine the final recognition result, and the steps include:

[0043] The first recognition result and the second recognition result are stored in frames,

[0044] Trigger a sliding window, the length of which is N frames, and keep track of the current stored frame;

[0045] The recognition results in the sliding window are counted, and the recognition result with the largest number is taken as the final recognition result.

[0046] On the other hand, the present invention discloses a multi-modal telescopic forklift attachment identification system, in which an electronic tag is provided on the attachment of the telescopic forklift, a reader and a camera are installed on the fixed arm, a control terminal is installed in the cockpit, and the reader and the camera communicate with the control terminal respectively;

[0047] The reader is used to read the electronic tag information and send the electronic tag information to the control terminal;

[0048] A camera, used for acquiring an image of an attachment and sending the image of the attachment to a control terminal;

[0049] The control terminal is used to identify the electronic tag information to obtain a first identification result, to visually identify the attachment image to obtain a second identification result, and to identify the attachment of the telescopic arm forklift according to the first identification result and the second identification result.

[0050] It can be seen from the above technical solutions that the present invention discloses a multimodal telescopic arm forklift attachment identification method and system, which, compared with the prior art, effectively solves the problem of short RFID recognition distance and the problem of insufficient reliability due to visual susceptibility to environmental influences, and realizes more efficient and accurate automatic identification of attachments.

[0051] The multimodal recognition result fusion method disclosed in the present invention can not only enable telescopic arm forklifts to cope with more complex and changeable working environments, but also ensure the reliability of accessory identification, thereby effectively avoiding safety accidents caused by accessory identification errors, especially forklift rollover accidents that may be caused by failure to timely replace the corresponding load map of the accessories. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0053] Figure 1 A flow chart of the multi-modal telescopic forklift attachment identification method provided by the present invention;

[0054] Figure 2 This is a structural diagram of the multi-modal telescopic arm forklift attachment identification system provided by the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Embodiment 1

[0057] The embodiment of the present invention discloses a multi-modal telescopic arm forklift attachment identification method. In order to overcome the defects of the single RFID identification technology, the present application takes into account the visual recognition technology and integrates it into the attachment identification process.

[0058] It should be noted that under poor lighting conditions or complex environments, the accuracy of visual recognition may be affected. Moreover, visual recognition has high requirements for computing resources and real-time processing capabilities may be limited. Therefore, this application integrates the label recognition method with the visual recognition method, so that the two can support and cooperate with each other to obtain more accurate recognition results.

[0059] Specifically, the multi-modal telescopic arm forklift attachment identification method of the present application is as follows: Figure 1 , the steps include:

[0060] An electronic tag is provided on the attachment, and a first identification result is obtained by identifying the electronic tag;

[0061] Acquire the attachment image, and obtain a second recognition result by visually recognizing the attachment image;

[0062] The attachment of the telescopic forklift is identified according to the first identification result and the second identification result.

[0063] In an exemplary embodiment,

[0064] The first step is to formulate different codes for different types of data, write the codes into electronic tags, and fix the electronic tags on forklift accessories to ensure that subsequent identification results are not repeated. The types of accessories described in the present invention include forks, hoppers, manned platforms, etc.

[0065] This embodiment uses EPCC1G2 tag memory. EPCC1G2 (EPC Class 1 Generation 2) is a specification of a radio frequency identification tag that supports the ISO 18000-6C protocol. Since the ISO 18000-6C protocol stipulates that hexadecimal is used to store information in the EPC area of ​​the tag, the binary code needs to be converted into hexadecimal and then written into the electronic tag.

[0066] Further, the electronic tag is preliminarily identified by using an RFID reader to obtain a binary code; in this embodiment, the RFID reader uses a UHF ultra-high frequency long-distance reader, which supports electronic tags that comply with ISO18000-6B and EPC CLASS1 G2 (ISO18000-6C) standards. The operating frequency is FCC 902-928MHz or CE 865-868MHz, the output power reaches 30dBm, and is equipped with a 12dbi gain antenna.

[0067] The RFID reader reads the current accessory tag information in real time, specifically extracting key information from the fifth byte of the hexadecimal data to obtain a string of binary codes. Further, the corresponding category is retrieved according to the binary code and the predefined mapping table to obtain the RFID accessory result C. RFID , where the mapping relationship table represents the correspondence between electronic tags and accessory categories.

[0068] Step 2: Visually recognize the attachment image to obtain a second recognition result;

[0069] Although models such as convolutional neural networks (CNN) have made significant breakthroughs in the field of image recognition in recent years, in the field of telescopic forklifts, since the arm span of a telescopic forklift can be extended to more than 20 meters at most, the attachment targets are too small when extended to a long distance, and too large when retracted to a close distance. This problem poses a great challenge to visual recognition.

[0070] Therefore, this application improves the adaptive attention mechanism for visual recognition; specifically, it includes:

[0071] 1) Convert the attachment image to the target scale through the scale calculation function. Assuming that the currently acquired image is I, use the scale calculation function Process I to obtain the target scale information S;

[0072]

[0073] In the formula, Indicates the attachment image. represents the scale calculation function, and Respectively represent the height and width of the target image, and Respectively represent the height and width of the attachment image.

[0074] 2) Use the convolutional neural network (CNN) to extract features from the tool image I and obtain the i-th layer feature map ; The public notice is:

[0075]

[0076] in It is the convolution operation of the i-th layer, including convolution layer, activation layer and pooling layer.

[0077] 3) Extract feature maps Conditional information related to the target scale;

[0078] Feature Map The size is height H×width W×number of channels C, and the formula of the extraction process is expressed as:

[0079]

[0080] In the formula, Represents the conditional information corresponding to the feature map of the i-th layer, represents the feature map of the i-th layer, Represents the feature map Perform global average pooling to obtain a 1×1×C vector, which represents the global features of the entire feature map. represents the concatenation operation, which concatenates the global average pooling result with the scale information S to form a vector containing the global information and scale information of the feature map. and Represent the weight matrices of the two fully connected layers, respectively, mapping the concatenated vector to the intermediate dimension and the final conditional information dimension, and Respectively represent the bias vectors of the corresponding fully connected layers, ReLU represents the ReLU activation function, which is used to introduce nonlinear characteristics. Represents the Sigmoid activation function, which maps the output value to between 0 and 1 as conditional information.

[0081] 4) Based on feature map and condition information Generate adaptive attention weights to adjust the feature area of ​​interest according to the size of the target, so that when the target is far away (small target), the attention to the features of the small target can be enhanced, and when the target is close (large target), the attention to the features of the large target can be enhanced;

[0082] Among them, the method for generating adaptive attention weights is:

[0083]

[0084] In the formula, represents the adaptive attention weight, and Represent conditional information and feature maps respectively The weight matrix is ​​used to project the conditional information and feature map into the same space or adapt the size. and Respectively represent condition information and feature map The bias vector of represents the weight matrix corresponding to the fully connected layer, It is a Softmax activation function, which is used to normalize the results so that the generated attention weights are between 0 and 1, and the sum is 1 to ensure the interpretability and applicability of the weights.

[0085] 5) After performing element-wise multiplication of the adaptive attention weight and the feature map, the adaptive attention weight is concatenated with the feature map to obtain the feature to be detected; that is,

[0086]

[0087] in represents the element-by-element multiplication operation. Then all adaptive feature maps Fusion is performed to obtain the final features to be detected .

[0088] 6) Identify the attachment according to the features to be detected to obtain the second identification result C object .

[0089] The present invention can significantly improve the accuracy and robustness of target detection by dynamically adjusting attention distribution.

[0090] To further optimize the above technical solution, the recognition method needs to be pre-trained before use;

[0091] In this step, various images of different accessories installed on the telescopic forklift are first collected. The accessory data needs to take into account the shape, size, color and other characteristics of different accessories, as well as possible factors such as occlusion, lighting, and weather changes. Each image is annotated to obtain the accessory dataset.

[0092] During training, the data was preprocessed using automatic orientation and resized to 640×640, and data augmentation used cropping in the range of 0% to 20% and brightness adjustment in the range of -15% to +15%.

[0093] The third step is to identify the attachment of the telescopic forklift according to the first recognition result and the second recognition result, including:

[0094] When the attachment is in the retracted state, the first recognition result is used as the final recognition result;

[0095] When the extension distance of the attachment exceeds the identification distance of the electronic tag, the second identification result is used as the final identification result;

[0096] When the attachment is in the process of extension and retraction and the distance does not exceed the electronic tag recognition distance, if the first recognition result is the same as the second recognition result, the current recognition result is used as the final recognition result; if the first recognition result is different from the second recognition result, the sliding window smoothing mechanism is triggered to determine the final recognition result.

[0097] In this step, when the telescopic arm of the telescopic forklift is in the retracted state for replacing the attachment and the line of sight is limited, only the RFID reading is used to obtain the final result C final =C RFID When the telescopic forklift attachment needs to be extended and replaced and exceeds the RFID reading distance, only the image recognition result is used to obtain the final result C final =C object .

[0098] When both recognition results exist, when C RFID = C object When there is C final =C RFID =C object , when C RFID ≠C object When , the following sliding window smoothing mechanism is used to determine the final recognition result.

[0099] In this embodiment, the first recognition result and the second recognition result are stored in frames.

[0100] Then, a sliding window is triggered, wherein the sliding window takes the first frame as a starting point and the end point is tracked to the currently stored frame; or the length of the sliding window is N frames, and the currently stored frame is tracked at all times;

[0101] The recognition results in the sliding window are counted, and the recognition result with the largest number is taken as the final recognition result.

[0102] In practical applications, the length of the sliding window will be gradually increased from 0 to the current frame stored in order to correspond to the situation where the number of frames is small at the beginning of the detection;

[0103] Or when storing the corresponding category in each frame, set a sliding window from the frame to the previous frame with a length of N frames. In these N frames, count the number of times all categories appear (None also belongs to a category), and then output the category with the largest number of occurrences; assuming C final (t) is the detection result at the 𝑡th frame, and the sliding window length is 𝑁 frames. At this moment, the sliding window will contain the detection results of N frames from t −N+1 to t. For each category in the window, count its occurrence times.

[0104] Let f(x,t) be the number of occurrences of category x in the sliding window corresponding to the tth frame, and let C output (t) is the final output category at the 𝑡th frame, then no matter what detection method is used, there is , the final recognition result can be obtained.

[0105] Furthermore, according to the attachment type identification result, the forklift is matched with the corresponding attachment load map and automatically switched.

[0106] The smooth switching target scale detection strategy proposed in the present invention eliminates the uncertainty of the recognition result at a single moment by smoothing the signals from the visual and RFID recognition modules and calculating the recognition results over a period of time, thereby improving the final recognition accuracy. This strategy can effectively filter out instantaneous noise or erroneous data (such as RFID tag interference, visual misrecognition, etc.), ensure the accuracy and stability of accessory recognition, and reduce recognition errors.

[0107] Embodiment 2

[0108] This embodiment discloses a multi-modal telescopic forklift attachment recognition system, in which, Figure 2 , an electronic tag 4 is provided on the attachment of the telescopic forklift, a reader / writer 1 and a camera 2 are installed on the fixed arm, a control terminal 3 is installed in the cockpit, and the reader / writer 1 and the camera 2 communicate with the control terminal respectively;

[0109] In an exemplary embodiment, the reader is an RFID reader that supports RS232, RS485 and RJ45 network ports, and is directly connected to a development board in a control terminal through the network port. The development board is responsible for processing data from the RFID reader, listening to the preset port 8088, and receiving data packets transmitted by the RFID reader through the TCP protocol, using non-blocking I / O operations.

[0110] The camera is connected to the development board via USB and transmits images to the development board in real time.

[0111] The reader is used to read the electronic tag information and send the electronic tag information to the control terminal;

[0112] A camera, used for acquiring an image of an attachment and sending the image of the attachment to a control terminal;

[0113] The control terminal is used to identify the electronic tag information to obtain a first identification result, visually identify the attachment image to obtain a second identification result, and identify the attachment of the telescopic forklift according to the first identification result and the second identification result. This part of the identification process adopts the same method as the first embodiment, so it will not be repeated here.

[0114] The method provided by the present invention is not only applicable to situations where automatic identification is required after an unmanned forklift replaces accessories, but also applicable to situations where the load map is not replaced in time or is forgotten after accessories are replaced manually. It can effectively avoid accidents such as forklift rollovers caused by performance parameters such as maximum load and arm span not being matched in time during subsequent forklift operations, thereby improving the safety of forklift operations.

[0115] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0116] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-modal telescopic forklift attachment identification method, characterized in that: The attachment is provided with an electronic tag, and a first identification result is obtained by identifying the electronic tag; Acquiring an attachment image and obtaining a second recognition result by visually recognizing the attachment image; the steps include: The tool image is converted to the target scale through the scale calculation function. The feature of the attachment image is extracted through a convolutional neural network to obtain a feature map; Extract conditional information related to the target scale in the feature map; Generate adaptive attention weights based on feature maps and conditional information; After performing element-wise multiplication of the adaptive attention weight and the feature map, the adaptive attention weight and the feature map are concatenated to obtain the feature to be detected; Identify the attachment according to the feature to be detected to obtain a second identification result; Identifying the attachment of the telescopic forklift according to the first identification result and the second identification result; including: When the attachment is in the retracted state, the first recognition result is used as the final recognition result; When the extension distance of the attachment exceeds the identification distance of the electronic tag, the second identification result is used as the final identification result; When the attachment is in the process of extension and retraction and the distance does not exceed the electronic tag recognition distance, if the first recognition result is the same as the second recognition result, the current recognition result is used as the final recognition result; if the first recognition result is different from the second recognition result, the sliding window smoothing mechanism is triggered to determine the final recognition result.

2. A multi-modal telescopic forklift attachment identification method according to claim 1, characterized in that: Different accessories have different codes. When writing the code into the electronic tag, Convert the binary code into hexadecimal, and the converted hexadecimal data is written into the EPC area of ​​the tag; When identifying an electronic tag, Extract key information starting from the fifth byte of the hexadecimal data. And according to the key information, in combination with the preset mapping relationship table, a first recognition result is obtained; The mapping relationship table represents the corresponding relationship between the electronic tags and the accessory categories.

3. A multi-modal telescopic forklift attachment identification method according to claim 1, characterized in that: The attachment image is converted to the target scale using the following formula; ； In the formula, Indicates the attachment image. represents the scale calculation function, and Represent the height and width of the target image respectively, and Respectively represent the height and width of the attachment image.

4. A multi-modal telescopic forklift attachment identification method according to claim 1, characterized in that: The method to extract conditional information related to the target scale in the feature map is: ； In the formula, Represents the conditional information corresponding to the feature map of the i-th layer, represents the feature map of the i-th layer, Represents the feature map Perform global average pooling, Represents a splicing operation, and Represent the weight matrices of the two fully connected layers, and Respectively represent the bias vectors of the corresponding fully connected layers, Represents the Sigmoid activation function.

5. The multi-modal telescopic forklift attachment identification method according to claim 1, characterized in that: The method for generating adaptive attention weights based on feature maps and conditional information is: ； In the formula, represents the adaptive attention weight, Indicates conditional information The weight matrix of Representation feature map The weight matrix of and Respectively represent condition information and feature map The bias vector of Represents the weight matrix corresponding to the fully connected layer.

6. A multi-modal telescopic forklift attachment identification method according to claim 1, characterized in that: Triggering the sliding window smoothing mechanism to determine the final recognition result, the steps include: The first recognition result and the second recognition result are stored in frames, Triggering a sliding window, wherein the sliding window starts from the first frame and ends at the stored current frame; The recognition results in the sliding window are counted, and the recognition result with the largest number is taken as the final recognition result.

7. A multi-modal telescopic forklift attachment identification method according to claim 1, characterized in that: Triggering the sliding window smoothing mechanism to determine the final recognition result, the steps include: The first recognition result and the second recognition result are stored in frames, Trigger a sliding window, the length of which is N frames, and keep track of the current stored frame; The recognition results in the sliding window are counted, and the recognition result with the largest number is taken as the final recognition result.

8. A multi-modal telescopic forklift attachment identification system, characterized in that: The attachments of the telescopic forklift are equipped with electronic tags, the fixed arm is equipped with a reader and a camera, and the cockpit is equipped with a control terminal, which communicates with the control terminal respectively; The reader is used to read the electronic tag information and send the electronic tag information to the control terminal; A camera, used for acquiring an image of an attachment and sending the image of the attachment to a control terminal; The control terminal is used to identify the electronic tag information to obtain a first identification result, visually identify the attachment image to obtain a second identification result, and identify the attachment of the telescopic forklift according to the first identification result and the second identification result; Performing visual recognition on the attachment image to obtain a second recognition result; the steps include: The tool image is converted to the target scale through the scale calculation function. The feature of the attachment image is extracted through a convolutional neural network to obtain a feature map; Extract conditional information related to the target scale in the feature map; Generate adaptive attention weights based on feature maps and conditional information; After performing element-wise multiplication of the adaptive attention weight and the feature map, the adaptive attention weight and the feature map are concatenated to obtain the feature to be detected; Identify the attachment according to the feature to be detected to obtain a second identification result; Identifying the attachment of the telescopic forklift according to the first identification result and the second identification result; including: When the attachment is in the retracted state, the first recognition result is used as the final recognition result; When the extension distance of the attachment exceeds the identification distance of the electronic tag, the second identification result is used as the final identification result; When the attachment is in the process of extension and retraction and the distance does not exceed the electronic tag recognition distance, if the first recognition result is the same as the second recognition result, the current recognition result is used as the final recognition result; if the first recognition result is different from the second recognition result, the sliding window smoothing mechanism is triggered to determine the final recognition result.

Images