Method, device and medium for visual identification of auxiliary equipment type based on telescopic arm forklift

Abstract

The present invention discloses a method, device and medium for visually identifying the type of auxiliary equipment based on a telescopic forklift. When a change in the auxiliary equipment of a target telescopic forklift is detected, an image of the auxiliary equipment of the target telescopic forklift is collected through a dual-spectrum visual device to obtain a current collected image; the current collected image is processed through a data processing method to obtain a current key point mathematical descriptor and a current region feature code; the current key point mathematical descriptor and the current region feature code are input into a pre-trained auxiliary equipment type visual recognition model for recognition to obtain an auxiliary equipment type recognition result; the auxiliary equipment type recognition result is input into the auxiliary equipment flag receiving control module corresponding to the target telescopic forklift to implement the adjustment processing operation of the vehicle working state. The problem of error in determining the auxiliary equipment type caused by the need for human participation in the change of the auxiliary equipment of the telescopic forklift is solved, and the accuracy of auxiliary equipment type recognition is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device and medium for visually identifying the type of auxiliary equipment based on a telescopic arm forklift. Background Art

[0002] Changing the type of auxiliary implement on a telescopic handler involves three steps: replacing the implement, identifying the implement, and adjusting the overall vehicle's auxiliary implement position. These steps require manual intervention. Each time the auxiliary implement type is changed, the driver must manually adjust the auxiliary implement knob to the appropriate position to complete the gear change.

[0003] During the development of the present invention, the inventors discovered the following deficiencies in the prior art: Currently, because each step in the process of changing an auxiliary tool on a telescopic forklift requires varying degrees of human interaction, equipment failures and safety incidents caused by human factors are unavoidable. For example, forgetting to adjust the auxiliary tool type after changing the auxiliary tool, incorrectly adjusting the auxiliary tool type after changing the auxiliary tool, and the driver's inability to observe the specific auxiliary tool type can all lead to damage to the vehicle or even serious construction accidents. Summary of the Invention

[0004] The present invention provides a method, device and medium for visually identifying the type of assistive device based on a telescopic arm forklift, so as to improve the accuracy of assistive device type identification and effectively protect the life and property safety of users.

[0005] According to one aspect of the present invention, a method for visually identifying the type of auxiliary equipment of a telescopic forklift is provided, which includes:

[0006] When a change in the auxiliary equipment of the target telescopic forklift is detected, an image of the auxiliary equipment of the target telescopic forklift is captured by a pre-set dual-spectrum vision device to obtain a current captured image;

[0007] Wherein, a dual-spectrum vision device is added to the end of the telescopic arm corresponding to the target telescopic forklift, and a mechanical anti-shake device is installed between the end of the arm corresponding to the target telescopic forklift and the dual-spectrum vision device;

[0008] Processing the current captured image using a pre-set data processing method to obtain a current key point mathematical descriptor and a current region feature code;

[0009] Input the current key point mathematical descriptor and the current region feature code into a pre-trained assistive device type visual recognition model for recognition, thereby obtaining an assistive device type recognition result;

[0010] The auxiliary tool type identification result is input into the auxiliary tool flag receiving control module corresponding to the target telescopic forklift, so as to implement the adjustment processing operation of the vehicle working state according to the received auxiliary tool type identification result.

[0011] According to another aspect of the present invention, a device for visually identifying the type of auxiliary equipment for a telescopic forklift is provided, comprising:

[0012] a current captured image determination module configured to capture an image of the target telescopic forklift's auxiliary equipment by using a pre-set dual-spectrum vision device to obtain a current captured image when a change in the auxiliary equipment of the target telescopic forklift is detected;

[0013] Wherein, a dual-spectrum vision device is added to the end of the telescopic arm corresponding to the target telescopic forklift, and a mechanical anti-shake device is installed between the end of the arm corresponding to the target telescopic forklift and the dual-spectrum vision device;

[0014] A current key point mathematical descriptor and current region feature code determination module, configured to process the current captured image using a preset data processing method to obtain a current key point mathematical descriptor and a current region feature code;

[0015] an assistive device type recognition result determination module, configured to input the current key point mathematical descriptor and the current region feature code into a pre-trained assistive device type visual recognition model for recognition, thereby obtaining an assistive device type recognition result;

[0016] The auxiliary tool type identification result input module is used to input the auxiliary tool type identification result into the auxiliary tool flag receiving control module corresponding to the target telescopic arm forklift, so as to adjust the vehicle working state according to the received auxiliary tool type identification result.

[0017] According to another aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for visually identifying the type of auxiliary equipment based on a telescopic forklift as described in any embodiment of the present invention is implemented.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the auxiliary tool type visual recognition method based on a telescopic arm forklift as described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention is to, when a change in the auxiliary equipment of the target telescopic forklift is detected, capture an image of the auxiliary equipment of the target telescopic forklift through a pre-set dual-spectrum vision device to obtain a current captured image; process the current captured image through a pre-set data processing method to obtain a current key point mathematical descriptor and a current region feature code; input the current key point mathematical descriptor and the current region feature code into a pre-trained auxiliary equipment type visual recognition model for recognition to obtain an auxiliary equipment type recognition result; and input the auxiliary equipment type recognition result into the auxiliary equipment flag receiving control module corresponding to the target telescopic forklift to implement adjustment processing operations for the vehicle working state based on the received auxiliary equipment type recognition result. This solves the problem of auxiliary equipment type determination errors caused by the need for human intervention in the auxiliary equipment changes of the telescopic forklift, improves the accuracy of auxiliary equipment type recognition, and ensures the life and property safety of users.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of a method for visually identifying the type of auxiliary equipment based on a telescopic forklift according to the first embodiment of the present invention;

[0023] Figure 2 This is a flowchart of the training of the assistive device type visual recognition model in the method provided in the second embodiment of the present invention;

[0024] Figure 3 2 is a schematic structural diagram of a visual identification device for an auxiliary tool type based on a telescopic forklift according to a third embodiment of the present invention;

[0025] Figure 4 It is a structural diagram of an electronic device provided according to the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "target", "current", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Example 1

[0029] Figure 1 A flowchart of a method for visually identifying the type of auxiliary equipment based on a telescopic forklift is provided for the first embodiment of the present invention. This embodiment is applicable to situations where the type of auxiliary equipment of a telescopic forklift is to be identified. The method can be performed by a visual identification device for the type of auxiliary equipment based on a telescopic forklift, and the visual identification device for the type of auxiliary equipment based on a telescopic forklift can be implemented in the form of hardware and / or software.

[0030] Correspondingly, such as Figure 1 As shown, the method includes:

[0031] S110 : When a change in an auxiliary device of a target telescopic forklift is detected, an image of the auxiliary device of the target telescopic forklift is captured by a pre-set dual-spectrum vision device to obtain a current captured image.

[0032] Among them, a dual-spectrum vision device is added to the end of the telescopic arm corresponding to the target telescopic forklift, and a mechanical anti-shake device is installed between the end of the arm corresponding to the target telescopic forklift and the dual-spectrum vision device.

[0033] The dual-spectrum vision device may be composed of a visible light camera and a near-infrared camera. The auxiliary equipment of the target telescopic handler may include forks, a hook, a manned platform, or a bucket. The currently captured image may be an image captured by the dual-spectrum vision device.

[0034] In this embodiment, the dual-spectrum vision device can be used to capture the surface texture, color, and morphological features of the auxiliary equipment on the target telescopic forklift.

[0035] In addition, a mechanical anti-shake device is installed between the end of the arm corresponding to the target telescopic forklift and the dual-spectrum vision device, and the mechanical anti-shake device uses a three-axis gyroscope to stabilize the gimbal.

[0036] The advantages of this setting are: by setting up a mechanical anti-shake device, the vibration of the arm end when the vehicle is running can be alleviated or avoided; by using a three-axis gyroscope to stabilize the gimbal, it can be used to compensate for the vibration of the arm end that occurs during the driving process of the vehicle or the extension and retraction of the arm, so that the dual-spectrum vision device can capture stable and clear image information.

[0037] In this embodiment, when a change in the auxiliary equipment of the target telescopic forklift is detected, it indicates that a processing operation is required to adjust the corresponding vehicle working state according to the change in the auxiliary equipment. Therefore, it is necessary to collect images through a dual-spectrum vision device to obtain the corresponding current collected image.

[0038] In another preferred implementation manner of this embodiment, for the dual-spectrum vision device, it can also be a visible light spectrum camera and a laser radar, which can generate corresponding three-dimensional image point clouds based on the collected images, and can realize the assistive device type prediction function of multimodal learning recognition.

[0039] S120 , processing the current captured image using a preset data processing method to obtain a current key point mathematical descriptor and a current region feature code.

[0040] The data processing method is a method for obtaining feature data corresponding to an image through various image processing methods. The current keypoint mathematical descriptor can be a descriptor that describes the image using a multidimensional vector. The current region feature code can be a feature code obtained by dividing the image into different regions and extracting features.

[0041] Specifically, the currently acquired image is processed by a preset data processing method to obtain the current key point mathematical descriptor, including: using a preset auxiliary tool morphological feature RGB conversion formula: , processing the current acquired image to obtain a single-channel grayscale image of the current assistive device morphological features; wherein, is a single-channel grayscale image of the current assistive device morphological feature; R is the red pixel value of the current acquired image; G is the green pixel value of the current acquired image; B is the blue pixel value of the current acquired image; the single-channel grayscale image of the assistive device morphological feature is denoised by a pre-set non-local mean denoising method to obtain the current assistive device morphological feature map; the corner key point information and edge intersection key point information of different assistive devices in the current assistive device morphological feature map are detected by a pre-set SIFT algorithm to obtain the current key point mathematical descriptor.

[0042] Among them, the SIFT (Scale Invariant Feature Transform) algorithm is a local feature description algorithm in the field of image processing.

[0043] In this embodiment, the current captured image is an RGB three-channel image, which can be converted to RGB using the auxiliary morphological feature RGB conversion formula: , to process the RGB three-channel image to obtain a single-channel grayscale image of the current assistive device's morphological characteristics. For example, if the red pixel value of the current captured image is 128, the green pixel value is 64, and the blue pixel value is 32, the pixel value of the single-channel grayscale image of the current assistive device's morphological characteristics can be calculated to be 80. This can significantly reduce the amount of data and increase the calculation speed. The RGB conversion formula for assistive device morphological characteristics can highlight the morphological characteristics of the assistive device, retain key feature information such as the object's outline and texture, and weaken the interference of light color temperature.

[0044] Furthermore, the single-channel grayscale image of the auxiliary tool morphological feature is denoised by the non-local mean denoising method. During the denoising process, median filtering is mainly used to eliminate random noise in the image. The non-local mean denoising method can also retain the sharpness of the auxiliary tool edge, thereby avoiding blurring of image information of key parts such as the tooth tip, fork tip, and hook of the auxiliary tool, and the current auxiliary tool morphological feature map can be further obtained.

[0045] Correspondingly, the SIFT algorithm can also be used to detect the corner key point information and edge intersection key point information of different assistive devices in the current assistive device morphological feature map to obtain the current key point mathematical descriptor, wherein the current key point mathematical descriptor can be a 128-dimensional vector descriptor.

[0046] Specifically, the current captured image is processed by a preset data processing method to obtain a current area feature code, including: dividing the current assistive device morphological feature map into feature areas through a preset feature area detection algorithm to obtain at least one current feature area; extracting the area of ​​interest from each current feature area to obtain a current area feature code.

[0047] The feature region detection algorithm can be based on a convolutional neural network or a capsule network. Assuming the capsule network algorithm can transmit information through vectorized capsules, it can preserve the posture and rotation properties of different assistive devices. Furthermore, when the captured assistive device images undergo translation or rotation changes, the data changes are more stable.

[0048] In this embodiment, different auxiliary tools have different characteristics. Specifically, a fork auxiliary tool is characterized by having two protruding tips, and a hook auxiliary tool is characterized by a smaller hook. Therefore, a feature region detection algorithm is required to divide the current auxiliary tool morphological feature map into feature regions, extract the region of interest from each current feature region, and obtain the current region feature code. This allows computing power to be concentrated in the key feature regions, reducing the impact of background, mud, damage, and occlusion.

[0049] S130: Input the current key point mathematical descriptor and the current region feature code into a pre-trained assistive device type visual recognition model for recognition, to obtain an assistive device type recognition result.

[0050] The assistive device type visual recognition model may be a model for recognizing the assistive device type, and the assistive device type recognition result may be the type of the assistive device that has been changed.

[0051] In this embodiment, the assistive device type visual recognition model may perform recognition processing operations based on the current key point mathematical descriptor and the current region feature code to obtain a specific assistive device type.

[0052] In addition, assuming that the assistive device type visual recognition model does not identify a specific assistive device type recognition result, the current captured image needs to be fed back to humans, and humans need to perform recognition processing operations, and the assistive device type visual recognition model can be retrained through manually marked assistive device type labels.

[0053] S140: Input the auxiliary tool type identification result into the auxiliary tool flag receiving control module corresponding to the target telescopic forklift, so as to implement an adjustment processing operation of the vehicle working state according to the received auxiliary tool type identification result.

[0054] In this embodiment, assuming that the identified auxiliary tool type is a hook, the hook needs to be input into the auxiliary tool flag receiving control module corresponding to the target telescopic forklift, which indicates that the hook has changed, so the vehicle working status of the target telescopic forklift needs to be adjusted.

[0055] The technical solution of the embodiment of the present invention is to, when a change in the auxiliary equipment of the target telescopic forklift is detected, capture an image of the auxiliary equipment of the target telescopic forklift through a pre-set dual-spectrum vision device to obtain a current captured image; process the current captured image through a pre-set data processing method to obtain a current key point mathematical descriptor and a current region feature code; input the current key point mathematical descriptor and the current region feature code into a pre-trained auxiliary equipment type visual recognition model for recognition to obtain an auxiliary equipment type recognition result; and input the auxiliary equipment type recognition result into the auxiliary equipment flag receiving control module corresponding to the target telescopic forklift to implement adjustment processing operations for the vehicle working state based on the received auxiliary equipment type recognition result. This solves the problem of auxiliary equipment type determination errors caused by the need for human intervention in the auxiliary equipment changes of the telescopic forklift, improves the accuracy of auxiliary equipment type recognition, and ensures the life and property safety of users.

[0056] Example 2

[0057] Figure 2 A flowchart for training an auxiliary tool type visual recognition model in a method for visually identifying auxiliary tool types for a telehandler is provided for the second embodiment of the present invention. This embodiment is an optimization based on the above embodiments. Before acquiring an image of the target telehandler using a pre-set dual-spectral vision device to obtain the current captured image, this embodiment also includes training the auxiliary tool type visual recognition model.

[0058] Correspondingly, such as Figure 2 As shown, the method includes:

[0059] S210: Acquire multiple historical assistive tool samples and the historical assistive tool type corresponding to each of the historical assistive tool image samples.

[0060] The historical assistive device samples may be image samples of various assistive device types. The historical assistive device types may be labeled for each historical assistive device sample, and the specific assistive device type may be determined by the label.

[0061] S220 , processing each of the historical auxiliary tool image samples using a preset data processing method to obtain corresponding historical key point mathematical descriptors and historical area feature codes.

[0062] The historical key point mathematical descriptor may be a descriptor that describes each historical auxiliary tool sample using a multi-dimensional vector, and the current region feature code may be a feature code obtained by dividing each historical auxiliary tool sample into different regions and extracting features.

[0063] S230 , processing each of the historical key point mathematical descriptors respectively through a preset feature integration and mapping processing method to obtain a historical key point integrated mapping mathematical descriptor.

[0064] The feature integration and mapping processing method may be a method for integrating and mapping the extracted features of the mathematical descriptors of the historical key points.

[0065] Optionally, the method uses a pre-set feature integration and mapping processing method to process each of the historical key point mathematical descriptors separately to obtain a historical key point integrated mapping mathematical descriptor, including: performing weighted summation and activation function processing on each of the historical key point mathematical descriptors through the neurons of the hidden layer in the feature integration and mapping processing method to obtain each historical key point processed mathematical descriptor; and performing feature integration and mapping processing on each of the historical key point processed mathematical descriptors to obtain a historical key point integrated mapping mathematical descriptor.

[0066] For example, assuming the mathematical descriptor of historical key points is a 128-dimensional vector descriptor, it is first necessary to perform linear and nonlinear transformations on the 128-dimensional vector descriptor. Then, the neurons in the hidden layer of the feature integration and mapping method use weighted summation and activation functions to process the input 128-dimensional vector descriptor, and gradually extract high-level features from the data. The different hidden layers can learn features at different levels and levels of abstraction. The features extracted by the hidden layers are then integrated and mapped to obtain the integrated and mapped mathematical descriptor of historical key points. This can be better used for training assistive device type visual recognition models.

[0067] S240: Input the historical key point integration mapping mathematical descriptors, the historical area feature codes, and the historical assistive device types into a pre-built initial assistive device type visual recognition model for training to obtain a currently trained assistive device type visual recognition model.

[0068] S250. Randomly obtain a group of historical assistive device verification samples and historical assistive device test types, input the historical assistive device verification samples into the currently trained assistive device type visual recognition model to obtain the model-predicted assistive device type, and combine the historical assistive device test types to determine whether the training of the assistive device type visual recognition model is complete.

[0069] In this embodiment, a group of historical assistive device verification samples and historical assistive device test types are randomly obtained from the test set samples, and used to verify whether the assistive device type visual recognition model is trained, that is, to determine whether the recognition accuracy of the assistive device type visual recognition model meets the requirements.

[0070] Optionally, the historical assistive device verification sample is input into the currently trained assistive device type visual recognition model to obtain the model-predicted assistive device type, and combined with the historical assistive device test type, it is determined whether the training of the assistive device type visual recognition model is completed, including: inputting the historical assistive device verification sample into the currently trained assistive device type visual recognition model to obtain the model-predicted assistive device type, and combined with the historical assistive device test type to calculate the input and output mean square error value; obtaining a preset input and output mean square error value threshold, and judging whether the input and output mean square error value meets the requirement of the input and output mean square error value threshold; if so, determining that the training of the assistive device type visual recognition model is completed, and determining the target parameter weight and target parameter threshold corresponding to the assistive device type visual recognition model.

[0071] The input / output mean square error (MSE) value can be used to describe the difference between the model-predicted assistive device type and the historical assistive device test type, reflecting the accuracy of the assistive device type prediction by the assistive device type visual recognition model. The input / output mean square error (MSE) value threshold can be a pre-set threshold value for the required input / output mean square error value.

[0072] In this embodiment, if the input and output mean square error values ​​meet the requirements of the input and output mean square error value threshold, it means that the prediction accuracy of the trained assistive device type visual recognition model meets the requirements, so the training of the assistive device type visual recognition model can be stopped, that is, it is determined that the training of the assistive device type visual recognition model is completed.

[0073] Furthermore, after the training of the assistive device type visual recognition model is completed, it is also necessary to determine the target parameter weights and target parameter thresholds corresponding to the assistive device type visual recognition model.

[0074] Optionally, after determining whether the input-output mean square error value meets the requirement of the input-output mean square error value threshold, it also includes: if not, adjusting the target parameter weight and the target parameter threshold respectively through a preset back propagation method, and returning the operation of integrating the historical key point mapping mathematical descriptor and the historical area feature code, as well as the historical assistive device type, and inputting them into the pre-built initial assistive device type visual recognition model for training, until it is determined that the training of the assistive device type visual recognition model is completed.

[0075] In this embodiment, if the input-output mean square error value does not meet the input-output mean square error value threshold requirement, it means that the prediction accuracy of the trained assistive device type visual recognition model does not meet the requirement, and it is necessary to use the back propagation method to adjust the target parameter weight and the target parameter threshold respectively, and retrain the assistive device type visual recognition model.

[0076] Specifically, in the process of executing the back-propagation method, it is necessary to analyze the target parameter weights and target parameter thresholds, and also require the steps of layered gradient normalization, adaptive gradient clipping, and updating the target parameter weights and target parameter thresholds.

[0077] In detail, the analysis of target parameter weights and target parameter thresholds is to calculate the gradient of the input and output mean square error values ​​of the assistive device type visual recognition model to these parameters under the current parameter settings, and to guide how to update the target parameter weights and target parameter thresholds based on the input and output mean square error values ​​so that the input and output mean square error values ​​of the model meet the requirements of the input and output mean square error value thresholds.

[0078] Furthermore, during layer-wise gradient normalization, the gradients of different layers need to be normalized so that the gradients of each layer are within a desired range. This can minimize the problems of vanishing or exploding gradients, accelerate the convergence of the assistive device visual recognition model, and ensure training stability.

[0079] Accordingly, during adaptive gradient clipping, the gradient is clipped to prevent excessive gradients from causing large parameter updates and destabilizing the training process. This adaptive approach allows for dynamic adjustment of the clipping threshold based on actual training conditions.

[0080] In the process of updating the target parameter weights and target parameter thresholds, the stochastic gradient descent algorithm is used to update the target parameter weights and target parameter thresholds of the assistive device type visual recognition model based on the calculated gradient information, so that the assistive device type visual recognition model can reduce the difference between the model-predicted assistive device type and the historical assistive device test type in subsequent predictions, and gradually improve the accuracy of the prediction.

[0081] S260: When a change in the auxiliary equipment of the target telescopic forklift is detected, an image of the auxiliary equipment of the target telescopic forklift is captured by a pre-set dual-spectrum vision device to obtain a current captured image.

[0082] Among them, a dual-spectrum vision device is added to the end of the telescopic arm corresponding to the target telescopic forklift, and a mechanical anti-shake device is installed between the end of the arm corresponding to the target telescopic forklift and the dual-spectrum vision device.

[0083] S270 , processing the current captured image using a preset data processing method to obtain a current key point mathematical descriptor and a current region feature code.

[0084] S280: Input the current key point mathematical descriptor and the current region feature code into a pre-trained assistive device type visual recognition model for recognition, to obtain an assistive device type recognition result.

[0085] S290: Input the auxiliary tool type identification result into the auxiliary tool flag receiving control module corresponding to the target telescopic forklift, so as to implement an adjustment processing operation of the vehicle working state according to the received auxiliary tool type identification result.

[0086] The technical solution of the embodiment of the present invention trains an assistive device type visual recognition model. When a change in the assistive device of a target telescopic forklift is detected, the trained assistive device type visual recognition model is used to identify the assistive device type of the current captured image. The assistive device type recognition result is input into the assistive device flag receiving control module corresponding to the target telescopic forklift, thereby adjusting the vehicle's operating status based on the received assistive device type recognition result. This solves the problem of assistive device type determination errors caused by the need for human intervention in assistive device changes on telescopic forklifts. By using the trained assistive device type visual recognition model, the accuracy of assistive device type recognition is improved, thereby reducing labor costs, improving the efficiency and accuracy of adjusting the vehicle's operating status due to assistive device changes, and ensuring the safety of users' lives and property.

[0087] Example 3

[0088] Figure 3 This is a schematic diagram of the structure of a telescopic forklift-based auxiliary tool type visual recognition device provided in the third embodiment of the present invention. The telescopic forklift-based auxiliary tool type visual recognition device provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or server to implement a telescopic forklift-based auxiliary tool type visual recognition method in the embodiment of the present invention. Figure 3 As shown, the device includes: a current captured image determination module 310, a current key point mathematical descriptor and current region feature code determination module 320, an assistive device type recognition result determination module 330 and an assistive device type recognition result input module 340.

[0089] The current captured image determination module 310 is configured to capture an image of the target telescopic forklift's auxiliary equipment through a pre-set dual-spectrum vision device to obtain a current captured image when a change in the auxiliary equipment of the target telescopic forklift is detected.

[0090] Wherein, a dual-spectrum vision device is added to the end of the telescopic arm corresponding to the target telescopic forklift, and a mechanical anti-shake device is installed between the end of the arm corresponding to the target telescopic forklift and the dual-spectrum vision device;

[0091] The current key point mathematical descriptor and current region feature code determination module 320 is used to process the current captured image using a preset data processing method to obtain the current key point mathematical descriptor and the current region feature code;

[0092] An assistive device type recognition result determination module 330 is configured to input the current key point mathematical descriptor and the current region feature code into a pre-trained assistive device type visual recognition model for recognition, thereby obtaining an assistive device type recognition result;

[0093] The auxiliary tool type identification result input module 340 is used to input the auxiliary tool type identification result into the auxiliary tool flag receiving control module corresponding to the target telescopic forklift, so as to adjust the vehicle working state according to the received auxiliary tool type identification result.

[0094] The technical solution of the embodiment of the present invention is that when a change in the auxiliary equipment of the target telescopic forklift is detected, an image of the auxiliary equipment of the target telescopic forklift is collected through a pre-set dual-spectrum vision device to obtain a current collected image; the current collected image is processed through a pre-set data processing method to obtain a current key point mathematical descriptor and a current region feature code; the current key point mathematical descriptor and the current region feature code are input into a pre-trained auxiliary equipment type visual recognition model for recognition to obtain an auxiliary equipment type recognition result; the auxiliary equipment type recognition result is input into the auxiliary equipment flag receiving control module corresponding to the target telescopic forklift to implement the adjustment processing operation of the vehicle working state according to the received auxiliary equipment type recognition result. This solves the problem of auxiliary equipment type determination errors caused by the need for human participation in the change of the auxiliary equipment of the telescopic forklift, improves the accuracy of auxiliary equipment type recognition, and ensures the life and property safety of users.

[0095] On the basis of the above embodiments, the current key point mathematical descriptor and current region feature code determination module 320 can be specifically used to: convert the auxiliary tool morphological feature RGB into the preset formula: , processing the current acquired image to obtain a single-channel grayscale image of the current assistive device morphological features; wherein, is a single-channel grayscale image of the current assistive device morphological feature; R is the red pixel value of the current acquired image; G is the green pixel value of the current acquired image; B is the blue pixel value of the current acquired image; the single-channel grayscale image of the assistive device morphological feature is denoised by a pre-set non-local mean denoising method to obtain the current assistive device morphological feature map; the corner key point information and edge intersection key point information of different assistive devices in the current assistive device morphological feature map are detected by a pre-set SIFT algorithm to obtain the current key point mathematical descriptor.

[0096] On the basis of the above embodiments, the current key point mathematical descriptor and current area feature code determination module 320 can also be specifically used to: divide the current assistive device morphological feature map into feature areas through a pre-set feature area detection algorithm to obtain at least one current feature area; extract the area of ​​interest from each current feature area to obtain the current area feature code.

[0097] On the basis of the above embodiments, it also includes an assistive device type visual recognition model training module, which can specifically include: a historical assistive device sample and historical assistive device type acquisition unit, which is used to acquire multiple historical assistive device samples and historical assistive device types corresponding to each of the historical assistive device image samples before acquiring the current acquired image by the preset dual-spectrum vision device to acquire the image of the target telescopic arm forklift; a historical key point mathematical descriptor and historical area feature code acquisition unit, which is used to process each of the historical assistive device image samples through a preset data processing method to respectively obtain the corresponding historical key point mathematical descriptor and historical area feature code; a historical key point integrated mapping mathematical descriptor determination unit, which is used to determine the historical key point through a preset feature integration and mapping processing method. , processing each of the historical key point mathematical descriptors respectively to obtain a historical key point integrated mapping mathematical descriptor; an assistive device type visual recognition model training unit, used to input the historical key point integrated mapping mathematical descriptor and the historical area feature code, as well as the historical assistive device type, into a pre-built initial assistive device type visual recognition model for training to obtain a currently trained assistive device type visual recognition model; an assistive device type visual recognition model training completion unit, used to randomly obtain a group of historical assistive device verification samples and historical assistive device test types, input the historical assistive device verification samples into the currently trained assistive device type visual recognition model to obtain a model-predicted assistive device type, and determine whether the training of the assistive device type visual recognition model is completed in combination with the historical assistive device test type.

[0098] Based on the above embodiments, the historical key point integrated mapping mathematical descriptor determination unit can be specifically used to: perform weighted summation and activation function processing on each historical key point mathematical descriptor through the neurons of the hidden layer in the feature integration and mapping processing method to obtain each historical key point processing mathematical descriptor; perform feature integration and mapping processing on each of the historical key point processing mathematical descriptors to obtain a historical key point integrated mapping mathematical descriptor.

[0099] On the basis of the above embodiments, the assistive device type visual recognition model training completion unit can be specifically used to: input the historical assistive device verification sample into the currently trained assistive device type visual recognition model to obtain the model-predicted assistive device type, and calculate the input and output mean square error value in combination with the historical assistive device test type; obtain a preset input and output mean square error value threshold, and determine whether the input and output mean square error value meets the requirements of the input and output mean square error value threshold. If so, it is determined that the training of the assistive device type visual recognition model is completed, and the target parameter weight and target parameter threshold corresponding to the assistive device type visual recognition model are determined.

[0100] On the basis of the above embodiments, the assistive device type visual recognition model training completion unit can also be specifically used for: after judging whether the input and output mean square error value meets the requirement of the input and output mean square error value threshold, if not, adjusting the target parameter weight and the target parameter threshold respectively through a preset back propagation method, and returning the operation of integrating the historical key point mapping mathematical descriptor and the historical area feature code, as well as the historical assistive device type, and inputting them into the pre-built initial assistive device type visual recognition model for training, until it is determined that the training of the assistive device type visual recognition model is fully completed.

[0101] The device for visually identifying the type of auxiliary equipment based on a telescopic forklift provided in an embodiment of the present invention can execute the method for visually identifying the type of auxiliary equipment based on a telescopic forklift provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0102] Example 4

[0103] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement the fourth embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0104] like Figure 4As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0105] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0106] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method for visually identifying the type of auxiliary implement for a telehandler.

[0107] In some embodiments, the telehandler-based assistive device type visual identification method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the telehandler-based assistive device type visual identification method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the telehandler-based assistive device type visual identification method through any other suitable means (e.g., via firmware).

[0108] The method includes: when a change in the auxiliary equipment of a target telescopic forklift is detected, an image of the auxiliary equipment of the target telescopic forklift is collected by a pre-set dual-spectrum vision device to obtain a current collected image; wherein, a dual-spectrum vision device is added to the end of the telescopic arm corresponding to the target telescopic forklift, and a mechanical anti-shake device is installed between the end of the arm corresponding to the target telescopic forklift and the dual-spectrum vision device; the current collected image is processed by a pre-set data processing method to obtain a current key point mathematical descriptor and a current area feature code; the current key point mathematical descriptor and the current area feature code are input into a pre-trained auxiliary equipment type visual recognition model for recognition to obtain an auxiliary equipment type recognition result; the auxiliary equipment type recognition result is input into an auxiliary equipment flag receiving control module corresponding to the target telescopic forklift, so as to implement an adjustment processing operation of the vehicle working state according to the received auxiliary equipment type recognition result.

[0109] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0111] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0114] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0115] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0116] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

[0117] Example 5

[0118] Embodiment 5 of the present invention further provides a method comprising a computer-readable storage medium, wherein the computer-readable instructions, when executed by a computer processor, are used to perform a method for visually identifying an auxiliary device type based on a telescopic forklift, the method comprising: when a change in an auxiliary device of a target telescopic forklift is detected, capturing an image of the auxiliary device of the target telescopic forklift using a pre-set dual-spectrum vision device to obtain a current captured image; wherein a dual-spectrum vision device is added to the end of the telescopic arm corresponding to the target telescopic forklift, and a mechanical anti-shake device is installed between the end of the boom corresponding to the target telescopic forklift and the dual-spectrum vision device; processing the current captured image using a pre-set data processing method to obtain a current key point mathematical descriptor and a current region feature code; inputting the current key point mathematical descriptor and the current region feature code into a pre-trained auxiliary device type visual recognition model for recognition to obtain an auxiliary device type recognition result; and inputting the auxiliary device type recognition result into an auxiliary device flag receiving control module corresponding to the target telescopic forklift to implement an adjustment processing operation for the vehicle working state based on the received auxiliary device type recognition result.

[0119] Of course, the computer-readable storage medium provided in an embodiment of the present invention has computer-executable instructions that are not limited to the method operations described above, and can also execute related operations in the visual recognition of auxiliary tool types based on telescopic arm forklifts provided in any embodiment of the present invention.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware. Of course, it can also be implemented with hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0121] It is worth noting that in the above-mentioned embodiment of visual identification of auxiliary equipment types based on telescopic arm forklifts, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0122] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for visually identifying the type of auxiliary equipment for a telescopic forklift, characterized in that: include: When a change in the auxiliary equipment of the target telescopic forklift is detected, an image of the auxiliary equipment of the target telescopic forklift is captured by a pre-set dual-spectrum vision device to obtain a current captured image; Wherein, a dual-spectrum vision device is added to the end of the telescopic arm corresponding to the target telescopic forklift, and a mechanical anti-shake device is installed between the end of the arm corresponding to the target telescopic forklift and the dual-spectrum vision device; Processing the current captured image using a pre-set data processing method to obtain a current key point mathematical descriptor and a current region feature code; Input the current key point mathematical descriptor and the current region feature code into a pre-trained assistive device type visual recognition model for recognition, thereby obtaining an assistive device type recognition result; Inputting the auxiliary tool type identification result into the auxiliary tool flag receiving control module corresponding to the target telescopic forklift, so as to adjust the vehicle working state according to the received auxiliary tool type identification result; Wherein, before acquiring an image of the target telescopic forklift auxiliary device by using a pre-set dual-spectrum vision device to obtain a current acquired image, the method further includes: Acquire a plurality of historical assistive device samples and a historical assistive device type corresponding to each of the historical assistive device image samples; Processing each of the historical auxiliary tool image samples using a pre-set data processing method to obtain corresponding historical key point mathematical descriptors and historical area feature codes; By using a preset feature integration and mapping processing method, each of the historical key point mathematical descriptors is processed respectively to obtain a historical key point integrated mapping mathematical descriptor; Inputting the historical key point integrated mapping mathematical descriptors, the historical region feature codes, and the historical assistive device types into a pre-built initial assistive device type visual recognition model for training to obtain a currently trained assistive device type visual recognition model; A group of historical assistive device verification samples and historical assistive device test types are randomly obtained, and the historical assistive device verification samples are input into the currently trained assistive device type visual recognition model to obtain the model-predicted assistive device type. Combined with the historical assistive device test types, it is determined whether the training of the assistive device type visual recognition model is completed.

2. The method according to claim 1, characterized in that The currently acquired image is processed by a preset data processing method to obtain a current key point mathematical descriptor, including: Through the preset auxiliary morphological feature RGB conversion formula: , processing the current acquired image to obtain a single-channel grayscale image of the current assistive device morphological features; in, is a single-channel grayscale image of the current auxiliary device morphological features; R is the red pixel value of the current acquired image; G is the green pixel value of the current acquired image; B is the blue pixel value of the current acquired image; Denoising the single-channel grayscale image of the assistive device morphology feature using a preset non-local mean denoising method to obtain a current assistive device morphology feature image; The preset scale-invariant feature transform (SIFT) algorithm is used to detect the key point information of corner points and edge intersection points of different assistive devices in the current assistive device morphological feature map to obtain a current key point mathematical descriptor.

3. The method according to claim 2, characterized in that The processing of the current captured image by a preset data processing method to obtain a current region feature code includes: Performing feature region division on the current assistive device morphology feature map using a preset feature region detection algorithm to obtain at least one current feature region; Extracting the region of interest from each of the current feature regions to obtain a current region feature code.

4. The method according to claim 1, wherein The method of processing each of the historical key point mathematical descriptors by a preset feature integration and mapping processing method to obtain the historical key point integrated mapping mathematical descriptor includes: Through the neurons of the hidden layer in the feature integration and mapping processing method, each historical key point mathematical descriptor is subjected to weighted summation and activation function processing to obtain the processed mathematical descriptors of each historical key point; Feature integration and mapping processing are performed on each of the historical key point processing mathematical descriptors to obtain a historical key point integrated mapping mathematical descriptor.

5. The method according to claim 4, characterized in that Inputting the historical assistive device verification sample into the currently trained assistive device type visual recognition model to obtain the model-predicted assistive device type, and combining the historical assistive device test type to determine whether the training of the assistive device type visual recognition model is fully completed, includes: Input the historical assistive device verification samples into the currently trained assistive device type visual recognition model to obtain the model-predicted assistive device type, and combine the historical assistive device test types to calculate the input-output mean square error value; Obtain a preset input-output mean square error value threshold, and determine whether the input-output mean square error value meets the requirement of the input-output mean square error value threshold; if so, determine that the training of the assistive device type visual recognition model is completed, and determine the target parameter weight and target parameter threshold corresponding to the assistive device type visual recognition model.

6. The method according to claim 5, characterized in that After determining whether the input-output mean square error value meets the requirement of the input-output mean square error value threshold, the method further includes: If not satisfied, the target parameter weight and target parameter threshold are adjusted respectively by a preset back propagation method, and the operation of integrating the historical key points and mapping the mathematical descriptors, the historical area feature codes, and the historical assistive device types is returned and input into a pre-built initial assistive device type visual recognition model for training until it is determined that the training of the assistive device type visual recognition model is completed.

7. A visual recognition device for the type of auxiliary equipment based on a telescopic forklift, characterized in that: include: a current captured image determination module configured to capture an image of the target telescopic forklift's auxiliary equipment by using a pre-set dual-spectrum vision device to obtain a current captured image when a change in the auxiliary equipment of the target telescopic forklift is detected; Wherein, a dual-spectrum vision device is added to the end of the telescopic arm corresponding to the target telescopic forklift, and a mechanical anti-shake device is installed between the end of the arm corresponding to the target telescopic forklift and the dual-spectrum vision device; A current key point mathematical descriptor and current region feature code determination module, configured to process the current captured image using a preset data processing method to obtain a current key point mathematical descriptor and a current region feature code; an assistive device type recognition result determination module, configured to input the current key point mathematical descriptor and the current region feature code into a pre-trained assistive device type visual recognition model for recognition, thereby obtaining an assistive device type recognition result; An auxiliary tool type identification result input module is used to input the auxiliary tool type identification result into the auxiliary tool flag receiving control module corresponding to the target telescopic forklift, so as to adjust the vehicle working state according to the received auxiliary tool type identification result; Among them, it also includes the assistive device type visual recognition model training module, including: a historical assistive device sample and historical assistive device type acquisition unit, configured to acquire a plurality of historical assistive device samples and a historical assistive device type corresponding to each of the historical assistive device image samples before acquiring an image of the target telescopic forklift using the pre-set dual-spectrum vision device and obtaining a current acquired image; A unit for obtaining mathematical descriptors of historical key points and characteristic codes of historical regions, configured to process each of the historical auxiliary tool image samples using a preset data processing method to obtain corresponding mathematical descriptors of historical key points and characteristic codes of historical regions; A historical key point integrated mapping mathematical descriptor determination unit is used to process each of the historical key point mathematical descriptors using a preset feature integration and mapping processing method to obtain a historical key point integrated mapping mathematical descriptor; an assistive device type visual recognition model training unit, configured to input the historical key point integrated mapping mathematical descriptors, the historical region feature codes, and the historical assistive device types into a pre-built initial assistive device type visual recognition model for training, thereby obtaining a currently trained assistive device type visual recognition model; The assistive device type visual recognition model training completion unit is used to randomly obtain a group of historical assistive device verification samples and historical assistive device test types, input the historical assistive device verification samples into the currently trained assistive device type visual recognition model, obtain the model-predicted assistive device type, and combine the historical assistive device test types to determine whether the training of the assistive device type visual recognition model is completed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for visually identifying the type of auxiliary equipment based on a telescopic forklift according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a method for visually identifying the type of an auxiliary tool based on a telescopic forklift according to any one of claims 1 to 6.

Images