Intelligent forklift control system and method based on smart transportation

Abstract

This application relates to the field of smart transportation, specifically to an intelligent forklift control system and method based on smart transportation. This system uses deep learning-based computer vision technology to detect and extract image features from images in front of the forklift, capturing the visual feature representation of the target object. This system then enhances the accuracy of the feature representation by further enhancing pixel-level semantic associations within the visual feature representation of the target object, thereby intelligently identifying whether the target object is an obstacle and making obstacle avoidance decisions. This effectively enhances the forklift's autonomous navigation and obstacle avoidance capabilities in complex operating environments, improving operational safety and efficiency.

Description

Technical Field

[0001] The present application relates to the field of smart transportation, and specifically to an intelligent forklift control system and method based on smart transportation. Background Art

[0002] With the rapid development of modern logistics and manufacturing, forklifts, as essential tools for material handling, are playing an increasingly crucial role in warehouses, factories, and other locations. Forklifts not only significantly improve material handling efficiency but also effectively reduce worker workload. However, traditional forklift operations are highly manual, resulting in low efficiency and the potential for safety accidents due to human error. For example, due to negligence and errors in operation, forklifts can collide during operation, causing not only severe physical damage to the forklift itself but also the dropping of cargo. This not only directly causes damage and loss to the cargo but also potentially triggers other potential hazards during the drop, leading to even more serious and unpredictable safety accidents. These issues not only severely impact business operations but also threaten the lives of workers.

[0003] In the context of the rapid development of modern manufacturing, forklifts are indispensable equipment in industries such as logistics and warehousing. Improving their intelligence level is of great significance for improving operational efficiency and reducing safety accidents. Therefore, an intelligent forklift control system and method based on smart transportation is expected. Summary of the Invention

[0004] This application is made in consideration of the above problems. One purpose of this application is to provide an intelligent forklift control system and method based on smart transportation.

[0005] The embodiment of the present application provides an intelligent forklift control system based on smart transportation, which includes:

[0006] A forklift front image acquisition module is used to acquire an image in front of the forklift collected by a camera deployed on the forklift;

[0007] a target object feature extraction module, configured to extract target object features from the image in front of the forklift to obtain a target object feature map;

[0008] A semantic association enhancement module, configured to perform pixel-level contextual semantic association enhancement on the target object feature map to obtain an enhanced target object feature map;

[0009] a target object recognition module, configured to determine a recognition result of the target object based on the enhanced target object feature map;

[0010] The obstacle avoidance module is configured to generate an obstacle avoidance instruction in response to the recognition result that the target object in the image in front of the forklift is an obstacle.

[0011] For example, according to an intelligent forklift control system based on smart transportation according to an embodiment of the present application, the target object feature extraction module includes:

[0012] An area of interest extraction unit, configured to extract an area of interest from the image in front of the forklift to obtain an area of interest image of a target object;

[0013] The region of interest image feature extraction unit is used to extract image features of the target object region of interest image to obtain the target object feature map.

[0014] For example, according to an intelligent forklift control system based on smart transportation according to an embodiment of the present application, the region of interest extraction unit is used to:

[0015] The forklift front image is input into a region of interest recommender based on an R-CNN network to obtain the target object region of interest image.

[0016] For example, according to an intelligent forklift control system based on smart transportation according to an embodiment of the present application, the region of interest image feature extraction unit is used to:

[0017] The target object region of interest image is input into a target object feature extractor based on a hole convolutional neural network model to obtain the target object feature map.

[0018] For example, according to an intelligent forklift control system based on smart transportation according to an embodiment of the present application, the semantic association enhancement module includes:

[0019] a pixel-level feature dispersion unit, configured to disperse the target object feature map along a channel dimension to obtain a set of target object channel local feature vectors;

[0020] A semantic association enhancement unit, configured to input the set of target object channel local feature vectors into semantic association enhancement based on feature energy level metric association radiation to obtain a set of enhanced target object channel local feature vectors;

[0021] The feature shape reshaping unit is used to perform feature shape reshaping on the set of local feature vectors of the enhanced target object channel to obtain the enhanced target object feature map.

[0022] For example, according to an intelligent forklift control system based on smart transportation according to an embodiment of the present application, the semantic association enhancement unit includes:

[0023] a characteristic energy level calculation subunit, configured to calculate a characteristic energy level coefficient of each target object channel local feature vector in the set of target object channel local feature vectors to obtain a set of characteristic energy level coefficients;

[0024] a one-dimensional arrangement subunit, configured to arrange the set of characteristic energy level coefficients into a characteristic energy level coefficient input vector;

[0025] a neighborhood correlation coding subunit, configured to perform one-dimensional convolution coding on the characteristic energy level coefficient input vector to obtain a characteristic energy level radiation correlation vector;

[0026] A weighting subunit, configured to perform point convolution coding and probabilistic mapping based on a Sigmoid function on the characteristic energy level radiation association vector to obtain a characteristic energy level radiation association weight vector;

[0027] The weighted enhancement subunit is used to perform weighted processing and residual processing on the set of local feature vectors of the target object channel using the characteristic energy level radiation associated weight vector as the weight vector to obtain the set of enhanced local feature vectors of the target object channel.

[0028] For example, according to an intelligent forklift control system based on smart transportation according to an embodiment of the present application, the characteristic energy level calculation subunit is used to:

[0029] Calculating a weighted sum of the variance of the local feature vector of the target object channel and a bias term to obtain a first intermediate term, and calculating a weighted sum of the square of the difference between the maximum value and the mean of the local feature vector of the target object channel, the variance of the local feature vector of the target object channel, and the bias term to obtain a second intermediate term;

[0030] The first intermediate term is divided by the second intermediate term to obtain the characteristic energy level coefficient.

[0031] For example, in an intelligent forklift control system based on smart transportation according to an embodiment of the present application, the target object recognition module is used to:

[0032] The enhanced target object feature map is input into a classifier-based obstacle identifier to obtain the recognition result, where the recognition result is used to indicate whether the target object in the target object region of interest image is an obstacle.

[0033] The embodiment of the present application further provides an intelligent forklift control method based on smart transportation, which includes:

[0034] Acquire an image in front of the forklift collected by a camera deployed on the forklift;

[0035] Performing target object feature extraction on the image in front of the forklift to obtain a target object feature map;

[0036] Performing pixel-level context semantic association enhancement on the target object feature map to obtain an enhanced target object feature map;

[0037] Determining a recognition result of the target object based on the enhanced target object feature map;

[0038] In response to the recognition result that the target object in the image in front of the forklift is an obstacle, an obstacle avoidance instruction is generated.

[0039] For example, according to an embodiment of the present application, an intelligent forklift control method based on smart transportation, wherein target object feature extraction is performed on the image in front of the forklift to obtain a target object feature map, including:

[0040] Extracting a region of interest from the image in front of the forklift to obtain a region of interest image of the target object;

[0041] Image features of the target object region of interest image are extracted to obtain the target object feature map.

[0042] According to the intelligent forklift control system and method based on smart transportation of the embodiments of the present application, it can effectively enhance the forklift's autonomous navigation and obstacle avoidance capabilities in complex working environments, and improve the safety and efficiency of operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings of the embodiments of the present application. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0044] Figure 1 The following is a schematic diagram showing the application architecture of an intelligent forklift control system based on smart transportation in an embodiment of the present application;

[0045] Figure 2 The figure shows a schematic diagram of the structure of an intelligent forklift control system based on smart transportation in an embodiment of the present application;

[0046] Figure 3 The following is a schematic diagram showing the structure of a target object feature extraction module of an intelligent forklift control system based on smart transportation in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of a semantic association enhancement module of an intelligent forklift control system based on smart transportation in an embodiment of the present application is shown;

[0048] Figure 5 A flow chart showing an intelligent forklift control method based on smart transportation in an embodiment of the present application is shown; and

[0049] Figure 6 The following diagram shows an application scenario of an intelligent forklift control system based on smart transportation in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts also fall within the scope of protection of this application.

[0051] The terms used in this specification are those commonly used in the art currently in consideration of the functions of the present application, but these terms may vary according to the intentions of those skilled in the art, precedents, or new technologies in the art. In addition, specific terms may be selected, and in such cases, their detailed meanings will be described in the detailed description of the present application. Therefore, the terms used in the specification should not be understood as simple names, but rather as the meaning of the terms and the overall description of the present application.

[0052] Although the present application makes various references to certain modules in the system according to embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are illustrative only, and different aspects of the system and method can use different modules.

[0053] Flowcharts are used throughout this application to illustrate the operations performed by the systems of the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps may be processed in reverse order or simultaneously, as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0054] Figure 1 A schematic diagram of the application architecture of an intelligent forklift control system based on smart transportation in an embodiment of the present application is shown, including a server 100 and a terminal device 200.

[0055] The terminal device 200 and the server 100 can be connected via the Internet to enable communication between them. Optionally, the Internet utilizes standard communication technologies and / or protocols. The Internet is typically the Internet, but may also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired, or wireless network, a private network, or any combination of a virtual private network. In some embodiments, technologies and / or formats such as Hypertext Markup Language (HTML) and Extensible Markup Language (XML) are used to represent data exchanged over the network. Conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPN), and Internet Protocol Security (IPsec) may also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies may be used in place of or in addition to the aforementioned data communication technologies.

[0056] Server 100 can provide various network services to terminal device 200. Server 100 can be a single server, a server cluster consisting of multiple servers, or a cloud computing center. Specifically, server 100 may include a processor 110 (Center Processing Unit, CPU), memory 120, input devices 130, and output devices 140. Input devices 130 may include a keyboard, mouse, touch screen, etc. Output devices 140 may include display devices such as a liquid crystal display (LCD) or a cathode ray tube (CRT).

[0057] The memory 120 may include a read-only memory (ROM) and a random access memory (RAM), and provides program instructions and data stored in the memory 120 to the processor 110. In the embodiment of the present application, the memory 120 may be used to store the corresponding method of the intelligent forklift control system based on smart transportation in the embodiment of the present application.

[0058] The processor 110 calls the program instructions stored in the memory 120, and the processor 110 is used to execute the steps of any one of the intelligent forklift control methods based on smart transportation in the embodiments of the present application according to the obtained program instructions.

[0059] In addition, the application architecture diagram in the embodiment of the present application is intended to more clearly illustrate the technical solution in the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Of course, for other application architectures and business applications, the technical solution provided in the embodiment of the present application is also applicable to similar problems.

[0060] The following non-restrictive description of the intelligent forklift control system based on smart transportation provided according to at least one embodiment of the present application is given through several examples or embodiments. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also fall within the scope of protection of this application.

[0061] To address the above technical issues, the technical concept of this application is to use deep learning-based computer vision technology to detect and extract image features from images in front of a forklift to capture the visual feature representation of the target object. This technology is then further enhanced by pixel-level semantic association of the visual feature representation of the target object to improve the accuracy of the feature representation. This allows for intelligent identification of whether the target object is an obstacle, enabling obstacle avoidance decisions to be made. This effectively enhances the forklift's autonomous navigation and obstacle avoidance capabilities in complex operating environments, improving operational safety and efficiency.

[0062] Based on this, Figure 2 The schematic diagram of the structure of an intelligent forklift control system 800 based on smart transportation in an embodiment of the present application is shown. The intelligent forklift control system 800 based on smart transportation includes: a forklift front image acquisition module 810 for acquiring an image in front of the forklift captured by a camera deployed on the forklift; a target object feature extraction module 820 for performing target object feature extraction on the image in front of the forklift to obtain a target object feature map; a semantic association enhancement module 830 for performing pixel-level contextual semantic association enhancement on the target object feature map to obtain an enhanced target object feature map; a target object recognition module 840 for determining a target object recognition result based on the enhanced target object feature map; and an obstacle avoidance module 850 for generating an obstacle avoidance instruction in response to the recognition result that the target object in the image in front of the forklift is an obstacle.

[0063] It should be understood that the core of the forklift front image acquisition module 810 is a camera, which is installed in front of the forklift and is responsible for capturing real-time images of the forklift during its travel. These images provide the system with necessary visual data and are the basis for subsequent processing and analysis. In the target object feature extraction module 820, the system uses image processing technology to identify various target objects in the image, such as pedestrians, other vehicles, cargo, etc. Through algorithms, the system can extract the key features of these target objects from the image, such as shape, color, size, etc., and represent these features in the form of a feature map. The function of the semantic association enhancement module 830 is to further analyze the feature map. It can enhance the features of the target object through contextual semantic analysis. For example, if a pedestrian walks in front of the forklift, the system not only recognizes the pedestrian's features, but also understands the pedestrian's possible movement direction and speed, thereby generating a more accurate enhanced feature map. The target object recognition module 840 uses advanced recognition algorithms, such as deep learning, to analyze the enhanced feature map and determine the specific type of target object in the image. For example, the system can identify pedestrians, vehicles, etc. in the image, and may further identify the type of vehicle or the specific behavior of the pedestrian. Finally, when the object recognition module identifies an obstacle ahead, the obstacle avoidance module 850 generates avoidance instructions based on this information. These instructions may include slowing down, stopping, or changing the driving path to ensure the forklift safely avoids the obstacle and continues its mission. The entire system is designed to achieve intelligent operation of the forklift, reduce human error, and improve operational efficiency and safety. Through the collaborative operation of these modules, the forklift can autonomously navigate complex traffic environments, identify and respond to various situations, thereby enhancing the level of automation in operations.

[0064] Specifically, in the technical solution of this application, an image of the forklift's front is first acquired by a camera deployed on the forklift. It should be understood that the camera, as a visual sensor, can capture real-time environmental information in front of the forklift. Acquiring images in front of the forklift helps to timely understand the position and shape of surrounding objects, providing a basic data source for obstacle identification in the forklift's path.

[0065] Next, considering that in practical applications, directly extracting overall image features from the image in front of the forklift may result in low computational efficiency and low obstacle recognition accuracy due to redundant information in the image. Therefore, this application first uses a region of interest (ROI) recommender based on the R-CNN (Region-based Convolutional Neural Networks) network to process the image in front of the forklift, filter out the region containing the target object from the image in front of the forklift, and generate an image of the target object region of interest. Among them, the R-CNN network can quickly and accurately locate the region of interest (RoI) that may contain the target object in the image by combining the three steps of candidate region generation, feature extraction, and classifier decision, thereby reducing the amount of image data for subsequent processing and improving the overall efficiency and real-time performance of data processing.

[0066] Then, the target object region of interest image is input into the target object feature extractor of the dilated convolutional neural network model to obtain a target object feature map. In other words, in order to more fully extract the deep-level features of the target object, this application introduces a dilated convolutional neural network (DCNN) as a target object feature extractor to perform feature extraction on the target object region of interest image. This fully utilizes the characteristics of dilated convolution to effectively increase the network's receptive field without reducing image resolution, retaining more spatial hierarchical information, thereby obtaining a target object feature map and providing a richer visual feature representation for subsequent obstacle recognition.

[0067] Accordingly, if Figure 3 As shown, the target object feature extraction module 820 includes: a region of interest extraction unit 821, which is used to extract the region of interest of the forklift front image to obtain a target object region of interest image; and a region of interest image feature extraction unit 822, which is used to extract image features of the target object region of interest image to obtain the target object feature map.

[0068] It should be understood that the target object feature extraction module 820 is a key component of the intelligent forklift control system. It is responsible for extracting useful information from the image in front of the forklift for further analysis and processing. The region of interest extraction unit 821 identifies and extracts specific areas in the image relevant to forklift operation. This unit uses image segmentation techniques to determine which parts of the image require special attention during forklift operation, such as possible obstacles, pedestrians, and cargo. By identifying these areas, the system can ignore irrelevant information in the image and focus on areas that directly affect forklift operation. The region of interest image feature extraction unit 822 extracts key visual features from the identified regions of interest. Once the regions of interest are extracted, this unit further analyzes these regions to extract features that are helpful for target object recognition. These features may include edges, textures, colors, shapes, etc. The extracted features are organized into a target object feature map, which provides input for subsequent target object recognition. The collaborative operation of these two units enables the system to more efficiently process image data and focus on information that is practically relevant to forklift operation. In this way, the system can improve the accuracy and efficiency of target object recognition, providing a solid foundation for intelligent forklift control.

[0069] The region of interest extraction unit 821 is configured to input the forklift front image into a region of interest recommender based on an R-CNN network to obtain the target object region of interest image. The region of interest image feature extraction unit 822 is configured to input the target object region of interest image into a target object feature extractor based on a dilated convolutional neural network model to obtain the target object feature map.

[0070] Secondly, considering the complex spatial dependencies and semantic correlations between the pixels in the target object feature map, it may be difficult to accurately capture the local detail features and contextual information of the target object by directly using the target object feature map for obstacle recognition. Therefore, in order to further improve the feature representation capability and strengthen the correlation and discrimination between the features of each pixel in the target object feature map, in this application, first, the target object feature map is feature-dispersed along the channel dimension, and each pixel in the spatial dimension of the target object feature map is split into independent feature representations to obtain the global channel features of each pixel, and generate a set of target object channel local feature vectors. In this way, by decomposing the spatial structure of the target object feature map, the global features are refined to local areas, so as to more carefully capture the local detail information of the target object.

[0071] Furthermore, in order to enhance the intrinsic connection between pixels in the image of the target object's region of interest, fully understand the spatial context information of the image, and improve the accuracy of obstacle recognition, the present application introduces a semantic association enhancement module based on feature energy level metric associated radiation to enhance the set of local feature vectors of the target object channel. Specifically, the semantic association enhancement module quantifies the feature information density of each pixel and its influence on other pixels in the surrounding neighborhood by calculating the feature energy level of each local feature vector of the target object channel, and captures the local dependency relationship between pixels by performing a one-dimensional convolution operation on each feature energy level coefficient, thereby generating attention weights, and performing weighted processing and residual processing on the set of local feature vectors of the target object channel to obtain a set of enhanced local feature vectors of the target object channel. That is, by evaluating the energy level differences and mutual radiation effects between the local feature vectors of the target object channel, the key information in the set is enhanced, while the interference of non-key information is suppressed, and the discriminative ability of feature representation is further improved.

[0072] Next, the set of local feature vectors of the enhanced target object channel is reshaped to obtain an enhanced target object feature map. That is, the set of local feature vectors of the enhanced target object channel is reorganized into a feature map in a feature-dispersed manner, restoring its original spatial structure, making the feature representation of the target object more accurate and comprehensive, thereby obtaining an enhanced target object feature map.

[0073] Accordingly, if Figure 4 As shown, the semantic association enhancement module 830 includes: a pixel-level feature discretization unit 831, which is used to perform feature discretization on the target object feature map along the channel dimension to obtain a set of target object channel local feature vectors; a semantic association enhancement unit 832, which is used to input the set of target object channel local feature vectors into the semantic association enhancement based on feature energy level measurement association radiation to obtain a set of enhanced target object channel local feature vectors; a feature shape reshaping unit 833, which is used to perform feature shape reshaping on the set of enhanced target object channel local feature vectors to obtain the enhanced target object feature map.

[0074] Among them, the semantic association enhancement unit 832 includes: a feature energy level calculation subunit, which is used to calculate the feature energy level coefficients of each target object channel local feature vector in the set of target object channel local feature vectors to obtain a set of feature energy level coefficients; a one-dimensional arrangement subunit, which is used to arrange the set of feature energy level coefficients into a feature energy level coefficient input vector; a neighborhood association coding subunit, which is used to perform one-dimensional convolution coding on the feature energy level coefficient input vector to obtain a feature energy level radiation association vector; a weighting subunit, which is used to perform point convolution coding and probabilistic mapping based on the Sigmoid function on the feature energy level radiation association vector to obtain a feature energy level radiation association weight vector; a weighted enhancement subunit, which is used to use the feature energy level radiation association weight vector as a weight vector to perform weighted processing and residual processing on the set of target object channel local feature vectors to obtain the set of enhanced target object channel local feature vectors.

[0075] Specifically, the characteristic energy level calculation subunit is used to: calculate the weighted sum between the variance of the local feature vector of the target object channel and the bias term to obtain a first intermediate term, and calculate the square of the difference between the maximum value and the mean of the local feature vector of the target object channel, the variance of the local feature vector of the target object channel, and the bias term to obtain a second intermediate term; calculate the first intermediate term divided by the second intermediate term to obtain the characteristic energy level coefficient.

[0076] In a specific example, the semantic association enhancement unit 832 is configured to process the set of target object channel local feature vectors using the following semantic association enhancement formula to obtain the set of enhanced target object channel local feature vectors, wherein the semantic association enhancement formula is: ;in, represents the set of local feature vectors of the target object channel, The value of is the number of feature vectors in the set of local feature vectors of the target object channel, 、 、 and Respectively represent the first, second, and third local feature vectors of the target object channel. and target object channel local feature vector, For the said The feature mean of the local feature vector of the target object channel, For the said The characteristic variance of the local feature vector of the target object channel, Indicates the The maximum value of each eigenvalue in the local eigenvector of the target object channel, is the bias term, 、 、 and Respectively represent the first, second, and third local feature vectors of the target object channel. and The characteristic energy level coefficient of the local characteristic vector of the target object channel, represents the characteristic energy level coefficient input vector, represents a one-dimensional convolution operation, is the scale of the one-dimensional convolution kernel, represents the characteristic energy level radiation correlation vector, represents the point convolution operation, express function, represents the characteristic energy level radiation association weight vector, represents dot product, Represents a set of local feature vectors of the enhanced target object channel.

[0077] The enhanced target object feature map is then input into a classifier-based obstacle identifier to obtain a recognition result, which indicates whether the target object in the target object region of interest image is an obstacle. Specifically, a classifier is used as an obstacle identifier to further analyze and determine whether the target object is an obstacle in the forklift's forward path by analyzing the enhanced target object feature map. Specifically, the obstacle identifier learns and extracts key features from the enhanced target object feature map and maps them into a specific classification space to obtain a recognition result indicating whether the target object is an obstacle. Furthermore, in response to the recognition result that the target object in the target object region of interest image is an obstacle, an obstacle avoidance command is generated to control the forklift to perform the corresponding obstacle avoidance operation, ensuring that the forklift can safely and efficiently avoid obstacles.

[0078] Accordingly, the target object recognition module 840 is configured to input the enhanced target object feature map into a classifier-based obstacle identifier to obtain the recognition result, wherein the recognition result is used to indicate whether the target object in the target object region of interest image is an obstacle.

[0079] It's easy to understand that a classifier uses given categories and known training data to learn classification rules and classifiers, and then classify (or predict) unknown data. Logistic regression and SVM are commonly used to solve binary classification problems. For multi-class classification, logistic regression or SVM can also be used, but multiple binary classifications are required to form a multi-class classification. However, this approach is error-prone and inefficient. A commonly used multi-classification method is the Softmax classification function.

[0080] Here, the present application takes into account that the target object feature map expresses the image semantic features of the target object region of interest image. After the features are discretized along the channel dimension and the distribution energy level metric association is enhanced based on the spatial distribution of the channel-by-channel image semantic features, the enhanced target object feature map obtained will also have an expression offset relative to the source image feature semantics of the target object region of interest image, thereby causing category offset.

[0081] Therefore, the present application hopes to compensate for the category deviation caused by the imbalance of feature correspondence ratio under the fine-grained distribution of image semantics by fusing the target object feature map and the enhanced target object feature map, thereby optimizing the enhanced target object feature map to improve the accuracy of the classification results obtained by the optimized enhanced target object feature map.

[0082] Specifically, the target object recognition module 840 includes: an optimization unit for optimizing the enhanced target object feature map to obtain an optimized enhanced target object feature map; a classification unit for inputting the optimized enhanced target object feature map into the classifier-based obstacle identifier to obtain the recognition result, and the recognition result is used to indicate whether the target object in the target object region of interest image is an obstacle.

[0083] Wherein, the optimization unit is used to: calculate the sum of the eigenvalues of the mean feature maps of the target object feature map and the enhanced target object feature map and the square root of the sum of the squares of the eigenvalues; calculate the weighted sum of the inverse of the square root of the sum of the squares of the eigenvalues and the sum of the eigenvalues, and perform a dot multiplication with the dot addition and summation feature map of the target object feature map and the enhanced target object feature map to obtain a first enhanced target object sub-feature map; perform a dot addition on the dot product feature map of the target object feature map and the enhanced target object feature map and the square root of the scale of the enhanced target object feature map to obtain a second enhanced target object sub-feature map; and calculate the weighted dot addition sum of the first enhanced target object sub-feature map and the second enhanced target object sub-feature map to obtain the optimized enhanced target object feature map.

[0084] In one example, the optimization unit is configured to optimize the enhanced target object feature map using the following optimization formula to obtain the optimized enhanced target object feature map; wherein the optimization formula is expressed as: ;in, and are the target object feature map and the enhanced target object feature map, is the mean feature map of the target object feature map and the enhanced target object feature map, and denote the sum of the eigenvalues and the square root of the sum of the squares of the eigenvalues of the mean feature map, respectively. is the scale of the enhanced target object feature map, i.e., width times height times number of channels, and and is the weighted sum weight as a hyperparameter, Indicates adding by position point, Indicates point multiplication by position, Represents the optimized enhanced target object feature map.

[0085] That is, based on the constrained representation of the structured foreground and background distinction of the supermanifold of the target object feature map and the enhanced target object feature map based on the mean feature map norm between the target object feature map and the enhanced target object feature map, the feature-level key correspondence between the feature maps of the target object feature map and the enhanced target object feature map is modeled, and the global correlation relationship between the corresponding features is adjusted, so as to make positive fine-grained correspondence suggestions by controlling the imbalance ratio between the corresponding feature values of the target object feature map and the enhanced target object feature map, so as to avoid the feature map-level category imbalance of the enhanced target object feature map by focusing on the focus, thereby improving the accuracy of the recognition results obtained by the obstacle identifier based on the classifier of the enhanced target object feature map.

[0086] Based on the above embodiments, see Figure 5 As shown in FIG, a flow chart of an intelligent forklift control method based on smart transportation in an embodiment of the present application. For example, the intelligent forklift control method based on smart transportation can be executed by a server, which can be Figure 1 The server 100 shown in FIG. Figure 5As shown, the intelligent forklift control method based on smart transportation according to the embodiment of the present application includes the following steps: S510, obtaining an image in front of the forklift collected by a camera deployed on the forklift; S520, performing target object feature extraction on the image in front of the forklift to obtain a target object feature map; S530, performing pixel-level context semantic association enhancement on the target object feature map to obtain an enhanced target object feature map; S540, determining the recognition result of the target object based on the enhanced target object feature map; S550, generating an obstacle avoidance instruction in response to the recognition result that the target object in the image in front of the forklift is an obstacle.

[0087] In one example, in step S520, target object features are extracted from the image in front of the forklift to obtain a target object feature map, including: performing region of interest extraction on the image in front of the forklift to obtain a target object region of interest image; and extracting image features of the target object region of interest image to obtain the target object feature map.

[0088] Here, those skilled in the art will understand that the specific operations of each step in the above-mentioned intelligent forklift control method based on smart transportation have been referred to above. Figures 2 to 4 It has been introduced in detail in the description of the intelligent forklift control system 800 based on smart transportation, and therefore, its repeated description will be omitted.

[0089] Figure 6 FIG is an application scenario diagram of an intelligent forklift control system based on smart transportation according to an embodiment of the present application. Figure 6 As shown, in this application scenario, first, an image in front of the forklift captured by a camera deployed on the forklift is obtained (for example, Figure 6 Then, the image in front of the forklift is input to a server (for example, Figure 6 In S) shown in , the server can use the intelligent forklift control algorithm based on smart transportation to process the image in front of the forklift to obtain an identification result indicating whether the target object in the image of the target object's area of interest is an obstacle.

[0090] Based on the above embodiments, the present application also provides another exemplary embodiment of an electronic device. In some possible implementations, the electronic device in the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the intelligent forklift control method based on smart transportation in the above embodiments can be implemented.

[0091] For example, in the case of electronic equipment Figure 1Taking the server 100 in the example for explanation, the processor in the electronic device is the processor 110 in the server 100, and the memory in the electronic device is the memory 120 in the server 100.

[0092] Embodiments of the present application also provide a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed by a processor, the intelligent forklift control method based on smart transportation according to the embodiments of the present application described with reference to the above figures can be executed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, flash memory, etc.

[0093] Embodiments of the present application also provide a computer program product or computer program, which includes computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to implement the intelligent forklift control method based on smart transportation according to an embodiment of the present application.

[0094] Those skilled in the art will appreciate that the contents disclosed in this application may be subject to various modifications and improvements. For example, the various devices or components described above may be implemented through hardware, software, firmware, or a combination of some or all of the three.

[0095] Furthermore, although the present application makes various references to certain units in the system according to embodiments of the present application, any number of different units may be used and run on the client and / or server. The units are illustrative only, and different aspects of the system and method may use different units.

[0096] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software functional modules. This application is not limited to any particular form of combination of hardware and software.

[0097] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or highly formal sense, unless expressly defined as such herein.

[0098] The above is an explanation of the present application and should not be considered as limiting thereof. Although several exemplary embodiments of the present application have been described, those skilled in the art will readily appreciate that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present application.

Claims

1. An intelligent forklift control system based on smart transportation, characterized in that: include: A forklift front image acquisition module is used to acquire an image in front of the forklift collected by a camera deployed on the forklift; a target object feature extraction module, configured to extract target object features from the image in front of the forklift to obtain a target object feature map; A semantic association enhancement module, configured to perform pixel-level contextual semantic association enhancement on the target object feature map to obtain an enhanced target object feature map; a target object recognition module, configured to determine a recognition result of the target object based on the enhanced target object feature map; an obstacle avoidance module, configured to generate an obstacle avoidance instruction in response to the recognition result that the target object in the image in front of the forklift is an obstacle; The semantic association strengthening module includes: a pixel-level feature dispersion unit, configured to disperse the target object feature map along a channel dimension to obtain a set of target object channel local feature vectors; A semantic association enhancement unit, configured to input the set of target object channel local feature vectors into semantic association enhancement based on feature energy level metric association radiation to obtain a set of enhanced target object channel local feature vectors; a feature shape reshaping unit, configured to perform feature shape reshaping on the set of local feature vectors of the enhanced target object channel to obtain the enhanced target object feature map; Wherein, the semantic association strengthening unit includes: a characteristic energy level calculation subunit, configured to calculate a characteristic energy level coefficient of each target object channel local feature vector in the set of target object channel local feature vectors to obtain a set of characteristic energy level coefficients; a one-dimensional arrangement subunit, configured to arrange the set of characteristic energy level coefficients into a characteristic energy level coefficient input vector; a neighborhood correlation coding subunit, configured to perform one-dimensional convolution coding on the characteristic energy level coefficient input vector to obtain a characteristic energy level radiation correlation vector; A weighting subunit, configured to perform point convolution coding and probabilistic mapping based on a Sigmoid function on the characteristic energy level radiation association vector to obtain a characteristic energy level radiation association weight vector; a weighted enhancement subunit, configured to perform weighted processing and residual processing on the set of local feature vectors of the target object channel using the characteristic energy level radiation association weight vector as a weight vector to obtain the set of enhanced local feature vectors of the target object channel; Wherein, the characteristic energy level calculation subunit is used to: Calculating a weighted sum of the variance of the local feature vector of the target object channel and a bias term to obtain a first intermediate term, and calculating a weighted sum of the square of the difference between the maximum value and the mean of the local feature vector of the target object channel, the variance of the local feature vector of the target object channel, and the bias term to obtain a second intermediate term; The first intermediate term is divided by the second intermediate term to obtain the characteristic energy level coefficient.

2. The intelligent forklift control system based on smart transportation according to claim 1 is characterized in that: The target object feature extraction module includes: An area of interest extraction unit, configured to extract an area of interest from the image in front of the forklift to obtain an area of interest image of a target object; The region of interest image feature extraction unit is used to extract image features of the target object region of interest image to obtain the target object feature map.

3. The intelligent forklift control system based on smart transportation according to claim 2 is characterized in that: The region of interest extraction unit is used to: The forklift front image is input into a region of interest recommender based on an R-CNN network to obtain the target object region of interest image.

4. The intelligent forklift control system based on smart transportation according to claim 3 is characterized in that: The region of interest image feature extraction unit is used to: The target object region of interest image is input into a target object feature extractor based on a hole convolutional neural network model to obtain the target object feature map.

5. The intelligent forklift control system based on smart transportation according to claim 4 is characterized in that: The target object recognition module is used to: The enhanced target object feature map is input into a classifier-based obstacle identifier to obtain the recognition result, where the recognition result is used to indicate whether the target object in the target object region of interest image is an obstacle.

6. An intelligent forklift control method based on smart transportation, using the intelligent forklift control system based on smart transportation according to claim 1, characterized in that: include: Acquire an image in front of the forklift collected by a camera deployed on the forklift; Performing target object feature extraction on the image in front of the forklift to obtain a target object feature map; Performing pixel-level context semantic association enhancement on the target object feature map to obtain an enhanced target object feature map; Determining a recognition result of the target object based on the enhanced target object feature map; In response to the recognition result that the target object in the image in front of the forklift is an obstacle, an obstacle avoidance instruction is generated.

7. The intelligent forklift control method based on smart transportation according to claim 6 is characterized in that: Extracting target object features from the image in front of the forklift to obtain a target object feature map includes: Extracting a region of interest from the image in front of the forklift to obtain a region of interest image of the target object; Image features of the target object region of interest image are extracted to obtain the target object feature map.

Images