Scooter obstacle avoidance method and system based on machine vision

Abstract

The invention relates to the field of machine vision, in particular to a forklift obstacle avoidance method and system based on machine vision. Surrounding obstacles are recognized and positioned in real time through a machine vision technology, and the driving direction of the forklift is controlled, so that the driving path is dynamically adjusted, and the obstacles are effectively avoided. According to the invention, rich visual information can be provided through a machine vision technology, and complex and small object target obstacles can be identified more accurately. Compared with a single physical sensor, the machine vision system can ensure that the forklift can better adapt to changes of complex environments, and robustness and flexibility of the system are improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision, and in particular to a method and system for avoiding obstacles for a scraper based on machine vision. Background Art

[0002] In the field of modern engineering and logistics, scrapers are important material handling equipment and are widely used in warehousing, construction, mining and other scenarios. These environments often have a large number of dynamic and static obstacles, such as other mechanical equipment, construction materials, personnel, etc., which pose challenges to the safe operation of scrapers.

[0003] Traditional obstacle avoidance methods for scrapers mainly rely on physical sensors such as ultrasonic sensors and radars to detect obstacles, which have limitations in detection distance, accuracy, and the ability to detect small objects. In addition, traditional obstacle avoidance methods are also easily restricted in complex environments and cannot effectively deal with complex and dynamically changing obstacles.

[0004] Therefore, an optimized obstacle avoidance solution for scraper is needed to solve the technical problem. Summary of the invention

[0005] This application is made in view of the above problems. One object of this application is to provide a method and system for avoiding obstacles for a forklift based on machine vision.

[0006] An embodiment of the present application provides a method for avoiding obstacles for a forklift truck based on machine vision, which includes: Acquire a forward moving image collected by a camera deployed on the loader; Inputting the forward traveling image into an object of interest identifier based on a Yolo model to obtain at least one object of interest image; Performing multi-scale feature extraction on the at least one object of interest image to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest; Inputting the shallow feature map of the object of interest and the structural feature map of the object of interest into a feature joint interaction module based on an attention mechanism to obtain a shallow-structural joint salient perceptual feature map of the object of interest; Based on the shallow-structure joint salient perceptual feature map of the object of interest, it is determined whether the object of interest is an obstacle, and whether a control instruction to adjust the direction of travel is generated.

[0007] For example, according to the machine vision-based shovel loader obstacle avoidance method of the present application, multi-scale feature extraction is performed on the at least one object of interest image to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest, including: inputting the at least one object of interest image into a pyramid network-based object multi-scale feature extractor to obtain the shallow feature map of the object of interest and the structural feature map of the object of interest.

[0008] For example, according to the machine vision-based obstacle avoidance method for a forklift truck of the present application, the shallow feature map of the object of interest and the structural feature map of the object of interest are input into a feature joint interaction module based on an attention mechanism to obtain a shallow-structural joint significant perception feature map of the object of interest, including: Performing feature shape reshaping on the shallow feature map of the object of interest and the structural feature map of the object of interest to obtain a shallow feature shape reshaping matrix of the object of interest and a structural feature shape reshaping matrix of the object of interest; Inputting the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest into a feature channel-by-channel interactive perception module to obtain a shallow-structural feature dependency matrix of the object of interest and a structural-shallow feature dependency matrix of the object of interest; Inputting the shallow-structure feature dependency matrix of the object of interest and the structure-shallow feature dependency matrix of the object of interest into a random dropout module to obtain a pruned optimized shallow-structure feature dependency matrix of the object of interest and a pruned optimized structure-shallow feature dependency matrix of the object of interest; Performing matrix multiplication of the pruning optimization object of interest shallow-structural feature dependency matrix and the object of interest structural feature shape reshaping matrix to obtain a dependency optimization object of interest structure-shallow feature dependency matrix; Multiplying the pruning optimized object of interest structure-shallow feature dependency matrix with the object of interest shallow feature shape reshaping matrix to obtain a dependency optimized object of interest shallow-structure feature dependency matrix; Reshaping the dependency relationship optimization object of interest structure-shallow feature dependency relationship matrix and the dependency relationship optimization object of interest shallow-structural feature dependency relationship matrix to obtain an optimized object of interest shallow feature map and an optimized object of interest structural feature map; A weighted sum of the optimized shallow feature map of the object of interest and the optimized structural feature map of the object of interest is calculated to obtain the shallow-structural joint salient perceptual feature map of the object of interest.

[0009] For example, according to the machine vision-based obstacle avoidance method for a scraper of the present application, the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest are input into a feature channel-by-channel interactive perception module to obtain a shallow-structural feature dependency matrix of the object of interest and a structural-shallow feature dependency matrix of the object of interest, including: Performing matrix multiplication of the transposed matrix of the shallow feature shape reshaping matrix of the object of interest and the shape reshaping matrix of the structural feature of the object of interest, and then dividing the matrix by the scale of the shallow-structural feature dependency matrix of the object of interest by position to obtain the shallow-structural feature interaction matrix of the object of interest; The transposed matrix of the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest are matrix-multiplied, and then the matrix is ​​divided by the scale of the structural feature shape reshaping matrix of the object of interest by position to obtain the structure-shallow feature interaction matrix of the object of interest; The shallow-structural feature interaction matrix of the object of interest and the structural-shallow feature interaction matrix of the object of interest are normalized respectively to obtain the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest.

[0010] For example, according to the machine vision-based shovel loader obstacle avoidance method of the present application, wherein the shallow-structural feature interaction matrix of the object of interest and the structural-shallow feature interaction matrix of the object of interest are respectively normalized to obtain the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest, including: using the Softmax function to perform soft maximum normalization processing on the shallow-structural feature interaction matrix of the object of interest and the structural-shallow feature interaction matrix of the object of interest to obtain the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest.

[0011] For example, according to the machine vision-based obstacle avoidance method for a forklift truck of the present application, determining whether the object of interest is an obstacle based on the shallow-structural joint significant perception feature map of the object of interest, and determining whether to generate a control instruction to adjust the direction of travel include: Inputting the shallow-structural joint salient perceptual feature map of the object of interest into a classifier-based object of interest identifier to obtain a recognition result, wherein the recognition result is used to indicate whether the object of interest is an obstacle; Based on the recognition result, it is determined whether to generate a control instruction for adjusting the traveling direction.

[0012] For example, according to the machine vision-based obstacle avoidance method for a loader of the present application, determining whether to generate a control instruction for adjusting the direction of travel is based on the recognition result, including: in response to the recognition result that the object of interest is an obstacle, generating a control instruction for adjusting the direction of travel.

[0013] The embodiment of the present application also provides a forklift obstacle avoidance system based on machine vision, which includes: An image acquisition module, used to acquire a forward moving image collected by a camera deployed on the scraper; An object of interest recognition module, configured to input the forward traveling image into an object of interest identifier based on a Yolo model to obtain at least one object of interest image; A multi-scale feature extraction module, configured to perform multi-scale feature extraction on the at least one object of interest image to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest; A salient feature joint module, used for inputting the shallow feature map of the object of interest and the structural feature map of the object of interest into a feature joint interaction module based on an attention mechanism to obtain a shallow-structural joint salient perceptual feature map of the object of interest; A control module is used to determine whether the object of interest is an obstacle based on the shallow-structure joint significant perception feature map of the object of interest, and determine whether to generate a control instruction to adjust the direction of travel.

[0014] For example, according to the machine vision-based shovel loader obstacle avoidance system of the present application, the multi-scale feature extraction module is used to: input the at least one object of interest image into the pyramid network-based object multi-scale feature extractor to obtain the shallow feature map of the object of interest and the structural feature map of the object of interest.

[0015] For example, according to the machine vision-based obstacle avoidance system for a forklift truck of the present application, the salient feature combination module is used to: Performing feature shape reshaping on the shallow feature map of the object of interest and the structural feature map of the object of interest to obtain a shallow feature shape reshaping matrix of the object of interest and a structural feature shape reshaping matrix of the object of interest; Inputting the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest into a feature channel-by-channel interactive perception module to obtain a shallow-structural feature dependency matrix of the object of interest and a structural-shallow feature dependency matrix of the object of interest; Inputting the shallow-structure feature dependency matrix of the object of interest and the structure-shallow feature dependency matrix of the object of interest into a random dropout module to obtain a pruned optimized shallow-structure feature dependency matrix of the object of interest and a pruned optimized structure-shallow feature dependency matrix of the object of interest; Performing matrix multiplication of the pruning optimization object of interest shallow-structural feature dependency matrix and the object of interest structural feature shape reshaping matrix to obtain a dependency optimization object of interest structure-shallow feature dependency matrix; Multiplying the pruning optimized object of interest structure-shallow feature dependency matrix with the object of interest shallow feature shape reshaping matrix to obtain a dependency optimized object of interest shallow-structure feature dependency matrix; Reshaping the dependency relationship optimization object of interest structure-shallow feature dependency relationship matrix and the dependency relationship optimization object of interest shallow-structural feature dependency relationship matrix to obtain an optimized object of interest shallow feature map and an optimized object of interest structural feature map; A weighted sum of the optimized shallow feature map of the object of interest and the optimized structural feature map of the object of interest is calculated to obtain the shallow-structural joint salient perceptual feature map of the object of interest.

[0016] The present application has at least the following technical effects: Compared with the prior art, the machine vision-based obstacle avoidance method and system for a shovel loader according to the embodiment of the present application collects the forward moving image through monitoring by the camera deployed on the shovel loader, and introduces an image processing and analysis algorithm based on artificial intelligence and machine vision technology at the back end to analyze the forward moving image, thereby learning and capturing the multi-scale joint feature representation of the target object of interest in the image, so as to effectively identify obstacles, and control the driving direction of the shovel loader based on the recognition results, so as to dynamically adjust the driving path and achieve effective avoidance of obstacles. This vision-based obstacle avoidance method not only achieves more accurate and reliable obstacle detection and avoidance, improves the safety of the shovel loader, but also enhances its adaptability in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0018] Figure 1 A schematic diagram of the application architecture of the method for avoiding obstacles for a forklift truck based on machine vision in an embodiment of the present application is shown; Figure 2 A flow chart of a method for avoiding obstacles for a forklift truck based on machine vision in an embodiment of the present application is shown; Figure 3 A flowchart of sub-step S540 of the method for avoiding obstacles for a forklift truck based on machine vision in an embodiment of the present application is shown; Figure 4A flowchart of sub-step S550 of the method for avoiding obstacles for a forklift truck based on machine vision in an embodiment of the present application is shown; Figure 5 A structural schematic diagram of a forklift obstacle avoidance system based on machine vision in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0020] The terms used in this specification are those common terms currently widely used in the art in consideration of the functions of the present application, but these terms may vary according to the intentions of those skilled in the art, precedents, or new technologies in the art. In addition, specific terms may be selected, and in this case, their detailed meanings will be described in the detailed description of the present application. Therefore, the terms used in the specification should not be understood as simple names, but rather as a general description based on the meaning of the terms and the present application.

[0021] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0022] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0023] Figure 1 A schematic diagram of the application architecture of a method for avoiding obstacles for a loader based on machine vision in an embodiment of the present application is shown, including a server 100 and a terminal device 200.

[0024] The terminal device 200 and the server 100 can be connected via the Internet to achieve mutual communication. Optionally, the above-mentioned Internet uses standard communication technology and / or protocol. The Internet is usually the Internet, but it can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network. In some embodiments, the data exchanged through the network can be represented by technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0025] The server 100 can provide various network services for the terminal device 200, wherein the server 100 can be a single server, a server cluster consisting of several servers, or a cloud computing center. Specifically, the server 100 may include a processor 110 (Center Processing Unit, CPU), a memory 120, an input device 130, and an output device 140, etc. The input device 130 may include a keyboard, a mouse, a touch screen, etc., and the output device 140 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0026] The memory 120 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 110 with program instructions and data stored in the memory 120. In the embodiment of the present application, the memory 120 may be used to store the program of the machine vision-based forklift obstacle avoidance method in the embodiment of the present application.

[0027] The processor 110 calls the program instructions stored in the memory 120, and the processor 110 is used to execute the steps of any one of the machine vision-based forklift obstacle avoidance methods in the embodiments of the present application according to the obtained program instructions.

[0028] In addition, the application architecture diagram in the embodiment of the present application is intended to more clearly illustrate the technical solution in the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Of course, for other application architectures and business applications, the technical solution provided in the embodiment of the present application is also applicable to similar problems.

[0029] The following is a non-restrictive description of the machine vision-based obstacle avoidance method for a shovel loader provided according to at least one embodiment of the present application through several examples or embodiments. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also fall within the scope of protection of the present application.

[0030] In response to the above technical problems, in the technical solution of this application, a method for shovel loader obstacle avoidance based on machine vision is proposed, which can use machine vision technology to identify and locate surrounding obstacles in real time, and control the driving direction of the shovel loader, so as to dynamically adjust the driving path, thereby achieving effective avoidance of obstacles. This application can provide rich visual information through machine vision technology, and identify complex and small object target obstacles more intelligently and accurately. Compared with a single physical sensor, the machine vision system can ensure that the shovel loader can better adapt to changes in complex environments and improve the robustness and flexibility of the system.

[0031] Specifically, the technical concept of this application is to monitor and collect the forward images through the camera deployed on the loader, and introduce the image processing and analysis algorithm based on artificial intelligence and machine vision technology in the back end to analyze the forward images, so as to learn and capture the multi-scale joint feature representation of the target object of interest in the image, so as to effectively identify obstacles, and control the driving direction of the loader based on the recognition results, so as to dynamically adjust the driving path and achieve effective avoidance of obstacles. This vision-based obstacle avoidance method not only achieves more accurate and reliable obstacle detection and avoidance, improves the safety of the loader, but also enhances its adaptability in complex environments.

[0032] Figure 2 The flowchart of the method for avoiding obstacles for a forklift truck based on machine vision in the embodiment of the present application is shown. For example, the method for avoiding obstacles for a forklift truck based on machine vision can be executed by a server, which can be Figure 1 The server 100 shown in FIG. Figure 2As shown, according to the machine vision-based obstacle avoidance method for a shovel loader according to an embodiment of the present application, the method includes the following steps: S510, obtaining a forward moving image collected by a camera deployed on the shovel loader; S520, inputting the forward moving image into an object of interest identifier based on a Yolo model to obtain at least one object of interest image; S530, performing multi-scale feature extraction on the at least one object of interest image to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest; S540, inputting the shallow feature map of the object of interest and the structural feature map of the object of interest into a feature joint interaction module based on an attention mechanism to obtain a shallow-structure joint significant perception feature map of the object of interest; S550, determining whether the object of interest is an obstacle based on the shallow-structure joint significant perception feature map of the object of interest, and determining whether to generate a control instruction to adjust the direction of travel.

[0033] Specifically, in the technical solution of the present application, first, a forward moving image collected by a camera deployed on a shovel loader is obtained. Then, the forward moving image is input into an object of interest identifier based on the Yolo model for processing, so as to quickly identify objects of interest (such as pedestrians, other vehicles and static obstacles), and provide a basis for subsequent feature extraction and classification. Among them, the Yolo model can complete the target detection task in a single reasoning process without the need for multiple stages of processing, so it is very suitable for application scenarios that require real-time feedback, such as autonomous driving vehicles, automatically navigated shovel loader, etc. At the same time, the Yolo model can simultaneously predict all bounding boxes and category probabilities by performing a forward propagation on a single image, which means that it can complete the detection process very quickly, which is very important for fast-moving shovel loader. Therefore, through the processing of the object of interest identifier based on the Yolo model, one or more objects of interest in the forward moving image can be identified and monitored, thereby obtaining at least one object of interest image.

[0034] Then, the at least one object of interest image is input into the pyramid network-based object multi-scale feature extractor to obtain the object of interest shallow feature map and the object of interest structural feature map. Through the processing of the pyramid network-based object multi-scale feature extractor, feature information of different levels and scales in the at least one object of interest image can be extracted respectively; wherein the object of interest shallow feature map contains shallow features such as edges, colors and textures of the object of interest in the image, and the object of interest shallow feature map contains more detailed information (corresponding to smaller objects). The object of interest structural feature map contains more abstract features and coarser contextual information (corresponding to larger objects), which can characterize the overall shape, structure and category information of the object. Structural features are critical for identifying specific object categories because they contain higher-level semantic information, which helps in subsequent obstacle recognition and loader obstacle avoidance control.

[0035] Accordingly, in step S530, multi-scale feature extraction is performed on the at least one object of interest image to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest, including: inputting the at least one object of interest image into an object multi-scale feature extractor based on a pyramid network to obtain the shallow feature map of the object of interest and the structural feature map of the object of interest.

[0036] Furthermore, considering that the shallow feature map of the object of interest and the structural feature map of the object of interest respectively contain feature information of different levels and different aspects about the object of interest. The shallow feature map of the object of interest emphasizes the shallow features such as texture and edge that are more obvious in the object of interest, while the structural feature map focuses more on the structural features and semantic information that are more critical for identifying the object of interest. Therefore, in order to effectively fuse the shallow features and structural features of the object of interest, so as to better perceive and understand the object of interest, so as to accurately identify obstacles and effectively control the direction of travel of the loader, in the technical solution of the present application, the shallow feature map of the object of interest and the structural feature map of the object of interest are further input into the feature joint interaction module based on the attention mechanism to obtain the shallow-structure joint significant perception feature map of the object of interest. Among them, the feature joint interaction module based on the attention mechanism is a feature fusion technology, which enhances the interaction between the shallow feature map of the object of interest and the structural feature map of the object of interest through multiple steps, and finally obtains a shallow-structure joint significant perception feature map of the object of interest that combines the advantages of both.

[0037] Specifically, first, the feature matrix along the channel dimension in the input shallow feature map of the object of interest and the structural feature map of the object of interest is expanded into a feature vector to facilitate the subsequent channel-by-channel interactive perception operation. Then, the dependency relationship between the two sets of feature vectors of the shallow feature and the structural feature of the object of interest is calculated to obtain the shallow-structural feature and the structural-shallow feature dependency matrix of the object of interest. Then, the dependency matrix is ​​sparsely processed by pruning operation to enhance the generalization ability of the model. Then, the pruned dependency matrix is ​​used to weight the original shallow and structural features of the object of interest, strengthen the features of the most relevant parts of the perception and recognition tasks of the object of interest, and suppress unimportant features. Finally, the optimized feature matrix is ​​reshaped into a feature map form, and the weighted sum is calculated to obtain the shallow-structural joint significant perception feature map of the object of interest. In this way, the intuitive shallow information in the shallow salient feature map of the object of interest and the structural semantic information in the structural feature map can be fully utilized to form a more comprehensive and rich feature representation. In addition, the feature joint interaction module based on the attention mechanism can strengthen the key information in the shallow feature map of the object of interest and the structural feature map of the object of interest through the dependency matrix, and remove redundant parts at the same time, further improve the feature quality, and improve the model's sensitivity and generalization ability to input data, so that the model is more accurate and efficient when performing obstacle recognition and detection and shovel loader direction control and obstacle avoidance tasks.

[0038] Accordingly, in step S540, if Figure 3As shown, the shallow feature map of the object of interest and the structural feature map of the object of interest are input into a feature joint interaction module based on an attention mechanism to obtain a shallow-structure joint significant perception feature map of the object of interest, including: S541, reshaping the shallow feature map of the object of interest and the structural feature map of the object of interest to obtain a shallow feature shape reshaping matrix of the object of interest and a structural feature shape reshaping matrix of the object of interest; S542, inputting the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest into a feature channel-by-channel interaction perception module to obtain a shallow-structure feature dependency matrix of the object of interest and a structure-shallow feature dependency matrix of the object of interest; S543, inputting the shallow-structure feature dependency matrix of the object of interest and the structure-shallow feature dependency matrix of the object of interest into a random dropout module to obtain a pruning optimization shallow-structure feature dependency matrix of the object of interest and a pruning optimization structure matrix of the object of interest. S544, matrix multiplying the pruned optimized object of interest shallow-structural feature dependency matrix with the object of interest structural feature shape reshaping matrix to obtain the dependency optimized object of interest structure-shallow feature dependency matrix; S545, multiplying the pruned optimized object of interest structure-shallow feature dependency matrix with the object of interest shallow feature shape reshaping matrix to obtain the dependency optimized object of interest shallow-structural feature dependency matrix; S546, reshaping the dependency optimized object of interest structure-shallow feature dependency matrix and the dependency optimized object of interest shallow-structural feature dependency matrix to obtain an optimized object of interest shallow feature map and an optimized object of interest structural feature map; S547, calculating the weighted sum of the optimized object of interest shallow feature map and the optimized object of interest structural feature map to obtain the object of interest shallow-structural joint salient perceptual feature map.

[0039] Among them, in step S542, the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest are input into the feature channel-by-channel interaction perception module to obtain the shallow-structural feature dependency matrix of the object of interest and the structure-shallow feature dependency matrix of the object of interest, including: performing matrix multiplication on the transposed matrix of the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest, and then dividing the matrix by the scale of the shallow-structural feature dependency matrix of the object of interest by position to obtain the shallow-structural feature interaction matrix of the object of interest; performing matrix multiplication on the transposed matrix of the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest, and then dividing the matrix by the scale of the structural feature shape reshaping matrix of the object of interest by position to obtain the structure-shallow feature interaction matrix of the object of interest; and normalizing the shallow-structural feature interaction matrix of the object of interest and the structure-shallow feature interaction matrix of the object of interest respectively to obtain the shallow-structural feature dependency matrix of the object of interest and the structure-shallow feature dependency matrix of the object of interest.

[0040] Specifically, the shallow-structural feature interaction matrix of the object of interest and the structural-shallow feature interaction matrix of the object of interest are respectively normalized to obtain the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest, including: using a Softmax function to perform soft maximum normalization processing on the shallow-structural feature interaction matrix of the object of interest and the structural-shallow feature interaction matrix of the object of interest to obtain the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest.

[0041] In one example, the shallow feature map of the object of interest and the structural feature map of the object of interest are input into a feature joint interaction module based on an attention mechanism to obtain a shallow-structure joint significant perception feature map of the object of interest, including: inputting the shallow feature map of the object of interest and the structural feature map of the object of interest into the feature joint interaction module based on the attention mechanism to process them with the following significant feature joint perception formula to obtain a shallow-structure joint significant perception feature map of the object of interest; wherein the significant feature joint perception formula is: in, is the shallow feature map of the object of interest, is the structural feature map of the object of interest, represents the feature shape reshaping process, and are the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest, T represents transposition, is the scale of the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest, for function, is matrix multiplication, and They are respectively the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest, represents pruning optimization processing, and They are respectively pruning optimization of the shallow-structural feature dependency matrix of the object of interest and pruning optimization of the structural-shallow feature dependency matrix of the object of interest. and The dependency relationship optimizes the shallow-structural feature dependency matrix of the object of interest and the dependency relationship optimizes the structural-shallow feature dependency matrix of the object of interest, respectively. and They are respectively used to optimize the shallow feature map of the object of interest and the structural feature map of the object of interest. and are all weighted hyperparameters, It is a shallow-structure joint salient perceptual feature map of the object of interest.

[0042] Then, the shallow-structural joint significant perceptual feature map of the object of interest is input into the classifier-based object of interest identifier to obtain a recognition result, and the recognition result is used to indicate whether the object of interest is an obstacle. That is, the joint significant perceptual feature information between the shallow features and structural features of the object of interest is used for classification processing, so as to effectively judge whether the object of interest is an obstacle. Furthermore, in response to the recognition result that the object of interest is an obstacle, a control instruction for adjusting the direction of travel is generated, so as to dynamically adjust the driving path and achieve effective avoidance of obstacles. This vision-based obstacle avoidance method not only achieves more accurate and reliable obstacle detection and avoidance, improves the safety of the scraper, but also enhances its adaptability in complex environments.

[0043] Accordingly, in step S550, if Figure 4As shown, based on the shallow-structure joint salient perceptual feature map of the object of interest, determining whether the object of interest is an obstacle and determining whether to generate a control instruction to adjust the direction of travel include: S551, inputting the shallow-structure joint salient perceptual feature map of the object of interest into a classifier-based object of interest identifier to obtain a recognition result, and the recognition result is used to indicate whether the object of interest is an obstacle; S552, based on the recognition result, determining whether to generate a control instruction to adjust the direction of travel.

[0044] Wherein, in step S552, based on the recognition result, determining whether to generate a control instruction for adjusting the traveling direction includes: in response to the recognition result that the object of interest is an obstacle, generating a control instruction for adjusting the traveling direction.

[0045] Specifically, the role of the classifier is to use the given categories and known training data to learn classification rules and classifiers, and then classify (or predict) unknown data. Logistic regression, SVM, etc. are often used to solve binary classification problems. For multi-class classification problems, logistic regression or SVM can also be used, but multiple binary classifications are required to form a multi-classification. However, this is prone to errors and is inefficient. Commonly used multi-classification methods include the Softmax classification function.

[0046] In a preferred example, considering that the shallow feature map of the object of interest and the structural feature map of the object of interest respectively represent the cross-scale and cross-depth image semantic features of the image of the object of interest based on the pyramid network, therefore, when performing attention-enhanced salient feature joint perception, the attention weight difference caused by the difference in the expression of cross-scale and cross-depth image semantic features will lead to the diversified set expression distribution of the image semantic features of the shallow-structural joint salient perception feature map of the object of interest. Therefore, it is expected to improve the long-range class regression understanding ability of the shallow-structural joint salient perception feature map of the object of interest when the classification regression is performed by the classifier-based object of interest identifier, thereby improving the accuracy of the recognition result.

[0047] Based on this, in the preferred example, inputting the shallow-structural joint salient perceptual feature map of the object of interest into a classifier-based object of interest identifier to obtain a recognition result specifically includes: Determine the number of zero eigenvalues ​​in the shallow-structure joint salient perceptual feature map of the object of interest, and respectively calculate the reciprocal of the logarithm of the number of zero eigenvalues ​​with base two and the exponential value of the reciprocal of the number of zero eigenvalues ​​with base natural constants to obtain a first shallow-structure joint salient perceptual representation value of the object of interest and a second shallow-structure joint salient perceptual representation value of the object of interest; Expanding the shallow-structure joint salient perceptual feature map of the object of interest into a shallow-structure joint salient perceptual feature vector of the object of interest; Calculate a power function of each eigenvalue of the shallow-structure joint salient perceptual eigenvector of the object of interest with the inverse of the number of zero eigenvalues ​​as an exponent to obtain a shallow-structure joint salient perceptual structural state vector of the object of interest; Performing a dot multiplication of the shallow-structure joint salient perceptual structure state vector of the object of interest and the first shallow-structure joint salient perceptual representation value of the object of interest to obtain a shallow-structure joint salient perceptual information representation vector of the object of interest; Performing dot multiplication of the autocorrelation matrix of the shallow-structure joint salient perception structure state vector of the object of interest with the shallow-structure joint salient perception representation value and the weight hyperparameter of the second object of interest respectively to obtain a shallow-structure joint salient perception regression understanding matrix of the object of interest; Performing matrix multiplication of the shallow-structure joint salient perception information representation vector of the object of interest and the shallow-structure joint salient perception regression understanding matrix of the object of interest to obtain an optimized shallow-structure joint salient perception feature vector of the object of interest; The optimized shallow-structure joint salient perceptual feature vector of the object of interest is input into the classifier-based object of interest identifier to obtain the recognition result.

[0048] That is, the shallow-structure joint salient perceptual representation value of the first object of interest and the shallow-structure joint salient perceptual representation value of the second object of interest are expressed as: in, is the number of zero eigenvalues ​​in the shallow-structural joint salient perceptual eigenvector of the object of interest, is the shallow-structure joint salient perceptual representation value of the first object of interest, is the shallow-structure joint salient perceptual representation value of the second object of interest, It is an exponential function with a natural constant as its base.

[0049] And the optimized shallow-structure joint salient perceptual feature vector of the object of interest It is expressed as: in, is the shallow-structural joint salient perceptual feature vector of the object of interest, is the weight hyperparameter, is the shallow-structure joint salient perceptual regression understanding matrix of the object of interest, is the shallow-structure joint salient perceptual information representation vector of the object of interest, is the number of zero eigenvalues ​​in the shallow-structural joint salient perceptual eigenvector of the object of interest, is the shallow-structure joint salient perceptual representation value of the first object of interest, is the shallow-structure joint salient perceptual representation value of the second object of interest, represents a power function in which each eigenvalue of the shallow-structural joint salient perceptual eigenvector of the object of interest is calculated with the inverse of the number of zero eigenvalues ​​as the exponent, represents the transpose of a vector, It is the point multiplication by position. is matrix multiplication, is the optimized shallow-structure joint salient perceptual feature vector of the object of interest.

[0050] Based on this, the present application realizes effective long-range modeling of the shallow-structure joint significant perception feature map of the object of interest under the field linear complexity by modeling the vector field zero dimension of the shallow-structure joint significant perception feature vector of the object of interest obtained after the shallow-structure joint significant perception feature map of the object of interest is expanded as the global field redundant dependency, so as to perform structured correlation self-distillation between information representation and class understanding in the selective state space based on feature distribution in the feature regression process of the shallow-structure joint significant perception feature map of the object of interest, thereby maintaining the overall information complexity of the feature distribution of the shallow-structure joint significant perception feature map of the object of interest while promoting the long-range dynamic class understanding balance of its feature convergence, so as to improve the class understanding consistency between the classification regression and feature extraction processes, improve the accuracy of the recognition result obtained by the shallow-structure joint significant perception feature map of the object of interest input into the classifier-based object of interest identifier, and can more accurately identify obstacles effectively, and control the driving direction of the shovel loader based on the recognition result, so as to dynamically adjust the driving path and achieve effective avoidance of obstacles.

[0051] Based on the above embodiments, see Figure 5As shown, it is a structural schematic diagram of a machine vision-based obstacle avoidance system 800 for a loader in an embodiment of the present application. The machine vision-based obstacle avoidance system 800 for a loader includes: an image acquisition module 810, which is used to acquire a forward moving image collected by a camera deployed on the loader; an object of interest recognition module 820, which is used to input the forward moving image into an object of interest recognizer based on a Yolo model to obtain at least one object of interest image; a multi-scale feature extraction module 830, which is used to perform multi-scale feature extraction on the at least one object of interest image to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest; a significant feature combination module 840, which is used to input the shallow feature map of the object of interest and the structural feature map of the object of interest into a feature combination interaction module based on an attention mechanism to obtain a shallow-structure joint significant perception feature map of the object of interest; and a control module 850, which is used to determine whether the object of interest is an obstacle based on the shallow-structure joint significant perception feature map of the object of interest, and determine whether to generate a control instruction to adjust the direction of travel.

[0052] In one example, in the above-mentioned machine vision-based loader obstacle avoidance system 800, the multi-scale feature extraction module 830 is used to: input the at least one object of interest image into the pyramid network-based object multi-scale feature extractor to obtain the shallow feature map of the object of interest and the structural feature map of the object of interest.

[0053] In one example, in the above-mentioned machine vision-based loader obstacle avoidance system 800, the salient feature combination module 840 is used to: perform feature shape reshaping on the shallow feature map of the object of interest and the structural feature map of the object of interest to obtain a shallow feature shape reshaping matrix of the object of interest and a structural feature shape reshaping matrix of the object of interest; input the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest into the feature channel-by-channel interactive perception module to obtain a shallow-structural feature dependency matrix of the object of interest and a structural-shallow feature dependency matrix of the object of interest; input the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest into the random dropout module to obtain a pruned optimized shallow-structural feature dependency matrix of the object of interest and a pruned optimized structural-shallow feature dependency matrix of the object of interest. Matrix; matrix multiply the pruned optimized object of interest shallow-structural feature dependency matrix with the object of interest structural feature shape reshaping matrix to obtain the dependency optimized object of interest structure-shallow feature dependency matrix; multiply the pruned optimized object of interest structure-shallow feature dependency matrix with the object of interest shallow feature shape reshaping matrix to obtain the dependency optimized object of interest shallow-structural feature dependency matrix; perform feature shape reshaping on the dependency optimized object of interest structure-shallow feature dependency matrix and the dependency optimized object of interest shallow-structural feature dependency matrix to obtain an optimized object of interest shallow feature map and an optimized object of interest structural feature map; calculate the weighted sum of the optimized object of interest shallow feature map and the optimized object of interest structural feature map to obtain the object of interest shallow-structural joint significant perceptual feature map.

[0054] Specifically, the specific functions and operations of each module in the above-mentioned machine vision-based forklift obstacle avoidance system 800 have been referred to above. Figures 2 to 4 The invention has been introduced in detail in the description of the machine vision-based obstacle avoidance method for a loader, and therefore, its repeated description will be omitted.

[0055] Based on the above embodiment, another exemplary embodiment of an electronic device is also provided in the embodiments of the present application. In some possible implementations, the electronic device in the embodiments of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the steps of the machine vision-based forklift obstacle avoidance method in the above embodiment when executing the program.

[0056] For example, in the case of electronic equipment Figure 1Taking the server 100 in the example as an example, the processor in the electronic device is the processor 110 in the server 100, and the memory in the electronic device is the memory 120 in the server 100.

[0057] The embodiment of the present application also provides a computer-readable storage medium, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the obstacle avoidance method for a forklift based on machine vision according to the embodiment of the present application described with reference to the above figures can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0058] The embodiment of the present application also provides a computer program product or a computer program, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the machine vision-based forklift obstacle avoidance method according to the embodiment of the present application.

[0059] Those skilled in the art will appreciate that the contents disclosed in this application may be subject to various variations and improvements. For example, the various devices or components described above may be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0060] In addition, although the present application makes various references to certain units in the system according to embodiments of the present application, any number of different units can be used and run on the client and / or server. The units are only illustrative, and different aspects of the system and method can use different units.

[0061] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present application is not limited to any particular form of combination of hardware and software.

[0062] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0063] The above is an explanation of the present application and should not be considered as limiting thereof. Although several exemplary embodiments of the present application have been described, those skilled in the art will readily appreciate that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present application.

Claims

1. A method for avoiding obstacles for a scraper based on machine vision, characterized in that: include: Acquire a forward moving image collected by a camera deployed on the loader; Inputting the forward traveling image into an object of interest identifier based on a Yolo model to obtain at least one object of interest image; Performing multi-scale feature extraction on the at least one object of interest image to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest; Inputting the shallow feature map of the object of interest and the structural feature map of the object of interest into a feature joint interaction module based on an attention mechanism to obtain a shallow-structural joint salient perceptual feature map of the object of interest; Based on the shallow-structure joint salient perceptual feature map of the object of interest, it is determined whether the object of interest is an obstacle, and whether a control instruction to adjust the direction of travel is generated.

2. The method for avoiding obstacles for a forklift truck based on machine vision according to claim 1, characterized in that: Performing multi-scale feature extraction on the at least one object of interest image to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest, comprising: inputting the at least one object of interest image into an object multi-scale feature extractor based on a pyramid network to obtain the shallow feature map of the object of interest and the structural feature map of the object of interest.

3. The method for avoiding obstacles for a forklift truck based on machine vision according to claim 2, characterized in that: Inputting the shallow feature map of the object of interest and the structural feature map of the object of interest into a feature joint interaction module based on an attention mechanism to obtain a shallow-structural joint salient perceptual feature map of the object of interest, including: Performing feature shape reshaping on the shallow feature map of the object of interest and the structural feature map of the object of interest to obtain a shallow feature shape reshaping matrix of the object of interest and a structural feature shape reshaping matrix of the object of interest; Inputting the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest into a feature channel-by-channel interactive perception module to obtain a shallow-structural feature dependency matrix of the object of interest and a structural-shallow feature dependency matrix of the object of interest; Inputting the shallow-structure feature dependency matrix of the object of interest and the structure-shallow feature dependency matrix of the object of interest into a random dropout module to obtain a pruned optimized shallow-structure feature dependency matrix of the object of interest and a pruned optimized structure-shallow feature dependency matrix of the object of interest; Performing matrix multiplication of the pruning optimization object of interest shallow-structural feature dependency matrix and the object of interest structural feature shape reshaping matrix to obtain a dependency optimization object of interest structure-shallow feature dependency matrix; Multiplying the pruning optimized object of interest structure-shallow feature dependency matrix with the object of interest shallow feature shape reshaping matrix to obtain a dependency optimized object of interest shallow-structure feature dependency matrix; Reshaping the dependency relationship optimization object of interest structure-shallow feature dependency relationship matrix and the dependency relationship optimization object of interest shallow-structural feature dependency relationship matrix to obtain an optimized object of interest shallow feature map and an optimized object of interest structural feature map; A weighted sum of the optimized shallow feature map of the object of interest and the optimized structural feature map of the object of interest is calculated to obtain the shallow-structural joint salient perceptual feature map of the object of interest.

4. The method for avoiding obstacles for a forklift truck based on machine vision according to claim 3, characterized in that: Inputting the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest into a feature channel-by-channel interactive perception module to obtain a shallow-structural feature dependency matrix of the object of interest and a structural-shallow feature dependency matrix of the object of interest, including: Performing matrix multiplication of the transposed matrix of the shallow feature shape reshaping matrix of the object of interest and the shape reshaping matrix of the structural feature of the object of interest, and then dividing the matrix by the scale of the shallow-structural feature dependency matrix of the object of interest by position to obtain the shallow-structural feature interaction matrix of the object of interest; The transposed matrix of the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest are matrix-multiplied, and then the matrix is ​​divided by the scale of the structural feature shape reshaping matrix of the object of interest by position to obtain the structure-shallow feature interaction matrix of the object of interest; The shallow-structural feature interaction matrix of the object of interest and the structural-shallow feature interaction matrix of the object of interest are normalized respectively to obtain the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest.

5. The method for avoiding obstacles for a forklift truck based on machine vision according to claim 4, characterized in that: The shallow-structural feature interaction matrix of the object of interest and the structural-shallow feature interaction matrix of the object of interest are respectively normalized to obtain the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest, including: using a Softmax function to perform soft maximum normalization processing on the shallow-structural feature interaction matrix of the object of interest and the structural-shallow feature interaction matrix of the object of interest to obtain the shallow-structural feature dependency matrix of the object of interest and the structural-shallow feature dependency matrix of the object of interest.

6. The method for avoiding obstacles for a forklift truck based on machine vision according to claim 5, characterized in that: Based on the shallow-structure joint salient perception feature map of the object of interest, determining whether the object of interest is an obstacle, and determining whether to generate a control instruction to adjust the direction of travel, including: Inputting the shallow-structural joint salient perceptual feature map of the object of interest into a classifier-based object of interest identifier to obtain a recognition result, wherein the recognition result is used to indicate whether the object of interest is an obstacle; Based on the recognition result, it is determined whether to generate a control instruction for adjusting the traveling direction.

7. The method for avoiding obstacles for a forklift truck based on machine vision according to claim 6, characterized in that: Based on the recognition result, determining whether to generate a control instruction for adjusting the traveling direction includes: in response to the recognition result that the object of interest is an obstacle, generating a control instruction for adjusting the traveling direction.

8. The obstacle avoidance system for scraper based on machine vision is characterized by: include: An image acquisition module, used to acquire a forward moving image collected by a camera deployed on the scraper; An object of interest recognition module, configured to input the forward traveling image into an object of interest identifier based on a Yolo model to obtain at least one object of interest image; A multi-scale feature extraction module, configured to perform multi-scale feature extraction on the at least one object of interest image to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest; A salient feature joint module, used for inputting the shallow feature map of the object of interest and the structural feature map of the object of interest into a feature joint interaction module based on an attention mechanism to obtain a shallow-structural joint salient perceptual feature map of the object of interest; A control module is used to determine whether the object of interest is an obstacle based on the shallow-structure joint significant perception feature map of the object of interest, and determine whether to generate a control instruction to adjust the direction of travel.

9. The machine vision-based obstacle avoidance system for scraper according to claim 8, characterized in that: The multi-scale feature extraction module is used to: input the at least one object of interest image into a pyramid network-based object multi-scale feature extractor to obtain a shallow feature map of the object of interest and a structural feature map of the object of interest.

10. The machine vision-based obstacle avoidance system for a scrap loader according to claim 9, characterized in that: The salient feature combination module is used to: Performing feature shape reshaping on the shallow feature map of the object of interest and the structural feature map of the object of interest to obtain a shallow feature shape reshaping matrix of the object of interest and a structural feature shape reshaping matrix of the object of interest; Inputting the shallow feature shape reshaping matrix of the object of interest and the structural feature shape reshaping matrix of the object of interest into a feature channel-by-channel interactive perception module to obtain a shallow-structural feature dependency matrix of the object of interest and a structural-shallow feature dependency matrix of the object of interest; Inputting the shallow-structure feature dependency matrix of the object of interest and the structure-shallow feature dependency matrix of the object of interest into a random dropout module to obtain a pruned optimized shallow-structure feature dependency matrix of the object of interest and a pruned optimized structure-shallow feature dependency matrix of the object of interest; Performing matrix multiplication of the pruning optimization object of interest shallow-structural feature dependency matrix and the object of interest structural feature shape reshaping matrix to obtain a dependency optimization object of interest structure-shallow feature dependency matrix; Multiplying the pruning optimized object of interest structure-shallow feature dependency matrix with the object of interest shallow feature shape reshaping matrix to obtain a dependency optimized object of interest shallow-structure feature dependency matrix; Reshaping the dependency relationship optimization object of interest structure-shallow feature dependency relationship matrix and the dependency relationship optimization object of interest shallow-structural feature dependency relationship matrix to obtain an optimized object of interest shallow feature map and an optimized object of interest structural feature map; A weighted sum of the optimized shallow feature map of the object of interest and the optimized structural feature map of the object of interest is calculated to obtain the shallow-structural joint salient perceptual feature map of the object of interest.

Images