Target detection method for forklift operation environment based on video surveillance technology

Abstract

The present application provides a target detection method for a forklift operating environment based on video surveillance technology, which includes: acquiring multiple forklift operating environment images taken by multiple cameras installed at different positions on a forklift; processing each of the forklift operating environment images based on a trained real-time target detection model to detect targets existing in the forklift operating environment in real time; tracking the position of the target in real time to obtain real-time position coordinates, and comparing the real-time position coordinates with a set forklift working safety area to determine whether the existence of the target will bring safety risks to the safe operation of the forklift, thereby enhancing the safety performance of the forklift and ensuring the safe and efficient conduct of industrial production.

Description

Technical Field

[0001] The present application relates to the field of safety technology, and in particular to a target detection method for a forklift operating environment based on video monitoring technology. Background Art

[0002] As an indispensable key equipment in warehousing and logistics, forklifts are particularly suitable for the operation requirements of indoor factory environments. However, compared with conventional cars, forklifts are usually designed with only forward, reverse and braking functions, and their steering performance is relatively limited. In addition, the complexity of the internal environment of the factory further increases the risk of forklift operation. The complex corners formed by shelves and various equipment, as well as the blind spots in the driving process, especially under load, may induce emergencies.

[0003] It is particularly noteworthy that if the forklift is driven by an inexperienced logistics personnel, their bad driving habits may not be able to effectively deal with emergencies, which may cause the forklift weighing about 9,000 pounds to tip over or lose control, causing incalculable losses. Therefore, innovative technologies are urgently needed to achieve target detection of the forklift operating environment to enhance the safety performance of the forklift and ensure safe and efficient industrial production. Summary of the invention

[0004] The purpose of this application is to propose a target detection method for a forklift operating environment based on video surveillance technology, which is used to solve or alleviate the technical problems existing in the prior art.

[0005] The embodiment of the present application provides a target detection method for a forklift operating environment based on video monitoring technology, which includes:

[0006] Acquire multiple images of the forklift working environment captured by multiple cameras installed at different positions on the forklift;

[0007] Based on the trained real-time target detection model, each of the forklift operating environment images is processed to detect targets existing in the forklift operating environment in real time;

[0008] The position of the target is tracked in real time to obtain real-time position coordinates, and the real-time position coordinates are compared with a set forklift working safety area to determine whether the existence of the target will bring safety risks to the safe operation of the forklift.

[0009] In the technical solution provided by the present application, since multiple cameras are installed on the forklift, images of the forklift operating environment can be captured from different angles. This helps to obtain a more comprehensive perspective and reduce the blind spots that may be caused by a single camera. In addition, the collected images are processed using the trained real-time target detection model. The model can identify potential risk targets such as pedestrians, obstacles, shelves, etc. in the image, and realize real-time detection of targets in the forklift operating environment. Furthermore, the detected targets are tracked in real time to obtain their real-time position coordinates in the forklift operating environment. This helps to continuously monitor the dynamic changes of the target and provide data support for subsequent risk assessment. Finally, the target position coordinates obtained in real time are compared with the preset forklift working safety area. If the target enters or approaches the safety area, the system can determine whether this will bring risks to the safe operation of the forklift. And once the system determines that there is a safety risk, it can immediately issue an early warning to prompt the driver to pay attention or take avoidance measures. In some cases, the system can even automatically intervene, such as slowing down or changing the direction of travel, to avoid potential collisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0011] Figure 1 The present invention is a flowchart of a method for detecting a target in a forklift operating environment based on video surveillance technology according to an embodiment of the present application.

[0012] Figure 2 This is a structural schematic diagram of a target detection model in a target detection method for a forklift operating environment based on video surveillance technology in an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0014] Figure 1 The present invention is a flowchart of a method for detecting a target in a forklift operating environment based on video surveillance technology according to an embodiment of the present application. Figure 2This is a structural schematic diagram of a target detection model in a target detection method for a forklift operating environment based on video surveillance technology in an embodiment of the present application.

[0015] like Figure 1 , 2 As shown in FIG. 1 , the target detection method for the forklift operation environment based on video surveillance technology includes:

[0016] Acquire multiple images of the forklift working environment captured by multiple cameras installed at different positions on the forklift;

[0017] Based on the trained real-time target detection model, each of the forklift operating environment images is processed to detect targets existing in the forklift operating environment in real time;

[0018] The position of the target is tracked in real time to obtain real-time position coordinates, and the real-time position coordinates are compared with a set forklift working safety area to determine whether the existence of the target will bring safety risks to the safe operation of the forklift.

[0019] In the technical solution provided by the present application, since multiple cameras are installed on the forklift, images of the forklift operating environment can be captured from different angles. This helps to obtain a more comprehensive perspective and reduce the blind spots that may be caused by a single camera. In addition, the collected images are processed using the trained real-time target detection model. The model can identify potential risk targets such as pedestrians, obstacles, shelves, etc. in the image, and realize real-time detection of targets in the forklift operating environment. Furthermore, the detected targets are tracked in real time to obtain their real-time position coordinates in the forklift operating environment. This helps to continuously monitor the dynamic changes of the target and provide data support for subsequent risk assessment. Finally, the target position coordinates obtained in real time are compared with the preset forklift working safety area. If the target enters or approaches the safety area, the system can determine whether this will bring risks to the safe operation of the forklift. And once the system determines that there is a safety risk, it can immediately issue an early warning to prompt the driver to pay attention or take avoidance measures. In some cases, the system can even automatically intervene, such as slowing down or changing the direction of travel, to avoid potential collisions.

[0020] Optionally, the number of the cameras is at least four, wherein at least one camera is respectively installed on the front, rear, left and right sides of the forklift to capture images of the forklift's operating environment in the four directions of the forklift's travel.

[0021] To this end, at least one camera is installed at the front, rear, left and right sides of the forklift to capture a panoramic image of the forklift's operating environment in four directions, front, back, left and right. This has the following technical benefits:

[0022] (1) By installing cameras on the front, rear, left and right sides of the forklift, all-round monitoring of the forklift's surroundings can be achieved. This layout ensures that no matter where the target is located on the forklift, it can be captured by at least one camera, thereby greatly reducing blind spots.

[0023] (2) Images captured by multiple cameras from different angles provide rich visual information. This information helps the real-time object detection model to more accurately identify potential risk targets such as pedestrians, obstacles, and shelves in the image. Images from multiple perspectives can also verify each other, improving the confidence of object detection.

[0024] (3) In complex forklift operating environments, such as light changes, shadows, and occlusions, a single camera may not be able to stably capture the target. The layout of multiple cameras can improve the robustness of the system. Even if one camera is disturbed, other cameras can still provide valid image information.

[0025] (4) Comprehensive field of view coverage and accurate target detection provide a basis for the development of advanced driver assistance functions. For example, the system can realize automatic obstacle avoidance, path planning, driver assistance prompts and other functions of the forklift based on the image information of multiple cameras, further improving the safety and operating efficiency of the forklift.

[0026] (5) By capturing images of the forklift’s surroundings in real time using multiple cameras, more information about the target’s position, speed, and trajectory can be obtained. This information helps to more accurately assess the target’s impact on the safe operation of the forklift, thereby making more reliable early warning and intervention decisions.

[0027] Optionally, the real-time target detection model includes a first processing network 1, a second processing network 2 and a third processing network 3;

[0028] The real-time target detection model based on the training is used to process each of the forklift operating environment images to detect targets in the forklift operating environment in real time, including:

[0029] Based on the first processing network 1, each of the forklift working environment images is subjected to a step-by-step channel number up-dimensionality processing and a spatial resolution down-dimensionality processing to obtain an output feature map of the first processing network 1;

[0030] Based on the second processing network 2, the output feature map of the first processing network 1 is subjected to a step-by-step channel number dimensionality reduction process and a spatial resolution dimensionality increase process to obtain an output feature map of the second processing network 2;

[0031] Based on the third processing network 3, the output feature map of the second processing network 2 is processed by step-by-step channel number up-dimensionality processing and spatial resolution down-dimensionality processing to obtain the output feature map of the third processing network 3, so as to detect the targets existing in the forklift operating environment in real time.

[0032] To this end, the real-time target detection model includes a first processing network 1, a second processing network 2 and a third processing network 3, which has the following technical benefits:

[0033] (1) Through the first processing network 1, the model can gradually deepen the feature extraction of the input image by gradually increasing the number of channels and reducing the spatial resolution. This helps to capture richer image details and contextual information, providing a solid foundation for subsequent object detection.

[0034] (2) The second processing network 2 performs step-by-step dimension reduction of the number of channels and dimension increase of the spatial resolution on the output feature map of the first processing network 1, thereby achieving feature fusion and enhancement. This process helps the model integrate and refine features at different scales, improving the expressiveness and robustness of the features.

[0035] (3) The third processing network 3 further performs step-by-step channel dimension increase and spatial resolution dimension reduction processing on the output feature map of the second processing network 2 to detect the targets in the forklift operating environment in real time. This design enables the model to achieve accurate target detection while maintaining a high processing speed. Through step-by-step processing, the model can gradually focus on the target area, reduce background interference, and improve the accuracy and stability of detection.

[0036] (4) The design of multiple processing networks in cascade makes the model more scalable and flexible. The structure and parameters of each processing network can be adjusted according to actual needs to adapt to different forklift operating environments and detection tasks.

[0037] (5) Through the step-by-step dimension increase and decrease processing, the model can reasonably allocate computing resources at different stages. In the feature extraction stage, the model can make full use of computing resources to capture rich image features; in the target detection stage, the model can efficiently use the refined features for fast and accurate detection.

[0038] Optionally, the first processing network 1 includes: an initial convolution layer 11, a plurality of cascaded first feature extraction and fusion modules 12, and a pooling module 13. Based on the first processing network 1, each of the forklift working environment images is subjected to step-by-step channel number dimension increase processing and spatial resolution dimension reduction processing to obtain an output feature map of the first processing network 1. The initial convolution layer 11 is used to perform initial convolution processing on each of the forklift working environment images to obtain an initial feature map. The plurality of cascaded first feature extraction and fusion modules 12 perform step-by-step channel number dimension increase processing and spatial resolution dimension reduction processing on the initial feature map to obtain a first-level feature map. The pooling module 13 performs pooling processing on the first-level feature map to obtain the output feature map of the first processing network 1. The number of channels of the first-level feature map is greater than the number of channels of the corresponding forklift working environment image, and the spatial resolution of the first-level feature map is less than the spatial resolution of the corresponding forklift working environment image.

[0039] To this end, the first processing network 1 includes an initial convolution layer 11, a plurality of cascaded first feature extraction and fusion modules 12, and a pooling module 13, which has the following technical benefits:

[0040] (1) The initial convolution layer 11 performs initial convolution processing on each forklift operating environment image to obtain an initial feature map. This step can effectively extract low-level features in the image, such as edges, textures, etc., providing a basis for subsequent feature extraction and fusion.

[0041] (2) The cascaded first feature extraction and fusion modules 12 perform step-by-step channel number up-dimensioning and spatial resolution down-dimensioning on the initial feature map to obtain a first-level feature map. This design can gradually deepen the extraction and fusion of features, capture richer image information and contextual relationships, and improve the expressiveness and robustness of features.

[0042] (3) Through the step-by-step dimension increase of the number of channels and the dimensionality reduction of the spatial resolution, the model can reasonably allocate computing resources at different stages. In the feature extraction stage, the model can make full use of computing resources to capture rich image features, while in the subsequent processing stages it can focus on more advanced feature fusion and object detection tasks.

[0043] (4) The number of channels of the first-level feature map is greater than that of the corresponding forklift operating environment image, which means that the model can extract more feature information. At the same time, the spatial resolution of the first-level feature map is smaller than that of the corresponding forklift operating environment image, which helps to reduce the amount of calculation and improve the processing speed while maintaining the representativeness of the features.

[0044] (5) Through the first processing network 1, the model can process the forklift operating environment image in real time and accurately detect the existing targets. The step-by-step feature extraction and fusion enable the model to better cope with complex environments and changing targets, and improve the accuracy and stability of target detection.

[0045] Optionally, when the output feature map of the first processing network 1 is subjected to step-by-step channel number dimensionality reduction processing and spatial resolution dimensionality reduction processing based on the second processing network 2, part of the first intermediate-level feature map obtained by performing step-by-step channel number dimensionality increase processing and spatial resolution dimensionality reduction processing on each forklift working environment image is obtained, and the obtained part of the second intermediate-level feature map obtained by performing step-by-step channel number dimensionality reduction processing and spatial resolution increase processing on the output feature map of the first processing network 1 is merged with the part of the second intermediate-level feature map obtained by performing step-by-step channel number dimensionality reduction processing and spatial resolution increase processing on the output feature map of the first processing network 1, so as to obtain the output feature map of the second processing network 2.

[0046] To this end, based on the second processing network 2, when performing step-by-step dimension reduction processing of the number of channels and step-by-step dimension increase processing of the spatial resolution on the output feature map of the first processing network 1, part of the first intermediate-level feature map is also obtained and fused with part of the second intermediate-level feature map, which has the following technical benefits:

[0047] (1) By obtaining part of the first intermediate-level feature map in the first processing network 1 and fusing it with part of the second intermediate-level feature map in the second processing network 2, feature reuse and enhancement are achieved. This design enables the model to make full use of feature information at different stages and improve the expressiveness and robustness of features.

[0048] (2) The first intermediate-level feature map usually contains more fine-grained features, such as edges and textures, while the second intermediate-level feature map contains more coarse-grained features, such as shapes and structures. By fusing these two types of features, the model can capture both the detailed information and the overall structure of the image, thereby improving the accuracy of object detection.

[0049] (3) Feature fusion helps the model maintain stable performance in different scenarios and lighting conditions. By reusing and enhancing features, the model can better cope with the complex changes in the forklift operating environment and improve generalization ability.

[0050] (4) Through feature fusion, the model can share and transfer useful feature information at different stages, thereby reducing repeated calculations. This design helps optimize computing resources and improve processing speed.

[0051] (5) Fusion of feature maps from different stages enables the model to detect targets of different scales simultaneously. This is especially important for targets of various sizes that may appear in a forklift operating environment (such as pedestrians, obstacles, shelves, etc.).

[0052] Optionally, the second processing network 2 includes: a plurality of feature pyramid construction modules 21, a plurality of second feature extraction and fusion modules 22, a second feature extraction and fusion module 22 (C3_1) is arranged between every two feature pyramid construction modules 21 so that the second processing network 2 is a cascade structure, and based on the second processing network 2, when the output feature map of the first processing network 1 is subjected to the step-by-step dimension reduction processing of the number of channels and the dimension increase processing of the spatial resolution, the first-level feature pyramid construction module 21 performs the first-level dimension reduction processing of the number of channels and the first-level dimension increase processing of the spatial resolution on the output feature map of the first processing network 1 to obtain the first The output feature map of the first-level feature pyramid construction module 21, the second-level second feature extraction and fusion module 22 performs multi-scale feature extraction and intra-group fusion on the output feature map of the first-level feature pyramid construction module 21, and performs cross-layer fusion processing on part of the first intermediate-level feature maps obtained by performing step-by-step channel number up-dimensionality processing and spatial resolution down-dimensionality processing on each forklift operating environment image to obtain the output feature map of the second-level second feature extraction and fusion module 22, and so on, through the cooperation of the subsequent-level feature pyramid construction module 21 and the second feature extraction and fusion module 22 to obtain the output feature map of the second processing network 2.

[0053] Referring to the above-mentioned second feature extraction and fusion module 22 (C3_1), the first feature extraction and fusion module 12 may be composed of a convolution layer (Conv) and a second feature extraction and fusion module 22 (C3_1).

[0054] To this end, the second processing network 2 performs a cascade structure of multiple feature pyramid construction modules 21 and a plurality of second feature extraction and fusion modules 22 to perform a step-by-step dimension reduction process on the output feature map of the first processing network 1 in terms of the number of channels and a step-by-step dimension increase process in terms of the spatial resolution, which has the following technical advantages:

[0055] (1) The feature pyramid construction module 21 can extract features of different scales from the input feature map and generate a feature pyramid with multi-scale information. This design enables the model to capture objects of different scales in the image and improve the robustness of object detection. The second feature extraction and fusion module 22 further extracts and fuses these multi-scale features to enhance the expressiveness and robustness of the features.

[0056] (2) Through cross-layer fusion processing, the model can effectively fuse feature maps at different stages to achieve feature reuse and enhancement. This design helps the model make full use of feature information at different stages and improve the accuracy of target detection.

[0057] (3) The design of the cascade structure enables the model to reasonably allocate computing resources at different stages. In the feature extraction stage, the model can make full use of computing resources to capture rich image features; while in the feature fusion and object detection stages, the model can focus on more advanced tasks and improve processing speed.

[0058] (4) Through multi-scale feature extraction and fusion, as well as cross-layer fusion processing, the model can better cope with complex changes in the forklift operating environment, such as changes in lighting and target scale. This design improves the generalization ability of the model and enables the model to maintain stable performance in different scenarios.

[0059] (5) The design of the second processing network 2 enables the model to achieve accurate target detection while maintaining a high processing speed. Through the step-by-step channel number reduction and spatial resolution dimension increase processing, the model can gradually focus on the target area, reduce background interference, and improve the accuracy and stability of detection.

[0060] Optionally, each feature pyramid construction module 21 includes a convolution layer (Conv), an upsampling layer (Upsample), and a fusion layer (Concate), and the convolution layer, the upsampling layer, and the fusion layer form a cascade structure in sequence, the convolution layer is used to extract features from the input feature map, the upsampling layer is used to upsample the output feature map obtained by feature extraction from the convolution layer to increase the spatial resolution of the feature map and keep the number of channels unchanged, and the fusion layer is used to fuse the output feature map obtained by upsampling the upsampling layer with the first-level feature map output by the first feature extraction and fusion module 12 to obtain a second intermediate-level feature map.

[0061] To this end, each feature pyramid construction module 21 includes a convolution layer, an upsampling layer, and a fusion layer, and forms a sequential cascade structure, which has the following technical benefits:

[0062] (1) The convolution layer can extract features from the input feature map and capture the key information in the image. Through the convolution operation, the model can learn low-level features such as edges and textures of the image, providing a basis for subsequent object detection tasks.

[0063] (2) The upsampling layer can upsample the feature map output by the convolutional layer, increase the spatial resolution of the feature map and keep the number of channels unchanged. This step helps the model recover the detailed information in the image and improve the accuracy of target detection. Especially when dealing with small targets, upsampling can make the model pay more attention to the detailed features of the target.

[0064] (3) The fusion layer can fuse the feature map output by the upsampling layer with the feature map output by the first feature extraction and fusion module 12 to obtain a second intermediate feature map. This feature fusion method helps the model to make full use of feature information at different stages and improve the expressiveness and robustness of the features. At the same time, feature reuse can also reduce the amount of model calculation and improve processing speed.

[0065] (4) Through the design of the feature pyramid construction module 21, the model can generate feature maps with multi-scale information. This multi-scale feature representation helps the model capture objects of different scales in the image and improves the robustness of object detection. Especially in the forklift operating environment, the size and shape of the object may vary greatly, and the multi-scale feature representation can make the model more adaptable to such changes.

[0066] (5) The design of the feature pyramid construction module 21 enables the model to maintain stable performance in different scenes and lighting conditions. Through steps such as feature extraction, upsampling, and feature fusion, the model can learn more robust feature representations and improve generalization capabilities.

[0067] Optionally, based on the third processing network 3, the output feature map of the second processing network 2 is subjected to step-by-step channel dimensionality reduction processing and spatial resolution dimensionality reduction processing to obtain the output feature map of the third processing network 3, so as to detect the target existing in the forklift operating environment in real time, obtain part of the second intermediate-level feature map obtained by performing step-by-step channel dimensionality reduction processing and spatial resolution up-dimension processing on the output feature map of the first processing network 1, and fuse them with part of the third intermediate-level feature map obtained by performing step-by-step channel dimensionality up-dimension processing and spatial resolution down-dimension processing on the output feature map of the second processing network 2, so as to obtain the output feature map of the third processing network 3.

[0068] To this end, the third processing network 3 performs step-by-step channel dimension increase processing and spatial resolution dimension reduction processing on the output feature map of the second processing network 2, and obtains part of the second intermediate level feature map and part of the third intermediate level feature map of the third processing network 3 for fusion, and finally obtains the output feature map of the third processing network 3, which has the following technical advantages:

[0069] (1) By obtaining part of the second intermediate-level feature map output by the first processing network 1 and fusing it with part of the third intermediate-level feature map of the third processing network 3, cross-stage reuse of feature information is achieved. This design enables the model to make full use of feature information at different processing stages and improve the expressiveness and robustness of features.

[0070] (2) The second intermediate level feature map usually contains more fine-grained features, such as edges and textures, while the third intermediate level feature map contains more coarse-grained features after further processing. By fusing these two types of features, the model can capture both the detailed information and the overall structure of the image, thus improving the accuracy of object detection.

[0071] (3) Fusion of feature maps from different stages enables the model to detect targets of different scales simultaneously. This is especially important for targets of various sizes that may appear in a forklift operating environment, such as pedestrians, obstacles, and shelves. Through feature fusion, the model can better capture the features of these targets and improve detection accuracy.

[0072] Optionally, the third processing network 3 includes a plurality of cascaded cross-layer feature extraction and fusion modules 31, the first-level cross-layer feature extraction and fusion module 31 performs channel number reduction processing and spatial resolution increase processing on the output feature map of the second processing network 2 to obtain the output feature map of the first-level feature pyramid construction module 21, the second-level second feature extraction and fusion module 22 performs multi-scale feature extraction and intra-group fusion on the output feature map of the first-level feature pyramid construction module 21, and performs cross-layer fusion processing with part of the second intermediate-level feature map to obtain the output feature map of the second-level second feature extraction and fusion module 22, and so on, to obtain the output feature map of the third processing network 3, the number of channels of the output feature map of the third processing network 3 is equal to the number of channels of the output feature map of the first processing network 1, and the spatial resolution of the output feature map of the third processing network 3 is equal to the spatial resolution of the output feature map of the first processing network 1.

[0073] Optionally, when the first processing network 1 performs step-by-step channel dimensionality upscaling and spatial resolution dimensionality reduction processing, the number of channels is doubled according to six gradients, and the spatial resolution is halved according to five gradients; when the second processing network 2 performs step-by-step channel dimensionality reduction processing and spatial resolution upscaling processing, the number of channels is halved according to three gradients, and the spatial resolution is doubled according to four gradients; when the third processing network 3 performs step-by-step channel dimensionality upscaling and spatial resolution reduction processing, the number of channels is doubled according to four gradients, and the spatial resolution is halved according to three gradients.

[0074] Similarly, the last cross-layer feature extraction and fusion module 31 adopts the second feature extraction and fusion module 22 (C3_1), and the remaining cross-layer feature extraction and fusion modules 31 each consist of a second feature extraction and fusion module 22 (C3_1), a convolutional layer (Conv), and a fusion layer (Concate). The detailed working principle can be found in the above scheme description.

[0075] To this end, the third processing network 3 performs channel number up-dimensioning and spatial resolution down-dimensioning processing on the output feature map of the second processing network 2 through a cascade of multiple cross-layer feature extraction and fusion modules 31, and performs cross-layer fusion with part of the second intermediate-level feature map, and finally obtains an output feature map with the same number of channels and spatial resolution as the output feature map of the first processing network 1, which has the following technical advantages:

[0076] (1) The third processing network 3 realizes multi-level feature extraction and fusion of the feature map output by the second processing network 2 through multiple cross-layer feature extraction and fusion modules 31. This design enables the model to make full use of feature information at different levels and improve the expressiveness and robustness of the features.

[0077] (2) Through cross-layer fusion processing, the model can effectively combine the coarse-grained features output by the second processing network 2 with the fine-grained features in some of the second intermediate-level feature maps. This combination helps the model capture both the detailed information and the overall structure of the image, thereby improving the accuracy of target detection.

[0078] (3) The output feature map of the third processing network 3 has the same number of channels and spatial resolution as the output feature map of the first processing network 1. This consistency design enables the model to maintain the consistency of feature maps at different stages, which is convenient for subsequent object detection tasks.

[0079] (4) Through feature fusion and cross-layer processing, the model can share and transfer useful feature information at different stages, thereby reducing repeated calculations. This design helps optimize computing resources and improve processing speed, enabling the model to detect targets in the forklift operating environment in real time.

[0080] (5) Feature fusion and cross-layer processing help the model maintain stable performance in different scenarios and lighting conditions. By reusing and enhancing features, the model can better cope with complex changes in the forklift operating environment, such as lighting changes and target scale changes, and improve generalization and robustness.

[0081] (6) Through multi-level feature extraction and fusion, the model can simultaneously detect targets of different scales. This is especially important for targets of various sizes that may appear in the forklift operating environment. The model can better capture the features of these targets and improve the accuracy of detection.

[0082] Optionally, based on the first processing network 1, each of the forklift operating environment images is subjected to a step-by-step channel number up-dimensionality processing and a spatial resolution dimensionality reduction processing to obtain an output feature map of the first processing network 1 that satisfies the following rules: the number of channels of the forklift operating environment image is 3, and the spatial resolution is 320*320; the output feature map of the first processing network 1 has a channel number of 1024 and a spatial resolution of 20*20; the first intermediate feature map has a channel number of 128, 256, or 512, and a spatial resolution of 160*160, 80*80, or 20*20;

[0083] Based on the second processing network 2, the output feature map of the first processing network 1 is subjected to a step-by-step channel number dimensionality reduction process and a spatial resolution dimensionality increase process to obtain an output feature map of the second processing network 2 that satisfies the following rules: the output feature map of the second processing network 2 has a channel number of 256 and a spatial resolution of 160*160, and the second intermediate-level feature map has a channel number of 512, 256, and 128, and a spatial resolution of 20*20, 40*40, and 80*80;

[0084] Based on the third processing network 3, the output feature map of the second processing network 2 is processed by step-by-step channel number up-dimensionality processing and spatial resolution down-dimensionality processing to obtain the output feature map of the third processing network 3, so as to detect in real time the targets existing in the forklift operating environment, and satisfy the following rules: the output feature map of the third processing network 3 has a channel number of 1024 and a spatial resolution of 20*20. When the output feature map of the second processing network 2 is processed by step-by-step channel number up-dimensionality processing and spatial resolution down-dimensionality processing to obtain the output feature map of the third processing network 3, several third intermediate-level feature maps are obtained, whose channel numbers are 128, 256, and 512, and whose spatial resolutions are 160*160, 80*80, and 40*40.

[0085] For this purpose, see Figure 2 , based on the configuration of the above rules, the second processing network 2 is fused with the feature map output by the first processing network 1 when processing the feature map, and the third processing network 3 is fused with the feature map output by the second processing network 2 when processing the feature map, and finally the following feature maps are obtained, so as to further generate target detection frames based on these feature maps to detect the target:

[0086] The output feature map of the third processing network 3 suitable for large object detection has 1024 channels and a spatial resolution of 20*20 to generate a large object detection frame;

[0087] Using three third-level feature maps:

[0088] The first third intermediate level feature map suitable for small target detection: the number of channels is 128 and the spatial resolution is 160*160 to generate the first small target detection frame;

[0089] The second third intermediate level feature map for small target detection: the number of channels is 256, and the spatial resolution is 80*80, to generate the second small target detection frame. The first third intermediate level feature map is especially suitable for small target detection, mainly used to detect relatively small pedestrians and special situations such as sideways, occlusion and half squatting;

[0090] The third intermediate-level feature map suitable for mid-range target detection has 512 channels and a spatial resolution of 40*40 to generate a mid-range target detection box.

[0091] The coordinate information of the target detection frame is compared with the set forklift working safety area to determine whether the existence of the target will bring safety risks to the safe operation of the forklift.

[0092] To this end, the output feature map of the third processing network 3 has 1024 channels and a spatial resolution of 20*20, which enables it to capture rich global features. In the forklift scene, large targets such as large cargo or obstacles usually occupy most of the image, so global features are crucial for their detection. Through the high-dimensional feature representation of 1024 channels, the model is able to more accurately describe the complex features of large targets, thereby improving the accuracy of large target detection. In addition, the third third intermediate-level feature map has 512 channels and a spatial resolution of 40*40, which is suitable for the detection of medium targets. In the forklift scene, medium targets such as medium-sized cargo or personnel usually occupy a moderate area of ​​the image. Through the feature representation of 512 channels, the model is able to capture more detailed information while maintaining global features, which is very important for the accurate detection of medium targets. Using the third third intermediate-level feature map to generate a medium target detection box can accurately locate medium-sized objects in the image, providing key information for the safe operation of the forklift. Finally, the first and second third-level feature maps have 128 channels and 160*160 spatial resolution, and 256 channels and 80*80 spatial resolution, respectively, which are particularly suitable for the detection of small targets. In the forklift scene, small targets such as small cargo, pedestrians or obstacles usually occupy a small part of the image. Moreover, the second third-level feature map generates a small target detection frame, which can also accurately locate small objects in the image, especially for relatively small pedestrians and special situations such as sideways, occlusion and half-squatting, providing important guarantees for the safe operation of forklifts.

[0093] An embodiment of the present application further provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer executable program, and the processor is used to execute the computer executable program to implement the method described in any one of the above embodiments.

[0094] An embodiment of the present application also provides a forklift safety monitoring system, which includes: multiple cameras installed at different positions on the forklift and a background server, the multiple cameras are used to shoot the forklift's working environment to obtain multiple forklift working environment images; a trained real-time target detection model is deployed on the background server, and each of the forklift working environment images is processed based on the trained real-time target detection model to detect targets existing in the forklift working environment in real time; and the position of the target is tracked in real time to obtain real-time position coordinates, and the real-time position coordinates are compared with the set forklift working safety area to determine whether the existence of the target will bring safety risks to the safe operation of the forklift.

[0095] For an exemplary description of the forklift safety monitoring system for detecting targets in the forklift operating environment in real time based on the trained real-time target detection model, see the above Figure 1 Records of.

[0096] The expressions "first", "second", "the first" or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and / or importance, but these expressions do not limit the corresponding components. The above expressions are only configured for the purpose of distinguishing an element from other elements. For example, a first user device and a second user device represent different user devices, although both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0097] When one element (e.g., a first element) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another element (e.g., a second element), it is understood that the one element is directly connected to the other element or the one element is indirectly connected to the other element via yet another element (e.g., a third element). Conversely, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (the second element), no element (e.g., a third element) is interposed between the two.

[0098] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A target detection method for a forklift operating environment based on video monitoring technology, characterized in that: include: Acquire multiple images of the forklift working environment captured by multiple cameras installed at different positions on the forklift; Based on the trained real-time target detection model, each of the forklift operating environment images is processed to detect targets existing in the forklift operating environment in real time; Tracking the position of the target in real time to obtain real-time position coordinates, and comparing the real-time position coordinates with a set forklift working safety area to determine whether the presence of the target will bring safety risks to the safe operation of the forklift; Wherein, the real-time target detection model includes a first processing network, a second processing network and a third processing network; The real-time target detection model based on the training is used to process each of the forklift operating environment images to detect targets in the forklift operating environment in real time, including: Based on the first processing network, each of the forklift working environment images is subjected to a step-by-step channel number up-dimensionality processing and a spatial resolution down-dimensionality processing to obtain an output feature map of the first processing network; Based on the second processing network, performing a channel number dimensionality reduction process and a spatial resolution dimensionality increase process on the output feature map of the first processing network step by step to obtain an output feature map of the second processing network; Based on the third processing network, the output feature map of the second processing network is subjected to a step-by-step channel number up-dimensionality processing and a spatial resolution down-dimensionality processing to obtain an output feature map of the third processing network, so as to detect targets existing in the forklift operating environment in real time; Wherein, when the first processing network performs step-by-step channel number up-dimension processing and spatial resolution down-dimension processing, the number of channels is doubled in sequence according to six gradients, and the spatial resolution is halved in sequence according to five gradients; when the second processing network performs step-by-step channel number down-dimension processing and spatial resolution up-dimension processing, the number of channels is halved in sequence according to three gradients, and the spatial resolution is doubled in sequence according to four gradients; when the third processing network performs step-by-step channel number up-dimension processing and spatial resolution down-dimension processing, the number of channels is doubled in sequence according to four gradients, and the spatial resolution is halved in sequence according to three gradients; Wherein, when performing step-by-step channel number up-dimensionality processing and spatial resolution down-dimensionality processing on each of the forklift operating environment images based on the second processing network, a part of the first intermediate-level feature map obtained by performing step-by-step channel number up-dimensionality processing and spatial resolution down-dimensionality processing on each of the forklift operating environment images is obtained, and the part of the second intermediate-level feature map obtained by performing step-by-step channel number down-dimensionality processing and spatial resolution up-dimensionality processing on the output feature map of the first processing network is merged with the part of the second intermediate-level feature map obtained by performing step-by-step channel number down-dimensionality processing and spatial resolution up-dimensionality processing on the output feature map of the first processing network, so as to obtain the output feature map of the second processing network; Wherein, based on the third processing network, the output feature map of the second processing network is subjected to step-by-step channel dimension increase processing and spatial resolution dimension reduction processing to obtain the output feature map of the third processing network, so as to detect the target existing in the forklift operating environment in real time, obtain part of the second intermediate-level feature map obtained by step-by-step channel dimension reduction processing and spatial resolution dimension increase processing on the output feature map of the first processing network, and fuse them with part of the third intermediate-level feature map obtained by step-by-step channel dimension increase processing and spatial resolution dimension reduction processing on the output feature map of the second processing network, so as to obtain the output feature map of the third processing network; Among them, the number of channels of the output feature map of the third processing network is equal to the number of channels of the output feature map of the first processing network, and the spatial resolution of the output feature map of the third processing network is equal to the spatial resolution of the output feature map of the first processing network.

2. The target detection method for a forklift operating environment based on video monitoring technology according to claim 1 is characterized in that: The number of the cameras is at least four, wherein at least one camera is installed at the front, rear, left and right sides of the forklift respectively, to capture images of the forklift operating environment along the front, back, left and right directions of the forklift.

3. The target detection method for a forklift operating environment based on video monitoring technology according to claim 2 is characterized in that: The first processing network includes: an initial convolution layer, a plurality of cascaded first feature extraction and fusion modules, and a pooling module. Based on the first processing network, each of the forklift operating environment images is subjected to step-by-step channel number dimension increase processing and spatial resolution dimension reduction processing to obtain an output feature map of the first processing network. The initial convolution layer is used to perform initial convolution processing on each of the forklift operating environment images to obtain an initial feature map. The plurality of cascaded first feature extraction and fusion modules perform step-by-step channel number dimension increase processing and spatial resolution dimension reduction processing on the initial feature map to obtain a first-level feature map. The pooling module performs pooling processing on the first-level feature map to obtain an output feature map of the first processing network.

4. The target detection method for a forklift operating environment based on video monitoring technology according to claim 3 is characterized in that: The second processing network includes: multiple feature pyramid construction modules, several second feature extraction and fusion modules, a second feature extraction and fusion module is set between every two feature pyramid construction modules so that the second processing network is a cascade structure, based on the second processing network, when the output feature map of the first processing network is subjected to step-by-step dimension reduction processing of the number of channels and the dimension increase processing of the spatial resolution, the first-level feature pyramid construction module performs the first-level dimension reduction processing of the number of channels and the first-level dimension increase processing of the spatial resolution on the output feature map of the first processing network to obtain the output feature map of the first-level feature pyramid construction module, the second-level second feature extraction and fusion module performs multi-scale feature extraction and intra-group fusion on the output feature map of the first-level feature pyramid construction module, and performs cross-layer fusion processing with part of the first intermediate-level feature maps obtained by step-by-step dimension increase processing of the number of channels and the dimension reduction processing of the spatial resolution on each forklift working environment image to obtain the output feature map of the second-level second feature extraction and fusion module, and so on, after the feature pyramid construction module and the second feature extraction and fusion module of the subsequent level cooperate to obtain the output feature map of the second processing network.

5. The target detection method for a forklift operating environment based on video monitoring technology according to claim 4 is characterized in that: Each feature pyramid construction module includes a convolution layer, an upsampling layer, and a fusion layer. The convolution layer, the upsampling layer, and the fusion layer form a cascade structure in sequence. The convolution layer is used to extract features from the input feature map. The upsampling layer is used to upsample the output feature map obtained by feature extraction from the convolution layer to increase the spatial resolution of the feature map and keep the number of channels unchanged. The fusion layer is used to fuse the output feature map obtained by upsampling the upsampling layer with the first-level feature map output by the first feature extraction and fusion module to obtain a second intermediate-level feature map.

6. The target detection method for a forklift operating environment based on video monitoring technology according to claim 5 is characterized in that: The third processing network includes a plurality of cascaded cross-layer feature extraction and fusion modules. The first-level cross-layer feature extraction and fusion module performs channel number reduction processing and spatial resolution increase processing on the output feature map of the second processing network to obtain the output feature map of the first-level feature pyramid construction module. The second-level second feature extraction and fusion module performs multi-scale feature extraction and intra-group fusion on the output feature map of the first-level feature pyramid construction module, and performs cross-layer fusion processing with part of the second intermediate-level feature map to obtain the output feature map of the second-level second feature extraction and fusion module, and so on, to obtain the output feature map of the third processing network.

7. The target detection method for a forklift operating environment based on video monitoring technology according to claim 6 is characterized in that: Based on the first processing network, each of the forklift operating environment images is subjected to a step-by-step channel number up-dimensionality processing and a spatial resolution dimensionality reduction processing, so as to obtain an output feature map of the first processing network that satisfies the following rules: the number of channels of the forklift operating environment image is 3, and the spatial resolution is 320*320; the output feature map of the first processing network has a channel number of 1024 and a spatial resolution of 20*20; the first intermediate feature map has a channel number of 128, 256, or 512, and a spatial resolution of 160*160, 80*80, or 20*20; Based on the second processing network, the output feature map of the first processing network is subjected to a step-by-step dimension reduction process of the number of channels and a dimension increase process of the spatial resolution, so as to obtain the output feature map of the second processing network, which satisfies the following rules: the output feature map of the second processing network has a channel number of 256 and a spatial resolution of 160*160, and the second intermediate-level feature map has a channel number of 512, 256, and 128, and a spatial resolution of 20*20, 40*40, and 80*80; Based on the third processing network, the output feature map of the second processing network is processed by step-by-step channel dimensionality increase and spatial resolution dimensionality reduction to obtain the output feature map of the third processing network, so as to detect in real time the targets existing in the forklift operating environment, satisfying the following rules: the output feature map of the third processing network has a channel number of 1024 and a spatial resolution of 20*20, and the output feature map of the second processing network is processed by step-by-step channel dimensionality reduction and spatial resolution dimensionality reduction to obtain the output feature map of the third processing network, thereby obtaining a plurality of third intermediate-level feature maps, whose channel numbers are 128, 256, 512, and spatial resolutions are 160*160, 80*80, and 40*40.

Images