Forklift active safety control method and system based on AI vision and VCU control

Abstract

The invention discloses a forklift active safety control method and system based on AI vision and VCU control, and particularly relates to the technical field of forklifts. According to the forklift active safety control method provided by the invention, the system response delay is less than or equal to 200ms through multi-thread parallel processing: multi-path camera image acquisition, AI visual inspection, distance measurement calculation and VCU control instruction issuing and parallel execution; furthermore, a user is supported to customize a safe distance threshold value and a speed limit threshold value after alarming; through multi-camera fusion, a lightweight YOLOv8-NCNN model detects personnel in real time, and a geometric projection ranging algorithm and a three-level alarm mechanism are combined to realize control of the forklift speed. The risk of human-vehicle collision is obviously reduced, and meanwhile the working efficiency of the forklift is guaranteed; according to the active safety control system for the forklift, flexible configuration of alarm parameters is supported through modular design, and the differential requirements of complex scenes such as storage and logistics are met; and the safety of man-vehicle collaborative operation is ensured.

Description

Technical Field

[0001] This invention relates to the field of forklift technology, and more specifically, to a forklift active safety control method and system based on AI vision and VCU control. Background Technology

[0002] It is known that in warehousing, logistics and other scenarios, forklifts often work together with personnel. If forklifts lack safety control measures, collision accidents are likely to occur. Therefore, existing forklift safety control technologies mainly rely on millimeter-wave radar or lidar.

[0003] However, in practical applications, if millimeter-wave radar is used, it needs to be used at close range to measure width or narrowness to achieve good results; if lidar is used, it is costly and bulky, making it difficult to adapt to small and medium-sized forklifts; if traditional vision solutions are used, they are greatly affected by lighting and occlusion, have low ranging accuracy, and high response latency (>300ms), which cannot meet the requirements of real-time control. Therefore, a forklift active safety control method and system based on AI vision and VCU control is proposed as a further improvement. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a forklift active safety control method and system based on AI vision and VCU control to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a forklift active safety control method based on AI vision and VCU control, comprising the following steps:

[0006] S1: Capture 360° environmental images around the forklift using a camera;

[0007] S2: Based on the lightweight YOLOv8-NCNN model, the environmental image acquired by S1 is processed in real time to detect personnel targets and output the corresponding bounding box coordinates;

[0008] S3: Based on the camera intrinsic parameter matrix in S1 and the bounding box coordinates provided in S2, calculate the real-time distance between the personnel and the forklift using a geometric projection algorithm;

[0009] S4: Based on the preset safe distance threshold, compare it with the real-time distance provided by S3, and trigger the corresponding alarm level;

[0010] S5: Based on the alarm level of S4, execute alarm actions and dynamically adjust the forklift travel speed.

[0011] Furthermore, in S4, the alarm levels include: a security status level, a yellow alarm level, and a red alarm level;

[0012] The safe distance thresholds include: a yellow alarm threshold and a red alarm threshold;

[0013] The safety status level is set as follows: personnel distance ≥ yellow alarm threshold;

[0014] The yellow alarm level is set as follows: yellow alarm threshold > personnel distance ≥ red alarm threshold;

[0015] The red alarm level is set as follows: personnel distance < red alarm threshold.

[0016] Furthermore, in S5,

[0017] If the alarm level is a safe status level, and the personnel distance is greater than or equal to the yellow alarm threshold, the alarm action to be performed is: dynamically adjust the forklift to run at full speed through the VCU control module;

[0018] If the alarm level is yellow, and the yellow alarm threshold > personnel distance ≥ red alarm threshold, the alarm action is to dynamically adjust the forklift speed limit to the preset yellow alarm speed through the VCU control module and trigger the lights to flash.

[0019] If the alarm level is red, and the personnel distance is less than the red alarm threshold, the alarm action is as follows: the forklift speed limit is dynamically adjusted to the preset red alarm speed through the VCU control module, and voice prompts and flashing lights are triggered simultaneously.

[0020] Furthermore, in S1, the camera used to collect 360° environmental images around the forklift is deployed at least at the front left, front right, and rear of the forklift.

[0021] Furthermore, the camera has a field of view ≥120°, a resolution of not less than 1280×720, and a coverage radius of at least 10 meters.

[0022] Furthermore, in S3, the calculation formula for the geometric projection algorithm is as follows:

[0023]

[0024] Among them, g x and g y The pixel width and height of the human detection bounding box in the image plane are respectively; f x and f y p represents the X-axis and Y-axis components of the camera's focal length; w and p h α and β are the average height and shoulder width of a person in the application scenario, respectively; β is the proximity coefficient; and α is the unit coefficient.

[0025] Furthermore, in S2, the YOLOv8-NCNN model is quantized using INT8, with a model size ≤ 6MB and an inference frame rate ≥ 30FPS.

[0026] A forklift active safety control system based on AI vision and VCU control, including the aforementioned forklift active safety control method, comprising:

[0027] Multi-camera module, which captures 360° environmental images around the forklift via cameras;

[0028] The AI ​​vision processing unit is equipped with a lightweight YOLOv8-NCNN model for detecting human targets and outputting the corresponding bounding box coordinates. This unit processes environmental images acquired by the multi-camera module in real time.

[0029] The ranging calculation module collects the camera intrinsic parameter matrix of the multi-camera module and the bounding box coordinates of the AI ​​vision processing unit, and calculates the real-time distance between the person and the forklift through a geometric projection algorithm.

[0030] The alarm execution module presets a safe distance threshold. This threshold is compared with the real-time distance provided by the ranging calculation module, triggering the corresponding alarm level. Based on the alarm level, it executes an alarm action.

[0031] The VCU control module adjusts the forklift's travel speed based on the alarm level transmitted by the alarm execution module.

[0032] Furthermore, the intrinsic parameters of the multi-camera module are obtained using the Zhang Zhengyou calibration method.

[0033] The technical effects and advantages of this invention are as follows:

[0034] Compared with existing technologies, the forklift active safety control method proposed in this invention achieves a system response latency of ≤200ms through multi-threaded parallel processing: multi-camera image acquisition, AI visual detection, distance calculation, and VCU control command issuance are executed in parallel. Furthermore, it supports user-defined safety distance thresholds and alarm-triggered speed limit thresholds. Specifically, it uses multi-camera fusion and a lightweight YOLOv8-NCNN model to detect personnel in real time, combined with a geometric projection ranging algorithm and a three-level alarm mechanism to achieve dynamic control of forklift speed, realizing high-precision (distance error <10cm), low-latency (≤200ms) personnel detection and dynamic protection. The combination of a tiered alarm mechanism and user-defined control significantly reduces the risk of collisions between people and vehicles while ensuring forklift operation efficiency. The forklift active safety control system proposed in this invention supports flexible configuration of alarm parameters through modular design, adapting to the differentiated needs of complex scenarios such as warehousing and logistics, and ensuring the safety of human-vehicle collaborative operations. Attached Figure Description

[0035] Figure 1 This is a system block diagram of the present invention.

[0036] Figure 2 This is a flowchart of an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram illustrating the principle of the distance measurement calculation module of the present invention.

[0038] Figure 4 This is a schematic diagram of the CAN bus circuit of the present invention.

[0039] The following are the reference numerals in the instruction manual's accompanying drawings: 1. Multi-camera module; 2. AI vision processing unit; 3. Distance calculation module; 4. Alarm execution module; 41. Audible and visual alarm device; 42. Voice prompt unit; 5. VCU control module; 6. Motor controller; Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0041] A forklift active safety control method based on AI vision and VCU control includes the following steps:

[0042] S1: Capture 360° environmental images around the forklift using cameras; for example, by deploying three wide-angle cameras at the left front, right front, and rear of the forklift, capture 360° environmental images around the forklift in real time.

[0043] S2: Based on the lightweight YOLOv8-NCNN model, the environmental image acquired by S1 is processed in real time to detect human targets and output the corresponding bounding box coordinates; where the bounding box coordinates are the pixel coordinates of the bottom center point of the bounding box;

[0044] S3: Based on the camera intrinsic parameter matrix in S1 and the bounding box coordinates provided in S2, calculate the real-time distance between the personnel and the forklift using a geometric projection algorithm;

[0045] S4: Compare the preset safe distance threshold with the real-time distance provided by S3 to trigger the corresponding alarm level; for example, two preset safe distance thresholds are used to divide the distance into three safe distances, and the three safe distances correspond to three alarm levels; and the two safe distance thresholds correspond to two speed thresholds respectively.

[0046] S5: Based on the alarm level of S4, execute alarm actions and dynamically adjust the forklift travel speed.

[0047] For example, the forklift speed can be dynamically adjusted through the VCU control module 5; that is, the three alarm levels correspond to the forklift speed, which can be dynamically adjusted according to the actual working conditions of the forklift. Specifically, before use, the forklift driver inputs the corresponding safe distance threshold through the vehicle system. The safe distance threshold is transmitted to the VCU control module 5 through the CAN bus. The VCU control module 5 updates and saves the safe distance threshold and sends a speed limit command to the forklift's motor controller 6. The alarm signal is sent to the alarm execution module 4 in real time. The forklift's motor controller 6 executes the corresponding command, and the alarm execution module 4 executes the alarm action.

[0048] like Figure 2 The flowchart shown divides the scene into three safe distances by setting two preset safety distance thresholds. This three-level alarm mechanism cyclically detects the minimum recognition distance in each camera's view. For example, in a system using Android 12 and an RK3588J chip, the overall time from the appearance of a pedestrian in the image to the vehicle's response is less than 200ms. Specifically:

[0049] Step 1: First, detect personnel targets based on the acquired environmental images and output their coordinates;

[0050] Step 2: Calculate the minimum distance between personnel and forklifts based on the coordinates;

[0051] Step 3: Determine if the minimum distance is less than the preset yellow safety distance threshold: if yes, proceed to step 4; if no, the system is in a safe state and proceed to step 6.

[0052] Step 4: Determine if the minimum distance is less than the preset red safety distance threshold: If yes, proceed to step 5; if no, the VCU control module 5 controls the motor controller 6 to drop to the yellow alarm speed threshold and uses the audible and visual alarm device 41 to flash; then proceed to step 6.

[0053] Step 5: The VCU control module 5 controls the motor controller 6 to drop to the red alarm speed threshold and uses the audible and visual alarm device 41 to flash.

[0054] Step 6: Repeat step 1 to achieve cyclic detection.

[0055] The forklift active safety control method disclosed in this invention achieves a system response delay of ≤200ms through multi-threaded parallel processing. For example, multiple images are simultaneously acquired and preprocessed by a multi-camera module 1; the distance calculation of personnel detection and distance calculation module 3 is executed in parallel by an AI vision processing unit 2; the alarm execution module 4 triggers the corresponding alarm level based on the distance comparison; the VCU control module 5 sends a speed limit command to the forklift's motor controller 6 via the CAN bus according to the alarm level, and feeds back the alarm signal to the alarm execution module 4 in real time; the forklift's motor controller 6 executes the corresponding command, and the alarm execution module 4 executes the alarm action.

[0056] In a preferred embodiment, in S4, the alarm levels include: a security status level, a yellow alarm level, and a red alarm level;

[0057] The safe distance thresholds include: yellow alarm thresholds and red alarm thresholds;

[0058] The safety status level is set as follows: personnel distance ≥ yellow alarm threshold;

[0059] The yellow alarm level is set as follows: Yellow alarm threshold > Personnel distance ≥ Red alarm threshold;

[0060] The red alarm level is set to: personnel distance < red alarm threshold.

[0061] For example: the three levels of alarm levels:

[0062] 1. Safety Status Level: Personnel distance ≥ yellow alarm threshold (default 3 meters), forklift running at full speed;

[0063] 2. Yellow alarm level: Yellow alarm threshold > personnel distance ≥ red alarm threshold (default 1.5-3 meters), VCU control module 5 limits the speed to the yellow alarm speed (default 70%), and the alarm action executed by synchronous alarm execution module 4 is to control the yellow warning box and light flashing by the sound and light alarm device 41;

[0064] 3. Red Alarm: When the distance between personnel and the red alarm threshold (default 1.5 meters), the VCU control module 5 forcibly limits the speed to less than the red alarm speed (default 30% or 0.5 m / s), and the alarm execution module 4 performs the following alarm actions: the voice prompt unit 42 gives a voice prompt "Danger behind", and the sound and light alarm device 41 flashes the LED at high frequency.

[0065] In a preferred embodiment, in S5, if the alarm level is a safe state level, and the personnel distance is greater than or equal to the yellow alarm threshold, the alarm action is to dynamically adjust the forklift to run at full speed through the VCU control module 5.

[0066] If the alarm level is yellow, and the yellow alarm threshold > personnel distance ≥ red alarm threshold, the alarm action is to dynamically adjust the forklift speed limit to the preset yellow alarm speed through the VCU control module 5 and trigger the lights to flash.

[0067] If the alarm level is red, and the personnel distance is less than the red alarm threshold, the alarm action is as follows: the forklift speed limit is dynamically adjusted to the preset red alarm speed through the VCU control module 5, and a voice prompt and flashing lights are triggered simultaneously.

[0068] The alarm execution module 4 includes an audible and visual alarm device 41 for flashing lights and a voice prompt unit 42 for voice prompts. The audible and visual alarm device 41 can be an audible and visual alarm, which uses LED flashing and a buzzer to provide multi-level warnings.

[0069] For example, when the VCU control module 5 is under a red alarm level, it will forcibly limit the forklift speed and feed back to the voice prompt unit 42 of the alarm execution module 4 to execute the alarm action, which is a voice prompt: "Danger! Please stop!"

[0070] In a preferred embodiment, in S1, the camera used to collect 360° environmental images around the forklift is deployed at least at the front left, front right, and rear of the forklift.

[0071] Furthermore, the camera's field of view must be ≥120°, its resolution must be no less than 1280×720, and its coverage radius must be at least 10 meters. For example, if the camera uses three wide-angle cameras, at least three cameras must be used to complete the acquisition of 360° environmental images around the forklift.

[0072] In a preferred embodiment, such as Figure 3 As shown, in S3, the calculation formula for the geometric projection algorithm is:

[0073]

[0074] Among them, g x and g y The pixel width and height of the human detection bounding box in the image plane are respectively; f x and f y p represents the X-axis and Y-axis components of the camera's focal length; w and p h α and β are the average height and shoulder width of a person in the application scenario, respectively; β is the proximity coefficient; and α is the unit coefficient.

[0075] In a preferred embodiment, in S2, the YOLOv8-NCNN model is quantized using INT8, the model size is ≤6MB, and the inference frame rate is ≥30FPS.

[0076] As attached Figure 1 As shown, a forklift active safety control system based on AI vision and VCU control includes a forklift active safety control method, comprising:

[0077] Multi-camera module 1 captures 360° environmental images around the forklift using cameras. For example, the multi-camera module includes three wide-angle cameras deployed at the front left, front right, and rear of the forklift. Each camera has a field of view ≥120° and a resolution of at least 1280×720, thus covering the forklift's 360° blind spot and an area within a 10-meter radius. Furthermore, the captured environmental images are time-stamped to ensure synchronized processing of multiple perspectives.

[0078] AI vision processing unit 2 is equipped with a lightweight YOLOv8-NCNN model for detecting people and outputting the corresponding bounding box coordinates. This unit processes environmental images acquired by the multi-camera module 1 in real time to achieve real-time person detection. For example, the YOLOv8-NCNN model has an input resolution of 416×416, an inference frame rate of ≥30FPS, and compresses the model size to less than 6MB through INT8 quantization to adapt to corresponding devices.

[0079] The ranging calculation module 3 collects the camera intrinsic parameter matrix of the multi-camera module 1 and the bounding box coordinates of the AI ​​vision processing unit 2, and calculates the real-time distance between the person and the forklift through a geometric projection algorithm.

[0080] Alarm execution module 4: This module presets a safe distance threshold, compares the preset safe distance threshold with the real-time distance provided by the ranging calculation module 3, and triggers the corresponding alarm level. It then executes an alarm action based on the alarm level.

[0081] The alarm execution module 4 includes an audible and visual alarm device 41 for flashing lights and a voice prompt unit 42 for voice prompts. The audible and visual alarm device 41 can be an audible and visual alarm, which uses LED flashing and a buzzer to provide multi-level warnings.

[0082] Among them, the alarm execution module 4 supports user-defined safe distance threshold and speed threshold. The speed threshold is the forklift travel speed threshold controlled by the VCU control module 5 through the forklift motor controller 6.

[0083] In addition, there is the VCU control module 5, which adjusts the forklift speed according to the alarm level transmitted by the alarm execution module 4.

[0084] Among them, the VCU control module 5 adjusts the speed of the forklift drive motor by controlling the motor controller 6 of the forklift, thereby adjusting the forklift travel speed;

[0085] For example, if the VCU control module 5 receives an alarm level of red from the alarm execution module 4, it will send a speed limit command to the motor controller 6 via the CAN bus to forcibly limit the forklift speed; and it will also provide feedback that the alarm action performed by the voice prompt unit 42 of the alarm execution module 4 is a voice prompt: "Danger! Please stop!"; and the audible and visual alarm device 41 of the alarm execution module 4 is linked with the voice prompt unit 42.

[0086] In a preferred embodiment, as shown in the appendix Figure 1 As shown, the intrinsic parameters of the cameras in the multi-camera module 1 are obtained using the Zhang Zhengyou calibration method for ranging compensation and coordinate transformation.

[0087] In summary, the data flow of this forklift active safety control system is as follows: the images acquired by the multi-camera module 1 are transmitted to the AI ​​vision processing unit 2, the AI ​​vision processing unit 2 processes the images and transmits them to the ranging calculation module 3, the ranging calculation module 3 performs ranging calculations and transmits them to the alarm execution module 4, the alarm execution module 4 transmits the triggered alarm level information to the VCU control module 5, the VCU control module 5 feeds back to the alarm execution module 4, and the alarm execution module 4 executes the alarm action.

[0088] like Figure 4 As shown, the CAN bus loop consists of the VCU control module 5 and the CAN communication module. The CAN communication module mainly includes two CAN lines: CAN1 and CAN2. CAN1 is interconnected with the vehicle's infotainment system, enabling human-machine interaction through CAN signal exchange. CAN2 connects the VCU control module 5 to peripherals, including a motor controller 6, an audible and visual alarm device 41, and a voice prompt unit 42. This allows the VCU control module 5 to control the forklift motor speed, the LED strip output of the audible and visual alarm device 41, and the voice alarm of the voice prompt unit 42 via the motor controller 6, thus enabling information exchange between the peripherals and the VCU control module.

[0089] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0090] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A forklift active safety control method based on AI vision and VCU control, characterized in that: Includes the following steps: S1: Capture 360° environmental images around the forklift using a camera; S2: Based on the lightweight YOLOv8-NCNN model, the environmental image acquired by S1 is processed in real time to detect personnel targets and output the corresponding bounding box coordinates; S3: Based on the camera intrinsic parameter matrix in S1 and the bounding box coordinates provided in S2, calculate the real-time distance between the personnel and the forklift using a geometric projection algorithm; S4: Based on the preset safe distance threshold, compare it with the real-time distance provided by S3, and trigger the corresponding alarm level; S5: Based on the alarm level of S4, execute alarm actions and dynamically adjust the forklift travel speed.

2. The forklift active safety control method based on AI vision and VCU control according to claim 1, characterized in that: In S4, the alarm levels include: security status level, yellow alarm level, and red alarm level; The safe distance thresholds include: a yellow alarm threshold and a red alarm threshold; The safety status level is set as follows: personnel distance ≥ yellow alarm threshold; The yellow alarm level is set as follows: yellow alarm threshold > personnel distance ≥ red alarm threshold; The red alarm level is set as follows: personnel distance < red alarm threshold.

3. The forklift active safety control method based on AI vision and VCU control according to claim 2, characterized in that: In S5, If the alarm level is a safe status level, and the personnel distance is greater than or equal to the yellow alarm threshold, the alarm action to be performed is: dynamically adjust the forklift to run at full speed through the VCU control module; If the alarm level is yellow, and the yellow alarm threshold > personnel distance ≥ red alarm threshold, the alarm action is to dynamically adjust the forklift speed limit to the preset yellow alarm speed through the VCU control module and trigger the lights to flash. If the alarm level is red, and the personnel distance is less than the red alarm threshold, the alarm action is as follows: the forklift speed limit is dynamically adjusted to the preset red alarm speed through the VCU control module, and voice prompts and flashing lights are triggered simultaneously.

4. The forklift active safety control method based on AI vision and VCU control according to claim 3, characterized in that: In S1, the camera used to collect 360° environmental images around the forklift is deployed at least at the front left, front right, and rear of the forklift. Furthermore, the camera has a field of view ≥120°, a resolution of not less than 1280×720, and a coverage radius of at least 10 meters.

5. The forklift active safety control method based on AI vision and VCU control according to claim 4, characterized in that: In S3, the calculation formula for the geometric projection algorithm is as follows: Among them, g x and g y The pixel width and height of the human detection bounding box in the image plane are respectively; f x and f y p represents the X-axis and Y-axis components of the camera's focal length; w and p h α and β are the average height and shoulder width of a person in the application scenario, respectively; β is the proximity coefficient; and α is the unit coefficient.

6. The forklift active safety control method based on AI vision and VCU control according to claim 5, characterized in that: In S2, the YOLOv8-NCNN model is quantized using INT8, with a model size ≤ 6MB and an inference frame rate ≥ 30FPS.

7. A forklift active safety control system based on AI vision and VCU control, comprising the forklift active safety control method according to any one of claims 1-6, characterized in that: include: Multi-camera module, which captures 360° environmental images around the forklift via cameras; The AI ​​vision processing unit is equipped with a lightweight YOLOv8-NCNN model for detecting human targets and outputting the corresponding bounding box coordinates. This unit processes environmental images acquired by the multi-camera module in real time. The ranging calculation module collects the camera intrinsic parameter matrix of the multi-camera module and the bounding box coordinates of the AI ​​vision processing unit, and calculates the real-time distance between the person and the forklift through a geometric projection algorithm. The alarm execution module presets a safe distance threshold. This threshold is compared with the real-time distance provided by the ranging calculation module, triggering the corresponding alarm level. Based on the alarm level, it executes an alarm action. The VCU control module adjusts the forklift's travel speed based on the alarm level transmitted by the alarm execution module.

8. A forklift active safety control system based on AI vision and VCU control according to claim 7, characterized in that: The intrinsic parameters of the multi-camera module are obtained using the Zhang Zhengyou calibration method.

Images