A forklift recognition and processing method based on vision and image processing

Abstract

The present invention discloses a forklift recognition and processing method based on vision and image processing, which relates to the technical field of production environment monitoring, and includes the following steps: Step 1, obtain the images captured by the thermal imaging camera in real time, and identify whether there is a forklift in the images; Step 2, after identifying the position of the forklift, record the position coordinates of the forklift, and convert the visible light image into a two-dimensional coordinate V indicating whether the pixel contains a forklift according to the pixel points y×z . The forklift recognition and processing method provided by the present invention can improve the quality of thermal imaging early warning and alarm in the warehouse, avoid meaningless early warning and alarm behaviors caused by daily forklift operations, reduce the workload of safety management personnel and also avoid the generation of complacency about early warning and alarm, achieving that every alarm is real; in addition, customized safety management measures can be formulated for forklifts according to this method, and the feasible routes of forklifts can be monitored and controlled in cooperation with temperature zone division, standardizing safe operations.

Description

Technical Field

[0001] The present invention relates to the technical field of production environment monitoring, and particularly to a forklift recognition and processing method based on vision and image processing. Background Art

[0002] Thermal imaging technology is the core technology for monitoring the environment, equipment, and warehouse temperature in the current production environment of factories. A large number of factories use binocular thermal imaging cameras installed to detect the temperature in the environment and achieve real-time monitoring.

[0003] However, the existing thermal imaging technology lacks universality in meeting the industry's identification requirements for specific objects (such as forklifts) in specific scenarios. Forklifts are the main means of transportation used when goods are warehoused and shipped out in the factory area. The temperature of a forklift during operation can reach above 80 degrees Celsius. The warning temperature for temperature monitoring in most storage environments is below 80 degrees Celsius. Frequent forklift operations will cause the warning system to "false alarm" multiple times.

[0004] According to the relevant on-site safety personnel, more than 95% of the daily temperature alarms in the storage are meaningless alarms caused by forklift operations. The original alarm mode reduces the accuracy of the alarm and unconsciously reduces the high attention and high alertness of on-site safety personnel to the alarm behavior, resulting in ideological paralysis and safety problems. At the same time, the forklift itself as an object should also be monitored and managed to avoid safety accidents caused by forklift fires in the storage. Summary of the Invention

[0005] The main purpose of the present invention is to provide a forklift recognition and processing method based on vision and image processing, which combines thermal imaging technology to establish a separate temperature threshold for special recognition of forklifts to achieve early warning.

[0006] The object of the present invention can be achieved by adopting the following technical solutions:

[0007] A forklift recognition and processing method based on vision and image processing, comprising the following steps

[0008] Step 1: Real-time obtain the image captured by the thermal imaging camera, and identify whether there is a forklift in the image;

[0009] Step 2: After identifying the position of the forklift, record the position coordinates of the forklift, and convert the visible light image into a two-dimensional coordinate V indicating whether the pixel contains a forklift according to pixel points y×z ;

[0010] Step 3: Superimpose the thermal imaging frame temperature matrix H corresponding to the visible light image with the visible light image matrix V, and screen out the area matrix F containing the forklift in the superimposed image;

[0011] Step 4: Screen out the highest temperature and average temperature in matrix F and the remaining matrix H - F without forklifts respectively, and use them as the temperature recording alarm values of the forklift and the ambient temperature, and report them to the early warning and alarm system.

[0012] Preferably, in step 3, the two-dimensional coordinates H of the thermal imaging screen temperature matrix H m×n , where m and n represent the positions of the pixel coordinate axes of the thermal imaging screen, and t represents the temperature at that pixel position.

[0013]

[0014] Visible light screen image matrix V, where x and y represent the positions of the pixel coordinate axes of the visible light screen, 0 indicates that no forklift is recognized at that pixel point, and 1 indicates that it is a pixel point where a forklift is recognized.

[0015]

[0016] Preferably, when H and V in step 3 are superimposed, the starting position of H in V needs to be adjusted according to the lens offset parameters set in the thermal imaging camera before superimposing.

[0017] Record the t corresponding to the pixel coordinates with value 1 in V within H in F, and record the remaining matrix within H as H - F.

[0018] Calculate MAX(F), AVG(F), MAX(H - F), and AVG(H - F) respectively, and provide them as real-time temperature data values to the superior early warning and alarm system.

[0019] Preferably, before identifying whether there is a forklift in the picture in step 1, there is also a step of improving the recognition of the forklift, which is specifically as follows

[0020] Step A: Obtain the forklift data set, and collect multiple real-time monitoring visible light pictures and corresponding thermal imaging pictures and thermal imaging temperature dot matrix data in the warehouses from multiple factory areas.

[0021] Step B: Extract multiple visible light pictures containing forklift pictures by manual screening and annotation, split them into a training set and a test set according to a ratio of 8:2, and use the training set data to train the model.

[0022] Step C: Divide the forklifts in the pictures into four major feature areas: bucket, main body, wheels, and rear of the vehicle, and record the eigenvalue information through binarization.

[0023] Step D: After feature training, generate a forklift feature model, and test it through the test set, adjust the model, and iterate the training data to meet the recognition accuracy requirements.

[0024] Mount the forklift feature recognition model that meets the recognition accuracy requirements in step D into the thermal imaging camera in step 1 to achieve the purpose of forklift image recognition.

[0025] Preferably, in step C, special model processing is also included for special scenarios such as occlusion, small scenes in the distance, partial display scenes in the near vicinity, and corner scenes.

[0026] The beneficial technical effects of the present invention: The forklift recognition and processing method provided by the present invention can improve the quality of thermal imaging early warning and alarm in the warehouse, avoid meaningless early warning and alarm behaviors caused by daily forklift operations, reduce the workload of safety management personnel, and also avoid the emergence of complacency about early warning and alarm, achieving that every alarm is real; in addition, customized safety management measures can be formulated for forklifts according to this method, and the feasible routes of forklifts can be monitored and controlled in cooperation with temperature zone division to standardize safe operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic flowchart of the method according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To make the technical solutions of the present invention clearer and more definite for those skilled in the art, the present invention will be further described in detail below with reference to the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0029] As Figure 1 shown, the forklift recognition and processing method based on vision and image processing provided in this embodiment includes the following steps

[0030] Step 1: Real-time obtain the images captured by the binocular thermal imaging camera, that is, real-time collect and store the images of the repository, and identify whether there is a forklift in the images.

[0031] After identifying the position of the forklift, record the position coordinates of the forklift, and convert the visible light image into a two-dimensional coordinate V indicating whether each pixel contains a forklift according to the pixel points, as shown in Equation 2; y×z as shown in Equation 2;

[0032] Step 3: Superimpose the temperature matrix H of the thermal imaging image corresponding to the visible light image and the visible light image matrix V, and screen out the area matrix F containing the forklift in the superimposed image, which can effectively distinguish the temperature of the forklift itself from the surrounding environment. If there are multiple forklifts, they are respectively denoted as F 1 、F 2 ...;

[0033] Step 4: Screen out the highest temperature and average temperature in matrix F and the remaining matrix H - F without forklifts respectively, and use them as the temperature recording alarm values of the forklift and the ambient temperature and report them to the early warning and alarm system. The system is respectively built-in with the ambient temperature threshold and the early warning temperature threshold of the forklift. When the highest temperature of the forklift (matrix F) is higher than the early warning temperature threshold of the forklift or the highest temperature in matrix H - F is higher than the ambient temperature threshold, it is regarded as too high temperature and an external alarm is triggered.

[0034] In this embodiment, the two-dimensional coordinates H of the thermal imaging screen temperature matrix H m×n , where m and n represent the positions of the pixel coordinate axes of the thermal imaging screen, and t represents the temperature at this pixel position.

[0035]

[0036] Visible light screen image matrix V, where x and y represent the positions of the pixel coordinate axes of the visible light screen, 0 indicates that no forklift is recognized at this pixel point, and 1 indicates a pixel point where a forklift is recognized.

[0037]

[0038] When H and V are superimposed, the lens offset parameters set in the binocular thermal imaging camera need to be used. Since the resolutions of the binocular lenses are different, data superposition calculation needs to be performed after offsetting, and the starting position of H in V needs to be adjusted to ensure that the starting positions of H and V after adjustment are the same coordinates of the actual screen, and then superposition is performed.

[0039] Record the t values of the pixel coordinates corresponding to the 1 values in V included in H in F, and record the remaining matrix in H as H - F.

[0040] Calculate MAX(F), AVG(F), MAX(H - F), and AVG(H - F) respectively, and provide them as real-time temperature data values to the superior early warning and alarm system.

[0041] In this embodiment, the two-dimensional coordinate values of F and H are as follows:

[0042]

[0043]

[0044] Set the H corresponding to the position (3, 4) of F, then H - F is:

[0045]

[0046] Then through observation and mean calculation, it is obtained that:

[0047] MAX(F) = 88, that is, the highest temperature of matrix F is 88°C;

[0048] AVG(F) = 67, that is, the average temperature of matrix F is 67°C;

[0049] MAX(H - F) = 86, that is, the highest temperature of the remaining matrix H - F without forklift is 86°C;

[0050] AVG(H - F) = 47, that is, the average temperature of the remaining matrix H - F without forklift is 47°C.

[0051] In this embodiment, as Figure 1 shown, before identifying whether there is a forklift in the picture in step 1, there is also a step to improve the accuracy of forklift identification, specifically as follows

[0052] Step A: Obtain the forklift dataset, collect at least 2000 hours of real-time in-warehouse monitoring visible light pictures, corresponding thermal imaging pictures and thermal imaging temperature dot matrix data from the sites of multiple factories. A relatively rich database can be established through a large number of visible light pictures, corresponding thermal imaging pictures and thermal imaging temperature dot matrix data;

[0053] Step B: By means of manual screening and annotation, at least 240,000 visible light pictures containing forklift pictures are extracted, and are split into a training set and a test set according to a ratio of 8:2. Using the training set data to train the model, dividing the visible light pictures containing forklifts into a training set and a test set not only helps in the initial establishment of the recognition model, but also helps in testing the accuracy of the model;

[0054] Step C: Divide the forklift in the picture into four major feature areas: bucket, main body, wheels, and rear of the vehicle. Record the eigenvalue information through binarization, and make special model processing for special scenarios such as occlusion, small scenes in the distance, partially displayed scenes nearby, and corner scenes, which can effectively improve the recognition accuracy of forklifts in special scenarios;

[0055] Step D: After feature training, generate a forklift feature model and test it through the test set, adjust the model, and iterate the training data to meet the recognition accuracy requirements;

[0056] Install the forklift recognition feature model that meets the recognition accuracy requirements in step D into the thermal imaging camera in step 1 to achieve the purpose of effective recognition of forklift images.

[0057] In summary, in this embodiment, the forklift recognition processing method provided in this embodiment can improve the quality of thermal imaging early warning and alarm in the warehouse, avoid meaningless early warning and alarm behaviors caused by daily forklift operations, reduce the workload of safety management personnel and also avoid the emergence of a complacent attitude towards early warning and alarm, achieving that every alarm is real; in addition, customized safety management measures can be formulated for forklifts according to this method, and the feasible routes of forklifts can be monitored and controlled in cooperation with temperature area division to standardize safe operations.

[0058] As described above, it is only a further embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the scope disclosed by the present invention, according to the technical solution and concept of the present invention, makes equivalent substitutions or changes, and all belong to the protection scope of the present invention.

Claims

1. A forklift recognition and processing method based on vision and image processing, characterized in that: It includes the following steps Step 1: Obtain the images captured by the thermal imaging camera in real time, and identify whether there is a forklift in the images; Step 2: After identifying the forklift position, record the forklift position coordinates, and convert the visible light image into a two-dimensional coordinate V of whether the pixel contains the forklift according to the pixel points yxz ; Step 3: Superimpose the thermal imaging frame temperature matrix H corresponding to the visible light frame image and the visible light frame image matrix V, and screen out the area matrix F containing the forklift in the superimposed image; Step 4: Screen out the highest temperature and average temperature in the matrix F and the remaining matrix H - F without forklift respectively, record the alarm values as the forklift temperature and the ambient temperature, and report them to the early warning and alarm system; In the said step 3, the two-dimensional coordinates H of the temperature matrix H of the thermal imaging screen m x n , where m and n represent the pixel coordinate axis positions of the thermal imaging screen, and t represents the temperature at this pixel position. ； Visible light frame image matrix V, where x and y represent the pixel coordinate axes positions of the visible light frame, 0 indicates that no forklift is recognized at this pixel point, and 1 indicates that it is a pixel point with forklift recognition, ； When superimposing H and V in Step 3, it is necessary to adjust the starting position of H into V according to the lens offset parameter set in the thermal imaging camera before superimposing; Record the t corresponding to the pixel coordinates with value 1 in V within H into F, and record the remaining matrix within H as H - F; Calculate MAX(F), AVG(F), MAX(H - F) and AVG(H - F) respectively, and provide them to the superior early warning and alarm system as the temperature real-time data values.

2. The forklift recognition and processing method based on vision and image processing according to claim 1, characterized in that: Before identifying whether there is a forklift in the image in Step 1, there is also a step of improving the forklift recognition, specifically as follows Step A: Obtain the forklift data set, collect multiple in - warehouse real - time monitoring visible light images and corresponding thermal imaging images and thermal imaging temperature dot matrix data from the sites of multiple factories; Step B: Extract multiple visible light pictures containing forklift images by manual screening and annotation, split them into a training set and a test set according to the ratio of 8:2, and use the training set data to train the model; Step C: Divide the forklift in the picture into four major feature areas: bucket, main body, wheels, and rear of the vehicle, and record the eigenvalue information through binarization; Step D: Generate a forklift feature model after feature training, and test it through the test set, adjust the model, and iterate the training data to meet the recognition accuracy requirements; Load the forklift recognition feature model that meets the recognition accuracy requirements in Step D into the thermal imaging camera in Step 1 to achieve the purpose of forklift image recognition.

3. The forklift recognition and processing method based on vision and image processing according to claim 2, characterized in that: In Step C, special model processing is also included for special scenarios such as occlusion, small distant scenes, partial near - field display scenes, and corner scenes.

Images