A warehouse-in task flow acquisition method based on image recognition technology

Abstract

The application discloses a warehouse-in task flow acquisition method and system based on image recognition technology, and overcomes the problem of low precision of the warehouse-in flow process in the prior art, which adopts a YOLOV7 algorithm to detect large targets and has no pre-frame design. The automatic control method comprises the following steps: S1, dividing the warehouse-in task flow, acquiring warehouse-in task information and dividing loading and unloading areas; S2, enhancing the YOLOV7 model by using an auxiliary function, and identifying vehicles by using the enhanced YOLOV7 model; S3, screening the identification result and judging whether the identification result is correct; S4, acquiring a task start time, loading and unloading goods and warehousing the goods; S5, identifying vehicle license plate information, and recording the license plate information into the warehouse-in task that has been bound; and S6, judging whether the warehouse-in task is completed, and recording a task end time after the task is completed. The method enhances the large target detection precision of YOLOV7, binds the related transport vehicles involved through the license plate, and leaves reliable information for subsequent examination and accountability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security management technology, and in particular to a method for obtaining the data entry task process based on image recognition technology. Background Technology

[0002] For all types of warehouses, controlling the inbound task process is crucial for warehouse management. Currently, inbound control typically relies on manual work orders, which often results in inaccurate task data and significant time discrepancies, posing considerable difficulties for subsequent monitoring and querying.

[0003] With the continuous development of computer information technology, image recognition technology is becoming increasingly widespread and popular in various fields, and has even begun to permeate our daily lives. It has promising prospects such as greatly improving production efficiency and reducing labor costs. Furthermore, the development of warehouse management is becoming increasingly mature, and integrating image recognition and intelligent information technology into warehouse management systems can better meet user needs. Currently, image recognition technology typically relies on deep learning technology, acquiring image features through stacked convolutional layers. YOLOv7 is a novel object detection algorithm that offers faster speed and accuracy compared to other object detection algorithms such as YOLOv6. However, due to its design without pre-selected bounding boxes, its accuracy still has room for further improvement when the camera's viewpoint and location are fixed. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems of low accuracy in existing technologies that use the YOLOv7 algorithm to detect large targets in warehouse inbound processes without a pre-selection box. This invention provides a method for obtaining inbound task processes based on image recognition technology. By using an improved YOLOv7 algorithm, the inbound time nodes are automatically recorded, thereby obtaining the inbound process. This enhances the accuracy of YOLOv7 in detecting large targets and binds the relevant transport vehicles through license plates, leaving reliable information for subsequent review and accountability.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] S1: Divide the inbound task process, obtain inbound task information and divide the loading and unloading areas;

[0007] S2: Enhance the YOLOv7 model using auxiliary functions, and use the enhanced YOLOv7 model to identify vehicles;

[0008] S3: Filter the recognition results and determine whether the recognition results are correct;

[0009] S4: Obtain the task start time, load and unload goods, and put the goods into the warehouse;

[0010] S5: Identify vehicle license plate information and record the license plate information into the already bound inbound task;

[0011] S6: Determine whether the inbound task is completed, and record the task end time after the task is completed.

[0012] The identified vehicles are trucks and forklifts. This invention utilizes auxiliary functions to enhance the traditional YOLOv7 algorithm's ability to identify trucks and forklifts, and can filter the identification results to further improve the accuracy of identification. It can automatically record the warehouse entry time node and task completion time node, obtain the entry process, and can use the binding of relevant transportation vehicles involved to leave reliable information for subsequent review and accountability.

[0013] Preferably, step S1 further includes:

[0014] S1.1: Divide the warehousing task into a task start node and a task end node, and the time between the two nodes is the task duration; obtain the materials that need to be warehoused, and bind the task name and material list.

[0015] S1.2: Divide the area into rectangular loading and unloading zones, and mark vehicle parking positions within the loading and unloading zones. Install cameras in the loading and unloading zones to photograph the vehicles.

[0016] The warehousing task is divided into two key nodes: the task start node and the task end node. In real-world scenarios, the work order system obtains the materials required for the next day's task at least one day in advance, binding the task name and material list. This facilitates subsequent tracking of task completion status.

[0017] Preferably, step S1.2 further includes:

[0018] S1.2.1: Based on the characteristics of the warehouse and the parking location of the vehicles, a first monitoring camera is installed along the length of the vehicle so that the first monitoring camera covers the entire loading and unloading area;

[0019] S1.2.2: Install a second monitoring camera in the width direction of the vehicle;

[0020] S1.2.3: Use the vision system to determine four edge points in the loading and unloading area of ​​the image captured by the first monitoring camera, obtain the image coordinates of the four edge points, and determine the electronic loading and unloading area.

[0021] The first monitoring camera captures the side of the vehicle; the second monitoring camera, a zoom camera, captures the front of the vehicle. The horizontal area of ​​the electronic fence in the electronic loading and unloading area occupies approximately 1920 pixels.

[0022] Preferably, in step S2, the auxiliary function is:

[0023]

[0024] Where p is the predicted result and gt is the actual result. This is the loss function of the YOLOv7 model. τ is a hyperparameter used to avoid high uncertainty, and x is the uncertainty, defined as follows:

[0025] x = ReLU(w*x + b)

[0026] Where x is the feature map, p is the prediction result of the YOLOv7 model, w and b are learnable parameters, and ReLU is used to make α≥0.

[0027] YOLOv7, through its FPN module, can improve the AP (Action Point) for small and medium targets; however, testing showed that the AP actually decreased for large targets. The forklifts and trucks to be identified in this invention are both large targets. Using auxiliary loss can help achieve the ability to see all objects, thereby enhancing the detection capability for large targets.

[0028] Preferably, step S2 further includes training the enhanced YOLOv7 model:

[0029] A1: Obtain vehicle and corresponding license plate data from the network to perform coarse training on the enhanced YOLOv7 algorithm;

[0030] A2: Collect and label vehicle and corresponding license plate data independently, and perform fine training on the enhanced YOLOV7 algorithm. During fine training, data augmentation is performed using flipping, translation, radialization, and mix-up.

[0031] Mix-up is an algorithm used in computer vision to enhance images by mixing different classes, thereby expanding the training dataset. After fine training, a YOLOv7 model with the ability to detect forklifts, trucks, and license plates was obtained.

[0032] Preferably, step S3 is further expressed as:

[0033] S3.1: Based on the vehicle proportions, filter out results whose length and width ratios within the model prediction box do not meet the vehicle proportion requirements;

[0034] S3.2: Based on the width ratio between the video frame and the model prediction box, further filter out prediction boxes that do not meet the conditions.

[0035] A YOLOv7 model with an auxiliary loss function was trained using a vehicle-related dataset. However, the resulting YOLOv7 model still exhibits false positives in practical applications, misidentifying non-truck / forklift units as truck / forklift units, thus triggering the task incorrectly. Therefore, it is necessary to filter the identification results to determine whether trucks / forklifts have been correctly identified. The model's predicted bounding box is the rectangular box containing the vehicle in the image.

[0036] Preferably, step S4 is further expressed as:

[0037] If a truck target and a forklift target are detected simultaneously at a certain time, and the midpoint of the lower border of the detected target is in the loading and unloading area, then the loading and unloading state is triggered and recorded as the task start time.

[0038] Once the task begins, loading and unloading of goods commences, which is the process of warehousing goods.

[0039] Preferably, the task start time also includes:

[0040] The warehouse utilizes an RFID system. Upon triggering a loading / unloading status, the system retrieves newly added goods and compares them with all goods in the inbound task information. If a item belongs to a specific task that has not yet been triggered, the task's start time is the most recent task start time. This ensures precise task start time.

[0041] Preferably, step S5 is further expressed as:

[0042] S5.1: Capture an image of the vehicle's license plate. Use the enhanced YOLOv7 model to obtain the coordinates (x1, y1) of the top-left corner of the rectangular region R containing the license plate, as well as its width and height (w, h). Calculate the center coordinates (c). x ,c y );

[0043] S5.2: Focus on (c x ,c y The license plate image is re-captured in the area R centered on the vehicle, and the license plate is identified through the HyperLPR network. The license plate information is recorded in the bound task, and the images from the first and second captures are retained.

[0044] Linking tasks to license plates means linking them to the relevant transport vehicles involved, leaving reliable information for subsequent investigation and accountability.

[0045] Preferably, step S6 is further expressed as:

[0046] The RFID system is used to determine if all goods in the assigned task are within the warehouse RFID system. If so, the task end time is triggered, and the precise inbound end time is recorded. This completes the entire inbound task process.

[0047] Therefore, the present invention has the following beneficial effects: 1. It enhances the recognition ability of the traditional YOLOv7 algorithm for trucks and forklifts by using auxiliary functions, and can filter the recognition results to further improve the recognition accuracy; 2. It can automatically record the warehouse entry time node and the task end time node to obtain the entry process; 3. It can use the binding of relevant transportation vehicles involved to leave reliable information for subsequent review and accountability. Attached Figure Description

[0048] Figure 1 This is a flowchart of the steps of the present invention.

[0049] Figure 2 This is a schematic diagram of the loading and unloading area in this invention.

[0050] In the picture: 1. Loading and unloading area; 2. Forklift; 3. Truck; 4. First surveillance camera; 5. Second surveillance camera. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0052] Example 1:

[0053] This embodiment describes a method for obtaining the data entry task flow based on image recognition technology, such as... Figure 1 As shown, the process includes the following steps: First, divide the warehousing task flow, obtain warehousing task information, and divide the loading and unloading areas; Second, use auxiliary functions to enhance the YOLOv7 model, and use the enhanced YOLOv7 model to identify vehicles; Third, filter the identification results and determine whether the identification results are correct; Fourth, obtain the task start time, load and unload the goods, and put the goods into the warehouse; Fifth, identify the vehicle license plate information and record the license plate information in the bound warehousing task; Sixth, determine whether the warehousing task is completed, and record the task end time after the task is completed.

[0054] The warehousing process acquisition in this invention refers to obtaining the start time of the warehousing task, the end time of the warehousing task, and other information about the warehousing task itself. This invention utilizes auxiliary functions to enhance the traditional YOLOv7 algorithm's ability to identify trucks and forklifts, and can filter the identification results to further improve accuracy. It can automatically record the warehouse's warehousing time nodes and task end time nodes, acquire the warehousing process, and, by binding the relevant transport vehicles involved, leave reliable information for subsequent review and accountability.

[0055] The automatic control method of this application will be further explained below:

[0056] Step 1: Define the inbound task process, obtain inbound task information, and define loading and unloading areas.

[0057] 1. Divide the warehousing task into two key nodes: the warehousing task start node and the warehousing task end node. The time between the two nodes is the task duration; obtain the materials that need to be warehoused and bind the task name and material list.

[0058] 2. Divide the area into rectangular loading and unloading zones, and mark vehicle parking positions within these zones. Install cameras in the loading and unloading zones to photograph the vehicles: Based on the characteristics of the warehouse and the vehicle parking positions, install the first monitoring camera along the length of the vehicle to cover the entire loading and unloading area; install the second monitoring camera along the width of the vehicle.

[0059] like Figure 2 As shown, in this embodiment, a rectangular loading and unloading area 1 with a size of 6m*10m is defined. A large truck 3 is positioned below the rectangular loading and unloading area. Based on the warehouse characteristics, the trucks are 3.5-5 ton trucks with a length of 6-7m. A first monitoring camera 4 with a 4mm lens is installed 12m to the side, so that the first monitoring camera just covers the entire loading and unloading area. The first monitoring camera faces the longer side of the rectangular loading and unloading area, capturing the side view of the truck. Simultaneously, a second monitoring camera 5 faces the shorter side of the rectangular loading and unloading area, capturing the front view of the truck. The second monitoring camera is a zoom camera.

[0060] The vision system is used to identify four edge points in the loading and unloading area captured by the first monitoring camera, and the image coordinates of the four edge points are obtained to determine the electronic loading and unloading area.

[0061] In this embodiment, four points are drawn in the loading and unloading area of ​​the image captured by the first monitoring camera using OpenCV. The image coordinates of the four corner points are obtained, and this area on the image is the electronic loading and unloading area. At this time, the horizontal number of pixels occupied by the electronic fence is approximately 1920.

[0062] Step 2: Enhance the YOLOv7 model using auxiliary functions, and then use the enhanced YOLOv7 model to identify vehicles.

[0063] YOLOv7, through its FPN module, can improve the AP (Action Per Target) for small and medium targets; however, testing showed that the AP actually decreased for large targets. The forklifts and trucks to be detected in this method are both large targets. Using auxiliary loss can help achieve the ability to see all objects, thereby enhancing the detection capability for large targets.

[0064] The auxiliary function is:

[0065]

[0066] Where p is the predicted result and gt is the actual result. This is the loss function of the YOLOv7 model. τ is a hyperparameter used to avoid high uncertainty, and x is the uncertainty, defined as follows:

[0067] x = ReLU(w*x + b)

[0068] Where x is the feature map, p is the prediction result of the YOLOv7 model, w and b are learnable parameters, and ReLU is used to make α≥0.

[0069] Training the enhanced YOLOv7 model:

[0070] We collected 4,000 images each of forklifts, trucks, and license plates from the internet and used them for coarse training of the enhanced YOLOv7 algorithm. We also collected and labeled 1,000 images each of forklifts, trucks, and license plates and used them for fine training of the enhanced YOLOv7 algorithm. During fine training, we used flipping, translation, radialization, and mix-up techniques to augment the data.

[0071] Step 3: Filter the recognition results to determine whether they are correct.

[0072] Based on the vehicle proportions, results whose aspect ratios within the model's predicted bounding box do not meet the vehicle proportions are filtered out; based on the width ratio between the video frame and the model's predicted bounding box, prediction boxes that do not meet the conditions are further filtered out.

[0073] Specifically, this includes filtering the results of truck target recognition and filtering the results of forklift target recognition.

[0074] 1. Filtering of truck target recognition results.

[0075] A YOLOv7 model with an auxiliary loss function was trained using a truck-related dataset. However, the obtained YOLOv7 model still has false detections in actual use, that is, non-truck units are identified as truck units, thus triggering the task in a misaligned manner.

[0076] Taking the dimensions of commonly used 3.5T-8T trucks as an example, based on the proportional relationship, results whose aspect ratios of the model prediction box (the rectangle containing the vehicle) do not satisfy the following formula are filtered out:

[0077]

[0078] Where d is the error coefficient, which is set to 0.1 based on actual testing; w represents the width of the detection model's predicted bounding box; and h represents the height of the model's predicted bounding box.

[0079] Since a 4mm first monitoring camera is set up at 12m, and the horizontal field of view is 80 degrees, the length of the predicted frame should occupy 0.6-0.7 of the entire electronic loading and unloading area. Using this relationship, the following formula is used to further filter out predicted frames that do not meet the requirements:

[0080]

[0081] Where W is the width of the current video frame (the entire image), and w is the width of the prediction box.

[0082] 2. Filtering of forklift target recognition results.

[0083] YOLOv7 is trained using a forklift-related dataset to obtain a YOLOv7 model, but the obtained YOLOv7 model may also have false positives.

[0084] Based on the 2:1 aspect ratio of the forklifts in the warehouse in this embodiment, the following formula is used to retain the prediction box results that satisfy the following ratio:

[0085]

[0086] Step 4: Obtain the task start time, load and unload the goods, and put them into the warehouse.

[0087] If a truck target and a forklift target are detected simultaneously at a certain time, and the midpoint of the lower border of the detected target is in the loading and unloading area, then the loading and unloading state is triggered and recorded as the task start time.

[0088] The warehouse uses an RFID system to acquire newly added goods after the loading and unloading status is triggered. The newly added goods are then compared with all goods in the inbound task information. If the goods belong to a certain task and the task has not been triggered before, the start time of that task is the most recent task start time.

[0089] Step 5: Identify vehicle license plate information and record the license plate information in the already bound data entry task.

[0090] Capture an image of the vehicle's license plate. Use an enhanced YOLOv7 model to obtain the coordinates (x1, y1) of the top-left corner, and the width and height (w, h) of the rectangular region R containing the license plate. Calculate the center coordinates (c). x ,c y ):

[0091]

[0092]

[0093] Variable zoom camera focuses to (c x ,c yThe license plate image is re-captured in the area R centered on the vehicle, and the license plate is identified through the HyperLPR network. The license plate information is recorded in the bound task, and the images from the first and second captures are retained.

[0094] Step 6: Determine whether the data entry task is completed, and record the task end time after completion.

[0095] The system uses an RFID system to determine whether all goods in the assigned task are in the warehouse RFID system. If so, the task end time is triggered, and the precise warehousing end time is recorded.

[0096] The method of this invention enhances the detection capability of the existing YOLOv7 algorithm for large targets, can automatically obtain the start time and end time of a certain inbound task, and can bind the inbound task with the vehicle performing the inbound task so that the responsible party can be found in time when problems occur.

[0097] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Other variations and modifications are possible without departing from the technical solutions described in the claims.

Claims

1. A method for obtaining the data entry task flow based on image recognition technology, characterized in that, include: S1: Define the inbound task process, obtain inbound task information, and delineate loading and unloading areas: The warehousing task is divided into a task start node and a task end node, and the time between the two nodes is the task duration; the materials to be warehoused are obtained, and the task name and material list are bound together. A rectangular loading and unloading area is divided, and vehicle parking positions are marked within the loading and unloading area. Based on the characteristics of the warehouse and the vehicle parking positions, a first monitoring camera is installed along the length of the vehicle, covering the entire loading and unloading area; a second monitoring camera is installed along the width of the vehicle; using a vision system, four edge points are determined in the loading and unloading area captured by the first monitoring camera, and the image coordinates of the four edge points are obtained to determine the electronic loading and unloading area. S2: Enhance the YOLOv7 model using auxiliary functions, and use the enhanced YOLOv7 model to identify vehicles; S3: Filter the recognition results and determine whether the recognition results are correct; S4: Obtain the task start time, load and unload goods, and put the goods into the warehouse; S5: Identify vehicle license plate information and record it in the already bound database entry task: Take a picture of the vehicle license plate, use the enhanced YOLOv7 model to obtain the coordinates of the upper left corner, width, and height of the license plate rectangular region R, and calculate the center coordinates (c x ,c y ); Focusing on (c x , c y In the area R centered on ), the license plate image is re-captured, the license plate is identified through the HyperLPR network, the license plate information is recorded to the already bound task, and the images captured twice are retained; S6: Determine whether the inbound task is completed, and record the task end time after the task is completed.

2. The method for obtaining the data entry task flow based on image recognition technology according to claim 1, characterized in that, The auxiliary function is: Where p is the predicted result and gt is the actual result. This is the loss function of the YOLOv7 model. τ is a hyperparameter used to avoid high uncertainty, and x is the uncertainty, defined as follows: x = ReLU(w*m + b). Where m is the feature map, p is the prediction result of the YOLOv7 model, w and b are learnable parameters, and ReLU is used to make α≥0.

3. A method for obtaining a database entry task flow based on image recognition technology according to claim 1 or 2, characterized in that, Step S2 also includes training the enhanced YOLOv7 model: A1: Obtain vehicle and corresponding license plate data from the network to perform coarse training on the enhanced YOLOv7 algorithm; A2: Collect and label vehicle and corresponding license plate data independently, and perform fine training on the enhanced YOLOV7 algorithm. During fine training, data augmentation is performed using flipping, translation, radialization, and mix-up.

4. A method for obtaining a database entry task flow based on image recognition technology according to claim 1 or 2, characterized in that, Step S3 is further expressed as follows: S3.1: Based on the vehicle proportions, filter out results whose length and width ratios within the model prediction box do not meet the vehicle proportion requirements; S3.2: Based on the width ratio between the video frame and the model prediction box, further filter out prediction boxes that do not meet the conditions.

5. A method for obtaining a database entry task flow based on image recognition technology according to claim 1 or 2, characterized in that, Step S4 is further expressed as follows: If a truck target and a forklift target are detected simultaneously at a certain time, and the midpoint of the lower border of the detected target is in the loading and unloading area, then the loading and unloading state is triggered and recorded as the task start time.

6. The method for obtaining the data entry task flow based on image recognition technology according to claim 5, characterized in that, The task acquisition start time also includes: The warehouse uses an RFID system to acquire newly added goods after the loading and unloading status is triggered. The newly added goods are then compared with all goods in the inbound task information. If the goods belong to a certain task and the task has not been triggered before, the start time of that task is the most recent task start time.

7. A method for obtaining a database entry task flow based on image recognition technology according to claim 1 or 2, characterized in that, Step S5 is further expressed as follows: the horizontal coordinate of the center coordinate is the sum of half the width of the license plate rectangular area R and the horizontal coordinate of the upper left corner of the license plate rectangular area R, and the vertical coordinate of the center coordinate is the sum of half the height of the license plate rectangular area R and the vertical coordinate of the upper left corner of the license plate rectangular area R.

8. A method for obtaining a database entry task flow based on image recognition technology according to claim 1 or 2, characterized in that, Step S6 is further expressed as follows: The system uses an RFID system to determine whether all goods in the assigned task are in the warehouse RFID system. If so, the task end time is triggered, and the precise warehousing end time is recorded.

Images