Empty box identification method, device and equipment and storage medium

Abstract

The invention discloses an empty box recognition method, device and equipment and a storage medium, and relates to the technical field of logistics and cargo detection, the empty box recognition method comprises the steps that a real-time video stream of a to-be-recognized vehicle is acquired, and the vehicle comprises a box body capable of loading cargos; scanning the box body by using a laser radar to obtain three-dimensional point cloud data of the box body; on the basis of a pre-trained empty box recognition model, whether the box body is an empty box or not is determined according to the real-time video stream and the three-dimensional point cloud data of the box body, and the pre-trained visual empty box recognition model is obtained according to the images of the empty box body and the non-empty box body in the box door opening state and the three-dimensional point cloud data. And training the visual empty box identification model. According to the scheme, accurate recognition of the empty box is achieved through image and radar data association and decision fusion, the vehicle empty box recognition speed can be effectively increased, the workload of people is relieved, and the empty box recognition efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of logistics and cargo detection, and particularly to an empty container identification method, device, equipment, and storage medium. Background Art

[0002] The existing empty container detection technologies for freight cars mainly fall into two categories: X-ray based empty container detection technology or acoustic based empty container detection technology.

[0003] Although the X-ray based empty container detection technology has advantages such as high resolution, high penetration power, and the ability to detect metals and non-metals, it also has disadvantages such as radiation hazards, expensive equipment, and complex operation and maintenance. The acoustic based empty container detection technology uses a broadband high sound pressure level acoustic load to excite the container body, collects the acoustic vibration signals excited by the container body through a micro acoustic sensor, and analyzes the received and excited acoustic signals of empty and non-empty containers to determine whether the container is empty. However, it has problems such as a small number of detectable types, the ability to only identify standard containers, low success rate and accuracy of empty container detection.

[0004] Based on the above, how to improve the efficiency and accuracy of empty container identification is an urgent problem to be solved.

[0005] The above content is only used to assist in understanding the technical solution of the present application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of the present application is to provide an empty container identification method, device, equipment, and storage medium, aiming to solve the technical problem of how to improve the efficiency and accuracy of empty container identification.

[0007] To achieve the above purpose, the present application proposes an empty container identification method, and the method includes:

[0008] Obtain the real-time video stream of the vehicle to be identified, where the vehicle includes a container body that can load goods;

[0009] Use a lidar to scan the container body to obtain the three-dimensional point cloud data of the container body;

[0010] Based on a pre-trained empty container identification model, confirm whether the container body is an empty container according to the real-time video stream and the three-dimensional point cloud data of the container body. The pre-trained visual empty container identification model is trained according to the images and three-dimensional point cloud data of empty and non-empty container bodies in the open state of the container door. Among them, the visual empty container identification model includes an image recognition model and a container interlayer recognition algorithm.

[0011] In one embodiment, the laser radar includes a first laser radar and a second laser radar, and the step of scanning the box using the laser radar to obtain three-dimensional point cloud data of the box includes:

[0012] Scanning the outer contour of the box using the first laser radar to obtain three-dimensional point cloud data of the outer contour of the box;

[0013] The second laser radar is used to scan the interior of the box to obtain three-dimensional point cloud data inside the box.

[0014] In one embodiment, before the step of scanning the box using a laser radar to obtain three-dimensional point cloud data of the box, the following steps are included:

[0015] Based on a pre-trained empty box recognition model, the type of the box is identified according to the real-time video stream, and the estimated size of the box is obtained;

[0016] According to the estimated size of the box, the position and angle of the second laser radar relative to the box are adjusted.

[0017] In one embodiment, the step of confirming whether the box is empty based on the real-time video stream and the three-dimensional point cloud data of the box based on the pre-trained empty box recognition model includes:

[0018] Based on the box interlayer recognition algorithm, the box interlayer recognition result is determined according to the outer contour three-dimensional point cloud data and the box inner three-dimensional point cloud data;

[0019] Based on the pre-trained image recognition model, according to the real-time video stream and the three-dimensional point cloud data of the box, confirm whether there is any left-behind item inside the box, and identify the left-behind item to obtain an item recognition result;

[0020] Based on a preset first weight configuration, the box interlayer recognition result and the item recognition result are concatenated and merged to obtain an empty box recognition result.

[0021] In one embodiment, the step of determining the box interlayer recognition result based on the box interlayer recognition algorithm according to the outer contour three-dimensional point cloud data and the box inner three-dimensional point cloud data includes:

[0022] Based on the box interlayer recognition algorithm, the outer volume and inner volume of the box are obtained according to the outer contour three-dimensional point cloud data and the box interior three-dimensional point cloud data;

[0023] Performing a difference calculation on the outer volume and the inner volume of the box to determine the box wall thickness;

[0024] When the wall thickness of the box body is greater than a preset wall thickness threshold value, it is determined that the recognition result of the box body sandwich layer is that there is a sandwich layer in the box body;

[0025] When the wall thickness of the box body is not greater than the preset wall thickness threshold value, it is determined that the recognition result of the box body sandwich layer is that there is no sandwich layer in the box body.

[0026] In one embodiment, the step of obtaining the outer volume and the inner volume of the box body based on the box body sandwich layer recognition algorithm according to the outer contour three-dimensional point cloud data and the three-dimensional point cloud data inside the box includes:

[0027] Perform filter processing on the outer contour three-dimensional point cloud data and the three-dimensional point cloud data inside the box to obtain the preprocessed outer contour three-dimensional point cloud data and the three-dimensional point cloud data inside the box;

[0028] Perform plane fitting on the preprocessed outer contour three-dimensional point cloud data to construct a three-dimensional space model of the box body;

[0029] According to the three-dimensional space model of the box body, calculate the outer contour size of the box body to obtain the outer volume of the box body;

[0030] Based on the three-dimensional space model of the box body and the three-dimensional point cloud data inside the box, use the voxel grid method to calculate the inner volume of the box body.

[0031] In one embodiment, the image recognition model includes a first image recognition model and a second image recognition model. The step of confirming whether there are any left items inside the box body and recognizing the left items based on the pre-trained image recognition model according to the real-time video stream and the three-dimensional point cloud data of the box body to obtain the item recognition result includes:

[0032] Based on a preset clustering algorithm, perform point cloud rendering on the three-dimensional point cloud data of the box body to obtain the corresponding box body image;

[0033] Use the pre-trained first image recognition model to perform image analysis and item recognition on the box body image to obtain a first item recognition result;

[0034] Use the pre-trained second image recognition model to perform image analysis and item recognition on the real-time video stream to obtain a second item recognition result;

[0035] Based on a preset second weight configuration, perform hierarchical combination and merging on the first item recognition result and the second item recognition result to obtain the item recognition result.

[0036] In addition, to achieve the above object, the present application also proposes an empty box recognition device, and the empty box recognition device includes:

[0037] A video acquisition module, configured to acquire a real-time video stream of a vehicle to be identified, wherein the vehicle includes a box capable of loading cargo;

[0038] A point cloud data acquisition module, which uses a laser radar to scan the box to obtain three-dimensional point cloud data of the box;

[0039] The empty box recognition module is used to confirm whether the box is empty based on the pre-trained empty box recognition model according to the real-time video stream and the three-dimensional point cloud data of the box. The pre-trained visual empty box recognition model is obtained by training the visual empty box recognition model based on the images and three-dimensional point cloud data of the empty box and the non-empty box with the door open, wherein the visual empty box recognition model includes an image recognition model and a box interlayer recognition algorithm.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes an empty box identification device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the empty box identification method as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the empty box identification method described above are implemented.

[0042] One or more technical solutions proposed in this application have at least the following technical effects:

[0043] The empty box recognition method, apparatus, equipment and storage medium proposed in the embodiments of the present application are specifically implemented by obtaining a real-time video stream of a vehicle to be identified, wherein the vehicle includes a box body that can be loaded with goods; scanning the box body with a laser radar to obtain three-dimensional point cloud data of the box body; based on a pre-trained empty box recognition model, confirming whether the box body is empty based on the real-time video stream and the three-dimensional point cloud data of the box body, wherein the pre-trained visual empty box recognition model is obtained by training the visual empty box recognition model based on images and three-dimensional point cloud data of empty boxes and non-empty boxes with the box door open, wherein the visual empty box recognition model includes an image recognition model and a box interlayer recognition algorithm.

[0044] This application obtains the real-time video stream of the vehicle to be recognized, and at the same time uses lidar to obtain the three-dimensional point cloud data inside and outside the box; the image recognition model in the pre-trained empty box recognition model performs image recognition and analysis on the real-time video stream and the three-dimensional point cloud data, and uses the box sandwich recognition algorithm to perform box sandwich recognition on the box of the vehicle to be recognized, so as to confirm whether the box is an empty box. Through the vehicle empty box detection technology that fuses visual perception and lidar, accurate recognition of empty boxes can be achieved, which can effectively improve the speed of vehicle empty box recognition, reduce the manual workload, and improve the efficiency of empty box recognition. Description of the Drawings

[0045] The drawings here are incorporated into the specification and form a part of this specification, showing the embodiments that conform to this application, and are used together with the specification to explain the principles of this application.

[0046] In order to more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0047] Figure 1 It is a schematic flowchart provided for the first embodiment of the empty box recognition method of this application;

[0048] Figure 2 It is a schematic flowchart provided for the second embodiment of the empty box recognition method of this application;

[0049] Figure 3 It is a schematic flowchart provided for the third embodiment of the empty box recognition method of this application;

[0050] Figure 4 It is a schematic flowchart provided for the fourth embodiment of the empty box recognition method of this application;

[0051] Figure 5 It is a schematic diagram of the process of empty box recognition involved in the embodiment of this application;

[0052] Figure 6 It is an example diagram of the lidar three-dimensional point cloud of the empty box involved in the embodiment of this application;

[0053] Figure 7 It is a schematic diagram of the module structure of the empty box recognition device in the embodiment of this application;

[0054] Figure 8 It is a schematic diagram of the device structure of the hardware operating environment involved in the empty box recognition method in the embodiment of this application.

[0055] The realization of the purpose, functional features and advantages of this application will be further described with reference to the embodiments and the drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0057] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The main solution of the embodiment of the present application is: obtaining a real-time video stream of the vehicle to be identified, wherein the vehicle includes a box that can be loaded with goods; using a laser radar to scan the box to obtain three-dimensional point cloud data of the box; based on a pre-trained empty box recognition model, confirm whether the box is empty according to the real-time video stream and the three-dimensional point cloud data of the box, the pre-trained visual empty box recognition model is obtained by training the visual empty box recognition model based on images and three-dimensional point cloud data of empty boxes and non-empty boxes with the box door open, wherein the visual empty box recognition model includes an image recognition model and a box interlayer recognition algorithm.

[0059] Existing empty container recognition technologies typically use X-ray or acoustic detection. However, due to the radiation hazards and expensive equipment associated with X-ray detection, and the limitations of acoustic detection in terms of detectable types, real-world applications of acoustic detection suffer from low success rates and accuracy.

[0060] From the above analysis, it can be seen that the existing empty box recognition technology has the problem of low speed and accuracy of empty box recognition.

[0061] The present application provides a solution, which obtains real-time video stream of the vehicle to be identified and uses lidar to obtain three-dimensional point cloud data inside and outside the box; performs image recognition and analysis on the real-time video stream and three-dimensional point cloud data through the image recognition model in the pre-trained empty box recognition model, and uses the box interlayer recognition algorithm to perform box interlayer recognition on the box of the vehicle to be identified, so as to confirm whether the box is empty. The vehicle empty box detection technology that integrates visual perception and lidar can realize accurate recognition of empty boxes, which can effectively improve the speed of vehicle empty box recognition, reduce human workload, and improve the efficiency of empty box recognition.

[0062] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an empty box identification device capable of implementing the above functions. The following uses the empty box identification device as an example to illustrate this embodiment and the following embodiments.

[0063] Based on this, an embodiment of the present application provides an empty box recognition method. Refer to Figure 1 , Figure 1 which is a schematic flowchart of the first embodiment of the empty box recognition method of the present application.

[0064] In this embodiment, the empty box recognition method includes steps S110 to S140:

[0065] Step S110, obtain a real-time video stream of the vehicle to be recognized, where the vehicle includes a box for loading goods;

[0066] First, the empty box recognition device acquires the video stream of the vehicle to be recognized based on a pre-installed vision sensor (such as a camera) and transmits it to the empty box recognition device in real time through the RTSP protocol (Real-Time Streaming Protocol), so as to obtain a real-time video stream. In this embodiment, the vehicle targeted by the empty box recognition device is a vehicle including a box for loading goods, such as a box truck or a container truck, etc.

[0067] Step S120, use a lidar to scan the box to obtain three-dimensional point cloud data of the box;

[0068] Specifically, the empty box recognition device uses a lidar to scan the inside and outside of the box of the vehicle to be recognized, so as to obtain the three-dimensional point cloud data of the inside and outside of the box, so as to judge whether there is an illegally modified sandwich in the box subsequently.

[0069] In a feasible implementation manner, the lidar includes a first lidar and a second lidar, and step S120 may include steps A01 to A02:

[0070] Step A01, use the first lidar to scan the outer contour of the box to obtain the three-dimensional point cloud data of the outer contour of the box;

[0071] Step A02, use the second lidar to scan the inside of the box to obtain the three-dimensional point cloud data inside the box.

[0072] In this implementation manner, in order to accurately obtain the three-dimensional point cloud data of the inside and outside of the box of the vehicle to be recognized, two lidars are used to scan the outer contour and the inside of the box respectively.

[0073] It should be noted that, referring to Figure 5 the application schematic diagram of the empty box recognition device shown, the first lidar is the outer space volume measurement lidar numbered ③, which is installed in the upper left corner of the gantry. In order to ensure that the overall outer contour of the vehicle box is obtained, a certain angle needs to be maintained with the side of the vehicle box.

[0074] The second lidar is the empty container scanning lidar labeled ②, which is installed on the gantry lifting rod of the empty container identification device. When the second lidar works, it can descend to the exact rear of the container of the vehicle to be identified and automatically rise after the work is completed to prevent collision with the vehicle. The device indicated by label ① is the empty container identification camera.

[0075] Specifically, when the vehicle stops at the preset empty container identification position, the first lidar is used to scan the outer contour of the container to obtain the three-dimensional point cloud data of the outer contour of the container. Among them, the preset empty container identification position is a position preset by relevant personnel according to actual needs or experience.

[0076] At the same time, the second lidar is used to scan the inside of the container. When there are other items inside the container, the inner space of the container and the remaining items are accurately scanned, so as to obtain the three-dimensional point cloud data inside the container, that is, the three-dimensional point cloud data inside the box.

[0077] Step S130, based on the pre-trained empty container identification model, according to the real-time video stream and the three-dimensional point cloud data of the container, confirm whether the container is an empty container. The pre-trained visual empty container identification model is trained according to the images and three-dimensional point cloud data of empty and non-empty containers in the open state of the container door. Among them, the visual empty container identification model includes an image recognition model and a container sandwich identification algorithm.

[0078] It should be noted that the pre-trained empty container identification model refers to the model obtained by relevant personnel training the visual empty container identification model according to the images and three-dimensional point cloud data of empty and non-empty containers in the open state of the container door until the empty container recognition rate reaches the preset recognition threshold. Among them, the visual empty container identification model includes an image recognition model and a container sandwich identification algorithm. The image recognition model can be any one or a combination of multiple deep learning models based on convolutional neural networks, deep learning models based on Transformer networks, or machine learning models such as support vector machines and random forests.

[0079] Among them, the preset recognition threshold is the required value of the empty container recognition accuracy and efficiency preset by relevant personnel according to actual empty container recognition needs or experience.

[0080] Specifically, through the pre-trained empty container identification model, the above-obtained real-time video stream is analyzed for images to confirm whether the container of the vehicle to be identified is an empty container, and whether there is a sandwich in the container is confirmed according to the three-dimensional point cloud data of the container, and it is comprehensively judged whether the container is an empty container.

[0081] It should be understood that in order to ensure the successful identification of an empty box for the box body, before using the lidar to scan the box body, the empty box identification device can use a pre-trained empty box identification model to confirm whether the box door of the box body is in an open state. When the box door is closed, it can prompt relevant personnel (such as the vehicle driver) to open the box door of the box body of the vehicle to be identified.

[0082] In a feasible implementation manner, the step S130 may include steps B01 to B03:

[0083] Step B01, based on the box body sandwich layer identification algorithm, determine the box body sandwich layer identification result according to the outer contour three-dimensional point cloud data and the in-box three-dimensional point cloud data;

[0084] Step B02, based on a pre-trained image recognition model, confirm whether there are any remaining items inside the box body according to the real-time video stream and the three-dimensional point cloud data of the box body, and identify the remaining items to obtain the item identification result;

[0085] Step B03, based on a preset first weight configuration, perform cascaded combination of the box body sandwich layer identification result and the item identification result to obtain the empty box identification result.

[0086] Specifically, the empty box identification device first calculates the external volume of the box body according to the outer contour three-dimensional point cloud data of the box body through the box body sandwich layer identification algorithm, and at the same time calculates the internal volume of the box body according to the in-box three-dimensional point cloud data. It should be understood that when there are other items in the box body, the internal volume of the box body should include the sum of the volume of the empty space inside the box and the volume of other items.

[0087] Then, according to the difference between the external volume and the internal volume of the box body, confirm whether there is a sandwich layer in the box body, so as to obtain the box body sandwich layer identification result.

[0088] At the same time, apply a pre-trained image recognition model to perform image analysis and item identification on the box body image in the real-time video stream of the vehicle to be identified and the box body image obtained by performing point cloud rendering on the three-dimensional point cloud data of the box body, confirm whether there are any other remaining items inside the box body, and identify the types of the remaining items in the box body, and comprehensively obtain the item identification result.

[0089] Finally, according to the first weight ratio configured according to the actual detection scenario of vehicle empty box identification, perform cascading on the box body sandwich layer identification result and the item identification result, realize the association and decision fusion of image and radar data, and perform weighted summation to obtain the empty box identification result. Among them, the preset first weight configuration is pre-configured by relevant personnel according to the actual detection scenario of vehicle empty box identification, and is used to measure the importance of the box body sandwich layer identification result and the item identification result.

[0090] In this embodiment, when the identification result of the box sandwich is that there is a sandwich in the box or the item category in the item identification result does not belong to the portable items in the preset white list, it is confirmed that the empty box identification result is that the box is not an empty box, and relevant personnel are prompted that the box of the vehicle is not an empty box and manual verification is required. The preset white list is the portable items that can be carried in the cargo box set by relevant personnel according to industry regulations and actual needs.

[0091] This embodiment provides an empty box identification method. By obtaining the real-time video stream of the vehicle to be identified, where the vehicle includes a box for loading goods; using a lidar to scan the box to obtain the three-dimensional point cloud data of the box; based on a pre-trained empty box identification model, according to the real-time video stream and the three-dimensional point cloud data of the box, it is confirmed whether the box is an empty box. The pre-trained visual empty box identification model is trained according to the images and three-dimensional point cloud data of the empty box and non-empty box when the box door is open, where the visual empty box identification model includes an image recognition model and a box sandwich identification algorithm.

[0092] This application obtains the real-time video stream of the vehicle to be identified, and at the same time uses lidar to obtain the three-dimensional point cloud data inside and outside the box; through the image recognition model in the pre-trained empty box identification model, image recognition and analysis are performed on the real-time video stream and the three-dimensional point cloud data, and the box sandwich identification algorithm is used to identify the box sandwich of the vehicle to be identified, so as to confirm whether the box is an empty box. Through the vehicle empty box detection technology that combines visual perception and lidar, accurate identification of empty boxes can be achieved, which can effectively improve the vehicle empty box identification speed, reduce the manual workload, and improve the empty box identification efficiency.

[0093] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the above-mentioned embodiment one can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 2 Before the above step S120, steps S210 to S220 are further included:

[0094] Step S210, based on a pre-trained empty box identification model, identify the type of the box according to the real-time video stream, and obtain the estimated size of the box;

[0095] Step S220, according to the estimated size of the box, adjust the position and angle of the second lidar relative to the box.

[0096] It is understandable that, in order to meet the detection requirements of boxes with different sizes and shapes, before applying the lidar, the empty box recognition device can use a pre-trained empty box recognition model to evaluate the size and shape of the box of the vehicle to be recognized, so as to adaptively adjust the scanning range of the lidar and improve the quality and efficiency of the lidar's internal scanning of different types of vehicles and their boxes.

[0097] Specifically, before applying the lidar, the empty box recognition device can use a pre-trained empty box recognition model to perform image analysis on the real-time video stream of the box, confirm the type of the cargo box of the box, and obtain the estimated size of the box based on the box features extracted from the image.

[0098] Then, using the estimated size, adjust the position of the second lidar relative to the box in space so that the lidar can better aim at the target area. For example, when the estimated height of the box is higher than the average box height, the empty box recognition device can automatically lift the second lidar to a higher position to cover the internal space of the box.

[0099] At the same time, based on the estimated size, the scanning angle of the second lidar can be further adjusted so that the scanning beam can be more evenly distributed in the entire internal space of the box. For example, for a longer or wider box, the lidar may need to be tilted multiple times to ensure that the internal space of the box can be fully scanned.

[0100] This embodiment provides an empty box recognition method, which includes obtaining a real-time video stream of a vehicle to be recognized, where the vehicle includes a box for loading goods; based on a pre-trained empty box recognition model, identifying the type of the box according to the real-time video stream, and obtaining the estimated size of the box; according to the estimated size of the box, adjusting the position and angle of the second lidar relative to the box; using the first lidar to scan the outer contour of the box to obtain the three-dimensional point cloud data of the outer contour of the box; using the second lidar to scan the inside of the box to obtain the three-dimensional point cloud data inside the box; based on a pre-trained empty box recognition model, confirming whether the box is an empty box according to the real-time video stream and the three-dimensional point cloud data of the box. The pre-trained visual empty box recognition model is trained according to the images and three-dimensional point cloud data of empty and non-empty boxes in the state of the box door being opened, where the visual empty box recognition model includes an image recognition model and a box sandwich recognition algorithm.

[0101] This application uses a pre-trained empty box recognition model to identify the type of the box body according to the real-time video stream, so as to obtain the estimated size of the box body; according to the estimated size, the position and angle of the second lidar relative to the box body are adjusted, so that the lidar can be used to accurately scan the box body subsequently, and sufficient three-dimensional point cloud data of the outer contour of the box body and the three-dimensional point cloud data inside the box are obtained, so as to ensure the accuracy of the outer volume and the inner volume of the box body and improve the accuracy of box body sandwich recognition.

[0102] Based on the first embodiment and / or the second embodiment of this application, in the third embodiment of this application, the same or similar content as in the above-mentioned first embodiment and / or the second embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 3 , the step B01 includes steps S310 to S340:

[0103] Step S310, based on the box body sandwich recognition algorithm, obtain the outer volume and the inner volume of the box body according to the outer contour three-dimensional point cloud data and the three-dimensional point cloud data inside the box;

[0104] Specifically, the empty box recognition device first processes the collected three-dimensional point cloud data of the box body contour and the three-dimensional point cloud data inside the box according to the box body sandwich recognition algorithm to obtain the outer volume and the inner volume of the box body. For example, the three-dimensional point cloud data is divided into a series of non-overlapping triangular patches by the triangulation method, and then the volume enclosed by the triangular patches is calculated, or the three-dimensional point cloud data is fitted into geometric shapes (such as cuboids, cylinders, etc.) by the geometric fitting method, and then the volume of the above geometric shapes is calculated.

[0105] In a feasible implementation manner, the step S310 may include steps C01 to C04:

[0106] Step C01, perform filter processing on the outer contour three-dimensional point cloud data and the three-dimensional point cloud data inside the box to obtain the preprocessed outer contour three-dimensional point cloud data and the three-dimensional point cloud data inside the box;

[0107] Step C02, perform plane fitting on the preprocessed outer contour three-dimensional point cloud data to construct a three-dimensional space model of the box body;

[0108] Step C03, calculate the outer contour size of the box body according to the three-dimensional space model of the box body to obtain the outer volume of the box body;

[0109] Step C04, based on the three-dimensional space model of the box body and the three-dimensional point cloud data inside the box, calculate the inner volume of the box body by using the voxel grid method.

[0110] It can be understood that in order to reduce the impact of noise on data, the empty box recognition device needs to apply filters to the three-dimensional point cloud data of the outer contour of the box body and the three-dimensional point cloud data inside the box for noise reduction processing and optimization processing, remove irrelevant point cloud information, and obtain the preprocessed three-dimensional point cloud data of the outer contour and the three-dimensional point cloud data inside the box, so as to improve the signal-to-noise ratio and reliability of the data.

[0111] Then, with reference to Figure 6 , apply a preset plane fitting algorithm to the preprocessed three-dimensional point cloud data of the outer contour of the box to obtain the fitting results of the middle side and the top surface of the outer contour of the box body, and then use the fitting results to construct a three-dimensional space model of the box body.

[0112] Among them, the preset plane fitting algorithm is a plane fitting algorithm preset by relevant personnel, which can be the least squares method, the RANSAC (Random Sample Consensus) algorithm, etc.

[0113] Finally, using the three-dimensional space model of the box body established above, determine the outer contour dimensions of the box body, calculate the length, width and height dimensions of the box body, and then use the volume calculation formula to calculate the outer volume of the box body. When the box body is irregularly shaped, the integral method can be used to calculate the outer volume of the box body.

[0114] The inner volume of the box body is calculated based on the voxel grid method. Specifically, in order for the empty box recognition device to distinguish between calculating the inner volume and the outer volume of the box body, it can perform voxel grid division on the three-dimensional space model of the box body, divide the three-dimensional space model of the box body into a series of small cubes (i.e., voxels), and each voxel has a fixed size.

[0115] According to the three-dimensional point cloud data inside the box, determine whether each voxel is occupied, that is, judge by confirming whether each voxel is included in the three-dimensional point cloud data inside the box. Count the number of occupied voxels and multiply by the volume of a single voxel to finally obtain the inner volume of the box body.

[0116] Step S320, perform a difference operation on the outer volume and the inner volume of the box body to determine the wall thickness of the box body;

[0117] Step S330, when the wall thickness of the box body is greater than the preset wall thickness threshold, determine that the box body sandwich recognition result is that there is a sandwich in the box body;

[0118] Step S340, when the wall thickness of the box body is not greater than the preset wall thickness threshold, determine that the box body sandwich recognition result is that there is no sandwich in the box body.

[0119] It should be noted that the preset wall thickness threshold refers to the maximum allowable wall thickness preset by relevant personnel based on an understanding of the box body material, manufacturing process and expected sandwich thickness.

[0120] Specifically, a subtraction operation is performed on the outer volume and the inner volume of the box to obtain the volume difference. Since this volume difference is a function related to the wall thickness of the box, the wall thickness of the box can be further determined through this volume difference. For example, for a box with a regular shape (such as a cuboid), the total surface area can be divided by the volume difference to approximate the average wall thickness; for a box with a complex shape, the wall thickness of the box can be obtained by distance analysis based on the local normal direction.

[0121] When the wall thickness of the box is greater than the preset wall thickness threshold, it is determined that there is a sandwich layer in the box; when the wall thickness of the box is not greater than the preset wall thickness threshold, it is determined that there is no sandwich layer in the box.

[0122] This embodiment provides an empty box recognition method. Through the box sandwich layer recognition algorithm, based on the three-dimensional point cloud data of the outer contour of the box and the three-dimensional point cloud data inside the box, the outer volume and the inner volume of the box are calculated, and then the wall thickness of the box is obtained through the volume difference between the outer volume and the inner volume. Then, according to the preset wall thickness threshold, it is determined whether there is a sandwich layer in the box, effectively improving the efficiency and accuracy of box sandwich layer recognition.

[0123] Based on any of the above embodiments of the present application, in the fourth embodiment of the present application, the content that is the same as or similar to any of the above embodiments can be referred to the above introduction and will not be repeated hereinafter. On this basis, the image recognition model includes a first image recognition model and a second image recognition model. Please refer to Figure 4 and the step B02 includes steps S410 to S440:

[0124] Step S410, based on a preset clustering algorithm, perform point cloud rendering on the three-dimensional point cloud data of the box to obtain the corresponding box image;

[0125] Step S420, use the pre-trained first image recognition model to perform image analysis and item recognition on the box image to obtain the first item recognition result;

[0126] Step S430, use the pre-trained second image recognition model to perform image analysis and item recognition on the real-time video stream to obtain the second item recognition result;

[0127] Step S440, based on a preset second weight configuration, perform hierarchical combination of the first item recognition result and the second item recognition result to obtain the item recognition result.

[0128] It should be noted that the preset clustering algorithm refers to a clustering algorithm preset by relevant personnel, which is used to cluster based on the characteristics of the point cloud data, divide the point cloud data into different groups or clusters, and each cluster represents an object or a part of an object. Among them, the characteristics of the point cloud data include spatial position, normal, density, reflection intensity, etc. for clustering.

[0129] The first image recognition model is used to perform image analysis on a two-dimensional image obtained from three-dimensional point cloud data and identify items in the two-dimensional image. The first image recognition model can be any one or a combination of multiple deep learning models based on convolutional neural networks, deep learning models based on Transformer networks, or machine learning models such as support vector machines and random forests.

[0130] The second image recognition model is used to analyze a real-time video stream and identify items in the real-time video stream. The second image recognition model can be any one or a combination of multiple deep learning models based on convolutional neural networks, deep learning models based on Transformer networks, or machine learning models such as support vector machines and random forests.

[0131] First, the empty box recognition device needs to use a preset clustering algorithm to perform point cloud clustering on the three-dimensional point cloud data of the box body, group the points in the point cloud according to feature similarity, and at the same time use attributes such as the normal line and reflection intensity of the point cloud data to color and render the grouped point cloud data, and project the rendered point cloud data onto a two-dimensional plane to obtain the corresponding box body image.

[0132] Then, use the first image recognition model to perform image analysis and item recognition on the box body image, confirm whether there are any remaining items inside the box body, and obtain information such as the size characteristics, material characteristics, location, and quantity of the remaining items, so as to determine the category information and size information of the items, that is, the first item recognition result.

[0133] At the same time, use the second image recognition model to perform image analysis on each frame of the real-time video stream mentioned above, confirm whether there are any remaining items inside the box body, and obtain information such as the dynamic characteristics, appearance color characteristics, location, and quantity of the remaining items in the real-time video stream, and output the category information and appearance information of the items, that is, the second item recognition result.

[0134] Finally, according to the preset second weight configuration, the first item recognition result and the second item recognition result are cascaded and merged to realize the association of image and radar data, and the weighted sum is obtained to get the item recognition result. Among them, the preset second weight configuration is pre-configured by relevant personnel according to the actual detection scenario of vehicle empty box recognition, and is used to represent the reliability of the first item recognition result and the second item recognition result. For example, when detecting items with unclear appearance features such as grayish-white, it is necessary to adjust the preset second weight configuration and set a higher weight for the first item recognition result.

[0135] This embodiment provides an empty box recognition method. By obtaining the real-time video stream of the vehicle to be recognized, where the vehicle includes a box for loading goods; using a lidar to scan the box to obtain the three-dimensional point cloud data of the box; based on the box sandwich recognition algorithm, according to the outer contour three-dimensional point cloud data and the three-dimensional point cloud data inside the box, determining the box sandwich recognition result; based on a preset clustering algorithm, performing point cloud rendering on the three-dimensional point cloud data of the box to obtain the corresponding box image; using a pre-trained first image recognition model to perform image analysis and item recognition on the box image to obtain the first item recognition result; using a pre-trained second image recognition model to perform image analysis and item recognition on the real-time video stream to obtain the second item recognition result; based on a preset second weight configuration, performing hierarchical combination of the first item recognition result and the second item recognition result to obtain the item recognition result; based on a preset first weight configuration, performing hierarchical combination of the box sandwich recognition result and the item recognition result to obtain the empty box recognition result.

[0136] This application clusters the three-dimensional point cloud data of the outer contour of the box and the three-dimensional point cloud data inside the box through a clustering algorithm, and performs point cloud rendering to obtain a two-dimensional image of the point cloud data. Through the first image model for this two-dimensional image, it pays more attention to the geometric features of the items in the image data and outputs the first item recognition result; through the second image model for the real-time video stream, it pays more attention to the appearance and dynamic features of the items in the image data and outputs the second item recognition result; finally, according to the preset second weight configuration, the results output by the two models are weighted and combined to comprehensively obtain the item recognition result, realizing the recognition of the items left inside the box that can adapt to multiple scenarios, thereby improving the accuracy and efficiency of empty box recognition.

[0137] This application also provides an empty box recognition device. Please refer to Figure 7 , the empty box recognition device includes:

[0138] A video acquisition module 10 for obtaining the real-time video stream of the vehicle to be recognized, where the vehicle includes a box for loading goods;

[0139] A point cloud data acquisition module 20 that uses a lidar to scan the box to obtain the three-dimensional point cloud data of the box;

[0140] An empty box recognition module 30 for confirming whether the box is an empty box based on a pre-trained empty box recognition model according to the real-time video stream and the three-dimensional point cloud data of the box. The pre-trained visual empty box recognition model is trained based on the images and three-dimensional point cloud data of empty and non-empty boxes in the state of the box door being open. Among them, the visual empty box recognition model includes an image recognition model and a box sandwich recognition algorithm.

[0141] The empty container identification device provided by this application adopts the empty container identification method in the above embodiment, and can solve the technical problem of how to improve the efficiency and accuracy of empty container identification. Compared with the prior art, the beneficial effects of the empty container identification device provided by this application are the same as those of the empty container identification method provided by the above embodiment, and other technical features in the empty container identification device are the same as the features disclosed in the method of the above embodiment, which will not be elaborated here.

[0142] This application provides an empty container identification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the empty container identification method in the above embodiment.

[0143] Refer to the following Figure 8 , which shows a schematic structural diagram of an empty container identification device suitable for implementing the embodiments of this application. The empty container identification device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The empty container identification device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of this application.

[0144] As shown in Figure 8As shown, the empty box identification device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the empty box identification device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the empty box identification device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows an empty box identification device having various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems may be alternatively implemented or had.

[0145] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0146] The empty box identification device provided by the present application adopts the empty box identification method in the above embodiments, and can solve the technical problem of how to improve the efficiency and accuracy of empty box identification. Compared with the prior art, the beneficial effects of the empty box identification device provided by the present application are the same as those of the empty box identification method provided by the above embodiments, and other technical features in the empty box identification device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0147] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0148] As described above, this is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0149] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the empty box identification method in the above embodiments.

[0150] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0151] The above computer-readable storage medium can be included in the empty box identification device; it can also exist separately without being assembled into the empty box identification device.

[0152] The above computer-readable storage medium carries one or more programs, which, when executed by the empty container recognition device, cause the empty container recognition device to: obtain a real-time video stream of a vehicle to be recognized, where the vehicle includes a container capable of loading goods; use a lidar to scan the container to obtain three-dimensional point cloud data of the container; based on a pre-trained empty container recognition model, confirm whether the container is an empty container according to the real-time video stream and the three-dimensional point cloud data of the container. The pre-trained visual empty container recognition model is trained according to the images and three-dimensional point cloud data of empty and non-empty containers in the open state of the container door. Among them, the visual empty container recognition model includes an image recognition model and a container interlayer recognition algorithm.

[0153] Computer program code for performing the operations of the present application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the block may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0155] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0156] The readable storage medium provided by the present application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned empty container recognition method, which can solve the technical problem of how to improve the efficiency and accuracy of empty container recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the empty container recognition method provided by the above embodiments, and will not be elaborated here.

[0157] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent scope of the present application.

Claims

1. An empty box recognition method, characterized in that, The method includes: Obtaining a real-time video stream of a vehicle to be recognized, where the vehicle includes a box for loading goods; Scanning the box using a lidar to obtain three-dimensional point cloud data of the box; Based on a pre-trained empty box recognition model, confirming whether the box is an empty box according to the real-time video stream and the three-dimensional point cloud data of the box. The pre-trained visual empty box recognition model is trained based on images and three-dimensional point cloud data of empty and non-empty boxes in the state of the box door being open. Among them, the visual empty box recognition model includes an image recognition model and a box sandwich recognition algorithm.

2. The method according to claim 1, characterized in that, The lidar includes a first lidar and a second lidar. The step of scanning the box using the lidar to obtain the three-dimensional point cloud data of the box includes: Scanning the outer contour of the box using the first lidar to obtain the three-dimensional point cloud data of the outer contour of the box; Scanning the interior of the box using the second lidar to obtain the three-dimensional point cloud data inside the box.

3. The method according to claim 2, wherein Before the step of scanning the box using the lidar to obtain the three-dimensional point cloud data of the box, it includes: Based on a pre-trained empty box recognition model, identifying the type of the box according to the real-time video stream, and obtaining the estimated size of the box; Adjusting the position and angle of the second lidar relative to the box according to the estimated size of the box.

4. The method according to claim 2, wherein The step of confirming whether the box is an empty box based on a pre-trained empty box recognition model according to the real-time video stream and the three-dimensional point cloud data of the box includes: Based on the box sandwich recognition algorithm, determining the box sandwich recognition result according to the three-dimensional point cloud data of the outer contour and the three-dimensional point cloud data inside the box; Based on a pre-trained image recognition model, confirming whether there are any remaining items inside the box according to the real-time video stream and the three-dimensional point cloud data of the box, and identifying the remaining items to obtain an item recognition result; Based on a preset first weight configuration, performing a hierarchical combination of the box sandwich recognition result and the item recognition result to obtain an empty box recognition result.

5. The method according to claim 4, wherein The step of determining the box sandwich recognition result according to the three-dimensional point cloud data of the outer contour and the three-dimensional point cloud data inside the box based on the box sandwich recognition algorithm includes: Based on the box sandwich recognition algorithm, obtaining the outer volume and the inner volume of the box according to the three-dimensional point cloud data of the outer contour and the three-dimensional point cloud data inside the box; Performing a difference operation on the outer volume and the inner volume of the box to determine the wall thickness of the box; When the wall thickness of the box is greater than a preset wall thickness threshold, determining that the box sandwich recognition result is that there is a sandwich in the box; When the wall thickness of the box is not greater than the preset wall thickness threshold, determining that the box sandwich recognition result is that there is no sandwich in the box.

6. The method according to claim 5, wherein The step of obtaining the outer volume and the inner volume of the box according to the three-dimensional point cloud data of the outer contour and the three-dimensional point cloud data inside the box based on the box sandwich recognition algorithm includes: Perform filter processing on the three-dimensional point cloud data of the outer contour and the three-dimensional point cloud data inside the box to obtain the preprocessed three-dimensional point cloud data of the outer contour and the three-dimensional point cloud data inside the box; Perform plane fitting on the preprocessed three-dimensional point cloud data of the outer contour to construct a three-dimensional space model of the box body; According to the three-dimensional space model of the box body, calculate the outer contour size of the box body to obtain the outer volume of the box body; Based on the three-dimensional space model of the box body and the three-dimensional point cloud data inside the box, use the voxel grid method to calculate the inner volume of the box body.

7. The method according to claim 4, wherein The image recognition model includes a first image recognition model and a second image recognition model. The step of confirming whether there are any left items inside the box body and recognizing the left items based on the pre-trained image recognition model, the real-time video stream, and the three-dimensional point cloud data of the box body to obtain the item recognition result includes: Based on a preset clustering algorithm, perform point cloud rendering on the three-dimensional point cloud data of the box body to obtain a corresponding box body image; Use the pre-trained first image recognition model to perform image analysis and item recognition on the box body image to obtain a first item recognition result; Use the pre-trained second image recognition model to perform image analysis and item recognition on the real-time video stream to obtain a second item recognition result; Based on a preset second weight configuration, perform hierarchical combination and merging of the first item recognition result and the second item recognition result to obtain the item recognition result.

8. An empty box recognition device, characterized in that, The empty box recognition device includes: A video acquisition module for acquiring the real-time video stream of the vehicle to be recognized, where the vehicle includes a box body capable of loading goods; A point cloud data acquisition module that uses a lidar to scan the box body to acquire the three-dimensional point cloud data of the box body; An empty box recognition module for confirming whether the box body is an empty box based on a pre-trained empty box recognition model, the real-time video stream, and the three-dimensional point cloud data of the box body. The pre-trained visual empty box recognition model is trained based on the images and three-dimensional point cloud data of empty and non-empty box bodies in the open state of the box door. The visual empty box recognition model includes an image recognition model and a box body sandwich recognition algorithm.

9. An empty box recognition device, characterized in that, The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the empty box recognition method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the empty box recognition method according to any one of claims 1 to 7.

Images