A container state detection method, device and electronic equipment

Abstract

Embodiments of the present application disclose a container state detection method, device and electronic equipment, the method comprising: using an image similarity algorithm to compare images collected by an image sensor in real time, determining when a vehicle appears by triggering a laser radar to start collecting point cloud data based on the comparison result, and determining when the vehicle drives away to trigger the laser radar to end the collection; analyzing the point cloud data to generate a pcd file, and obtaining multiple frames of image data from the pcd file; and selecting the best frame from the multiple frames of image data for damage detection. The above scheme has less influence on light compared with existing image recognition algorithms, avoids high cost caused by the use of high-performance graphics cards due to deep learning, and can accurately identify the damage degree of a plane due to the unique 3D detection property of the laser radar, thereby avoiding inaccurate original data caused by changes in light.

Description

Technical Field

[0001] This invention relates to object detection technology, and more specifically, to a method, apparatus, and electronic device for detecting the condition of a container. Background Technology

[0002] Most existing port gates use a combination of cameras and deep learning to detect the degree of container damage, thereby obtaining an identification of the damage level. However, this approach also has the following problems: First, many existing algorithms rely on the port system to determine whether a vehicle has reached a designated location, or depend on deep learning for this judgment. This method requires collecting a large amount of data to train the model in the early stages, which is time-consuming. Second, from a perception and recognition perspective, cameras are sensitive to changes in light. In good daylight conditions, they can obtain high-quality images, but in low-light conditions such as at night or on cloudy days, cameras struggle to obtain high-quality images, affecting the algorithm's recognition. Summary of the Invention

[0003] In view of the above, the present invention provides the following technical solution:

[0004] A method for detecting the condition of a container, comprising:

[0005] An image similarity algorithm is used to compare images acquired by an image sensor in real time. Based on the comparison results, it is determined whether to trigger the lidar to start acquiring point cloud data. Specifically, image data and point cloud data of passing vehicles are obtained through an image sensor and a lidar, respectively. The image sensor periodically acquires images.

[0006] When a vehicle is detected in the image captured by the image sensor, the lidar is triggered to start collecting point cloud data. When the vehicle is detected to have moved away from the image captured by the image sensor, the lidar is triggered to end the data collection.

[0007] The point cloud data is parsed to generate a PCD file, and multiple frames of image data are obtained from the PCD file by dividing the time equally.

[0008] Based on the correspondence between the point cloud data and the image frames of the multi-frame image data, the best frame is selected from the multi-frame image data for damage detection based on the principle of minimum image similarity.

[0009] Optionally, the image sensor and the lidar are coaxial in the vertical direction, and the distance from the axis center to the road edge of the passing vehicle is 1.5 meters. The image sensor is 2.5 meters above the ground, and the lidar is 2 meters above the ground.

[0010] The image sensor has a horizontal detection field of view of 150° and a vertical detection field of view of 60°, while the lidar has a horizontal detection field of view of 360° and a vertical detection field of view of 90°.

[0011] Optionally, the step of using an image similarity algorithm to compare images acquired by an image sensor in real time, and determining whether to trigger the lidar to start acquiring point cloud data based on the comparison result, includes:

[0012] An image similarity algorithm is used to determine the starting image within the detection range where no vehicles are passing;

[0013] The first image acquired by the image sensor is periodically obtained, and the first similarity between the starting image and each of the first images is compared sequentially.

[0014] From the time when the first similarity is lower than the first threshold until the first similarity is no greater than the second threshold, the first variable is assigned a first value; and during the time when the first similarity is higher than the second threshold, the first variable is assigned a second value.

[0015] Optionally, the step of using an image similarity algorithm to determine the starting image where no vehicles are passing within the detection range includes:

[0016] Obtain N images acquired by the image sensor during the first time period, where N is a positive integer;

[0017] For the N images, the similarity between adjacent frames is compared one by one to obtain N-1 similarity values;

[0018] If the range of all similarity data is lower than the first similarity threshold, and the variance of all similarity data is lower than the second similarity threshold, then the last frame of the N images is determined as the starting image.

[0019] Optionally, the step of triggering the lidar to start collecting point cloud data when it is determined that a vehicle appears in the image collected by the image sensor, and triggering the lidar to stop collecting data when it is determined that the vehicle has driven away from the image collected by the image sensor, includes:

[0020] When the first variable is at a first value, the lidar is controlled to collect and save point cloud data; when the first variable is at a second value, the collection of point cloud data is stopped.

[0021] Optionally, the step of parsing the point cloud data to generate a PCD file, and obtaining multiple frames of image data from the PCD file by dividing the data into equal time segments, includes:

[0022] Based on the time difference between the first and last frames of the point cloud data, M time points are equally divided. M point cloud image frames corresponding to the M time points are then selected from the point cloud data, where M is a positive integer not less than 2.

[0023] Optionally, the step of selecting the best frame from the multi-frame image data for damage detection based on the correspondence between the point cloud data and the image frames of the multi-frame image data, according to the principle of minimum image similarity, includes:

[0024] The point cloud image frame with the lowest similarity to the starting image is selected from the M point cloud image frames and determined as the best point cloud image frame for damage detection.

[0025] Optionally, damage detection is performed on the optimal point cloud image frame, including:

[0026] A pass-through filtering algorithm is used to filter out point cloud data other than the detected object in the optimal point cloud image frame;

[0027] A corner detection algorithm is used to identify the corner data of the detected object and extract the box surface data of the detected object;

[0028] The box surface data is fitted using a plane fitting algorithm to establish a feature extraction model and extract planar features;

[0029] A decision tree algorithm is used to establish a recognition model, calculate the distance from the point cloud corresponding to the box surface data to the box surface data representation box surface, perform feature engineering, and determine the degree of damage of the detected target.

[0030] A container condition detection device, comprising:

[0031] The image processing module is used to compare the images acquired by the image sensor in real time using an image similarity algorithm, and determine whether to trigger the lidar to start acquiring point cloud data based on the comparison result. The image sensor and lidar respectively acquire image data and point cloud data of passing vehicles, and the image sensor periodically acquires images.

[0032] The radar control module is used to trigger the lidar to start collecting point cloud data when it determines that a vehicle appears in the image collected by the image sensor, and to trigger the lidar to stop collecting data when it determines that the vehicle has moved away from the image collected by the image sensor.

[0033] The point cloud processing module is used to parse the point cloud data to generate a PCD file, and to obtain multiple frames of image data from the PCD file by dividing the time equally.

[0034] The damage detection module is used to select the best frame from the multi-frame image data for damage detection based on the correspondence between the point cloud data and the image frames of the multi-frame image data and the principle of minimum image similarity.

[0035] An electronic device, comprising:

[0036] processor;

[0037] Memory for storing executable program instructions of the processor;

[0038] The executable program instructions include: using an image similarity algorithm to compare images acquired by an image sensor in real time; determining whether to trigger the lidar to start acquiring point cloud data based on the comparison results; wherein image data and point cloud data of passing vehicles are obtained by the image sensor and lidar respectively, and the image sensor periodically acquires images; when it is determined that a vehicle appears in the image acquired by the image sensor, the lidar is triggered to start acquiring point cloud data; when it is determined that the vehicle leaves the image acquired by the image sensor, the lidar is triggered to stop acquiring data; parsing the point cloud data to generate a PCD file; obtaining multiple frames of image data from the PCD file by dividing the time equally; and selecting the best frame from the multiple frames of image data for damage detection based on the principle of minimum image similarity, according to the correspondence between the point cloud data and the image frames of the multiple frames of image data.

[0039] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a container status detection method, device, and electronic device. The method includes: using an image similarity algorithm to compare images acquired by an image sensor in real time, and determining whether to trigger a lidar to start collecting point cloud data based on the comparison result. Specifically, image data and point cloud data of passing vehicles are obtained through an image sensor and a lidar, respectively, and the image sensor periodically acquires images. When it is determined that a vehicle appears in the image acquired by the image sensor, the lidar is triggered to start collecting point cloud data; when it is determined that the vehicle has left the image acquired by the image sensor, the lidar acquisition ends. The point cloud data is parsed to generate a PCD file, and multiple frames of image data are obtained from the PCD file through equal time division. Based on the correspondence between the point cloud data and the image frames of the multiple frames of image data, the best frame is selected from the multiple frames of image data for damage detection based on the principle of minimum image similarity. The above solution determines whether there are passing vehicles based on on-site images. If passing vehicles are present, the LiDAR is triggered to acquire point cloud data of the container. The damage detection of the container is then performed by processing and analyzing the point cloud data. Compared with existing image recognition algorithms, this solution is less affected by lighting conditions and avoids the high cost of purchasing high-performance graphics cards for deep learning. Furthermore, due to the unique 3D detection properties of LiDAR, it can accurately identify the degree of damage to a plane, avoiding inaccurate raw data caused by changes in lighting. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 This is a flowchart of a container condition detection method disclosed in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the hardware architecture of the container status detection scheme disclosed in an embodiment of the present invention;

[0043] Figure 3 This is an example diagram showing the sensor installation location and detection range disclosed in an embodiment of the present invention;

[0044] Figure 4 This is a flowchart of determining whether to trigger the operation of a lidar according to an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the point cloud data acquisition triggering process disclosed in an embodiment of the present invention;

[0046] Figure 6 This is a logic diagram illustrating the data acquisition process of the lidar disclosed in an embodiment of the present invention.

[0047] Figure 7 This is a schematic diagram of the image data and point cloud data saving process disclosed in an embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of the process for determining the optimal frame as disclosed in an embodiment of the present invention;

[0049] Figure 9 This is a schematic diagram of container residue detection disclosed in an embodiment of the present invention;

[0050] Figure 10 This is a schematic diagram of the structure of a container condition detection device disclosed in an embodiment of the present invention;

[0051] Figure 11 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] To better understand the implementation of this application, we will first introduce the application scenarios of the solution in this application.

[0054] The container condition detection method described in this application can be applied to port gates to detect whether containers loaded on trucks are damaged and the extent of the damage. The implementation of this solution requires image data and point cloud data; therefore, image acquisition devices and lidar need to be installed at the port gate to obtain image data and point cloud data.

[0055] Figure 1 This is a flowchart of a container condition detection method disclosed in an embodiment of the present invention. See also... Figure 1 As shown, the container condition detection method may include:

[0056] Step 101: Use an image similarity algorithm to compare the images acquired by the image sensor in real time, and determine whether to trigger the lidar to start acquiring point cloud data based on the comparison results. The image sensor and lidar are used to acquire image data and point cloud data of passing vehicles, respectively. The image sensor acquires images periodically.

[0057] The image sensor is installed at the port gate. Upon activation, it periodically acquires images within its acquisition range and transmits these images to the processing unit for analysis. By analyzing the continuously acquired images, the presence of a target, i.e., a passing vehicle, can be determined within the acquisition range.

[0058] In this embodiment, an image similarity algorithm can be used to analyze whether there are passing vehicles within the image acquisition range, and further, based on the analysis results, it can be determined whether to trigger the lidar to start acquiring point cloud data. That is, the working state of the lidar is indirectly controlled by determining whether there are vehicles passing within the gate area.

[0059] Step 102: When it is determined that a vehicle appears in the image acquired by the image sensor, the lidar is triggered to start acquiring point cloud data. When it is determined that the vehicle has driven away from the image acquired by the image sensor, the lidar is triggered to end the acquisition.

[0060] Understandably, when a vehicle is detected passing through, the LiDAR can be triggered to start collecting point cloud data; when there is no vehicle passing through, or when the vehicle leaves, the LiDAR does not need to work to collect point cloud data, thus avoiding waste of resources and preventing the LiDAR from collecting a lot of useless point cloud data, which would add interference and burden to subsequent identification and processing.

[0061] Step 103: Parse the point cloud data to generate a PCD file, and obtain multiple frames of image data from the PCD file by dividing the time equally.

[0062] The original point cloud data acquired by the lidar is in pcap file format, which is quite large. To facilitate analysis and processing, in this embodiment, the pcap file is parsed into a pcd file. Since the point cloud data is extensive, the pcd file will also contain a large amount of data. Therefore, not all data will be processed in the implementation. Instead, this embodiment obtains multiple frames of image data from the pcd file by dividing the data into equal time frames. The number of frames obtained can be determined based on the application scenario requirements and is not fixed; for example, it could be 10 frames, 15 frames, or 20 frames.

[0063] Step 104: Based on the correspondence between the point cloud data and the image frames of the multi-frame image data, the best frame is selected from the multi-frame image data for damage detection based on the principle of minimum image similarity.

[0064] The selection of the optimal frame is based on the principle of minimum image similarity, aiming to choose the image frame that best reflects the state of the container on the vehicle from multiple image frames. In one implementation, the optimal frame can be the image frame with the lowest similarity to an image of a vehicle not passing through the gate.

[0065] Damage inspection of containers can promptly transmit the results to the port control platform, reducing safety risks caused by severe container deformation. At the same time, early detection of container damage can assist the port in replacing containers in a timely manner before loading, improving operational efficiency.

[0066] The container status detection method described in this embodiment determines whether there are passing vehicles based on on-site images. If passing vehicles are present, a LiDAR is triggered to acquire point cloud data of the container. The container is then damaged by processing and analyzing the point cloud data. Compared with existing image recognition algorithms, this solution is less affected by lighting conditions and avoids the high costs associated with purchasing high-performance graphics cards for deep learning. Furthermore, due to the unique 3D detection properties of LiDAR, it can accurately identify the degree of damage to a plane, avoiding inaccurate raw data caused by changes in lighting.

[0067] Figure 2 This is a schematic diagram of the hardware architecture of the container status detection scheme disclosed in an embodiment of the present invention. See also... Figure 2 As shown, the main sensing sensors are camera 10 and lidar 20, which realize data collection of trucks and container surfaces; MEC30 (Multi-access Edge Computing) is used to trigger the sensors to collect data, process and identify the acquired data, and send the identified results to the designated server.

[0068] The camera and LiDAR acquire image data (I) and point cloud data (L) of vehicles passing through the port gate. This data is then transmitted to the MEC unit via FTP or MQTT for LiDAR PCAP (PCAP is the format for storing data acquired by LiDAR via TCP) packet parsing and image data similarity calculation. The PCAP file is used to store the LiDAR data. The MEC unit processes the point cloud data to determine the degree of container damage. The PCAP packet parsing steps may include, but are not limited to: parsing the PCAP header, performing byte order conversion and time order conversion on subsequent messages, and then saving the data in CSV format according to the LiDAR's scanning sequence.

[0069] In one implementation, the image sensor and the lidar are coaxial in the vertical direction, and the distance from the axis center to the road edge of the passing vehicle is 1.5 meters. The image sensor is 2.5 meters above the ground, and the lidar is 2 meters above the ground. The horizontal detection field of view of the image sensor is 150° and the vertical detection field of view is 60°. The horizontal detection field of view of the lidar is 360° and the vertical detection field of view is 90°.

[0070] Specifically, Figure 3 This is an example diagram illustrating the sensor installation location and detection range disclosed in an embodiment of the present invention. The sensors include a camera (image sensor) and a LiDAR. See also... Figure 3 As shown, the lidar and camera are coaxial in the vertical direction and installed on a vertical pole beside the lane at the port gate. The distance from the vertical pole to the curb is influenced by a combination of factors, including the installation environment, site conditions, and the height of the container truck rack. The detection angles of the camera and lidar are determined by the hardware itself and are unique attributes. The distance from the vertical pole to the curb is affected by both the container length and the camera's own field of view (FOV) parameters; to ensure complete container coverage, it is typically within 3 meters. The installation height of the camera and lidar is affected by both the container rack height on the container truck and the hardware's own FOV parameters; typically, the installation height does not exceed the maximum height of the upper edge of the container on the truck from the ground. In this application, the distance from the vertical pole to the curb is 1.5m, the camera installation height is 2.5m, and the lidar installation height is 2m. The camera's horizontal detection FOV is 150°, and its vertical detection FOV is 60°; the lidar's horizontal detection FOV is 360°, and its vertical detection FOV is 90°. When container trucks pass through port gates, the area where they can stop is limited due to card-swiping and gate restrictions, and they must stop and wait. The existing installation method can completely cover the side of the container.

[0071] It should be noted that, Figure 3 The installation locations and detection ranges of the cameras and lidar are provided as examples and do not constitute a limitation on the content.

[0072] Figure 4 This is a flowchart illustrating the process of determining whether to trigger the operation of a lidar system, as disclosed in an embodiment of the present invention. See also... Figure 4 As shown, in one implementation, the step of using an image similarity algorithm to compare images acquired by an image sensor in real time, and determining whether to trigger the LiDAR to start acquiring point cloud data based on the comparison result, may include:

[0073] Step 401: Use an image similarity algorithm to determine the starting image within the detection range where there are no passing vehicles.

[0074] If there are no passing vehicles, meaning no vehicles are within the image sensor's acquisition range, the background image without any passing vehicles is designated as the starting image. This image can then be used as a reference for subsequent comparisons to determine whether the acquisition range includes passing vehicles.

[0075] Specifically, the step of using an image similarity algorithm to determine the starting image where no vehicles are passing within the detection range may include: obtaining N images collected by the image sensor within a first time period, where N is a positive integer; comparing the similarity between adjacent frames of the N images one by one to obtain N-1 similarity values; if the range of all similarity data is lower than a first similarity threshold and the variance of all similarity data is lower than a second similarity threshold, then the last frame of the N images is determined as the starting image.

[0076] Figure 5 This is a schematic diagram of the point cloud data acquisition triggering process disclosed in an embodiment of the present invention. Combined with... Figure 5 As shown, an implementation may include the following steps:

[0077] Step 501: The camera continuously captures several images, periodically checking whether the capture time is greater than 3 minutes. If not, proceed to step 502; if yes, proceed to step 504.

[0078] Step 502: Continue acquiring images;

[0079] Step 503: Compare the similarity between the current frame and the previous frame one by one. The comparison method is not limited. It can be compared by a hash image similarity algorithm or by an image classification model trained by a CNN convolutional neural network in deep learning. Return to step 501.

[0080] Step 504: Calculate the range and variance of all similarities obtained within 3 minutes;

[0081] Step 505: Determine whether the similarity range and variance between the images collected in 3 consecutive minutes are within the thresholds TH_gap and TH_var, respectively. If yes, proceed to step 506; otherwise, return to step 501.

[0082] Step 506: Save the last frame image within 3 minutes as the start image BG (BEGIN image, i.e., the start image).

[0083] Since the similarity range and variance calculated within 3 minutes can represent the relative stable state of the current environment, and setting within the thresholds TH_gap and TH_var can determine whether a vehicle has arrived, thus becoming the start flag for whether data collection can be performed. Due to on-site environmental factors, including light, brightness, etc., it takes time to determine the thresholds, so the collection time is set to 3 minutes. Of course, in practical applications, the collection duration can also be set to other durations outside 3 minutes based on needs, and this application does not limit this.

[0084] Step 402: Periodically obtain the first image collected by the image sensor, and sequentially compare the first similarity between the start image and each of the first images.

[0085] Set the collection interval T (considering that the stop duration of the vehicle passing through the gate is short, the time interval preferably satisfies the following condition 1s < T < 5s). Every T duration, collect and extract a new picture and save it as IMAGE.

[0086] Compare the similarity between the picture BG and the picture IMAGE. The comparison method is not limited. For example, it can be compared through the Hash hash image similarity algorithm; it includes the gate background and vehicle information, etc.

[0087] Step 403: In the period from when the first similarity is lower than the first threshold to when the first similarity is not greater than the second threshold, assign the first value to the first variable. In the period when the first similarity is higher than the second threshold, assign the second value to the first variable.

[0088] Since the smaller the similarity, the greater the difference; set the threshold Th_1 to be suitable for the similarity value of the presence / absence of a vehicle, that is, below the threshold, there is a suitable difference value from the original background, and it is judged that a vehicle has passed within the camera detection range. If the similarity is less than the threshold Th_1, that is, when a vehicle has passed within the camera detection range, the camera continuously collects and saves the data image, records the current moment t_start, and outputs the variable Flag = True. This variable is used to trigger the lidar to collect data.

[0089] After triggering the lidar to collect data, when the similarity between the collected picture image(n) and the picture BG is greater than the threshold Th_2, stop continuously collecting camera data, record the current moment t_end, and output the variable Flag = False. This variable is used to terminate the lidar from collecting lidar data. Considering that the image is affected by natural light, assign the new Nth frame picture image(n) to the variable BG to ensure that the picture in the BG variable changes over time, which can avoid the lidar being mis-triggered due to the long-term non-update of the original picture.

[0090] Based on the above, the step of triggering the lidar to start collecting point cloud data when it is determined that a vehicle appears in the image collected by the image sensor, and triggering the lidar to stop collecting when it is determined that the vehicle leaves the image collected by the image sensor, may include: controlling the lidar to collect and save point cloud data when the first variable is a first value, and stopping the collection of point cloud data when the first variable is a second value.

[0091] The implementation logic for data acquisition by the lidar in this implementation is as follows: Figure 6 As shown. Combined with Figure 6 The lidar determines whether to collect data based on the trigger flag. Specifically, the implementation may include the following steps:

[0092] Step 601: Determine the value of the flag bit (Flag);

[0093] Step 602: When the received Flag = True, trigger the LiDAR to collect and save point cloud data; otherwise, do not collect point cloud data.

[0094] Step 603: After the data collection begins, obtain and determine the value of the new flag bit (Flag);

[0095] Step 604: When Flag = FALSE, stop collecting point cloud data; otherwise, continue collecting point cloud data. Save the collected data in pcap format for subsequent point cloud data parsing.

[0096] In one implementation, the step of parsing the point cloud data to generate a PCD file and obtaining multiple frames of image data from the PCD file by dividing the time equally can include: dividing the point cloud data into M time points based on the time difference between the first and last frames, and selecting M point cloud image frames corresponding to the M time points from the point cloud data, where M is a positive integer not less than 2.

[0097] Figure 7 This is a schematic diagram illustrating the image data and point cloud data saving process disclosed in an embodiment of the present invention. (Combined with...) Figure 7 As shown in the example, saving image data and point cloud data may include:

[0098] Step 701: Based on the time difference between the start time t_start and the end time t_end of the point cloud data collection, generate 10 time points t_1, t_2, ..., t_10.

[0099] Step 702: Analyze the 10 time points t_1, t_2, ..., t_10 corresponding to 10 frame files (point cloud data). At the same time, filter out the corresponding 10 frame image data.

[0100] Before or during the execution of steps S71 and S72, the following steps S73 and S74 may be executed.

[0101] Step 703: Generate a pcd file by parsing the original point cloud data pcap file using an algorithm;

[0102] Step 704: Save the image data corresponding to the pcd file.

[0103] Figure 8 This is a schematic diagram of the process for determining the optimal frame disclosed in an embodiment of the present invention. (In conjunction with...) Figure 8 As shown, the step of selecting the best frame from the multi-frame image data for damage detection based on the principle of minimum image similarity, according to the correspondence between the point cloud data and the image frames of the multi-frame image data, may include: selecting the point cloud image frame with the lowest similarity to the starting image from the M point cloud image frames and determining it as the best point cloud image frame for damage detection. Specifically, it may include:

[0104] Step 801: Calculate the similarity between the first image corresponding to each of the M point cloud image frames and the starting image frame by frame;

[0105] Step 802: Obtain the image index or corresponding time of the point cloud image frame based on the first image with the highest similarity;

[0106] Step 803: Index or time match M point cloud image frames and determine the best point cloud image frame from them.

[0107] Since the laser data corresponds to the image index (frame), the best frame is selected for damage detection based on the principle of minimum image similarity among the 10 generated PCD files. The similarity between each of the 10 image files and the image BG that triggered the LiDAR data acquisition is calculated. The image with the lowest similarity is selected, and the corresponding PCD file is found from the 10 PCD files based on the index or time of that image.

[0108] The defect detection of the optimal point cloud image frame may include: using a pass-through filtering algorithm to filter out point cloud data other than the detection object in the optimal point cloud image frame; using a corner recognition algorithm to identify the corner data of the detection object and extract the box surface data of the detection object; using a plane fitting algorithm to fit the box surface data, establish a feature extraction model, and extract planar features; using a decision tree algorithm to establish a recognition model, calculate the distance from the point cloud corresponding to the box surface data to the box surface representing the box surface, perform feature engineering, and determine the degree of defect of the detection target.

[0109] Figure 9 This is a schematic diagram of container residue detection disclosed in an embodiment of the present invention. (Combined with...) Figure 9 Container residue testing can include:

[0110] Step 901: Use a pass-through filtering algorithm to filter out most of the redundant point clouds in the pcd file according to the detection range;

[0111] Step 902: Use a corner recognition algorithm to identify the corners of the container side and extract the identified container surface data;

[0112] Step 903: Use a plane fitting algorithm to fit the box surface data, establish a feature extraction model, and extract the planar features;

[0113] Step 904: Use the machine learning algorithm - decision tree to build a recognition model, calculate the distance between the point cloud and the plane and perform feature engineering, and classify the damage level of the container according to the decision tree algorithm;

[0114] Step 905: The model outputs the recognition results, giving the corresponding degree of damage according to the location and number of boxes.

[0115] Finally, the generated damaged results and corresponding image data can be sent to a designated server via a protocol.

[0116] This application's solution utilizes image similarity-based LiDAR triggering. By detecting the environment through a camera, it can identify whether a vehicle has arrived at the detection location, triggering the LiDAR to periodically collect point cloud data, thus avoiding memory consumption caused by storing large amounts of invalid data. The computing unit analyzes the collected data, filters and saves the best container data, and extracts the optimal frame of container surface data based on calculated image similarity. The container surface data is fitted using an existing model to obtain planar feature information, and machine learning is used to determine the degree of container damage. Finally, the results and images are sent to a designated server according to the communication protocol, facilitating subsequent verification.

[0117] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0118] The methods described in the above-disclosed embodiments of the present invention are detailed in terms of their specific implementation. The methods of the present invention can be implemented using various forms of devices. Therefore, the present invention also discloses a device, and specific embodiments are given below for detailed description.

[0119] Figure 10 This is a schematic diagram of the structure of a container condition detection device disclosed in an embodiment of the present invention. See also... Figure 10 As shown, the container condition detection device 100 may include:

[0120] The image processing module 1001 is used to compare the images acquired by the image sensor in real time using an image similarity algorithm, and determine whether to trigger the lidar to start acquiring point cloud data based on the comparison result. The image sensor and lidar respectively acquire image data and point cloud data of passing vehicles, and the image sensor periodically acquires images.

[0121] The radar control module 1002 is used to trigger the lidar to start collecting point cloud data when it is determined that a vehicle appears in the image collected by the image sensor, and to trigger the lidar to end collecting data when it is determined that the vehicle has driven away from the image collected by the image sensor.

[0122] The point cloud processing module 1003 is used to parse the point cloud data to generate a PCD file and obtain multiple frames of image data from the PCD file by dividing the time equally.

[0123] The damage detection module 1004 is used to select the best frame from the multi-frame image data for damage detection based on the correspondence between the point cloud data and the image frames of the multi-frame image data and the principle of minimum image similarity.

[0124] The container status detection device described in this embodiment determines whether there are passing vehicles based on on-site images. If passing vehicles are present, it triggers the LiDAR to acquire point cloud data of the container. The container is then damaged by processing and analyzing the point cloud data. Compared with existing image recognition algorithms, this solution is less affected by lighting conditions and avoids the high costs associated with purchasing high-performance graphics cards for deep learning. Furthermore, due to the unique 3D detection properties of LiDAR, it can accurately identify the degree of damage to a plane, avoiding inaccurate raw data caused by changes in lighting.

[0125] For details on the specific implementation of each module of the container condition detection device and other possible implementations, please refer to the relevant sections of the method embodiments, which will not be repeated here.

[0126] Any of the container condition detection devices described in the above embodiments includes a processor and a memory. The image processing module, radar control module, point cloud processing module, damage detection module, etc. in the above embodiments are all stored as program modules in the memory, and the processor executes the above program modules stored in the memory to realize the corresponding functions.

[0127] The processor contains a kernel, which retrieves the corresponding program modules from memory. One or more kernels can be configured, and the processing of backtracking data can be achieved by adjusting kernel parameters.

[0128] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0129] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer, and contains software code. After being loaded and executed by the computer, the computer program can implement the steps shown in any embodiment of the container status detection method described above.

[0130] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by the computer, the computer program can implement the steps shown in any embodiment of the container status detection method described above.

[0131] Furthermore, embodiments of the present invention provide an electronic device. Figure 11 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. See also... Figure 11 As shown, the electronic device 1100 includes at least one processor 1101, at least one memory 1102 connected to the processor, and a bus 1103; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the above-mentioned container status detection method.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0133] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0135] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting the condition of a container, characterized in that, include: An image similarity algorithm is used to compare images acquired by an image sensor in real time. Based on the comparison results, it is determined whether to trigger the lidar to start acquiring point cloud data. Specifically, image data and point cloud data of passing vehicles are obtained through an image sensor and a lidar, respectively. The image sensor periodically acquires images. When a vehicle is detected in the image captured by the image sensor, the lidar is triggered to start collecting point cloud data. When the vehicle is detected to have moved away from the image captured by the image sensor, the lidar is triggered to end the data collection. The point cloud data is parsed to generate a PCD file, and multiple frames of image data are obtained from the PCD file by dividing the time equally. Based on the correspondence between the point cloud data and the image frames of the multi-frame image data, the best frame is selected from the multi-frame image data for damage detection based on the principle of minimum image similarity.

2. The container condition detection method according to claim 1, characterized in that, The image sensor and the lidar are coaxial in the vertical direction, and the distance from the axis center to the road edge of the passing vehicle is 1.5 meters. The image sensor is 2.5 meters above the ground, and the lidar is 2 meters above the ground. The image sensor has a horizontal detection field of view of 150° and a vertical detection field of view of 60°, while the lidar has a horizontal detection field of view of 360° and a vertical detection field of view of 90°.

3. The container condition detection method according to claim 1, characterized in that, The process of using an image similarity algorithm to compare images acquired by an image sensor in real time, and determining whether to trigger the lidar to start acquiring point cloud data based on the comparison results, includes: An image similarity algorithm is used to determine the starting image within the detection range where no vehicles are passing; The first image acquired by the image sensor is periodically obtained, and the first similarity between the starting image and each of the first images is compared sequentially. From the time when the first similarity is lower than the first threshold until the first similarity is no greater than the second threshold, the first variable is assigned a first value; and during the time when the first similarity is higher than the second threshold, the first variable is assigned a second value.

4. The container condition detection method according to claim 3, characterized in that, The step of using an image similarity algorithm to determine the starting image where no vehicles are passing within the detection range includes: Obtain N images acquired by the image sensor during the first time period, where N is a positive integer; For the N images, the similarity between adjacent frames is compared one by one to obtain N-1 similarity values; If the range of all similarity data is lower than the first similarity threshold, and the variance of all similarity data is lower than the second similarity threshold, then the last frame of the N images is determined as the starting image.

5. The container condition detection method according to claim 3, characterized in that, The process of triggering the LiDAR to start collecting point cloud data when a vehicle is detected in the image acquired by the image sensor, and triggering the LiDAR to stop collecting data when the vehicle is detected to have moved away from the image acquired by the image sensor, includes: When the first variable is at a first value, the lidar is controlled to collect and save point cloud data; when the first variable is at a second value, the collection of point cloud data is stopped.

6. The container condition detection method according to claim 3, characterized in that, The step of parsing the point cloud data to generate a PCD file, and obtaining multiple frames of image data from the PCD file by dividing the time equally, includes: Based on the time difference between the first and last frames of the point cloud data, M time points are equally divided. M point cloud image frames corresponding to the M time points are then selected from the point cloud data, where M is a positive integer not less than 2.

7. The container condition detection method according to claim 6, characterized in that, The step of selecting the best frame from the multi-frame image data for damage detection based on the correspondence between the point cloud data and the image frames of the multi-frame image data, according to the principle of minimum image similarity, includes: The point cloud image frame with the lowest similarity to the starting image is selected from the M point cloud image frames and determined as the best point cloud image frame for damage detection.

8. The container condition detection method according to claim 7, characterized in that, Defect detection is performed on the optimal point cloud image frame, including: A pass-through filtering algorithm is used to filter out point cloud data other than the detected object in the optimal point cloud image frame; A corner detection algorithm is used to identify the corner data of the detected object and extract the box surface data of the detected object; The box surface data is fitted using a plane fitting algorithm to establish a feature extraction model and extract planar features; A decision tree algorithm is used to establish a recognition model, calculate the distance from the point cloud corresponding to the box surface data to the box surface data representation box surface, perform feature engineering, and determine the degree of damage of the detected target.

9. A container condition detection device, characterized in that, include: The image processing module is used to compare the images acquired by the image sensor in real time using an image similarity algorithm, and determine whether to trigger the lidar to start acquiring point cloud data based on the comparison result. The image sensor and lidar respectively acquire image data and point cloud data of passing vehicles, and the image sensor periodically acquires images. The radar control module is used to trigger the lidar to start collecting point cloud data when it determines that a vehicle appears in the image collected by the image sensor, and to trigger the lidar to stop collecting data when it determines that the vehicle has moved away from the image collected by the image sensor. The point cloud processing module is used to parse the point cloud data to generate a PCD file, and to obtain multiple frames of image data from the PCD file by dividing the time equally. The damage detection module is used to select the best frame from the multi-frame image data for damage detection based on the correspondence between the point cloud data and the image frames of the multi-frame image data and the principle of minimum image similarity.

10. An electronic device, characterized in that, include: processor; Memory for storing executable program instructions of the processor; The executable program instructions include: using an image similarity algorithm to compare images acquired by an image sensor in real time; determining whether to trigger the lidar to start acquiring point cloud data based on the comparison results; wherein image data and point cloud data of passing vehicles are obtained by the image sensor and lidar respectively, and the image sensor periodically acquires images; when it is determined that a vehicle appears in the image acquired by the image sensor, the lidar is triggered to start acquiring point cloud data; when it is determined that the vehicle leaves the image acquired by the image sensor, the lidar is triggered to stop acquiring data; parsing the point cloud data to generate a PCD file; obtaining multiple frames of image data from the PCD file by dividing the time equally; and selecting the best frame from the multiple frames of image data for damage detection based on the principle of minimum image similarity, according to the correspondence between the point cloud data and the image frames of the multiple frames of image data.

Images