Loading state detection method, device and equipment and computer program product

Abstract

The embodiment of the invention is suitable for the technical field of Internet of Things, and provides a loading state detection method, device and equipment and a computer program product, and the method comprises the steps: obtaining image data and point cloud data collected for the tail of a carriage; according to the image data and the point cloud data, the loading state of the carriage is recognized; and outputting the loading rate of the compartment under the condition that the loading state is ready for loading, so that the compartment can be measured in real time to identify the loading state of the compartment, and if the loading state is ready for the device, the loading rate may change, so that the loading rate can be changed under the condition that the loading state of the compartment is ready for loading. And outputting the loading rate of the carriage, realizing automatic measurement of the loading state of the carriage, and automatically triggering and outputting the loading rate of the carriage.

Description

Technical Field

[0001] The embodiments of the present application belong to the technical field of Internet of Things, and in particular, relate to a loading status detection method, device, equipment and computer program product. Background Art

[0002] With the rapid development of e-commerce and other fields, the logistics industry has also ushered in great development. Therefore, the transportation management of tens of thousands of parcels has become a core issue facing the logistics industry. Among them, the logistics loading link is particularly important. Currently, the loading and unloading of vehicles basically rely on manual records. The accuracy of the data cannot be verified, the labor cost is high, and the timeliness of the data is poor. Therefore, the real-time scheduling of vehicles is difficult. Insufficient or excess volume in the loading process has a serious impact on production and manufacturing, and the efficiency is low.

[0003] In order to improve the feasibility of real-time vehicle dispatch and determine whether the vehicle has excess capacity, it is usually necessary to detect the vehicle's used loading capacity (or remaining capacity). Generally, detecting the loading capacity is a continuous process, and how to accurately start and end the loading capacity detection directly affects the effectiveness and energy consumption of the loading capacity detection. In some existing solutions, in order to balance the effectiveness and energy consumption of the loading capacity, it is necessary to make certain modifications to the vehicle, including installing a vehicle motion state detection device and a door state detection device in the vehicle, and combining the detection data of the above state detection devices to trigger the detection device in the vehicle to output the loading capacity. Since the vehicle hardware modification is required, the cost is high and it cannot be generally applied. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a loading status detection method, device, equipment and computer program product to improve the efficiency of detecting the loading status of a vehicle compartment.

[0005] A first aspect of an embodiment of the present application provides a loading status detection method, comprising: Acquire image data and point cloud data collected from the rear of the carriage; identifying a loading state of the carriage based on the image data and the point cloud data; When the loading status is ready to be loaded, the loading rate of the carriage is output.

[0006] In some implementations of the first aspect, the method further includes: When the loading status is ready to load, returning to the step of identifying the loading status of the carriage based on the image data and the point cloud data; When the loading status is loading completion, loading completion information is output.

[0007] In some implementations of the first aspect, identifying the loading state of the carriage based on the image data and the point cloud data includes: Performing a ready state detection based on the image data and the point cloud data; In the case where the ready state detection is completed, determining that the loading state of the carriage is ready to load, and performing an end state detection based on the image data and the point cloud data; When the end state detection is completed, the loading state of the carriage is updated to loading completed.

[0008] In some implementations of the first aspect, performing the ready state detection based on the image data and the point cloud data includes: Identifying pixel coordinates of corner points of the vehicle compartment in the image data; In a case where the corner point pixel coordinates are located in a preset corner point area of ​​the image data, determining a first distance between the tail and a preset position; When the first distance reaches a first threshold, determining whether there is an inner wall line in the image data, and determining whether the feature data is located in the same plane; the feature data is point cloud data between the corner points of the carriage; If the characteristic data are not located in the same plane, determining a target plane according to the characteristic data, and determining a second distance between the target plane and the preset position; When an inner wall line exists in the image data and a difference between the first distance and the second distance is greater than a second threshold, it is determined that the ready state detection is completed.

[0009] In some implementations of the first aspect, determining a first distance between the tail and a preset position includes: identifying the pixel coordinates of the license plate in the image data that matches the vehicle license plate located at the rear; Determine the current long side pixel distance and the current short side pixel distance according to the license plate pixel coordinates; Determine a first distance between the tail and a preset position according to the current long side pixel distance, the current short side pixel distance and preset mapping information; The mapping information records a plurality of different distance values, and a preset long-side pixel distance and a preset short-side pixel distance corresponding to each of the distance values.

[0010] In some implementations of the first aspect, the identifying the license plate pixel coordinates in the image data that match the vehicle license plate located at the rear portion includes: Determine the pixel coordinates of the corner point of the carriage located in the image data; Determining a recognition area in the image data according to the pixel coordinates of the corner points; The pixel coordinates of the license plate that matches the vehicle license plate located at the rear are identified in the identification area.

[0011] In some implementations of the first aspect, performing end state detection based on the image data and the point cloud data includes: Determine a third distance between the tail and a preset first position according to the image data and the point cloud data; If the characteristic data are located in the same plane, determining that the end state detection is completed; If the feature data are not located in the same plane, the point cloud distance is determined according to the feature data, and when the difference between the point cloud distance and the third distance is less than a third threshold, it is determined that the end state detection is completed.

[0012] In some implementations of the first aspect, determining, based on the image data and the point cloud data, a third distance between the tail and a preset first position includes: Determining a true width value of the tail according to the point cloud data; Determine the pixel width value of the tail according to the image data; A third distance between the tail and a preset first position is determined according to the real width value, the pixel width value and preset data acquisition parameters.

[0013] In some implementations of the first aspect, determining the true width value of the tail according to the point cloud data includes: Determine a left wall plane and a right wall plane, and a normal vector of the left wall plane or the right wall plane according to the point cloud data; Determine a fourth distance between the left wall plane and the preset origin coordinates, and a fifth distance between the right wall plane and the preset origin coordinates; The actual width value of the tail is determined according to the normal vector, the fourth distance, and the fifth distance.

[0014] A second aspect of an embodiment of the present application provides a loading status detection device, including: A data acquisition module, used to acquire image data and point cloud data collected from the rear of the carriage; A loading status recognition module, used to recognize the loading status of the carriage according to the image data and the point cloud data; The loading rate output module is used to output the loading rate of the carriage when the loading status is ready to be loaded.

[0015] A third aspect of an embodiment of the present application provides an electronic device, comprising a ToF module, a camera module, a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the loading status detection method as described in the first aspect above.

[0016] A fourth aspect of an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed, the loading status detection method described in the first aspect is executed.

[0017] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the loading status detection method as described in the first aspect above is implemented.

[0018] Compared with the prior art, the embodiments of the present application have the following beneficial effects: In the embodiment of the present application, image data and point cloud data are acquired by collecting data from the rear of a car; the loading status of the car is identified based on the image data and the point cloud data; and when the loading status is ready to load, the loading rate of the car is output, so that the car can be measured in real time to identify its loading status. If the loading status is ready to load, the loading rate may change. Therefore, when it is determined that the loading status of the car is ready to load, the loading rate of the car is output, thereby realizing automatic measurement of the loading status of the car and automatic triggering of the output of the loading rate of the car. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of some components of a measurement terminal provided in an embodiment of the present application is shown; Figure 2 This is a schematic diagram of a measurement terminal deployment provided in an embodiment of the present application; Figure 3 is a schematic diagram of a loading status detection method provided in an embodiment of the present application; Figure 4 This is a schematic diagram of another loading status detection method provided in an embodiment of the present application. Figure 5This is a ready state detection flow chart provided by an embodiment of the present application; Figure 6 It is a schematic diagram of partial image data when the tailgate of a vehicle compartment is not opened; Figure 7 It is a schematic diagram of partial image data after the tailgate of a vehicle compartment is opened; Figure 8 is an end state detection flow chart provided by an embodiment of the present application; Fig. 9 is a schematic diagram of a loading status detection device provided in an embodiment of the present application; Fig.10 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0022] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0023] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0025] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0027] Reference Figure 1 A schematic diagram of some components of a measurement terminal provided by an embodiment of the present application is shown, and the measurement terminal includes a ToF (Time of Flight) module for acquiring point cloud data, a camera module for acquiring image data, and a computing module for processing point cloud data and / or image data, including but not limited to. The lens baseline in the ToF module is parallel to the lens baseline in the camera module, and the ToF module and the camera module are arranged adjacent to each other to avoid measurement errors caused by mechanism deformation.

[0028] Reference Figure 2 , shows a schematic diagram of the deployment of a measurement terminal provided in an embodiment of the present application, and the measurement terminal is deployed at a specified position PT. If the vehicle travels to the specified area and faces the specified direction, the measurement terminal can face the rear of the vehicle's carriage and perform corresponding measurements. As an example, the measurement terminal is 1.8-2.2 meters above the bottom of the carriage, and the baseline of the lens remains parallel to the longitudinal direction of the carriage. When deploying the measurement terminal, the ToF field of view is maintained to cover more than half of the length of the longest carriage. When measuring vehicles with shorter carriages, the length of the ToF area in the longitudinal direction of the carriage is kept unchanged, and the distance between the camera areas in the longitudinal direction of the carriage is shortened.

[0029] After the measurement terminal is deployed, corresponding measurements can be performed on the vehicle. The technical solution of the present application is described below through specific embodiments.

[0030] Reference Figure 3 , shows a schematic diagram of a loading status detection method provided in an embodiment of the present application, which may specifically include the following steps: Step 301, acquiring image data and point cloud data collected from the rear of the carriage; After the deployment of the measurement terminal is completed, if the vehicle is located at the specified position and facing the specified direction, the ToF module and the camera module of the measurement terminal are facing the rear of the vehicle's upper compartment, so that the measurement terminal can use the camera module to collect data from the rear of the compartment to obtain image data, and use the ToF module to collect data from the rear of the compartment to obtain point cloud data.

[0031] Step 302, identifying the loading status of the carriage according to the image data and the point cloud data; Combine image data and point cloud data to identify the loading status of the carriage. The loading status of the carriage can be divided into ready to load and ready to load. Ready to load means that relevant personnel or equipment can start loading operations on the carriage, that is, placing loading objects in the carriage. The loading objects are objects to be transported by the vehicle. The loading operation can be performed by relevant personnel or equipment.

[0032] In some implementations, before determining that the carriage is in a loading state that is ready to load, the loading state of the carriage may be determined to be not ready to load, indicating that relevant personnel or equipment are currently unable to perform loading operations on the carriage.

[0033] Step 303: when the loading status is ready to load, output the loading rate of the carriage.

[0034] When the loading state is ready to load, it means that the relevant personnel or equipment can start or have started loading operations on the carriage, and the loading rate of the carriage has changed. At this time, the carriage can be checked in a preset manner to determine and output the current loading rate of the carriage.

[0035] The embodiment of the present application does not limit the implementation method of determining the carriage loading rate.

[0036] As an example, since point cloud data can determine the depth of the current ToF module at different positions in the car, and image data can determine whether and where there are loaded objects in the current car, the measurement terminal can combine the point cloud data and image data to generate the loading rate of the car at the current moment.

[0037] In an embodiment of the present application, image data and point cloud data are acquired by capturing the rear of a car; based on the image data and the point cloud data, the loading status of the car is identified; when the loading status is ready to load, the loading rate of the car is output, so that the car can be measured in real time to identify its loading status. If the loading status is ready to load, the loading rate may change. Therefore, when it is determined that the loading status of the car is ready to load, the loading rate of the car is output, thereby realizing automatic measurement of the loading status of the car and automatic triggering of the output of the loading rate of the car.

[0038] Since the measurement terminal is fixedly set up, if the measurement terminal cannot collect point cloud data or image data matching the rear of the vehicle, it can be determined that the current vehicle is not in the specified position. Therefore, the embodiment of the present application can also realize whether the parking position and parking direction of the vehicle meet the preset parking conditions, and the parking conditions include but are not limited to a specified area and a specified direction.

[0039] Reference Figure 4 , showing a schematic diagram of another loading status detection method provided in an embodiment of the present application. In some embodiments of the present application, the method further includes: when the loading status is ready to load, returning to the step of identifying the loading status of the carriage based on the image data and the point cloud data; step 304, when the loading status is loading completion, outputting loading completion information.

[0040] The loading state can also be loading completed, which means that the relevant personnel or equipment have completed the loading operation for the carriage and the vehicle can leave the current location. When the loading state of the current carriage is determined to be ready for loading, it means that the relevant personnel or equipment are operating, that is, the loading is not completed. In this case, after outputting the loading rate of the carriage, it is necessary to return to step 302 until the loading state is changed to the completed state.

[0041] When the loading status is loading completion, the loading completion information is output to prompt that the loading operation for the carriage has been completed. At this time, the loading rate will no longer change, and the generation and output of the carriage's loading rate can be stopped.

[0042] In some implementations, if it has been determined that the carriage is in a ready-to-load state and has not reached a loading completion state, the loading state of the carriage is determined to be loading in progress, indicating that relevant personnel or equipment are performing loading operations on the carriage. The loading rate of the carriage can be output at this time to determine the loading progress.

[0043] In some embodiments of the present application, step 302 includes: Performing a ready state detection based on the image data and the point cloud data; In the case where the ready state detection is completed, determining that the loading state of the carriage is ready to load, and performing an end state detection based on the image data and the point cloud data; When the end state detection is completed, the loading state of the carriage is updated to loading completed.

[0044] In the process of identifying the loading status of the carriage, a ready state detection is first performed based on the image data and the point cloud data. If the ready state detection is completed, the loading status is determined to be ready for loading; if the ready state detection is not completed, the loading status is determined to be not ready for loading.

[0045] Since the loading operation takes a certain amount of time, the loading status of "ready to load" may last for a while. At this time, it is necessary to continuously perform end status detection based on image data and point cloud data to determine whether the loading operation is completed.

[0046] When the end state detection is not completed, the loading state of the carriage is updated to ready for loading; when the end state detection is completed, the loading state of the carriage is updated to completed for loading.

[0047] As an example, when the ready state detection is completed but the end state detection is not completed, the loading state of the carriage may be determined as loading in progress to determine that relevant personnel or equipment are currently performing a loading operation.

[0048] Reference Figure 5 , shows a flowchart of a ready state detection provided by an embodiment of the present application. In some embodiments of the present application, the ready state detection based on the image data and the point cloud data includes: identifying the corner point pixel coordinates of the corner point of the carriage in the image data; when the corner point pixel coordinates are located in a preset corner point area of ​​the image data, determining a first distance between the tail and a preset position; When the first distance reaches a first threshold, determining whether there is an inner wall line in the image data, and determining whether the feature data is located in the same plane; the feature data is point cloud data between the corner points of the carriage; If the characteristic data are not located in the same plane, determining a target plane according to the characteristic data, and determining a second distance between the target plane and the preset position; When an inner wall line exists in the image data and a difference between the first distance and the second distance is greater than a second threshold, it is determined that the ready state detection is completed.

[0049] There are 4 preset corner point areas, and the positions of the preset corner point areas in the image data are preset, so as to determine whether the rear of the carriage is close to or located in the middle of the image data. The preset position is the position of the measurement terminal, and the first distance can be the distance between the rear of the carriage and the preset position in the horizontal direction.

[0050] In practical applications, the loading status of the carriage is ready for loading when conditions (1)-(3) are met. (1) The carriage is parked in the designated area and the door is open; (2) The straight line of the inner wall of the carriage can be detected; (3) The door of the carriage is open, and the distance between the point cloud distance measurement and the tailgate of the carriage is greater than the threshold.

[0051] Reference Figure 6 , showing a schematic diagram of partial image data when the tailgate of a vehicle compartment is not opened; referring to Figure 7 , showing a schematic diagram of partial image data after the tailgate of a vehicle is opened.

[0052] As an example, the ready state detection is performed through the following steps: First, the car door corner point detection is performed on the image data to confirm whether the four corner points of the rear door of the car can be detected (upper left, lower left, upper right, and lower right). If the four corner points are not detected or the corner point positions are not within the image coordinate threshold range, the detection continues. If the conditions are met, the license plate prior knowledge is used to quickly measure the distance to the rear of the car to determine the first distance. When the distance to the rear door of the car (the first distance) reaches the set threshold, it is determined that the car is parked in the designated area, and then the image data and point cloud data are processed separately. Image processing is performed on the image data within the corner point area, and the number of straight lines or edges in the area can be detected using technologies such as line detection or edge detection. If no straight line or edge is detected, it indicates that the rear door of the car is not open, then return to the previous step and continue detection. The point cloud data (i.e., feature data) in the corner point area is plane-divided. Calculate whether the feature data are in the same plane. If the feature data are in the same plane, it is determined that the rear door of the car is not open (such as Figure 6 As shown), return to continuous detection, otherwise determine whether the difference between the tail distance and the plane distance is greater than the threshold, if it is greater, it is determined that the car door is open (such as Figure 7 as shown).

[0053] In some embodiments of the present application, determining the first distance between the tail and the preset position includes: identifying the license plate pixel coordinates in the image data that matches the vehicle license plate located at the tail; determining the current long side pixel distance and the current short side pixel distance based on the license plate pixel coordinates; determining the first distance between the tail and the preset position based on the current long side pixel distance, the current short side pixel distance and preset mapping information; the mapping information records a plurality of different distance values, as well as a preset long side pixel distance and a preset short side pixel distance corresponding to each of the distance values.

[0054] In practice, a vehicle license plate is provided at the rear of the carriage, and the vehicle license plate has certain shape characteristics, and the first distance can be determined based on this.

[0055] Since the parking area of ​​the vehicle is pre-set, the physical size of the vehicle license plate is known information, and the deployment position of the measurement terminal is also known, the pixel coordinate range of the vehicle license plate in the image data collected by the camera module can be pre-measured under different distance values ​​of the vehicle distance measurement terminal. It can be known that under different distance values ​​of the vehicle distance measurement terminal, the lengths of the sides of the vehicle license plate in the image data, including the preset long side pixel distance of the longer side of the vehicle license plate and the preset short side pixel distance of the shorter side of the vehicle license plate, thereby obtaining mapping information.

[0056] Based on the above mapping information, after the image data is collected, the current long side pixel distance and the current short side pixel distance of the currently collected vehicle license plate can be determined by identifying the pixel coordinate range of the vehicle license plate in the image data. The distance value corresponding to the mapping information difference and the current long side pixel distance and the current short side pixel distance is the first distance.

[0057] In one implementation, the current long side pixel distance and the current short side pixel distance may be proportionally converted and corrected according to the actual vehicle license plate aspect ratio (2:1).

[0058] In some embodiments of the present application, the identifying of the license plate pixel coordinates in the image data that matches the vehicle license plate located at the tail includes: determining the corner point pixel coordinates of the vehicle compartment corner point in the image data; determining an identification area in the image data based on the corner point pixel coordinates; and identifying the license plate pixel coordinates that matches the vehicle license plate located at the tail in the identification area.

[0059] According to the relative position of the vehicle license plate and the corner points of the vehicle compartment, after determining the pixel coordinates of the corner points, an area larger than the vehicle license plate area can be determined in the image data as the recognition area. By determining the pixel coordinates of the license plate in the recognition area rather than in the entire range of the image data, the efficiency of recognizing the pixel coordinates of the license plate can be improved.

[0060] Among them, the pixel coordinates of the corner point A of the car are (xleft, yleft), and the pixel coordinates of the corner point B of the car are (xright, yright). It can be seen that the pixel coordinates of the corner points of the recognition area are C (xa, ya), D (xb, ya), E (xa, yb), and F (xb, yb). Among them, xa=xleft+xright / 2-Xthresh, xb=xleft+xright / 2+Xthresh, ya=yleft, yb=yleft+Ythresh, and Xthresh and Ythresh are pre-set empirical thresholds.

[0061] Reference Figure 8, showing a flow chart of attire status detection provided by an embodiment of the present application; in some embodiments of the present application, the end status detection is performed based on the image data and the point cloud data, including: determining a third distance between the tail and a preset first position based on the image data and the point cloud data; if the feature data are located in the same plane, determining that the end status detection is completed; if the feature data are not located in the same plane, determining the point cloud distance based on the feature data, and when the difference between the point cloud distance and the third distance is less than a third threshold, determining that the end status detection is completed.

[0062] In actual applications, if the carriage loading status is ready to load, the carriage tailgate is open, so the distance to the rear of the carriage can be accurately measured, and then the point cloud data is judged whether it is in a plane. If it is in a plane, it is determined that the carriage door is closed and loading is completed. Otherwise, it is judged whether the point cloud distance and the distance to the rear of the carriage are less than the third threshold. When the point cloud distance and the distance to the rear of the carriage are less than the third threshold, it is confirmed that the carriage is fully loaded. Otherwise, the current loading status is judged to be ready to load (or loading), and the test is returned to continue until the loading is completed.

[0063] In some embodiments of the present application, determining the third distance between the tail and the preset first position based on the image data and the point cloud data includes: Determine the true width value of the tail based on the point cloud data; determine the pixel width value of the tail based on the image data; and determine the third distance between the tail and a preset first position based on the true width value, the pixel width value and preset data acquisition parameters.

[0064] Since the point cloud data is collected for the rear part of the car, and the tailgate is open when the car is in a ready-to-load state, the distance between the inner walls of the car can be determined based on the depth features contained in the point cloud data, and then the actual width value of the car can be determined.

[0065] Since the image data is collected for the rear of the carriage, the number of pixels corresponding to the width of the rear can be determined by identifying the image data, and the pixel width value of the rear can be determined according to the preset unit length and the number of pixels. For example, one pixel can be used as the unit length, and the pixel width value of the rear is the number of pixels corresponding to the width.

[0066] The preset acquisition parameter is the focal length f of the camera module. The actual width of the tailgate is known to be , the pixel width on the image is , focal length f (obtained by camera calibration), then the distance from the tailgate to the camera is d (i.e. the third distance).

[0067]

[0068] In some embodiments of the present application, determining the true width value of the tail according to the point cloud data includes: Determine a left wall plane and a right wall plane, and a normal vector of the left wall plane or the right wall plane according to the point cloud data; Determine a fourth distance between the left wall plane and the preset origin coordinates, and a fifth distance between the right wall plane and the preset origin coordinates; The actual width value of the tail is determined according to the normal vector, the fourth distance, and the fifth distance.

[0069] The origin coordinates can be determined in advance for the point cloud data, and a three-dimensional coordinate system can be constructed for the origin coordinates. The position of each point cloud can be determined in the three-dimensional coordinate system, and then the planes where different point clouds are located can be determined.

[0070] The RANSAC (Random Sample Consistency) algorithm is used to perform point cloud plane detection to determine the left wall plane and the right wall plane, as well as the normal vector of the left wall plane or the right wall plane. Since the left wall plane and the right wall plane are parallel planes in three-dimensional coordinates, their normal vectors are the same.

[0071] According to the fourth distance between the left wall plane and the preset origin coordinates, the fifth distance between the right wall plane and the preset origin coordinates, and the length of the normal vector, the distance between the left wall plane and the right wall plane can be determined, and the distance is determined to be the actual width value of the carriage. .

[0072]

[0073] Among them, A, B, and C are normal vectors of two parallel planes, D1 is the fourth distance, and D2 is the fifth distance.

[0074] As an example, in the RANSAC algorithm, the number of iterations N and the threshold D can be set. In each iteration, three points are randomly selected as the sample set, and the normal vector of the plane and the distance from the origin are calculated using these three points to obtain the plane equation. For each point in the point cloud, the distance to the estimated plane is calculated. If the distance is less than the threshold D, the point is considered to belong to the plane, and all points belonging to the plane are used to re-estimate the plane model. Repeat the above until the preset number of iterations N is reached. The model containing the most internal points (points belonging to the plane) is selected from all iterations as the final plane model. Finally, the point cloud is segmented into plane points and non-plane points according to the model.

[0075] Furthermore, in order to speed up the detection speed of the point cloud plane, the point cloud data is preprocessed. Since the inner wall plane of the car is relatively regular, the point cloud data can be divided into multiple grids so that the grid size is adapted to the point cloud density and plane size. Independent plane detection is then performed in each grid to reduce the amount of global search calculations and speed up point cloud plane detection.

[0076] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0077] Reference Fig. 9 , shows a schematic diagram of a loading status detection device provided in an embodiment of the present application, which may specifically include a data acquisition module 901, a loading status identification module 902, and a loading rate output module 903, wherein: The data acquisition module 901 is used to acquire image data and point cloud data collected from the rear of the carriage; A loading state recognition module 902, used to recognize the loading state of the carriage according to the image data and the point cloud data; The loading rate output module 903 is used to output the loading rate of the carriage when the loading status is ready to load.

[0078] In some embodiments of the present application, the device further comprises: A continuous identification module, used for calling the loading state identification module 902 when the loading state is ready to load; The loading completion information output module is used to output loading completion information when the loading status is loading completion.

[0079] In some embodiments of the present application, the loading status identification module 902 includes: A ready state detection submodule, used for performing a ready state detection based on the image data and the point cloud data; an end state detection submodule, for determining that the loading state of the carriage is ready for loading when the ready state detection is completed, and performing an end state detection based on the image data and the point cloud data; The loading status updating submodule is used to update the loading status of the carriage to loading completed when the end status detection is completed.

[0080] In some embodiments of the present application, the ready state detection submodule includes: A corner pixel coordinate identification unit, used to identify the corner pixel coordinates of the car corner in the image data; a first distance determining unit, configured to determine a first distance between the tail and a preset position when the pixel coordinates of the corner point are located in a preset corner point area of ​​the image data; A first judgment unit is used to judge whether there is an inner wall line in the image data and whether the feature data are located in the same plane when the first distance reaches a first threshold; the feature data is point cloud data between the corner points of the carriage; a second distance determining unit, configured to determine a target plane according to the feature data, and determine a second distance between the target plane and the preset position if the feature data are not located in the same plane; The ready state detection completion unit is used to determine that the ready state detection is completed when an inner wall line exists in the image data and a difference between the first distance and the second distance is greater than a second threshold.

[0081] In some embodiments of the present application, the first distance determining unit includes: a license plate pixel coordinate determination subunit, for identifying the license plate pixel coordinates in the image data that matches the vehicle license plate located at the tail; A license plate pixel distance determination subunit, used to determine the current long side pixel distance and the current short side pixel distance according to the license plate pixel coordinates; A first distance determination subunit, configured to determine a first distance between the tail and a preset position according to the current long side pixel distance, the current short side pixel distance and preset mapping information; The mapping information records a plurality of different distance values, and a preset long-side pixel distance and a preset short-side pixel distance corresponding to each of the distance values.

[0082] In some embodiments of the present application, the license plate pixel coordinate determination subunit includes: A corner point pixel coordinate determination subunit, used to determine the corner point pixel coordinates of the car corner point in the image data; An identification region determination subunit, configured to determine an identification region in the image data according to the pixel coordinates of the corner points; The license plate pixel coordinate determination subunit is used to identify the license plate pixel coordinates matching the vehicle license plate located at the tail in the identification area.

[0083] In some embodiments of the present application, the end state detection submodule includes: A third distance determining unit, configured to determine a third distance between the tail and a preset first position based on the image data and the point cloud data; A first completion unit for end state detection is used to determine that the end state detection is completed if the feature data are located in the same plane; The second completion unit of the end state detection is used to determine the point cloud distance based on the feature data if the feature data are not located in the same plane, and determine that the end state detection is completed when the difference between the point cloud distance and the third distance is less than a third threshold.

[0084] In some embodiments of the present application, the third distance determination unit includes: A real width value determination subunit, used to determine the real width value of the tail according to the point cloud data; A pixel width value determining subunit, used for determining a pixel width value of the tail according to the image data; The third distance determination subunit is used to determine a third distance between the tail and a preset first position according to the real width value, the pixel width value and preset data acquisition parameters.

[0085] In some embodiments of the present application, the real width value determining subunit includes: A plane and normal vector determination subunit, used to determine a left wall plane and a right wall plane, and a normal vector of the left wall plane or the right wall plane according to the point cloud data; a fourth distance determining subunit, configured to determine a fourth distance between the left side wall plane and a preset origin coordinate; a fifth distance determining subunit, configured to determine a fifth distance between the right side wall plane and a preset origin coordinate; The real width value calculation subunit is used to determine the real width value of the tail according to the normal vector, the fourth distance, and the fifth distance.

[0086] An embodiment of the present application provides a loading status detection device, and by using the device, each step in the aforementioned method embodiments can be implemented.

[0087] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0088] Reference Fig.10 , shows a schematic diagram of an electronic device provided by an embodiment of the present application. Fig.10As shown, the electronic device 1000 in the embodiment of the present application includes: a ToF module, a camera module, a processor 1010, a memory 1020, and a computer program 1021 stored in the memory 1020 and executable on the processor 1010. When the processor 1010 executes the computer program 1021, the steps in each embodiment of the above-mentioned loading status detection method are implemented, for example Figure 3 Alternatively, when the processor 1010 executes the computer program 1021, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Fig. 9 Functions of modules 901 to 903 are shown.

[0089] Exemplarily, the computer program 1021 may be divided into one or more modules / units, which are stored in the memory 1020 and executed by the processor 1010 to complete the present application. The one or more modules / units may be a series of computer program instruction segments that can perform specific functions, and the instruction segments may be used to describe the execution process of the computer program 1021 in the electronic device 1000. For example, the computer program 1021 may be divided into a data acquisition module, a loading status identification module, and a loading rate output module, and the specific functions of each module are as follows: A data acquisition module, used to acquire image data and point cloud data collected from the rear of the carriage; A loading status recognition module, used to recognize the loading status of the carriage according to the image data and the point cloud data; The loading rate output module is used to output the loading rate of the carriage when the loading status is ready to be loaded.

[0090] The electronic device 1000 may be the detection device in each of the above embodiments. The electronic device 1000 may include, but is not limited to, a processor 1010 and a memory 1020. Those skilled in the art may understand that Fig.10 It is only an example of the electronic device 1000 and does not constitute a limitation of the electronic device 1000. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 1000 may also include input and output devices, network access devices, buses, etc.

[0091] The processor 1010 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0092] The memory 1020 may be an internal storage unit of the electronic device 1000, such as a hard disk or memory of the electronic device 1000. The memory 1020 may also be an external storage device of the electronic device 1000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 1000. Further, the memory 1020 may also include both an internal storage unit of the electronic device 1000 and an external storage device. The memory 1020 is used to store the computer program 1021 and other programs and data required by the electronic device 1000. The memory 1020 may also be used to temporarily store data that has been output or is to be output.

[0093] The embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the loading status detection method as described in the above-mentioned embodiments is implemented.

[0094] An embodiment of the present application further discloses a computer program product, including a computer program. When the computer program is executed, the loading status detection method as described in the above embodiments is executed.

[0095] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A loading status detection method, characterized in that: include: Acquire image data and point cloud data collected from the rear of the carriage; identifying a loading state of the carriage based on the image data and the point cloud data; When the loading status is ready to be loaded, the loading rate of the carriage is output.

2. The method according to claim 1, characterized in that The method further comprises: When the loading status is ready to load, returning to the step of identifying the loading status of the carriage based on the image data and the point cloud data; When the loading status is loading completion, loading completion information is output.

3. The method according to claim 1, characterized in that The identifying the loading state of the carriage according to the image data and the point cloud data includes: Performing a ready state detection based on the image data and the point cloud data; In the case where the ready state detection is completed, determining that the loading state of the carriage is ready to load, and performing an end state detection based on the image data and the point cloud data; When the end state detection is completed, the loading state of the carriage is updated to loading completed.

4. The method according to claim 3, characterized in that The performing of the ready state detection based on the image data and the point cloud data includes: Identifying pixel coordinates of corner points of the vehicle compartment in the image data; In a case where the corner point pixel coordinates are located in a preset corner point area of ​​the image data, determining a first distance between the tail and a preset position; When the first distance reaches a first threshold, determining whether there is an inner wall line in the image data, and determining whether the feature data is located in the same plane; the feature data is point cloud data between the corner points of the carriage; If the characteristic data are not located in the same plane, determining a target plane according to the characteristic data, and determining a second distance between the target plane and the preset position; When an inner wall line exists in the image data and a difference between the first distance and the second distance is greater than a second threshold, it is determined that the ready state detection is completed.

5. The method according to claim 4, characterized in that The determining a first distance between the tail and a preset position includes: Determine the pixel coordinates of the corner point of the carriage located in the image data; Determining a recognition area in the image data according to the pixel coordinates of the corner points; Identifying, in the identification area, the pixel coordinates of the license plate that matches the vehicle license plate located at the rear; Determine the current long side pixel distance and the current short side pixel distance according to the license plate pixel coordinates; Determine a first distance between the tail and a preset position according to the current long side pixel distance, the current short side pixel distance and preset mapping information; The mapping information records a plurality of different distance values, and a preset long-side pixel distance and a preset short-side pixel distance corresponding to each of the distance values.

6. The method according to any one of claims 3 to 5, characterized in that: The performing end state detection based on the image data and the point cloud data includes: Determining a true width value of the tail according to the point cloud data; Determine the pixel width value of the tail according to the image data; Determine a third distance between the tail and a preset first position according to the real width value, the pixel width value and preset data acquisition parameters; If the characteristic data are located in the same plane, determining that the end state detection is completed; If the feature data are not located in the same plane, the point cloud distance is determined according to the feature data, and when the difference between the point cloud distance and the third distance is less than a third threshold, it is determined that the end state detection is completed.

7. The method according to claim 6, characterized in that Determining the true width value of the tail according to the point cloud data includes: Determine a left wall plane and a right wall plane, and a normal vector of the left wall plane or the right wall plane according to the point cloud data; Determine a fourth distance between the left wall plane and the preset origin coordinates, and a fifth distance between the right wall plane and the preset origin coordinates; The actual width value of the tail is determined according to the normal vector, the fourth distance, and the fifth distance.

8. A loading status detection device, characterized in that: include: A data acquisition module, used to acquire image data and point cloud data collected from the rear of the carriage; A loading status recognition module, used to recognize the loading status of the carriage according to the image data and the point cloud data; The loading rate output module is used to output the loading rate of the carriage when the loading status is ready to be loaded.

9. An electronic device, characterized in that: The electronic device comprises a ToF module, a camera module, a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the method according to any one of claims 1 to 7 to be performed.

Images