A method, device, equipment and computer program product for detecting a loading state

Abstract

The embodiments of the present application are applicable to the field of Internet of Things technology, and provide a method, device, equipment and computer program product for detecting the loading state. The method includes: acquiring image data and point cloud data collected for the tail of the carriage; identifying the loading state of the carriage based on the image data and the point cloud data; and outputting the loading rate of the carriage when the loading state is that the loading is ready. Thus, the carriage can be measured in real time to identify its loading state. If the loading state is that the device is ready, the loading rate may change. Therefore, when it is determined that the loading state of the carriage is that the loading is ready, the loading rate of the carriage is output, realizing automatic measurement of the loading state of the carriage and automatic triggering of the output of the loading rate of the carriage.

Description

Technical Field

[0001] The embodiments of the present application belong to the technical field of the Internet of Things, and particularly relate to a method, device, equipment, and computer program product for detecting the loading state. Background Art

[0002] With the rapid development of fields such as e-commerce, the logistics industry has also seen significant development. Therefore, the management of the transportation of tens of thousands of packages has become the core problem faced by the logistics industry. Among them, the logistics loading link is particularly important. Currently, the loading and unloading of vehicles basically rely on manual records, and the accuracy of the data cannot be verified, the labor cost is high, and the timeliness of the data is poor. Therefore, the real-time scheduling of vehicles is difficult, and insufficient or excessive volume during the loading process has a serious impact on production and manufacturing, and the efficiency is low.

[0003] To improve the feasibility of real-time vehicle scheduling and determine whether the volume of the vehicle is excessive, it is usually necessary to detect the used loading volume (or remaining volume) of the vehicle. Generally, detecting the loading volume is a continuous process, and how to accurately start and end the detection of the loading volume directly affects the effectiveness of the loading volume detection and energy consumption. In some existing solutions, to balance the effectiveness of the loading volume and energy consumption, it is necessary to transform the vehicle to a certain extent, including setting a vehicle motion state detection device and a door state detection device in the vehicle, and combining the detection data of the above state detection devices to trigger the detection device in the vehicle to output the loading volume. Since hardware transformation of the vehicle is required, the cost is high and it cannot be widely applied. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a method, device, equipment, and computer program product for detecting the loading state, so as to improve the detection efficiency of the loading state of the carriage.

[0005] The first aspect of the embodiments of the present application provides a method for detecting the loading state, including:

[0006] Obtaining image data and point cloud data collected for the tail of the carriage;

[0007] Identifying the loading state of the carriage based on the image data and the point cloud data;

[0008] Outputting the loading rate of the carriage when the loading state is that loading is ready.

[0009] In some implementation manners of the first aspect, the method further includes:

[0010] When the loading state is that loading is ready, returning to the step of identifying the loading state of the carriage based on the image data and the point cloud data;

[0011] When the loading state is the end of loading, output information indicating the end of loading.

[0012] In some implementations of the first aspect, identifying the loading state of the carriage according to the image data and the point cloud data includes:

[0013] Perform a readiness state detection according to the image data and the point cloud data;

[0014] When the readiness state detection is completed, determine that the loading state of the carriage is ready for loading, and perform an end state detection according to the image data and the point cloud data;

[0015] When the end state detection is completed, update the loading state of the carriage to the end of loading.

[0016] In some implementations of the first aspect, performing the readiness state detection according to the image data and the point cloud data includes:

[0017] Identify the corner pixel coordinates of the carriage corners in the image data;

[0018] When the corner pixel coordinates are located in a preset corner area of the image data, determine a first distance between the tail and a preset position;

[0019] When the first distance reaches a first threshold, determine whether there is an inner wall line in the image data, and determine whether the feature data is located on the same plane; the feature data is the point cloud data between the carriage corners;

[0020] If the feature data is not located on the same plane, determine a target plane according to the feature data, and determine a second distance between the target plane and the preset position;

[0021] When there is an inner wall line in the image data and the difference between the first distance and the second distance is greater than a second threshold, determine that the readiness state detection is completed.

[0022] In some implementations of the first aspect, determining the first distance between the tail and the preset position includes:

[0023] Identify the license plate pixel coordinates in the image data that match the vehicle license plate located at the tail;

[0024] According to the license plate pixel coordinates, determine the current long side pixel distance and the current short side pixel distance;

[0025] Determine a first distance between the tail and a preset position according to the current long-side pixel distance, the current short-side pixel distance, and preset mapping information;

[0026] The mapping information records different multiple distance values, and preset long-side pixel distances and preset short-side pixel distances corresponding to each of the distance values.

[0027] In some implementation manners of the first aspect, the identifying the license plate pixel coordinates in the image data that match the vehicle license plate located at the tail includes:

[0028] Determine the corner pixel coordinates of the carriage in the image data;

[0029] Determine an identification area in the image data according to the corner pixel coordinates;

[0030] Identify the license plate pixel coordinates in the identification area that match the vehicle license plate located at the tail.

[0031] In some implementation manners of the first aspect, the performing an end state detection according to the image data and the point cloud data includes:

[0032] Determine a third distance between the tail and a preset first position according to the image data and the point cloud data;

[0033] If the feature data is in the same plane, determine that the end state detection is completed;

[0034] If the feature data is not in the same plane, then determine a point cloud distance according to the feature data, and determine that the end state detection is completed when the difference between the point cloud distance and the third distance is less than a third threshold.

[0035] In some implementation manners of the first aspect, the determining a third distance between the tail and a preset first position according to the image data and the point cloud data includes:

[0036] Determine the true width value of the tail according to the point cloud data;

[0037] Determine the pixel width value of the tail according to the image data;

[0038] Determine a third distance between the tail and a preset first position according to the true width value, the pixel width value, and preset data acquisition parameters.

[0039] In some implementation manners of the first aspect, the determining the true width value of the tail according to the point cloud data includes:

[0040] Based on the point cloud data, determine the left side wall plane and the right side wall plane, as well as the normal vector of the left side wall plane or the right side wall plane;

[0041] Determine the fourth distance between the left side wall plane and the preset origin coordinates, and the fifth distance between the right side wall plane and the preset origin coordinates;

[0042] Based on the normal vector, the fourth distance, and the fifth distance, determine the true width value of the tail.

[0043] A second aspect of the embodiments of the present application provides a loading state detection device, including:

[0044] A data acquisition module, configured to acquire image data and point cloud data collected for the tail of the carriage;

[0045] A loading state recognition module, configured to recognize the loading state of the carriage based on the image data and the point cloud data;

[0046] A loading rate output module, configured to output the loading rate of the carriage when the loading state is ready for loading.

[0047] A third aspect of the embodiments of the present application provides an electronic device, including a ToF module, a camera module, a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the loading state detection method as described in the first aspect above.

[0048] A fourth aspect of the embodiments of the present application provides a computer program product, including a computer program. When the computer program is run, the loading state detection method as described in the first aspect above is executed.

[0049] A fifth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the loading state detection method as described in the first aspect above is implemented.

[0050] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0051] In the embodiment of the present application, image data and point cloud data collected for the tail of the carriage are obtained; according to the image data and the point cloud data, the loading state of the carriage is identified; when the loading state is that loading is ready, the loading rate of the carriage is output, so that the carriage can be measured in real time to identify its loading state. If the loading state is that the device is ready, the loading rate may change. Therefore, when it is determined that the loading state of the carriage is that loading is ready, the loading rate of the carriage is output, realizing automatic measurement of the loading state of the carriage and automatic triggering of the output of the carriage loading rate. Brief Description of the Drawings

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0053] Figure 1 Shows a schematic diagram of some components of a measurement terminal provided by an embodiment of the present application;

[0054] Figure 2 Is a schematic diagram of the deployment of a measurement terminal provided by an embodiment of the present application;

[0055] Figure 3 Is a schematic diagram of a loading state detection method provided by an embodiment of the present application;

[0056] Figure 4 Is another schematic diagram of a loading state detection method provided by an embodiment of the present application

[0057] Figure 5 Is a schematic flowchart of a ready state detection provided by an embodiment of the present application;

[0058] Figure 6 Is a schematic diagram of some image data when the rear door of the carriage is not opened;

[0059] Figure 7 Is a schematic diagram of some image data after the rear door of the carriage is opened;

[0060] Figure 8 Is a schematic flowchart of an end state detection provided by an embodiment of the present application;

[0061] Figure 9 Is a schematic diagram of a loading state detection device provided by an embodiment of the present application;

[0062] Figure 10 Is a schematic diagram of an electronic device provided by an embodiment of the present application. Detailed Implementation Modes

[0063] In the following description, for purposes of illustration and not limitation, specific details such as specific system architectures, technologies, etc. are set forth in order to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to impede the description of the present application with unnecessary details.

[0064] It should be understood that when used in the specification and claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0065] It should also be understood that the term "and / or" as used in the specification and claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0066] As used in the specification and claims of the present application, the term "if" can be interpreted as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted as meaning "once determined" or "in response to determining" or "once detected [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0067] In addition, in the description of the specification and claims of the present application, the terms "first", "second", "third", etc. are only used for differentiating descriptions and cannot be understood as indicating or implying relative importance.

[0068] The reference to "one embodiment" or "some embodiments" etc. described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0069] Refer to Figure 1The figure shows a schematic diagram of some components of a measurement terminal provided by an embodiment of the present application. The measurement terminal includes a ToF (Time of Flight) module for acquiring point cloud data, a camera module for acquiring image data, and a computing power module for processing the point cloud data and / or the image data, including but not limited to. The lens baseline in the ToF module is parallel to the lens baseline in the camera module, and the ToF module and the camera module are arranged adjacent to each other to avoid measurement errors caused by mechanism deformation.

[0070] Referring to Figure 2 , the figure shows a schematic diagram of the deployment of a measurement terminal provided by an embodiment of the present application. The measurement terminal is deployed at the specified position PT. If the vehicle travels to the specified area and faces the specified direction, the measurement terminal can face the tail of the carriage in the vehicle and perform corresponding measurements. As an example, the height of the measurement terminal from the carriage bottom surface is 1.8 - 2.2 meters, and the lens baseline is parallel to the longitudinal direction of the carriage. When deploying the measurement terminal, keep the ToF field of view angle capable of covering more than half of the length of the longest carriage. When measuring a vehicle with a shorter carriage, keep the length of the ToF area in the longitudinal direction of the carriage unchanged and shorten the distance of the camera area in the longitudinal direction of the carriage.

[0071] After the deployment of the measurement terminal is completed, corresponding measurements can be performed on the vehicle. The technical solution of the present application will be described below through specific embodiments.

[0072] Referring to Figure 3 , the figure shows a schematic diagram of a loading state detection method provided by an embodiment of the present application, which specifically may include the following steps:

[0073] Step 301, acquire image data and point cloud data collected for the tail of the carriage;

[0074] After the deployment of the measurement terminal is completed. If the vehicle is at the specified position and faces the specified direction, the lenses of the ToF module and the camera module of the measurement terminal face the tail of the carriage in the vehicle, so that the measurement terminal can use the camera module to collect image data for the tail of the carriage and use the ToF module to collect point cloud data for the tail of the carriage.

[0075] Step 302, identify the loading state of the carriage according to the image data and the point cloud data;

[0076] Combining the image data and the point cloud data, identify the loading state of the carriage. Among them, the loading state of the carriage can be divided into including loading ready. Loading ready means that relevant personnel or equipment can start the loading operation for the carriage, that is, place loading objects into the carriage, and the loading objects are the objects to be transported by the vehicle. The loading operation can be performed by relevant personnel or equipment.

[0077] In some implementations, before it is determined that the carriage is in a ready-to-load state, the loading state of the carriage can be determined as not ready to load, indicating that relevant personnel or equipment cannot currently perform loading operations on the carriage.

[0078] Step 303, when the loading state is ready to load, output the loading rate of the carriage.

[0079] When the loading state is ready to load, it means that relevant personnel or equipment can start or have started loading operations on the carriage, and the loading rate of the carriage changes. At this time, the carriage can be inspected in a preset manner to determine and output the current loading rate of the carriage.

[0080] The embodiments of the present application do not limit the implementation manner of determining the loading rate of the carriage.

[0081] As an example, since the point cloud data can determine the depths at different positions of the current ToF module in the carriage, and the image data can determine whether there are loading objects in the current carriage and where they are, the measurement terminal can combine the point cloud data and the image data to generate the loading rate of the carriage at the current moment.

[0082] In the embodiments of the present application, by acquiring image data and point cloud data collected for the tail of the carriage; based on the image data and the point cloud data, identifying the loading state of the carriage; when the loading state is ready to load, outputting the loading rate of the carriage, so that the carriage can be measured in real time to identify its loading state. If the loading state is ready to load, the loading rate may change. Therefore, when the loading state of the carriage is determined to be ready to load, the loading rate of the carriage is output, realizing automatic measurement of the loading state of the carriage and automatic triggering of the output of the loading rate of the carriage.

[0083] Since the measurement terminal is fixedly installed, if the measurement terminal cannot collect point cloud data or image data that matches the tail of the carriage, it can be determined that the current vehicle is not in the specified position. Therefore, the embodiments of the present application can also implement whether the parking position and parking orientation of the vehicle meet the preset parking conditions, and the parking conditions include but are not limited to the specified area and the specified direction.

[0084] Refer to Figure 4 , which shows a schematic diagram of another loading state detection method provided by the embodiments of the present application. In some embodiments of the present application, the method further includes: when the loading state is ready to load, returning to the step of identifying the loading state of the carriage based on the image data and the point cloud data; Step 304, when the loading state is ended, outputting loading end information.

[0085] The loading status can also be loading completed, which means that the relevant personnel or equipment have completed the loading operation for the carriage, and the vehicle can drive away from the current location. When it is determined that the loading status of the current carriage is loading ready, it means that the relevant personnel or equipment are performing operations, that is, the loading is not completed. In this case, after outputting the loading rate of the carriage, step 302 needs to be returned until the loading status changes to the end status.

[0086] When the loading status is loading completed, output the loading completed information to indicate that the loading operation for the carriage has been completed. At this time, the loading rate will no longer change, and the generation and output of the loading rate of the carriage can be stopped.

[0087] In some implementation manners, if it is determined that the carriage is in the loading ready state and has not reached the loading completed state, then determine the loading status of the carriage as loading in progress, which means that the relevant personnel or equipment are performing the loading operation for the carriage. At this time, the loading rate of the carriage can be output to determine the loading progress.

[0088] In some embodiments of the present application, step 302 includes:

[0089] Perform ready state detection based on the image data and the point cloud data;

[0090] When the ready state detection is completed, determine the loading status of the carriage as loading ready, and perform end state detection based on the image data and the point cloud data;

[0091] When the end state detection is completed, update the loading status of the carriage to loading completed.

[0092] During the process of identifying the loading status of the carriage, first perform ready state detection based on the image data and the point cloud data. When the ready state detection is completed, determine the loading status as loading ready; when the ready state detection is not completed, determine the loading status as not ready.

[0093] Since the execution process of the loading operation takes a certain amount of time, the situation where the loading status is loading ready may last for a period of time. At this time, it is necessary to continuously perform end state detection based on the image data and the point cloud data to determine whether the loading operation is completed.

[0094] When the end state detection is not completed, set the loading status of the carriage as loading ready; when the end state detection is completed, update the loading status of the carriage to loading completed.

[0095] As an example, when the readiness state detection is completed and the end state detection is not completed, the loading state of the carriage may be determined to be in progress of loading, so as to determine that relevant personnel or equipment are currently performing the loading operation.

[0096] Referring to Figure 5 , a flowchart of a readiness state detection provided by an embodiment of the present application is shown. In some embodiments of the present application, the readiness state detection based on the image data and the point cloud data includes: identifying the corner pixel coordinates of the carriage corner points in the image data; when the corner pixel coordinates are located in a preset corner point area of the image data, determining a first distance between the tail and a preset position;

[0097] When the first distance reaches a first threshold, determining whether there is an inner wall line in the image data and determining whether the feature data is located on the same plane; the feature data is the point cloud data between the carriage corner points;

[0098] If the feature data is not located on the same plane, determining a target plane according to the feature data and determining a second distance between the target plane and the preset position;

[0099] When there is an inner wall line in the image data and the difference between the first distance and the second distance is greater than a second threshold, it is determined that the readiness state detection is completed.

[0100] Among them, there are 4 preset corner point areas, and the positions of the preset corner point areas in the image data are preset in advance, so as to determine whether the carriage tail is close to or located in the middle of the image data. The preset position is the position of the measurement terminal, and the first distance may be the horizontal distance between the carriage tail and the preset position.

[0101] In practical applications, when conditions (1)-(3) are met, the loading state of the carriage is ready for loading. (1) The carriage is parked in the designated area and the carriage door is open; (2) The straight line of the carriage inner wall can be detected; (3) The carriage door is open, and the distance measured by the point cloud and the carriage tail door is greater than the threshold.

[0102] Referring to Figure 6 , a schematic diagram of partial image data when the carriage tail door is not opened is shown; referring to Figure 7 , a schematic diagram of partial image data after the carriage tail door is opened is shown.

[0103] As an example, the ready state detection is performed through the following steps: First, perform corner point detection on the image data to confirm whether the 4 corner points (upper left, lower left, upper right, lower right) of the rear door of the carriage can be detected. If the four corner points are not detected or the corner point positions are not within the image coordinate threshold range, continue the detection. If the conditions are met, use the prior knowledge of the license plate to quickly measure the distance to the rear of the carriage to determine the first distance. When the distance of the rear door of the carriage (the first distance) reaches the set threshold, it is determined that the carriage is parked in the designated area, and then the image data and the point cloud data are processed respectively. Image processing is performed on the image data within the corner point area. Techniques such as line detection or edge detection can be used to detect the number of lines or edges within the detection area. If no lines or edges are detected, it indicates that the rear door of the carriage is not opened, and then return to the previous step and continue the detection. Plane division is performed on the point cloud data (i.e., feature data) within the corner point area. Calculate whether the feature data is on the same plane. If the feature data is on the same plane, it is determined that the rear door of the carriage is not opened (as Figure 6 shown), return to continue the detection, otherwise determine whether the difference between the rear distance and the plane distance is greater than the threshold. If it is greater, it is determined that the carriage door has been opened (as Figure 7 shown).

[0104] In some embodiments of the present application, determining the first distance between the rear and the preset position includes: identifying the license plate pixel coordinates in the image data that match the vehicle license plate located at the rear; determining the current long side pixel distance and the current short side pixel distance based on the license plate pixel coordinates; determining the first distance between the rear and the preset position based on the current long side pixel distance, the current short side pixel distance, and the preset mapping information; the mapping information records different multiple distance values, and the preset long side pixel distance and the preset short side pixel distance corresponding to each of the distance values.

[0105] In practice, a vehicle license plate is provided at the rear of the carriage, and the vehicle license plate has certain shape characteristics, and the first distance can be determined based on this.

[0106] Since the parking area of the vehicle is preset in advance, the physical size of the vehicle license plate is known information, and the deployment position of the measurement terminal is also known. Therefore, it is possible to pre-measure the pixel coordinate range of the vehicle license plate in the image data collected by the camera module when the vehicle is at different distance values from the measurement terminal. In this way, it is possible to know the size of each side of the vehicle license plate in the image data when the vehicle is at different distance values from the measurement terminal, including the preset long side pixel distance of the longer side of the vehicle license plate and the preset short side pixel distance of the shorter side of the vehicle license plate, so as to obtain the mapping information.

[0107] Based on the above mapping information, after collecting the image data, the current long-side pixel distance and the current short-side pixel distance of the currently collected vehicle license plate can be determined by identifying the pixel coordinate range of the vehicle license plate in the image data. The distance value corresponding to the mapping information difference, the current long-side pixel distance, and the current short-side pixel distance is the first distance.

[0108] In one implementation, the current long-side pixel distance and the current short-side pixel distance can be proportionally converted and corrected according to the actual aspect ratio of the vehicle license plate (2:1).

[0109] In some embodiments of the present application, the identifying the license plate pixel coordinates in the image data that match the vehicle license plate located at the tail includes: determining the corner pixel coordinates of the carriage in the image data; determining an identification area in the image data based on the corner pixel coordinates; and identifying the license plate pixel coordinates in the identification area that match the vehicle license plate located at the tail.

[0110] According to the relative position of the vehicle license plate and the carriage corner points, after determining the corner pixel coordinates, a region larger than the vehicle license plate region can be determined in the image data as the identification area. By determining the license plate pixel coordinates in the identification area rather than in the entire range of the image data, the identification efficiency of the license plate pixel coordinates can be improved.

[0111] Among them, the pixel coordinates of carriage corner point A are (xleft, yleft), and the pixel coordinates of carriage corner point B are (xright, yright). Thus, it can be known that the pixel coordinates of the corners of the identification area are C(xa, ya), D(xb, ya), E(xa, yb), and F(xb, yb) respectively. Among them, xa = xleft + xright / 2 - Xthresh, xb = xleft + xright / 2 + Xthresh, ya = yleft, yb = yleft + Ythresh, where Xthresh and Ythresh are preset empirical thresholds.

[0112] Refer to Figure 8 , which shows a flowchart of a dressing state detection provided by an embodiment of the present application; in some embodiments of the present application, the performing the end state detection based on the image data and the point cloud data includes: determining a third distance between the tail and a preset first position based on the image data and the point cloud data; if the feature data is in the same plane, determining that the end state detection is completed; if the feature data is not in the same plane, then determining a point cloud distance based on the feature data, and determining that the end state detection is completed when the difference between the point cloud distance and the third distance is less than a third threshold.

[0113] In practical applications, if the loading status of the carriage is ready for loading and the rear door of the carriage is in the open state, the distance at the rear of the carriage can be accurately measured. Then, it is determined whether the point cloud data is on a plane. If it is on a plane, it is determined that the carriage door is closed and the loading is completed. Otherwise, it is determined whether the distance between the point cloud distance and the rear of the carriage is less than a third threshold. When the distance between the point cloud distance and the rear of the carriage is less than the third threshold, it is confirmed that the carriage is fully loaded. Otherwise, the current loading status is determined to be ready for loading (or in the process of loading), and the continuous detection is returned until the loading is completed.

[0114] In some embodiments of the present application, determining the third distance between the rear portion and the preset first position according to the image data and the point cloud data includes:

[0115] Determining the true width value of the rear portion according to the point cloud data; determining the pixel width value of the rear portion according to the image data; and determining the third distance between the rear portion and the preset first position according to the true width value, the pixel width value, and the preset data acquisition parameters.

[0116] Since the point cloud data is collected for the rear portion of the carriage, and when the carriage is in the loading status of being ready for loading, the rear door of the carriage is open. Therefore, the distance between the inner walls of the carriage can be determined according to the depth features included in the point cloud data, and then the true width value of the carriage can be determined.

[0117] Since the image data is collected for the rear portion of the carriage, the number of pixels corresponding to the width of the rear portion can be determined by identifying the image data. According to the preset unit length and the number of pixels, the pixel width value of the rear portion can be determined. For example: one pixel point can be used as the unit length, and the pixel width value of the rear portion is the number of pixels corresponding to the width.

[0118] The preset acquisition parameter is the focal length f of the camera module. It is known that the true width of the rear door of the carriage is and the pixel width on the image is , and the focal length f (obtained by camera calibration), then the distance d from the rear door of the carriage to the camera (i.e., the third distance).

[0119]

[0120] In some embodiments of the present application, determining the true width value of the rear portion according to the point cloud data includes:

[0121] Determining the left side wall plane and the right side wall plane according to the point cloud data, and the normal vector of the left side wall plane or the right side wall plane;

[0122] Determine the fourth distance between the left wall plane and the preset origin coordinates, and the fifth distance between the right wall plane and the preset origin coordinates;

[0123] Determine the true width value of the tail according to the normal vector, the fourth distance, and the fifth distance.

[0124] The origin coordinates can be determined in advance for the point cloud data, and a three-dimensional coordinate system can be constructed for the origin coordinates. The positions of each point cloud can be determined under this three-dimensional coordinate system, and then the planes where different point clouds are located can be determined.

[0125] Use the RANSAC (Random Sample Consensus) algorithm to detect the point cloud plane, determine the left wall plane and the right wall plane, and the normal vector of the left wall plane or the right wall plane. Since the left wall plane and the right wall plane are parallel planes in the three-dimensional coordinates, their normal vectors are the same.

[0126] According to the fourth distance between the left wall plane and the preset origin coordinates, the fifth distance between the right wall plane and the preset origin coordinates, and the length of the normal vector, the distance between the left wall plane and the right wall plane can be determined, and this distance can be determined as the true width value of the carriage 。

[0127]

[0128] Wherein, A, B, and C are the normal vectors of two parallel planes, D1 is the fourth distance, and D2 is the fifth distance.

[0129] As an example, in the RANSAC algorithm, the number of iterations N and the threshold D can be set. In each iteration, three points are randomly selected as the sample set, and the normal vector of the plane and the distance from the origin are calculated using these three points, so as to obtain the plane equation. For each point in the point cloud, calculate its distance to the estimated plane. If the distance is less than the threshold D, then this point is considered to belong to this plane. Use all the points belonging to this plane to re-estimate the plane model. Repeat the above until the preset number of iterations N is reached. Select the model containing the most inliers (points belonging to the plane) from all iterations as the final plane model, and finally segment the point cloud into plane points and non-plane points according to the model.

[0130] Furthermore, to accelerate the detection speed of the point cloud plane, preprocess the point cloud data. Since the inner wall plane of the carriage is relatively regular, the point cloud data can be divided into multiple grids, making the grid size adapt to the point cloud density and the plane size, and then perform independent plane detection within each grid to reduce the computational amount of global search and accelerate the detection of the point cloud plane.

[0131] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0132] Referring to Figure 9 , a schematic diagram of a loading state detection device provided by an embodiment of the present application is shown, which may specifically include a data acquisition module 901, a loading state recognition module 902, and a loading rate output module 903, where:

[0133] The data acquisition module 901 is configured to acquire image data and point cloud data collected for the tail of the carriage;

[0134] The loading state recognition module 902 is configured to recognize the loading state of the carriage based on the image data and the point cloud data;

[0135] The loading rate output module 903 is configured to output the loading rate of the carriage when the loading state is loading ready.

[0136] In some embodiments of the present application, the device further includes:

[0137] The continuous recognition module is configured to call the loading state recognition module 902 when the loading state is loading ready;

[0138] The loading end information output module is configured to output loading end information when the loading state is loading end.

[0139] In some embodiments of the present application, the loading state recognition module 902 includes:

[0140] The ready state detection sub-module is configured to perform ready state detection based on the image data and the point cloud data;

[0141] The end state detection sub-module is configured to determine that the loading state of the carriage is loading ready when the ready state detection is completed, and perform end state detection based on the image data and the point cloud data;

[0142] The loading state update sub-module is configured to update the loading state of the carriage to loading end when the end state detection is completed.

[0143] In some embodiments of the present application, the ready state detection sub-module includes:

[0144] The corner pixel coordinate recognition unit is configured to recognize the corner pixel coordinates of the carriage corners in the image data;

[0145] A first distance determination unit, configured to determine a first distance between the tail and a preset position when the corner pixel coordinates are located in a preset corner area of the image data;

[0146] A first judgment unit, configured to judge whether there is an inner wall line in the image data and whether the feature data is located in the same plane when the first distance reaches a first threshold; the feature data is the point cloud data between the carriage corners;

[0147] A second distance determination unit, configured to, if the feature data is not located in the same plane, determine a target plane according to the feature data and determine a second distance between the target plane and the preset position;

[0148] A ready state detection completion unit, configured to determine that the ready state detection is completed when there is an inner wall line in the image data and the difference between the first distance and the second distance is greater than a second threshold.

[0149] In some embodiments of the present application, the first distance determination unit includes:

[0150] A license plate pixel coordinate determination subunit, configured to identify license plate pixel coordinates in the image data that match the vehicle license plate located at the tail;

[0151] A license plate pixel distance determination subunit, configured to determine a current long side pixel distance and a current short side pixel distance according to the license plate pixel coordinates;

[0152] A first distance determination subunit, configured to determine a first distance between the tail and a preset position according to the current long side pixel distance, the current short side pixel distance, and preset mapping information;

[0153] The mapping information records different multiple distance values, and preset long side pixel distances and preset short side pixel distances corresponding to each of the distance values.

[0154] In some embodiments of the present application, the license plate pixel coordinate determination subunit includes:

[0155] A corner pixel coordinate determination subunit, configured to determine corner pixel coordinates of the carriage corner in the image data;

[0156] An identification area determination subunit, configured to determine an identification area in the image data according to the corner pixel coordinates;

[0157] A license plate pixel coordinate determination subunit, configured to identify license plate pixel coordinates in the identification area that match the vehicle license plate located at the tail.

[0158] In some embodiments of the present application, the end state detection sub-module includes:

[0159] A third distance determination unit, configured to determine a third distance between the tail and a preset first position according to the image data and the point cloud data;

[0160] An end state detection first completion unit, configured to determine that the end state detection is completed if the feature data is on the same plane;

[0161] An end state detection second completion unit, configured to, if the feature data is not on the same plane, determine a point cloud distance according to the feature data, and determine that the end state detection is completed when a difference between the point cloud distance and the third distance is less than a third threshold.

[0162] In some embodiments of the present application, the third distance determination unit includes:

[0163] A true width value determination subunit, configured to determine a true width value of the tail according to the point cloud data;

[0164] A pixel width value determination subunit, configured to determine a pixel width value of the tail according to the image data;

[0165] A third distance determination subunit, configured to determine a third distance between the tail and a preset first position according to the true width value, the pixel width value, and preset data acquisition parameters.

[0166] In some embodiments of the present application, the true width value determination subunit includes:

[0167] A plane and normal vector determination subunit, configured to determine a left side wall plane and a right side wall plane, and a normal vector of the left side wall plane or the right side wall plane according to the point cloud data;

[0168] A fourth distance determination subunit, configured to determine a fourth distance between the left side wall plane and a preset origin coordinate;

[0169] A fifth distance determination subunit, configured to determine a fifth distance between the right side wall plane and a preset origin coordinate;

[0170] A true width value calculation subunit, configured to determine a true width value of the tail according to the normal vector, the fourth distance, and the fifth distance.

[0171] A loading state detection device provided by an embodiment of the present application. By applying this device, each step in the foregoing method embodiments can be implemented.

[0172] For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For related parts, please refer to the description in the method embodiments section.

[0173] Referring to Figure 10 , a schematic diagram of an electronic device provided by an embodiment of the present application is shown. As Figure 10 shown, the electronic device 1000 in the embodiment of the present application includes: a ToF module, a camera module, a processor 1010, a memory 1020, and a computer program 1021 stored in the memory 1020 and executable on the processor 1010. When the processor 1010 executes the computer program 1021, the steps in each of the above-described loading state detection method embodiments are implemented, such as Figure 3 the steps 301 to 303 shown. Alternatively, when the processor 1010 executes the computer program 1021, the functions of each module / unit in each of the above-described device embodiments are implemented, such as Figure 9 the functions of the modules 901 to 903 shown.

[0174] Exemplarily, the computer program 1021 can be divided into one or more modules / units. The one or more modules / units are stored in the memory 1020 and executed by the processor 1010 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments can be used to describe the execution process of the computer program 1021 in the electronic device 1000. For example, the computer program 1021 can be divided into a data acquisition module, a loading state recognition module, and a loading rate output module. The specific functions of each module are as follows:

[0175] The data acquisition module is used to acquire image data and point cloud data collected for the tail of the carriage;

[0176] The loading state recognition module is used to recognize the loading state of the carriage based on the image data and the point cloud data;

[0177] The loading rate output module is used to output the loading rate of the carriage when the loading state is loading ready.

[0178] The electronic device 1000 can be the detection device in the foregoing various embodiments. The electronic device 1000 may include, but is not limited to, a processor 1010 and a memory 1020. Those skilled in the art can understand, Figure 10This is merely an example of the electronic device 1000 and does not constitute a limitation on the electronic device 1000. It may include more or fewer components than those shown in the figure, or combine certain components, or have different components. For example, the electronic device 1000 may further include input / output devices, network access devices, buses, etc.

[0179] The processor 1010 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0180] The memory 1020 may be an internal storage unit of the electronic device 1000, such as the hard disk or memory of the electronic device 1000. The memory 1020 may also be an external storage device of the electronic device 1000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 1000. Further, the memory 1020 may also include both the internal storage unit and the external storage device of the electronic device 1000. The memory 1020 is used to store the computer program 1021 and other programs and data required by the electronic device 1000. The memory 1020 may also be used to temporarily store data that has been output or is to be output.

[0181] The embodiments of the present application also disclose a computer-readable storage medium storing a computer program, which when executed by a processor implements the loading state detection method as described in the foregoing various embodiments.

[0182] The embodiments of the present application also disclose a computer program product including a computer program, which when run causes the loading state detection method as described in the foregoing various embodiments to be executed.

[0183] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limiting the same. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included within the protection scope of the present application.

Claims

1. A loading status detection method, characterized in that: include: Acquire image data and point cloud data collected from the rear of the carriage; identifying a loading state of the carriage based on the image data and the point cloud data; When the loading state is ready to load, outputting the loading rate of the carriage; The loading state of being ready to load is determined when a ready state detection based on the image data and the point cloud data is completed; The ready state detection includes: Identifying pixel coordinates of corner points of the vehicle compartment in the image data; In a case where the corner point pixel coordinates are located in a preset corner point area of ​​the image data, determining a first distance between the tail and a preset position; When the first distance reaches a first threshold, determining whether there is an inner wall line in the image data, and determining whether the feature data is located in the same plane; the feature data is point cloud data between the corner points of the carriage; If the characteristic data are not located in the same plane, determining a target plane according to the characteristic data, and determining a second distance between the target plane and the preset position; When an inner wall line exists in the image data and a difference between the first distance and the second distance is greater than a second threshold, it is determined that the ready state detection is completed.

2. The method according to claim 1, characterized in that The method further comprises: When the loading status is ready to load, returning to the step of identifying the loading status of the carriage based on the image data and the point cloud data; When the loading status is loading completion, loading completion information is output.

3. The method according to claim 1, characterized in that The identifying the loading state of the carriage according to the image data and the point cloud data includes: Performing a ready state detection based on the image data and the point cloud data; In the case where the ready state detection is completed, determining that the loading state of the carriage is ready to load, and performing an end state detection based on the image data and the point cloud data; When the end state detection is completed, the loading state of the carriage is updated to loading completed.

4. The method according to claim 1, characterized in that: The determining a first distance between the tail and a preset position includes: Determine the pixel coordinates of the corner point of the carriage located in the image data; Determining a recognition area in the image data according to the pixel coordinates of the corner points; Identifying, in the identification area, the pixel coordinates of the license plate that matches the vehicle license plate located at the rear; Determine the current long side pixel distance and the current short side pixel distance according to the license plate pixel coordinates; Determine a first distance between the tail and a preset position according to the current long side pixel distance, the current short side pixel distance and preset mapping information; The mapping information records a plurality of different distance values, and a preset long-side pixel distance and a preset short-side pixel distance corresponding to each of the distance values.

5. The method according to any one of claims 3-4, characterized in that: The performing end state detection based on the image data and the point cloud data includes: Determining a true width value of the tail according to the point cloud data; Determine the pixel width value of the tail according to the image data; Determine a third distance between the tail and a preset first position according to the real width value, the pixel width value and preset data acquisition parameters; If the characteristic data are located in the same plane, determining that the end state detection is completed; If the feature data are not located in the same plane, the point cloud distance is determined according to the feature data, and when the difference between the point cloud distance and the third distance is less than a third threshold, it is determined that the end state detection is completed.

6. The method according to claim 5, characterized in that Determining the true width value of the tail according to the point cloud data includes: Determine a left wall plane and a right wall plane, and a normal vector of the left wall plane or the right wall plane according to the point cloud data; Determine a fourth distance between the left wall plane and the preset origin coordinates, and a fifth distance between the right wall plane and the preset origin coordinates; The actual width value of the tail is determined according to the normal vector, the fourth distance, and the fifth distance.

7. A loading status detection device, characterized in that: include: A data acquisition module, used to acquire image data and point cloud data collected from the rear of the carriage; A loading status recognition module, used to recognize the loading status of the carriage according to the image data and the point cloud data; A loading rate output module, used for outputting the loading rate of the carriage when the loading state is ready to load; The loading state of being ready to load is determined when a ready state detection based on the image data and the point cloud data is completed; The loading status identification module comprises: A corner pixel coordinate identification unit, used to identify the corner pixel coordinates of the car corner in the image data; a first distance determining unit, configured to determine a first distance between the tail and a preset position when the pixel coordinates of the corner point are located in a preset corner point area of ​​the image data; A first judgment unit is used to judge whether there is an inner wall line in the image data and whether the feature data are located in the same plane when the first distance reaches a first threshold; the feature data is point cloud data between the corner points of the carriage; a second distance determining unit, configured to determine a target plane according to the feature data, and determine a second distance between the target plane and the preset position if the feature data are not located in the same plane; The ready state detection completion unit is used to determine that the ready state detection is completed when there is an inner wall line in the image data and the difference between the first distance and the second distance is greater than a second threshold.

8. An electronic device, characterized in that: The electronic device comprises a ToF module, a camera module, a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the method according to any one of claims 1 to 6 to be performed.

Images