COMPUTER-IMPLEMENTED METHOD FOR SELECTING SUITABLE IMAGES FOR VOLUME DETERMINATION

Abstract

A computer-implemented method 100 for selecting suitable images 136 for determining the volume 127 of a storage room 156 comprises a step of receiving 102 an image 136. Quality parameters are calculated for the image 136 and compared 114 with limit values. If limit values ​​are violated, the image 136 is rejected as unsuitable for the volume determination 127. Otherwise, a check is performed 120 to determine whether the storage room 156 is captured in the image 136. If this is the case, a check is performed 122 to determine whether a previous image was generated chronologically before the image 136, wherein the storage room 156 is captured in the previous image and no further previous image was generated with a shorter time interval than the image 136. The image 136 is then compared 128 with the previous image with regard to a loading state, if a previous image was generated. If no difference is found, the image 136 is accepted as suitable for volume determination 127 124.Otherwise, image 136 is rejected as unsuitable for volume determination 127.

Description

Technical area

[0001] The present invention relates to a method for assessing image quality. In particular, the present invention relates to a computer-implemented method for selecting suitable images for volume determination. Furthermore, the present invention relates to a device, a computer program product, a computer-readable storage medium, a data carrier signal, and a vehicle. Technical background and task

[0002] During loading and unloading processes of storage spaces such as trailers (for trucks, motor vehicles, trains or the like), containers, aircraft cargo holds or the like, one goal is often to fill the available space as efficiently as possible.

[0003] Systems exist that analyze and digitally document the loading status of a trailer. Sensors such as cameras and time-of-flight systems (lidar, radar, and the like) are used to determine the free volume in the trailer. These sensors are usually permanently installed. The installation can be mounted inside the trailer or on a loading ramp, for example. The sensor data is processed in a computing unit, allowing a statement to be made about the remaining loading volume of the trailer. This computing unit is either located directly on-site or provided as a backend or cloud service.

[0004] High data transmission rates are required to transmit the large volume of data from the sensors to the processing unit. To achieve this, the system, which is at least partially permanently installed, must have stable network access (e.g., internet, DSL, LAN). A disadvantage of these systems is the significant initial installation effort, as the alignment and configuration of the sensors must be handled by a specialist.

[0005] Another option for determining the remaining cargo space is systems that use mobile devices (such as smartphones). Here, individual camera images or a video are sent to a backend or server, where they are processed due to the high computational requirements for volume calculation. This approach also requires large data volumes to be transmitted to the backend or server, for example, via mobile communications.

[0006] However, a single image is not sufficient to make accurate statements about the loading status of a trailer, as it cannot determine the correct remaining space or volume in the trailer. While a volume can be calculated from a (two-dimensional) image, this calculation is based on assumptions about certain parts of the image. If some cargo is obscured by other cargo, these cannot be captured by the camera, and the assumptions about the obscured parts of the image may be incorrect, resulting in an incorrect volume calculation.

[0007] In "Simple and Effective Load Volume Estimation in Moving Trucks using LiDARs" (LL Amorim, F. Mutz, AF De Souza, C. Badue, and T. Oliveira-Santos in 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Rio de Janeiro, Brazil, 2019, pp. 210-217, doi: 10.1109 / SIBGRAPI.2019.00036), an automated system for volume estimation of bulk materials in moving trucks using two multi-layer LiDAR sensors is presented. The sensors are mounted so that vehicles can pass through without stopping. The system creates a 3D mesh of the load for volume determination and uses a simple heuristic for vehicle tracking.

[0008] DE 102018 006 765 A1 provides methods and systems for managing cargo vehicles. In one aspect, a method for determining the volume of the cargo space of a vehicle in real time is disclosed. The method comprises generating a first spatial model of the cargo space based on images from a plurality of cameras, including at least one depth and color camera, positioned in and around the cargo space, which generate an updated spatial model of the cargo space using the images from the plurality of cameras. After detecting items being loaded into or unloaded from the cargo space, estimating the volume of the loaded items in the updated spatial model and determining the remaining volume in the cargo space based on the estimated volume of the loaded items, and estimating the total volume of the cargo based on the first spatial model.

[0009] It is therefore the object of the present invention to provide computer-implemented methods for selecting suitable images for determining the volume of a storage space, which eliminates at least one of the aforementioned disadvantages. Furthermore, it is the object of the invention to provide a corresponding device, a computer program product, a computer-readable storage medium, a data carrier signal, and a vehicle. Disclosure of the invention

[0010] The object is achieved according to the invention by the features of the main claims. Advantageous embodiments can be found in the subclaims.

[0011] According to a first aspect of the invention, a computer-implemented method for selecting suitable images for determining the volume of a storage space comprises a step of receiving an image. A storage space can be, for example, the trailer of a truck, a trunk or rear seat of a motor vehicle, a cargo hold of an aircraft or a railway carriage, or the like. The image can be a single image or part of an image sequence or a video.

[0012] In a further step of the process, the image's quality parameters are calculated. Examples of quality parameters, which specifically indicate the quality of an image with regard to its suitability for successful further image processing, include its resolution, average / minimum / maximum sharpness, brightness, and homogeneity.

[0013] In a further step of the process, initial thresholds are provided for each quality parameter. For example, a minimum resolution or two minimum resolutions (one for each dimension of the image) are provided.

[0014] In a further step of the process, the image is rejected as unsuitable for volume determination if at least one quality parameter exceeds or falls below a first threshold assigned to it. The purpose of the first thresholds is to ensure that the image's quality is sufficient for volume determination. If this is not the case, based on the thresholds, the image is rejected as unsuitable.

[0015] In a further step of the process, it is checked whether the storage room is captured in the image. This step is only performed if no quality parameter exceeds or falls below an assigned first threshold; in other words, if the image quality is sufficient.

[0016] In an advantageous embodiment, when checking whether the storage room is captured in the image, it is also checked whether freight is captured in the image. Freight is designed for the storage of physical goods, such as pallets, boxes, containers, cartons, and the like. Some procedural steps are simplified if it is determined early in the process whether any freight is actually parked in the storage room.

[0017] In a further step of the process, it is checked whether a previous image was generated prior to the current image. This step is only performed if the storage room is included in the image. The storage room is included in the previous image, and no other previous image was generated with a shorter time lag.

[0018] A check is therefore carried out to determine whether further images have been generated in the past in which the storage room was captured. These further images can, for example, be viewed as an image series in which the further images are sorted chronologically. The first additional image in the image series is the one captured first. If the image is added to a non-empty image series, a predecessor image exists for the image, namely always the previous element in the image series (due to the chronological sorting). Typically, all further images in the image series have also undergone the process and have been accepted as suitable for volume determination.

[0019] In a further step of the process, the image is compared with the previous image with regard to the loading status. This step is only performed if a previous image has been generated. If no previous image exists, no meaningful comparison can be performed.

[0020] The loading status records the location of cargo in the storage area. It can also record the type of cargo, including dimensions, type, weight, and other information. This information can be read, for example, from a label attached to the cargo. Alternatively, the label may contain an identifier, allowing additional information about the cargo to be read from a database. Object detection and classification methods can be used to record the loading status.

[0021] Creating a digital model of the key parts of the storage room and its surroundings can also help determine the loading status. The images in the series can also be taken from different perspectives, as a camera used to capture the images should not be permanently installed but rather attached to a support that is involved in the loading or unloading process and has a good view of the storage room. For example, such a camera could be attached to a forklift, a person, a lifting platform, or a pallet truck. The different perspectives in the images can then be incorporated into the digital model. The position of the support must also be taken into account, as this determines the camera perspective.

[0022] The camera can be part of a computing unit located locally on the carrier. The computing unit can have a processor and memory to perform the necessary steps. A mobile device such as a smartphone can also have these features. However, the computing unit typically does not have sufficient computing power to perform volume determination.

[0023] In an advantageous embodiment, the image is accepted as suitable for volume determination if no previous image has been generated. In this case, a comparison is neither possible nor necessary. Since the image has already been previously checked for essential quality parameters, it can then be forwarded directly to volume determination. Volume determination typically does not take place on the same device as the selection process. The selection process ultimately aims to reduce the data volume to be transmitted, so that no unnecessary data is transmitted and evaluated.

[0024] In a further step of the process, the image is determined to be suitable for volume determination if a difference is detected during the comparison. If a difference in the loading condition is detected, cargo has been added to or removed from the storage space. This changes the available volume, and a new volume determination should be performed.

[0025] In a further step of the process, the image is discarded as unsuitable for volume determination if no difference is found during the comparison.

[0026] Optionally, you can check whether there are people in the image. This could potentially violate a regulation such as the GDPR, which can also be detected. The object detection performed previously makes this step easy to perform.

[0027] To accurately determine the free, available volume in the storage room, a large number of chronological images are analyzed, and the load differences are recorded. A camera records the images or a video and selects only those images for which a volume determination is likely to reveal a changed value. Superfluous images between which no load differences have occurred would only lead to redundant calculations.

[0028] The volume determination, i.e., the calculation of the available cargo space from the images, is realized using multiple neural networks and additional image processing steps and requires high computing power. The volume determination is therefore performed in a backend or server. For this purpose, the images, and thus large amounts of data, must be transmitted to the backend or server. The disclosed method advantageously reduces the amount of data by transmitting only relevant images.

[0029] In an advantageous embodiment, the image is reduced in size if the size exceeds a second threshold. The image size can be, for example, its dimensions in pixels or the image file size in bytes. Optionally, a copy of the image can also be stored on the processing unit. The original then remains in the processing unit's memory while the copy, possibly reduced in size, undergoes the process.

[0030] In an advantageous embodiment, a volume determination of the storage room is carried out using an accepted image.

[0031] Additionally, vehicle movement data can also be considered. For example, with a forklift, the period of time during which a reversing movement occurs during loading can be determined. Normally, no cargo is being transported during this period, thus avoiding obscurations in the image. This increases the chances of capturing a relevant image.

[0032] According to a second aspect of the invention, a computer program product comprises instructions which, when executed by a computer, cause the computer to execute a method as described above. The computer program product can be written in a programming language, for example, Python or C++.

[0033] According to a third aspect of the invention, a computer-readable storage medium comprises instructions that, when the program is executed by a computer, cause the computer to execute a method as described above. The computer-readable storage medium can be implemented, for example, as an SSD (solid-state disk) or as a flash memory. The computer-readable storage medium can also store other data, for example, sensor data and / or data that is (temporarily) stored during the execution of the method.

[0034] According to a fourth aspect of the invention, a data carrier signal transmits the computer program product as described above. The data carrier signal can be transmitted via a cable, for example, using a CAN bus cable, or wirelessly via Wi-Fi, Bluetooth, mobile radio, or the like. It can also be transmitted via a cable, for example, using CAN (Controller Area Network), FlexRay, Ethernet, LIN (Local Interconnect Network), or MOST (Media Oriented Systems Transport).

[0035] According to a fifth aspect of the invention, a device for selecting suitable images for determining the volume of a storage room comprises a camera that captures images in the form of image data. Furthermore, the device comprises an evaluation unit that is communicatively connected to the camera and is configured to execute a method as described above.

[0036] If there is no or insufficient connection to the backend, the images can be kept in memory until an adequate connection is established.

[0037] According to a sixth aspect of the invention, a vehicle comprises a device as described above. Summary of the characters

[0038] The invention is explained in more detail below using exemplary embodiments and figures. The figures show: Fig. 1: A flowchart of an embodiment of a computer-implemented method for selecting suitable images for determining the volume of a storage room; Fig. 2: An embodiment of a device which implements the method of Fig. 1 can perform; and Fig. 3: A view showing an example of application of the invention. Detailed description of the characters

[0039] Fig. 1 shows a flowchart of an embodiment of a computer-implemented method 100 for selecting suitable images 136 for a volume determination 127 of a storage room 156.

[0040] In a receiving step 102, an image 136 is received. The image 136 was captured by a camera 134, which in turn is attached to a support. The support, for example, a forklift, a loading ramp, or a vehicle 146, is typically involved in a loading or unloading operation of the storage space 156. Therefore, the camera 134 is potentially capable of generating images 136 of the storage space 156 depending on the orientation of its field of view 148. This typically depends on the orientation of the support in space.

[0041] In the example of Fig. 1, a copy of the image 136 is created and stored in a first storage step 104. This serves as a data backup. The method can be performed either with the original image 136 or the copy.

[0042] However, either image 136 or the copy remains stored unchanged to prevent possible data loss. The memory 140 used for this purpose should be located on the carrier to keep signal paths as short and interference-free as possible.

[0043] Now, in a size reduction step 108, the size of the image 136 is changed. For example, if the camera 134 has a resolution of 12 megapixels with page resolutions of 4048px x 3040px, the size can be reduced to 3.1 megapixels (2048px x 1536px), because this resolution may be sufficient to perform a volume determination 127. However, it can also be provided to use the reduced-size image 136 only to determine suitability with regard to a volume determination 127, and then use the unreduced image 136 for the volume determination 127.

[0044] Subsequently, in a calculation step 110, quality parameters of the image 136 are calculated. Quality parameters can be, for example, contrast, brightness, or sharpness.

[0045] In a provision step 112, threshold values ​​are provided for the quality parameters. For example, this can be used to specify the minimum contrast required to perform a volume determination 127. In the subsequent first test step 114, the quality parameters are checked for compliance with the threshold values. If the quality parameters violate threshold values, the image 136 is rejected in a rejection step 116 as unsuitable for volume determination 127 (t-branch in the first test step 114).

[0046] If there are no violations of the threshold values, a classification step 118 is executed (f-branch in the first test step 114). In this step, object detection and classification are performed on the image 136. In other words, objects in the image 136 are detected and classified. For this purpose, the person skilled in the art uses known methods, for example, Region Proposals (R-CNN, Fast R-CNN, Faster R-CNN), Single Shot MultiBox Detector (SSD), or You Only Look Once (YOLO).

[0047] After object recognition, a second test step 120 checks whether the image 136 has captured the storage room 156 and freight 158. If no freight 158 ​​is captured, no volume determination 127 needs to be performed because the storage room 156 is empty. If the storage room 156 is not captured, no volume calculation 127 can be performed using the image 136. In these cases, the image 136 is discarded as unsuitable for the volume determination 127 (f-branch at the second test step 120).

[0048] However, if cargo hold 156 and cargo means 158 are captured in image 136, a third check step 122 determines whether another image of cargo hold 156 is already available. For this purpose, a database containing image series (chronologically sorted sequences of images 136) for cargo holds 156 can be accessed, for example.

[0049] If no image 136 of the cargo hold 156 yet exists, i.e., if it is the first image 136 taken of the cargo hold 156, it is accepted as suitable for volume determination 127 in an acceptance step 124 (f-branch in the third test step 122). Subsequently, it can be stored in a second storage step 126, for example, in the aforementioned database for image series. A volume determination 127 can then be performed using the image 136.

[0050] However, if an image 136 of the cargo compartment 156 already exists, a comparison step 128 is executed (t-branch at the third test step 122). In this step, the image 136 is compared with the previous image with regard to the loading status. Known image processing methods such as histogram comparison, template matching, or feature matching can be used here. The creation of digital twins can also be used here. The comparison checks whether cargo 158 has been added to, removed from, or moved within the storage compartment 156. For this purpose, for example, positions of the cargo 158 in the storage compartment 156 are detected and compared.

[0051] In a fourth test step 130, a check is now performed to determine whether any differences were detected. If this is not the case, there has been no change in the loading state. In this case, a new volume determination 127 is not necessary, which is why the image 136 is rejected as unsuitable in the rejection step 116 (f-branch in the fourth test step 130). However, if differences are identified, the image 136 is accepted as suitable for the volume determination 127 in the acceptance step 124. It can be saved (see second storage step 126) and then used for the volume determination 127.

[0052] Fig. 2 shows an embodiment of a device 132 which implements the method 100 of Fig. 1. The device 132 has a camera 134 which can capture images 136. In addition, the device 132 has an evaluation unit 144 which compares the images 136 with the Fig. 1 described method 100.

[0053] For this purpose, the evaluation unit 144 has a processor 138 and a memory 140. A computer program product 142 is stored in the memory 140. Further data can be stored in the memory 140, for example, images 136 captured in the past or a database with image series for a plurality of storage rooms 156.

[0054] Fig. Figure 3 shows a view illustrating an application example of the invention. To illustrate the invention, the figure shows a vehicle 146 in the form of a forklift. A camera 134 is mounted on the vehicle, which can capture images 136 within a field of view 148.

[0055] In the example of Fig. 3 shows a loading ramp 152 to which a truck 154 is docked. This truck has a loading space 156 containing a cargo vehicle 158. During a loading or unloading process of the storage space 156 by the forklift 146, the camera 134 continuously records a video of the process.

[0056] The individual images 136 of the video are processed by an evaluation unit 144, which is arranged in the forklift 146 and receives the images 136 of the camera 134 by means of a data cable, using the method 100 from Fig. 1. The evaluation unit 144 corresponds to the Fig. 2.

[0057] After the images 136 have been evaluated by the method 100 using the evaluation unit 144, an image 136 that has been accepted as suitable for volume determination 127 is transmitted to a server 160 via communication means 150. The communication means 150 allow wireless transmission 162 of data, for example, via WLAN, mobile radio, or the like.

[0058] The server 160, in turn, has a second evaluation unit 164, which performs a volume determination 127 based on the image 136 transmitted by the first evaluation unit 144. The result can be fed into a database, which can be used to plan loading or unloading of the storage space 156. For example, the driver of the forklift 146 or, if the forklift 146 is automated, the forklift 146 itself can be instructed as to which next freight 158 ​​should be loaded into or removed from the cargo space 156. The instruction depends on how much volume the cargo space 156 still has and which freight 158 ​​still fits into this volume. List of reference symbols 100 procedures 102 Reception step 104 First storage step 108 size reduction step 110 Calculation step 112 Deployment step 114 First test step (Q-Params) 116 Discard step 118 Classification step 120 Second test step (storage room + cargo in picture?) 122 Third test step (predecessor?) 124 Acceptance step 126 Second storage step 127 Determination step (volume) 128 Comparison step 130 Fourth test step (difference?) 132 Device 134 Camera 136 Image 138 processor 140 storage 142 Computer program product 144 Evaluation unit 146 vehicles 148 field of view 150 means of communication 152 Loading ramp 154 trucks 156 storage room 158 freight vehicles 160 servers 162 Wireless Communication 164 Second evaluation unit

Claims

[1] Computer-implemented method (100) for selecting suitable images (136) for determining the volume (127) of a storage room (156), comprising the steps: a) receiving (102) an image (136), b) Calculation (110) of quality parameters of the image (136), c) providing (112) initial limit values ​​for each quality parameter, d) rejecting (116) the image (136) as unsuitable for the volume determination (127) if at least one quality parameter exceeds or falls below at least one first limit value assigned to it, e) checking (120) whether the storage room (156) is captured in the image (136) if no quality parameter exceeds or falls below a first limit value assigned to it, f) checking (122), if the storage room (156) is captured in the image (136), whether a previous image was generated before the image (136), wherein the storage room (156) is captured in the previous image and no further previous image was generated with a shorter time interval to the image (136), g) comparing (128) the image (136) with the previous image with regard to a loading state, if a previous image was generated, h) accepting (124) the image (136) as suitable for the volume determination (127) if a difference is found during the comparison (128), or i) Rejecting (116) the image (136) as unsuitable for volume determination (127) if no difference is found during the comparison (128). [2] Method according to claim 1, characterized in that during the check (120) as to whether the storage room (156) is captured in the image (136), it is additionally checked whether freight means (158) are captured in the image (136). [3] Method according to claim 1 or 2, characterized in that the image (136) is reduced in size if the size exceeds a second limit value. [4] Method according to one of the preceding claims, characterized in that the image (136) is accepted as suitable for the volume determination (127) if no previous image was generated. [5] Method according to one of the preceding claims, characterized in that a volume determination (127) of the storage room (156) is carried out using an accepted image (136). [6] A computer program product (142) comprising instructions which, when executed by a computer, cause the computer to carry out a method (100) according to any one of the preceding claims. [7] A computer-readable storage medium (140) comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method (100) according to any one of claims 1 to 5. [8] Device (132) for selecting suitable images (136) for determining the volume (127) of a storage room (156), comprising: a) A camera (134) which captures images (136), b) An evaluation unit (144) which is communicatively connected to the camera (134) and is designed to carry out a method (100) according to one of claims 1 to 5.

Images