Loading space recognition device, system, method, and program

Abstract

To provide a loading space recognition device capable of making a contribution to a determination on the possibility of occurrence of a load collapse due to movement of a load.SOLUTION: A loading space recognition device includes a load overall image inference unit that is configured to infer an overall image of a load loaded in a loading space on the basis of 3D data obtained by capturing an image of the load loading space from one direction and to output the inferred overall image as the inference result data, a voxel generation unit that is configured to convert the inference result data into voxels and output the voxels as voxel data, and a determination unit that is configured to, by comparing the voxel data at any reference time with the voxel data after the elapse of a prescribed or arbitrary time from the reference time, estimate a displacement volume or movement amount of the load, and, by comparing the estimated displacement volume or movement amount with a threshold, determine the presence / absence of the possibility of a load collapse.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] [Statement about related applications] The present invention is based on a claim of priority from Japanese Patent Application No. 2021-030741 (filed February 26, 2021), the entire contents of which are incorporated herein by reference. The present invention relates to a loading space recognition device, a system, a method, and a program. [Background technology]

[0002] In the logistics industry, cargo loaded in truck containers is often damaged and transport quality is reduced when the cargo falls during transport, etc. Therefore, technologies have been proposed to generate information for arranging cargo to prevent cargo from collapsing, and to quickly detect any cargo collapse that has actually occurred (see, for example, Patent Documents 1 to 6).

[0003] Patent Documents 1 and 2 disclose a shape information generation device as a technology for preventing cargo from falling, which device includes a first information acquisition unit that acquires three-dimensional information of a first region on the surface of a plurality of stacked items, obtained by imaging or scanning the plurality of items from a first location; a second information acquisition unit that acquires three-dimensional information of a second region on the surface of the plurality of items, obtained by imaging or scanning the plurality of items from a second location; and a synthesis unit that generates information indicating the three-dimensional shape of at least a portion of the surface of the plurality of items based on the three-dimensional information of the first region acquired by the first information acquisition unit and the three-dimensional information of the second region acquired by the second information acquisition unit, wherein the first location and the second location are located at different positions, and the synthesis unit complements one of the three-dimensional information of the first region and the three-dimensional information of the second region with the other to generate information indicating the three-dimensional shape of at least a portion of the surface of the plurality of items.

[0004] Patent Document 3 discloses a loading method for stacking a plurality of objects in a predetermined packing shape and with a predetermined weight as a technology for preventing cargo from falling over, the method comprising the steps of measuring the weight and packing shape of each of the objects, calculating the density of each of the objects from the weight and packing shape of each of the objects, and accumulating density information of each of the objects to calculate the loading position of each of the objects.

[0005] Patent Document 4 discloses a vehicle trunk monitoring device that is equipped with surveillance cameras mounted on the top surface of the trunk at a midpoint in the vehicle width direction and at two locations that divide the entire length of the trunk into thirds, as technology that allows the driver to check the status of luggage in the trunk at any time using a monitor (whether the luggage has collapsed or not), an information processing device that inputs image data of the interior of the trunk from the surveillance cameras, a monitor that inputs and displays image data of the interior of the trunk from the information processing device, and a communication device that inputs image data of the interior of the trunk from the information processing device and transmits it to a base station, and is configured so that the availability of loading space can be determined from images from the surveillance cameras.

[0006] Patent Document 5 discloses a cargo shift monitoring system for transport vehicles, as a technology that allows drivers and management centers to easily know of cargo shifts in transport vehicles and to respond quickly to cargo shifts. The system comprises an on-board terminal that monitors cargo shifts using a sensor attached to the loading platform of the transport vehicle and, when cargo shifts are detected, issues a warning to the driver indicating that cargo shifts have occurred, and a management center that receives a cargo shift monitoring signal indicating that cargo shifts have occurred, which is sent from the on-board terminal when the on-board terminal issues the warning to the driver, and collects and manages information about cargo shifts in transport vehicles.

[0007] Patent Document 6 discloses a method for detecting cargo collapse that can detect cargo collapse that occurs during transfer and after the transfer work is completed. The method involves taking images of the group of items stacked on a pallet from above before and after each transfer of an individual item, obtaining a first image before the items are transferred and a second image after the items are transferred, comparing the first and second images, and determining whether or not cargo collapse has occurred based on the degree of change in the area of ​​the items other than the area where the transferred items were located. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent No. 6511681 [Patent Document 2] Patent No. 6577687 [Patent Document 3] Japanese Patent Application Laid-Open No. 2002-29631 [Patent Document 4] Japanese Patent Application Laid-Open No. 2018-199489 [Patent Document 5] Japanese Patent Application Laid-Open No. 2005-018472 [Patent Document 6] Japanese Patent Application Laid-Open No. 2007-179301 Summary of the Invention [Problem to be solved by the invention]

[0009] The following analysis is provided by the present inventors.

[0010] However, the technologies for suppressing and preventing cargo shifting described in Patent Documents 1 to 3 do not completely prevent cargo shifting simply by loading the cargo so that it does not occur, and since cargo can move in all directions due to vehicle acceleration, braking, vibration, etc., which can lead to cargo shifting, it is not possible to determine the possibility of cargo shifting due to cargo movement.

[0011] Furthermore, in the technology for checking luggage described in Patent Document 4, the driver checks the status of the luggage in the luggage compartment on a monitor to become aware of any luggage collapse, making it difficult from a safety standpoint for the driver to determine the possibility of luggage collapse due to luggage movement while driving the vehicle.

[0012] Furthermore, the technology for detecting cargo collapse described in Patent Document 5 determines whether cargo collapse has occurred based on whether or not a sensor receives signals such as infrared or ultrasonic waves, and therefore cannot detect cargo movement within a range where there is no change in the signal received by the sensor, and therefore may not be able to determine the possibility of cargo collapse due to cargo movement.

[0013] Furthermore, in the technology of Patent Document 6 that can detect cargo collapse that occurs during transfer and after the transfer operation is completed, edge images are generated from the first image taken before the transfer and the second image taken after the transfer, and the two images are compared to determine whether cargo collapse has occurred based on the degree of change in items other than the transferred items.However, comparison between edge images cannot detect movement of cargo that does not move at the edge position (movement of cargo towards or away from the image capture position), and there is a possibility that it will not be possible to determine the possibility of cargo collapse due to cargo movement.

[0014] A main object of the present invention is to provide a loading space recognition device, system, method, and program that can contribute to determining the possibility of cargo collapse due to movement of luggage. [Means for solving the problem]

[0015] The loading space recognition device according to the first viewpoint includes a luggage overall image estimation unit configured to estimate an overall image of luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the luggage loading space from one direction and output the estimation result data as estimation result data; a voxelization unit configured to voxelize the estimation result data and output the voxel data as voxel data; and a determination unit configured to estimate the amount of change in volume or movement of the luggage by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and to determine whether or not there is a possibility of luggage collapse by comparing the estimated amount of change in volume or movement with a threshold value.

[0016] The loading space recognition system relating to the second viewpoint includes a sensor that senses the surface of the luggage in the loading space and outputs the captured three-dimensional data, and a loading space recognition device relating to the first viewpoint.

[0017] A loading space recognition method according to a third aspect is a loading space recognition method that recognizes a loading space of luggage using hardware resources, and includes the steps of: estimating an overall image of luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the loading space from one direction and outputting the estimated result data as estimation result data; voxelizing the estimation result data and outputting the voxel data as voxel data; and estimating a volume change or movement amount of the luggage by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and comparing the estimated volume change or movement amount with a threshold value to determine whether or not there is a possibility of luggage collapse.

[0018] The program relating to the fourth viewpoint is a program that causes hardware resources to execute a process for recognizing a cargo loading space, and causes the hardware resources to execute the following processes: a process for estimating an overall image of the cargo loaded in the loading space based on three-dimensional data obtained by capturing an image of the loading space from one direction and outputting the estimated result data as estimation result data; a process for voxelizing the estimated result data and outputting the voxel data as voxel data; and a process for estimating the amount of change in volume or movement of the cargo by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and for determining whether or not there is a possibility of cargo collapse by comparing the estimated amount of change in volume or movement with a threshold value.

[0019] The program can be recorded on a computer-readable storage medium. The storage medium can be a non-transient medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present disclosure can also be embodied as a computer program product. The program is input to a computer device from an input device or an external device via a communication interface, stored in a storage device, and drives a processor according to predetermined steps or processes. The processing results, including intermediate states as needed, can be displayed at each stage on a display device, or the computer device can communicate with the outside world via the communication interface. For example, a computer device for this purpose typically includes a processor, a storage device, an input device, a communication interface, and, if necessary, a display device, all of which can be connected to each other via a bus. [Effects of the Invention]

[0020] The first to fourth aspects can contribute to determining the possibility of cargo collapse due to movement of cargo. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is an image diagram schematically illustrating an example of the configuration and usage of a loading space recognition system according to a first embodiment. [Figure 2] 1 is a block diagram schematically illustrating a configuration of a loading space recognition device in a loading space recognition system according to a first embodiment. [Figure 3] 4 is a flowchart schematically illustrating the operation of the loading space recognition device in the loading space recognition system according to the first embodiment. [Figure 4] 10A and 10B are conceptual diagrams schematically illustrating examples of cases in which there is a volume fluctuation when a difference between a reference voxel and a comparison voxel is extracted. [Figure 5] 10A and 10B are conceptual diagrams showing examples in which there is no volume change and there is a movement amount when a difference between a reference voxel and a comparison voxel is extracted. [Figure 6] FIG. 6 is an image diagram showing a transition from the group estimation to the longest movement amount estimation in Example 2-1 of FIG. 5. [Figure 7] FIG. 6 is an image diagram showing a transition from the group estimation to the longest movement amount estimation in Example 2-2 of FIG. 5. [Figure 8] FIG. 6 is an image diagram showing a transition from the group estimation to the longest movement amount estimation in Example 2-3 of FIG. 5. [Figure 9] FIG. 6 is an image diagram showing a transition from the group estimation to the longest movement amount estimation in Example 2-4 of FIG. 5. [Figure 10] 10A and 10B are conceptual diagrams schematically illustrating modified examples of the configuration and usage of the loading space recognition system according to the first embodiment. [Figure 11] FIG. 10 is a block diagram schematically illustrating the configuration of a loading space recognition device according to a second embodiment. [Figure 12] FIG. 2 is a block diagram illustrating a configuration of hardware resources. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, embodiments will be described with reference to the drawings. Note that, where reference numerals are used in this application, they are intended solely to facilitate understanding and are not intended to limit the present invention to the illustrated embodiments. Furthermore, the following embodiments are merely illustrative and do not limit the present invention. Furthermore, connecting lines between blocks in the drawings, etc., referred to in the following description, include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of main signals (data) and do not exclude bidirectionality. Furthermore, although not explicitly shown, input and output ports exist at the input and output ends of each connecting line in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, etc., shown in this disclosure. The same applies to input and output interfaces. A program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and, if necessary, a display device. The computer device is configured to communicate with internal or external devices (including computers) via the communication interface, whether wired or wireless.

[0023] [Embodiment 1] A loading space recognition system according to a first embodiment will be described with reference to the drawings. Fig. 1 is an image diagram that schematically shows an example of the configuration and usage of the loading space recognition system according to the first embodiment. Fig. 2 is a block diagram that schematically shows the configuration of a loading space recognition device in the loading space recognition system according to the first embodiment. Here, a container load on a truck will be described as an example.

[0024] The loading space recognition system 1 is a system that uses a sensor 10 to recognize changes (volume changes, distance changes) of objects (baggage 4 in FIG. 1) in a loading space (loading space 5 inside a container 3 of a truck 2 in FIG. 1) where the objects are loaded (see FIG. 1). In the loading space recognition system 1, the sensor 10 and a loading space recognition device 200 are connected to each other so that they can communicate (wired communication or wireless communication). The loading space recognition system 1 can be mounted on the truck 2. In the loading space recognition system 1, the loading space recognition device 200 recognizes changes in objects in the loading space 5 based on image data (100 in FIG. 2) captured by the sensor 10. Here, the image data 100 is three-dimensional data captured by the sensor 10 (data capturing three-dimensional elements such as point cloud data and depth data, as well as data reconstructed into a three-dimensional space from multiple images, etc.).

[0025] The sensor 10 senses and photographs (takes an image of) the surface of the luggage 4 in the loading space 5, which is the photographing area (see FIG. 1). The sensor 10 is communicably connected to the loading space recognition device 200. The sensor 10 outputs photographic data (100 in FIG. 2) created from three-dimensional data obtained by photographing the loading space 5 to the loading space recognition device 200. The sensor 10 can be selected according to photographing conditions such as the photographing distance required to detect the luggage 4, the angle of view, and the size of the container 3, as well as customer requests. In addition, products sold by various manufacturers can be used for the sensor 10. For example, a stereo camera, a ToF (Time of Flight) camera, a 3D-lidar (Three Dimensions - Light Detection and Ranging), a 2D-lidar (Two Dimensions - Light Detection and Ranging), a LIDAR (Laser Imaging Detection and Ranging), etc. can be used as the sensor 10. The sensor 10 can be attached at a position where it can photograph the loading space 5, which is the photographing area, and can be attached, for example, near the upper part of the cargo entrance side of the container 3. The sensor 10 can photograph the cargo 4 in the loading space 5 from above the cargo entrance side of the container 3. At least one sensor 10 is required in the loading space 5, but multiple sensors 10 may be provided. Furthermore, multiple sensors 10 may be used when the loading space 5 is large and it is difficult to photograph the entire space with a single sensor. Note that when there are multiple sensors 10 in the loading space 5, the sensors 10 may be of different types or from different manufacturers. Furthermore, when there are multiple sensors 10 in the loading space 5, the photographed data photographed by each sensor 10 can be combined by the loading space recognition device 200.

[0026] The loading space recognition device 200 recognizes changes (volume changes, distance changes) of the luggage 4 in the loading space 5 where the luggage 4 is loaded, based on the image data 100 from the sensor 10 (see FIGS. 1 and 2). The loading space recognition device 200 can be a device (computer) having functional units (e.g., a processor, a storage device, an input device, a communication interface, and a display device) that constitute a computer, such as a laptop personal computer, a smartphone, or a tablet terminal. The loading space recognition device 200 has a function to determine whether or not the luggage 4 is likely to collapse. The loading space recognition device 200 can also be used in situations such as pallet loading, where a certain amount of volume change is expected if luggage collapse occurs. The loading space recognition device 200 is implemented by a preprocessing unit 210, a package shape recognition unit 220, a determination unit 230, and a user interface unit 240 by executing a predetermined program.

[0027] The pre-processing unit 210 is a functional unit that performs pre-processing on the photographed data 100 in order to perform loading space recognition processing (see FIG. 2). The pre-processing unit 210 outputs pre-processed data 101, which is obtained by pre-processing the photographed data 100, to the package style recognition unit 220 and the user interface unit 240. The pre-processing unit 210 includes a format conversion unit 211 and a noise removal unit 212.

[0028] The format conversion unit 211 is a functional unit that converts the format of the photographed data 100 into a common format that can be commonly used in the loading space recognition device 200 as preprocessing (see FIG. 2). The format conversion unit 211 outputs the photographed data 100 in the common format to the noise removal unit 212. Note that if the format of the photographed data 100 is originally a common format, the processing of the format conversion unit 211 can be omitted (skip).

[0029] The noise removal unit 212 is a functional unit that removes noise (for example, point clouds unnecessary for loading space recognition) from the photographed data 100 from the format conversion unit 211 as preprocessing (see FIG. 2). The noise removal unit 212 outputs preprocessed data 101, from which noise has been removed from the photographed data 100, to the package overall image estimation unit 222 of the package shape recognition unit 220, and outputs the preprocessed data 101 to the display unit 241 of the user interface unit 240 as needed. Examples of noise removal methods include smoothing, filtering (for example, moving average filter processing, median filter processing, etc.), and outlier removal processing (for example, outlier removal processing using a chi-squared test). Note that if there is almost no noise, the processing by the noise removal unit 212 may be omitted (skip).

[0030] The preprocessed data 101 is output to the display unit 241, and the user can view it. In addition, since the photographed data 100 and the preprocessed data 101 are the basis for all processing, it is also possible to use them as confirmation by making them saveable.

[0031] The packaging style recognition unit 220 is a functional block that recognizes the current packaging style from the pre-processed data 101 (see FIG. 2). The packaging style recognition unit 220 outputs voxel data 102 relating to the recognized current packaging style to the determination unit 230. The packaging style recognition unit 220 includes an area designation unit 221, a luggage overall image estimation unit 222, and a voxelization unit 223. The packaging style recognition unit 220 measures information about the shape of the loaded luggage or free space from the pre-processed data 101 and provides quantitative information.

[0032] The area designation unit 221 is a functional unit that designates an area (loading area) within the container 3 where cargo 4 can be loaded (see FIG. 2). For example, if there is an area within the container 3 where cargo 4 cannot be loaded, the area designation unit 221 excludes that area when designating the area. The area designation unit 221 can allow the user to designate the loading area via the operation unit 242, and can also designate an area automatically in combination with a system that automatically acquires the area edge.

[0033] Furthermore, the area designation unit 221 may allow the user to designate a determination exclusion area to be excluded from the determination by the determination unit 230 via the operation unit 242. This prevents excessive determination of changes in packaging style of luggage 4 that cannot maintain a constant shape. The change determination exclusion area may be designated not only in area units, but also in voxel units or single / individual luggage units (object units).

[0034] The luggage overall image estimation unit 222 is a functional unit that estimates an overall image of the luggage 4 loaded in the loading area based on the preprocessed data 101 from the noise removal unit 212 of the preprocessed data 101 and the loading area specified by the area specification unit 221 (see FIG. 2). The luggage overall image estimation unit 222 creates estimation result data related to the overall image of the loaded luggage 4. The luggage overall image estimation unit 222 outputs the created estimation result data to the voxelization unit 223. The luggage overall image estimation unit 222 estimates not only luggage loaded on the surface or at the forefront of the luggage, but also luggage loaded in the back (occlusion area) that is blocked by them and cannot be seen by the sensor 10 (and gaps, if necessary). To estimate the luggage loaded in the occluded area, it is possible to use supplementary data accumulated in chronological order, or prior knowledge of the luggage such as its size and number, or to use a simple algorithm such as assuming that a coordinate point indicating the luggage exists on an extension of a straight line connecting the front coordinate point and the sensor, or to use pre-processed data 101 stored in chronological order, or to use machine learning.

[0035] An example of a method for collecting training data for machine learning is a method in which preprocessed data 101 is associated with the actual loading status and recorded in a database. For example, distance sensors are installed at predetermined intervals on the ceiling surface of a container, and the distance sensors measure the distance to the loaded cargo. This makes it possible to determine the cargo loading status, such as whether the cargo directly below the sensor is stacked close to the ceiling, about halfway up, or no cargo at all. By installing multiple sensors on the ceiling surface, it is possible to determine the cargo loading status throughout the container. In this way, machine learning can be performed by associating the preprocessed data 101 with the actual loading status. The machine learning method is not limited to this, and other methods may also be used. Furthermore, when machine learning is used, a function for estimating information and attributes other than shape, such as "weight" and "no overloading / no underloading," may be provided.

[0036] The voxelization unit 223 is a functional unit that voxels the estimation result data of the luggage overall image estimation unit 222 (see FIG. 2). The voxelization unit 223 creates voxel data 102 relating to the overall image of the luggage 4 based on the estimation result data of the luggage overall image estimation unit 222. The voxelization unit 223 creates voxel data 102 by treating areas where no loaded luggage 4 exists as empty space. The voxelization unit 223 outputs the created voxel data 102 to the reference determination unit 231 and difference extraction unit 232 of the determination unit 230. The voxelization process may involve estimation of occluded areas that are not visible from the 3D sensor, or interpolation using data accumulated in chronological order, or prior knowledge of the size, number, etc. of the loaded luggage may be used.

[0037] Here, the voxel data 102 is data that represents the overall image of the luggage 4 by combining multiple voxels (cubes) of a predetermined size. The voxel data 102 includes information related to the dimensions of each voxel and the position of each face. The voxel data 102 may be stored each time it is created. The voxel data 102 may also include information such as the number of luggage within one voxel, information on which multiple voxels one luggage spans, weight, shape, etc.

[0038] The determination unit 230 is a functional unit that estimates the amount of change in the volume or movement of the luggage by comparing the voxel data 102 (reference voxel data) at a certain reference time point with the voxel data 102 (contrast voxel data) at a predetermined or arbitrary time point, and determines whether or not there is a possibility of cargo collapse by comparing the estimated amount of change in volume or movement with a threshold (see FIG. 2). This makes it possible to prevent the luggage 4 from falling. Furthermore, the determination unit 230 may detect whether or not there is a possibility of cargo collapse not only by comparing the reference voxel data with the contrast voxel data, but also by comparing the contrast voxel data with other contrast voxel data created immediately before. This makes it possible not to determine that there is a possibility of cargo collapse when the amount of change between the contrast voxel data per unit time is smaller than the amount of change between the reference voxel data and the contrast voxel data. Furthermore, a detection unit 20 that detects shaking and sound may be attached to the truck 2 (a structure having a loading space) (see FIG. 10 ), and the determination unit 230 may acquire comparison voxel data from the voxelization unit 223 when the detection unit 20 detects shaking or sound of a certain level or greater, and determine whether or not there is a possibility of cargo collapse. Furthermore, to complement and verify the reliability of the movement estimation, information obtained from another system, such as whether there are many heavy loads or, conversely, many light loads, may be used. The determination unit 230 includes a reference determination unit 231, a difference extraction unit 232, a fluctuation volume estimation unit 233, a lump extraction unit 234, a combination estimation unit 235, a movement amount estimation unit 236, and a cargo collapse determination unit 237.

[0039] The reference determination unit 231 is a functional unit that determines the voxel data 102 at a certain reference point (reference time) from the voxelization unit 223 as a change reference point (see FIG. 2). In response to an instruction from the operation unit 242, the reference determination unit 231 stores the voxel data 102 at the reference point (instructed time) from the voxelization unit 223 and holds it as reference data for positional fluctuations. The reference determination unit 231 outputs the voxel data 102 at the reference point to the difference extraction unit 232.

[0040] The difference extraction unit 232 is a functional unit that extracts voxel differences (e.g., differences in positions in the depth direction) at corresponding positions on a surface of the package 4 viewed from a predetermined position (e.g., the rear of the truck 2, the position of the sensor 10) by comparing the voxel data 102 at an arbitrary reference time (reference voxel data) with the voxel data 102 at a point in time when a predetermined or arbitrary time has elapsed from the reference time (contrast voxel data) (see FIG. 2). In performing the difference extraction process, the difference extraction unit 232 obtains the reference voxel data from the reference determination unit 231 and obtains the contrast voxel data from the voxelization unit 223. In the difference extraction process, for example, if the position of the voxel's surface when viewed from behind the truck 2 changes toward the front of the truck 2 in the depth direction (for example, if the luggage 4 at the target voxel position moves to the left, right, or downward and a luggage 4 one level further back appears, or if the luggage 4 at the target voxel position moves toward the back, or if the occlusion portion decreases), a difference is extracted to represent a "decrease" in volume. Similarly, if the position of the voxel's surface changes toward the rear of the truck 2 (for example, if the luggage 4 at the target voxel position moves toward the front, or if another luggage 4 moves in front of the luggage 4 at the target voxel position, or if the occlusion portion increases), a difference is extracted to represent an "increase" in volume. Furthermore, in the difference extraction process, for example, differences can be extracted to represent the degree of decrease or increase in volume in stages depending on the distance by which the voxel's surface position has changed toward or behind the truck 2. The difference extraction unit 232 outputs the difference data extracted by the difference extraction process to the fluctuation volume estimation unit 233 and the lump extraction unit 234.

[0041] The volume fluctuation estimation unit 233 is a functional unit that estimates the volume fluctuation of the entire image of the luggage 4 (volume fluctuation) based on the difference data from the difference extraction unit 232 (see FIG. 2). The volume fluctuation can be expressed, for example, by the sum of the differences in the depth direction positions of voxels at corresponding positions on the surface of the luggage 4 when viewed from the rear of the truck 2. If a certain volume increases at one position and the same volume decreases at another position, the volume fluctuation will be zero (see movement examples 2-1 to 2-4 in FIG. 5). If a certain volume increases at one position and the volume decreases at another position by a greater amount than the increase, the volume fluctuation will be a negative value. If a certain volume increases at one position and the volume decreases at another position by a smaller amount than the increase, the volume fluctuation will be a positive value (see movement examples 1-1 to 1-3 in FIG. 4). If a certain volume increases at one position and the volume does not decrease at another position, the volume fluctuation will be a positive value. Furthermore, if a certain volume amount decreases at a certain position and there is no increase in the volume amount at another position, the fluctuation volume amount will be a negative value. The fluctuation volume amount estimation unit 233 outputs the estimated fluctuation volume amount estimation data to the cargo collapse determination unit 237.

[0042] The cluster extraction unit 234 is a functional unit that extracts clusters of voxels with no increase or decrease in volume that exist between voxels with an increased volume (volume-increased voxels) and voxels with a decreased volume (volume-decreased voxels) at the same horizontal position, based on the difference data from the difference extraction unit 232 (see FIG. 2). The cluster extraction unit 234 can assume that there is no change in the volume of the luggage 4. If the cluster extraction unit 234 is able to extract a cluster, it outputs difference data including the extracted cluster data to the combination estimation unit 235. If the cluster extraction unit 234 is unable to extract a cluster, it outputs the difference data from the difference extraction unit 232 to the combination estimation unit 235.

[0043] The combination estimation unit 235 is a functional unit that estimates combinations of volume-increasing voxels and volume-decreasing voxels based on the difference data from the cluster extraction unit 234 (including cluster data if clusters are extracted) (see FIG. 2). The combination estimation unit 235 can assume that there is no change in the volume of the luggage 4. In the combination estimation process, for example, if multiple combinations are possible, it is possible to prioritize combinations of volume-increasing voxels and volume-decreasing voxels that are in the same horizontal position. Furthermore, if multiple combinations are possible, it is possible to prioritize combinations of volume-increasing voxels and volume-decreasing voxels that are the furthest apart from each other. Furthermore, if multiple combinations are possible, it is possible to prioritize combinations of volume-increasing voxels and volume-decreasing voxels that maximize the total distance. Furthermore, a volume-decreasing voxel can be prevented from being combined with a volume-increasing voxel in the next higher row because, if the luggage 4 falls, it is impossible for the luggage 4 to move upward. A volume-increasing voxel can be prevented from being combined with a volume-decreasing voxel in the next lower row because, if the luggage 4 falls, it is impossible for the luggage 4 to move upward. If the combination estimation unit 235 is able to estimate a combination, it outputs difference data including the extracted combination data to the movement amount estimation unit 236. If the combination estimation unit 235 is unable to estimate a combination, it outputs difference data from the lump extraction unit 234 to the movement amount estimation unit 236.

[0044] The movement amount estimation unit 236 is a functional unit that estimates the amount of movement of the luggage 4 that has occurred between a reference time and a predetermined or arbitrary time period, based on the difference data (which may include the chunk data and the combination data) from the combination estimation unit 235 (see FIG. 2). The movement amount estimation unit 236 can assume that the volume of the luggage 4 does not change. When the difference data includes the chunk data and the combination data, the movement amount estimation unit 236 calculates the distance from the volume-decreasing voxel to the volume-increasing voxel in the combination data, calculates the length of the voxel in the chunk data that has no increase or decrease in volume, and estimates the value obtained by subtracting the calculated length from the calculated distance as the amount of movement of the luggage 4. When the difference data does not include the chunk data but includes the combination data, the movement amount estimation unit 236 calculates the distance from the volume-decreasing voxel to the volume-increasing voxel in the combination data, and estimates the calculated distance as the amount of movement of the luggage 4. Note that if the differential data does not include combination data (whether or not there is bundled data), the movement of the luggage 4 involves a change in volume, so the movement amount estimation unit 236 does not estimate the movement amount, and priority can be given to estimating the change in volume amount in the change volume amount estimation unit 233. Furthermore, in estimating the movement amount, corrections may be made so that different weights are assigned to horizontal movement and vertical movement (falling), for example, horizontal movement may be corrected to be smaller than the estimated movement amount so that no warning is issued even if there is a large movement, and vertical or diagonal movement may be corrected to be larger than the estimated movement amount so that even small movements are warned. If the differential data includes multiple combination data, the movement amount estimation unit 236 will estimate the movement amount for each combination data.

[0045] Furthermore, the movement distance estimation unit 236 selects the longest movement distance from the estimated movement distances and estimates the selected movement distance as the longest movement distance. Note that if there is only one estimated movement distance, that movement distance is estimated as the longest movement distance. The movement distance estimation unit 236 outputs the estimated longest movement distance estimation data to the cargo collapse determination unit 237.

[0046] Examples of the operations of the cluster extraction unit 234, combination estimation unit 235, and movement amount estimation unit 236 will be described later.

[0047] The cargo collapse determination unit 237 is a functional unit that determines whether or not there is a possibility of cargo collapse of the luggage 4 by comparing the fluctuation volume amount estimation data from the fluctuation volume amount estimation unit 233 or the longest movement amount estimation data from the movement amount estimation unit 236 with a preset threshold (a first threshold for the fluctuation volume amount or a second threshold for the longest movement amount) (see FIG. 2). The cargo collapse determination unit 237 determines that there is a possibility of cargo collapse of the luggage 4 when the fluctuation volume amount estimation data is greater than the first threshold (or may be equal to or greater than the first threshold). The cargo collapse determination unit 237 determines that there is no possibility of cargo collapse of the luggage 4 when the fluctuation volume amount estimation data is equal to or less than the first threshold (or may be less than the first threshold). The cargo collapse determination unit 237 determines that there is a possibility of cargo collapse of the luggage 4 when the longest movement amount is greater than a second threshold (or may be equal to or greater than the second threshold). The cargo collapse determination unit 237 determines that there is no possibility of cargo collapse of the luggage 4 when the longest movement amount is equal to or less than the second threshold (it can also be less than the second threshold). The cargo collapse determination unit 237 can arbitrarily use either the volume fluctuation amount or the longest movement amount to determine whether there is a possibility of cargo collapse of the luggage 4, but considering the data processing load, it can make the determination by using the volume fluctuation amount, which imposes a relatively small load. When it is determined that there is a possibility of cargo collapse of the luggage 4, the cargo collapse determination unit 237 outputs warning output instruction information to the warning output unit 243 to warn the user that an abnormality has occurred.

[0048] The user interface unit 240 is a functional unit equipped with a user interface (a function for exchanging information with the user) (see FIG. 2). The user interface unit 240 acts as an interface between the user and the loading space recognition device 200, allowing the user to operate each process and check the results of each process. The user interface unit 240 is equipped with a display unit 241, an operation unit 242, and a warning output unit 243.

[0049] The display unit 241 is a functional unit that displays the preprocessed data 101 and the like from the noise removal unit 212 (see FIG. 2). As the display unit 241, for example, a liquid crystal display, an organic EL (Electroluminescence) display, AR (Augmented Reality) glasses, etc. can be used.

[0050] The operation unit 242 is a functional unit that issues area designation and determination instructions to the area designation unit 221 and the reference determination unit 231 based on user operations (see FIG. 2). As the operation unit 242, for example, a touch panel, a mouse, a camera and software that recognizes gestures and eye movements, etc. can be used.

[0051] The warning output unit 243 is a functional unit that outputs a warning to the user based on warning output instruction information from the cargo collapse determination unit 237 (see FIG. 2). As the warning output unit 243, for example, a display that displays characters or images related to the warning, a speaker that outputs an alarm sound, a lamp that lights up an alarm, a communication unit that transmits warning output instruction information to other systems, etc. can be used.

[0052] Next, the operation of the loading space recognition device in the loading space recognition system according to embodiment 1 will be described with reference to the drawings. Fig. 3 is a flowchart schematically showing the operation of the loading space recognition device in the loading space recognition system according to embodiment 1. Note that for the components of the loading space recognition device and their details, please refer to Figs. 1 and 2 and their descriptions.

[0053] First, the format conversion unit 211 of the pre-processing unit 210 acquires, from the sensor 10, reference photographic data 100 (reference photographic data; three-dimensional data) of the luggage 4 in the loading space 5, which is the photographic area (step A1).

[0054] Next, the format conversion unit 211 of the pre-processing unit 210 converts the format of the reference imaging data 100 into a common format (step A2).

[0055] Next, the noise removal unit 212 of the pre-processing unit 210 removes noise from the reference photographic data 100 converted into the common format to create reference pre-processed data 101 (step A3).

[0056] Next, the overall image estimation unit 222 of the package shape recognition unit 220 estimates the overall image of the loaded luggage 4 on the loading area based on the reference pre-processing data 101 and the loading area specified by the area specification unit 221 (step A4).

[0057] Next, the voxelization unit 223 of the package style recognition unit 220 creates reference voxel data 102 (reference voxel data) relating to the overall image of the package 4 based on the estimation result data estimated in step A4 (step A5).

[0058] Next, the reference determination unit 231 of the determination unit 230 stores the reference voxel data created by the voxelization unit 223 as a reference value for the cargo collapse determination process by the user operating the operation unit 242 (step A6). The user can operate the operation unit 242, for example, after loading of the cargo 4 into the container 3 is completed and before delivery begins.

[0059] Next, the format conversion unit 211 of the pre-processing unit 210 acquires comparison photographic data 100 (comparison photographic data; three-dimensional data) from the sensor 10, which is photographic area, of the luggage 4 in the loading space 5, when a predetermined or arbitrary time has elapsed since acquiring the reference photographic data 100 (reference photographic data) (step A7).

[0060] Next, the format conversion unit 211 of the pre-processing unit 210 converts the format of the comparison imaging data 100 into a common format (step A8).

[0061] Next, the noise removal unit 212 of the pre-processing unit 210 removes noise from the photographed data 100 for comparison that has been converted into the common format, to create pre-processed data 101 for comparison (step A9).

[0062] Next, the overall image estimation unit 222 of the package shape recognition unit 220 estimates the overall image of the loaded luggage 4 on the loading area based on the pre-processed data 101 for comparison and the loading area specified by the area specification unit 221 (step A10).

[0063] Next, the voxelization unit 223 of the package style recognition unit 220 creates comparison voxel data 102 (comparison voxel data) relating to the overall image of the package 4 based on the estimation result data estimated in step A10 (step A11).

[0064] Next, the difference extraction unit 232 of the judgment unit 230 compares the reference voxel data stored in the reference determination unit 231 with the contrast voxel data created by the voxelization unit 223 to extract the voxel differences (e.g., differences in depth positions) at corresponding positions on the surface of the luggage 4 viewed from a predetermined position (e.g., the rear of the truck 2, the position of the sensor 10) (step A12).

[0065] Next, the volume fluctuation estimation unit 233 of the determination unit 230 estimates the volume fluctuation of the entire image of the baggage 4 (volume fluctuation) based on the difference data from the difference extraction unit 232 (step A13).

[0066] Next, the cargo collapse determination unit 237 of the determination unit 230 determines whether the fluctuation volume estimation data from the fluctuation volume estimation unit 233 is greater than a first threshold value for the fluctuation volume amount set in advance (step A14). If the fluctuation volume estimation is greater than the first threshold value (YES in step A14), it determines that there is a possibility of cargo collapse of the luggage 4, outputs warning output instruction information to the warning output unit 243, and proceeds to step A20.

[0067] If the estimated volume change is equal to or less than the first threshold (NO in step A14), the cluster extraction unit 234 of the determination unit 230 extracts clusters of voxels with no change in volume that are located between the volume-increasing voxel and the volume-decreasing voxel at the same horizontal position based on the difference data from the difference extraction unit 232 (step A15). If clusters cannot be extracted, the process is skipped.

[0068] Next, the combination estimation unit 235 of the determination unit 230 estimates a combination of volume-increasing voxels and volume-decreasing voxels based on the difference data from the cluster extraction unit 234 (including cluster data if clusters are extracted) (step A16).

[0069] Next, the movement amount estimation unit 236 of the determination unit 230 estimates the movement amount of the luggage 4 that has occurred between the reference time and the time when a predetermined or arbitrary time has elapsed, based on the difference data (which may include the aggregate data and the combination data) from the combination estimation unit 235 (step A17).

[0070] Here, in estimating the amount of movement, if the differential data includes block data and combination data, the distance from the volume-decreasing voxel to the volume-increasing voxel in the combined data is calculated, the length of the voxel in the block data with no increase or decrease in volume is calculated, and the value obtained by subtracting the calculated length from the calculated distance is estimated as the amount of movement of the luggage 4. In addition, in estimating the amount of movement, if the differential data does not include block data but includes combination data, the distance from the volume-decreasing voxel to the volume-increasing voxel in the combination data is calculated, and the calculated distance value is estimated as the amount of movement of the luggage 4. In addition, in estimating the amount of movement, if the differential data includes multiple combination data, the amount of movement is estimated for each combination data.

[0071] Next, the movement amount estimation unit 236 of the determination unit 230 selects the longest movement amount from the estimated movement amounts, and estimates the selected movement amount as the longest movement amount (step A18).

[0072] Next, the cargo collapse determination unit 237 of the determination unit 230 determines whether the longest movement amount estimation data from the movement amount estimation unit 236 is greater than a preset second threshold value for the longest movement amount (step A19). If the longest movement amount estimation data is greater than the second threshold value (YES in step A19), it determines that there is a possibility of cargo collapse of the cargo 4, outputs warning output instruction information to the warning output unit 243, and proceeds to step A20. If the longest movement amount estimation data is equal to or less than the second threshold value (NO in step A19), it determines that there is no possibility of cargo collapse of the cargo 4, ends one cycle, and repeats steps A7 to A20 until the user gives an instruction to end.

[0073] If the estimated fluctuation volume is greater than the first threshold (YES in step A14), or if the longest movement amount estimated data is greater than the second threshold (YES in step A19), the warning output unit 243 of the user interface unit 240 outputs a warning to the user (step A20) based on the warning output instruction information from the cargo collapse determination unit 237. After that, one cycle ends, and steps A7 to A20 are repeated until the user gives an instruction to end the cycle.

[0074] Next, the operations of the difference extraction unit, the volume fluctuation estimation unit, and the cargo collapse determination unit of the loading space recognition device in the loading space recognition system according to embodiment 1 will be described with reference to several examples and drawings. Fig. 4 is an image diagram that schematically shows several examples in which there is a volume fluctuation when the difference between the reference voxel and the comparison voxel is extracted.

[0075] When the reference voxel data when viewing the entire image of the cargo (4 in Figure 1) from the rear of the truck (2 in Figure 1) is in a state similar to the reference voxel data in Figure 4, if the reference voxel data in Figure 4 changes to the contrast voxel data according to Movement Example 1-1, the difference extraction data extracted by the difference extraction unit (232 in Figure 2) will be like the difference extraction data according to Movement Example 1-1 in Figure 4. In the difference extraction data according to Movement Example 1-1, the volume of only the four voxels on the top left increases by one level, and there is no increase or decrease in the other voxels. In other words, because there is only a one-level increase in volume for four voxels, the total volumes of the reference voxel data and the contrast voxel data are different. In this case, it is considered that the occlusion area (gaps or spaces) has increased behind the voxel whose volume has increased, so the distance from the surface position of the reference voxel data to the surface position of the increased volume in the comparison voxel data has moved, and the amount of change in the volume change estimation unit (233 in Figure 2) can estimate the amount of change in the volume change threshold, and the load collapse determination unit (237 in Figure 2) can determine the amount of change in the volume change threshold.

[0076] When the reference voxel data in FIG. 4 changes to the comparison voxel data according to Movement Example 1-2, the difference extraction data extracted by the difference extraction unit (232 in FIG. 2) becomes the difference extraction data according to Movement Example 1-2 in FIG. 4. In the difference extraction data according to Movement Example 1-2, the volume amounts of four voxels each decrease by one level, the volume amounts of another eight voxels each increase by one level, and there is no increase or decrease in the other voxels. In other words, although there are increases and decreases in volume, even when the increases and decreases in volume are offset, there are still four more voxels with an increase of one level in volume, so the total volume differs between the reference voxel data and the comparison voxel data. In this case, the luggage has moved overall, making it impossible to accurately calculate the amount of movement. In this case, the luggage 4 moves between four increases in volume and four decreases in volume, and the remaining four increases in volume move the distance from the surface position of the reference voxel data to the increased surface position of the comparison voxel data. The fluctuation volume estimation unit (233 in Figure 2) estimates the fluctuation volume, and the cargo collapse determination unit (237 in Figure 2) determines the threshold value of the fluctuation volume.

[0077] When the reference voxel data in Fig. 4 changes to the comparison voxel data according to Movement Examples 1-3, the difference extraction data extracted by the difference extraction unit (232 in Fig. 2) becomes the difference extraction data according to Movement Examples 1-3 in Fig. 4. In the difference extraction data according to Movement Examples 1-3, the volume amounts of eight voxels each decrease by one level, the volume amounts of another 12 voxels each increase by one level, and the volume amounts of four voxels each increase by two levels, while the other voxels do not increase or decrease. In other words, although there are increases and decreases in volume, even when the increases and decreases in volume are offset, there are four more voxels with an increase in volume equivalent to one level, and four more voxels with an increase in volume equivalent to two levels, so the total volumes of the reference voxel data and the comparison voxel data are different. In this case, the luggage 4 moves between an increase of 8 units in one step and a decrease of 8 units in one step, and the distance from the surface position of the reference voxel data to the increased surface position of the comparison voxel data moves by the remaining increase of 4 units in one step and an increase of 4 units in two steps. The change volume estimation unit (233 in Figure 2) estimates the change volume, and the cargo collapse determination unit (237 in Figure 2) determines the threshold value of the change volume.

[0078] Next, the operations of the difference extraction unit, the lump extraction unit, the combination estimation unit, the movement amount estimation unit, and the cargo collapse determination unit of the loading space recognition device in the loading space recognition system according to the first embodiment will be described using several examples and drawings. FIG. 5 is an image diagram that schematically shows several examples in which there is no volume fluctuation and there is a movement amount when the difference between the reference voxel and the comparison voxel is extracted. FIG. 6 is an image diagram that schematically shows the transition from the lump estimation to the longest movement amount estimation in Example 2-1 of FIG. 5. FIG. 7 is an image diagram that schematically shows the transition from the lump estimation to the longest movement amount estimation in Example 2-2 of FIG. 5. FIG. 8 is an image diagram that schematically shows the transition from the lump estimation to the longest movement amount estimation in Example 2-3 of FIG. 5. FIG. 9 is an image diagram that schematically shows the transition from the lump estimation to the longest movement amount estimation in Example 2-4 of FIG. 5.

[0079] When the reference voxel data when viewing the overall image of the cargo (4 in FIG. 1) from the rear of the truck (2 in FIG. 1) is in a state similar to the reference voxel data in FIG. 5, if the reference voxel data in FIG. 5 changes to the comparison voxel data according to Movement Example 2-1, the difference extraction data extracted by the difference extraction unit (232 in FIG. 2) will be like the difference extraction data according to Movement Example 2-1 in FIG. 5. In the difference extraction data according to Movement Example 2-1, the volume amounts of six voxels each increase by one level, the volume amounts of another six voxels each decrease by one level, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, and the increases and decreases in volume are offset by the same number and number of levels, so the total volumes of the reference voxel data and the comparison voxel data are the same. In this case, the cluster extraction unit (234 in Fig. 2) extracts two clusters of voxels that are located at the same horizontal position as shown in Fig. 6(A) and have no change in volume, and are located between a volume-increasing voxel and a volume-decreasing voxel. The combination estimation unit (235 in Fig. 2) estimates two combinations of linked clusters of volume-increasing voxels and clusters of volume-decreasing voxels, as shown in Fig. 6(B). The movement estimation unit (236 in Fig. 2) estimates the movement of the volume-increasing voxels from the volume-decreasing voxels involved in the combination, as shown in Fig. 6(C). The distance to the cell (see the two arrows in Figure 6(C)) is calculated, and the length of the voxel (not shown) with no increase or decrease in volume related to the block data is calculated. The distance calculated as in Figure 6(D) minus the calculated length is estimated as the amount of movement of the luggage 4 (see the two arrows in Figure 6(D)). The movement amount estimation unit (236 in Figure 2) estimates the longest movement amount among the movement amounts as the longest movement amount (see the circled arrow in Figure 6(D)), and the cargo collapse determination unit (237 in Figure 2) determines the threshold value of the longest movement amount.

[0080] When the reference voxel data in Fig. 5 changes to the comparison voxel data according to Movement Example 2-2, the difference extraction data extracted by the difference extraction unit (232 in Fig. 2) becomes the difference extraction data according to Movement Example 2-2 in Fig. 5. In the difference extraction data according to Movement Example 2-2, the volume amounts of eight voxels each increase by one step, the volume amounts of another eight voxels each decrease by one step, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, and the increases and decreases in volume are offset by the same number and number of steps, so the total volumes of the reference voxel data and the comparison voxel data are the same. In this case, the cluster extraction unit (234 in Fig. 2) extracts one cluster of voxels with no change in volume that exists between a volume-increasing voxel and a volume-decreasing voxel at the same horizontal position as shown in Fig. 7(A), and the combination estimation unit (235 in Fig. 2) estimates two combinations of a cluster of volume-increasing voxels and a cluster of volume-decreasing voxels that are linked as shown in Fig. 7(B). The movement estimation unit (236 in Fig. 2) calculates the distance from the volume-decreasing voxel to the volume-increasing voxel involved in the combination as shown in Fig. 7(C) (see the two arrows in Fig. 7(C)). The length of the voxel (one voxel not shown) with no increase or decrease in volume related to the block of data is calculated, and the value obtained by subtracting the calculated length from the calculated distance as shown in FIG. 7(D) is estimated as the amount of movement of the luggage 4 (see the two arrows in FIG. 7(D); one with deduction and the other without deduction). The movement amount estimation unit (236 in FIG. 2) estimates the longest movement amount among the movement amounts as the longest movement amount (see the circled arrows in FIG. 7(D); in this case, there are two, but either one is acceptable), and the cargo collapse determination unit (237 in FIG. 2) determines the threshold value of the longest movement amount.

[0081] When the reference voxel data in Fig. 5 changes to the comparison voxel data according to Movement Example 2-3, the difference extraction data extracted by the difference extraction unit (232 in Fig. 2) becomes the difference extraction data according to Movement Example 2-3 in Fig. 5. In the difference extraction data according to Movement Example 2-3, the volume amounts of eight voxels each increase by one step, the volume amounts of another eight voxels each decrease by one step, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, and the increases and decreases in volume are offset by the same number and number of steps, so the total volumes of the reference voxel data and the comparison voxel data are the same. In this case, the cluster extraction unit (234 in Fig. 2) cannot extract clusters of voxels with no change in volume that exist between a volume-increasing voxel and a volume-decreasing voxel at the same horizontal position as in Fig. 8(A), so it skips them. The combination estimation unit (235 in Fig. 2) estimates two combinations of clusters of volume-increasing voxels and clusters of volume-decreasing voxels that are linked as in Fig. 8(B), and the movement estimation unit (236 in Fig. 2) The distance from the volume-decreasing voxel to the volume-increasing voxel related to the combination is calculated as shown in Figure 8(C) (see the two arrows in Figure 8(C)), and the calculated distance is estimated as the movement amount of the luggage 4 as shown in Figure 8(D) (see the two arrows in Figure 8(D)).The movement amount estimation unit (236 in Figure 2) estimates the longest movement amount among the movement amounts as the longest movement amount (see the circled arrow in Figure 8(D)), and the cargo collapse determination unit (237 in Figure 2) determines the threshold value of the longest movement amount.

[0082] When the reference voxel data in Fig. 5 changes to the comparison voxel data according to Movement Example 2-4, the difference extraction data extracted by the difference extraction unit (232 in Fig. 2) becomes the difference extraction data according to Movement Example 2-4 in Fig. 5. In the difference extraction data according to Movement Example 2-4, the volume amounts of 12 voxels each increase by one step, the volume amounts of another 12 voxels each decrease by one step, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, and the increases and decreases in volume are offset by the same number and number of steps, so the total volumes of the reference voxel data and the comparison voxel data are the same. In this case, the cluster extraction unit (234 in Fig. 2) cannot extract clusters of voxels with no change in volume that exist between a volume-increasing voxel and a volume-decreasing voxel at the same horizontal position as in Fig. 9(A), so it skips them. The combination estimation unit (235 in Fig. 2) estimates three combinations of clusters of volume-increasing voxels and clusters of volume-decreasing voxels that are linked together as in Fig. 9(B), and the movement estimation unit (236 in Fig. 2) The distance from the volume-decreasing voxel to the volume-increasing voxel related to the combination is calculated as shown in Figure 9(C) (see the three arrows in Figure 9(C)), and the calculated distance is estimated as the movement amount of the luggage 4 as shown in Figure 9(D) (see the three arrows in Figure 9(D)).The movement amount estimation unit (236 in Figure 2) estimates the longest movement amount among the movement amounts as the longest movement amount (see the circled arrow in Figure 9(D)), and the cargo collapse determination unit (237 in Figure 2) determines the threshold value of the longest movement amount.

[0083] For the criteria for the cluster extraction process by the cluster extraction unit 234, the criteria for the combination estimation process by the combination estimation unit 235, and the criteria for the movement amount estimation process by the movement amount estimation unit 236 in the above movement examples 1-1 to 1-3 and 2-1 to 2-4, please refer to the detailed explanation in Figure 2.

[0084] According to embodiment 1, the difference between voxels at corresponding positions between the reference voxel data and the comparison voxel data is extracted, the amount of change in volume or the maximum amount of movement of the overall image of the luggage 4 is estimated, and a threshold judgment is performed, which can contribute to determining the possibility of cargo collapse due to luggage movement.

[0085] Furthermore, according to the first embodiment, even if an occlusion portion hidden by the luggage 4 occurs, the occlusion portion is estimated and the shape of the luggage 4 is grasped, so it is possible to take into account changes in the luggage 4 in the occlusion portion.

[0086] Furthermore, according to the first embodiment, by storing reference voxel data of the package shape at a specific time, such as before the truck 2 starts moving, and comparing it with contrast voxel data of the package shape after a predetermined or arbitrary time has passed from the reference time, it is possible to detect whether or not there is a possibility of cargo collapse, so even if there is cargo that has become unstable due to loading and unloading work, etc., and is in danger of falling, it is possible to detect this and notify the driver. This makes it possible to detect the possibility of cargo collapse of the cargo 4 early and prevent damage to the cargo 4, thereby contributing to preventing a decline in transportation quality.

[0087] [Embodiment 2] The loading space recognition device according to the second embodiment will be described with reference to the drawings. Fig. 11 is a block diagram showing a schematic configuration of the loading space recognition device according to the second embodiment.

[0088] The loading space recognition device 200 is a device that recognizes changes in the load (volume change, distance change) of the load in the load space where the load is loaded, based on the photographed data. The loading space recognition device 200 includes a load overall image estimation unit 222, a voxelization unit 223, and a determination unit 230.

[0089] The luggage overall image estimation unit 222 is configured to estimate an overall image of the luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the luggage loading space from a predetermined direction, and output the estimation result data.

[0090] The voxelization unit 223 is configured to voxelize the estimation result data and output it as voxel data.

[0091] The judgment unit 230 is configured to estimate the amount of change in volume or movement of the cargo by comparing voxel data at an arbitrary reference time with voxel data after a predetermined or arbitrary time has elapsed from the reference time, and to judge whether or not there is a possibility of cargo collapse by comparing the estimated amount of change in volume or movement with a threshold value.

[0092] According to embodiment 2, the difference between voxels at corresponding positions between voxel data at the reference time and voxel data other than the reference time is extracted, the amount of change in volume or the longest movement amount of the overall image of the luggage is estimated, and a threshold judgment is performed, which can contribute to determining the possibility of cargo collapse due to luggage movement.

[0093] The loading space recognition device according to the first and second embodiments can be configured by so-called hardware resources (information processing device, computer), and can use a device having the configuration exemplified in Fig. 12. For example, the hardware resource 1000 includes a processor 1001, a memory 1002, a network interface 1003, and the like, which are interconnected by an internal bus 1004.

[0094] 12 is not intended to limit the hardware configuration of the hardware resource 1000. The hardware resource 1000 may include hardware (e.g., an input / output interface) that is not shown. Furthermore, the number of units such as the processor 1001 included in the device is not intended to be limited to the example shown in FIG. 12, and for example, multiple processors 1001 may be included in the hardware resource 1000. The processor 1001 may be, for example, a central processing unit (CPU), a microprocessor unit (MPU), a graphics processing unit (GPU), or the like.

[0095] The memory 1002 may be, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), or a solid state drive (SSD).

[0096] The network interface 1003 may be, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, or the like.

[0097] The functions of the hardware resource 1000 are realized by the processing module described above. The processing module is realized, for example, by the processor 1001 executing a program stored in the memory 1002. The program can be updated by downloading it over a network or by using a storage medium storing the program. Furthermore, the processing module may be realized by a semiconductor chip. In other words, it is sufficient that the functions performed by the processing module can be realized by executing software on some kind of hardware.

[0098] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0099] [Appendix 1] a luggage overall image estimation unit configured to estimate an overall image of the luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the luggage loading space from a predetermined direction, and output the estimated result data; a voxelization unit configured to voxelize the estimation result data and output it as voxel data; a determination unit configured to estimate a volume change or movement amount of the cargo by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and to determine whether or not there is a possibility of cargo collapse by comparing the estimated volume change or movement amount with a threshold value; A loading space recognition device comprising: [Appendix 2] The determination unit a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point; a difference extraction unit configured to compare the voxel data at the reference time with the voxel data at a point in time when a predetermined or arbitrary time has elapsed from the reference time, thereby extracting differences between voxels at corresponding positions on a surface of the baggage viewed from a predetermined position, and outputting the difference data; a volume fluctuation estimation unit configured to estimate a volume fluctuation of the entire image of the package based on the difference data and output the volume fluctuation estimation data; a cargo collapse determination unit configured to determine whether or not there is a possibility of cargo collapse by comparing the fluctuation volume amount estimation data with a threshold value for fluctuation volume amount as the threshold value; 2. The loading space recognition device according to claim 1, comprising: [Appendix 3] The determination unit a cluster extraction unit configured to extract clusters of voxels with no change in volume that exist between a volume-increasing voxel whose volume has increased and a volume-decreasing voxel whose volume has decreased, at the same horizontal position, based on the difference data, and output the cluster data; a combination estimation unit configured to estimate a combination of the volume-increasing voxels and the volume-decreasing voxels based on the difference data and output the combination data as combination data; a movement amount estimation unit configured to estimate at least one movement amount of luggage that has occurred between the reference time and a predetermined or arbitrary time based on the difference data, the collection data, and the combined data, select the longest movement amount from the estimated movement amounts, estimate the selected movement amount as the longest movement amount, and output the longest movement amount estimated data; Furthermore, The load space recognition device according to Supplementary Note 2, wherein the cargo collapse determination unit is configured to determine whether or not there is a possibility of cargo collapse by comparing the fluctuation volume amount estimation data with a threshold value for fluctuation volume amount as the threshold value, and when it is determined that there is a possibility of cargo collapse, to determine whether or not there is a possibility of cargo collapse by comparing the longest movement amount estimation data with the threshold value for longest movement amount as the threshold value. [Appendix 4] The determination unit a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point; a difference extraction unit configured to compare the voxel data at the reference time with the voxel data at a point in time when a predetermined or arbitrary time has elapsed from the reference time, thereby extracting differences between voxels at corresponding positions on a surface of the baggage viewed from a predetermined position, and outputting the difference data; a cluster extraction unit configured to extract clusters of voxels with no change in volume that exist between a volume-increasing voxel whose volume has increased and a volume-decreasing voxel whose volume has decreased, at the same horizontal position, based on the difference data, and output the cluster data; a combination estimation unit configured to estimate a combination of the volume-increasing voxels and the volume-decreasing voxels based on the difference data and output the combination data as combination data; a movement amount estimation unit configured to estimate at least one movement amount of luggage that has occurred between the reference time and a predetermined or arbitrary time based on the difference data, the collection data, and the combined data, select the longest movement amount from the estimated movement amounts, estimate the selected movement amount as the longest movement amount, and output the longest movement amount estimated data; a cargo collapse determination unit configured to determine whether or not there is a possibility of cargo collapse by comparing the longest movement amount estimation data with a threshold value for the longest movement amount as the threshold value; 2. The loading space recognition device according to claim 1, comprising: [Appendix 5] an area designation unit configured to designate a cargo loading area in the loading space by a user's operation; the luggage overall image estimation unit is configured to estimate an overall image of luggage loaded in the loading area; 5. The loading space recognition device according to claim 1. [Appendix 6] the region designation unit is configured to designate, in response to a user operation, a determination exclusion region to be excluded from determination by the determination unit; the baggage overall image estimation unit is configured to estimate an overall image of the baggage loaded in the loading area by excluding the baggage loaded in the determination exclusion area; 6. The loading space recognition device according to claim 5. [Appendix 7] The vehicle further includes a detector attached to a structure having the loading space and detecting vibrations or sounds, the determination unit is configured to acquire the voxel data from the voxelization unit when the detection unit detects a certain level of shaking or sound, and estimate the amount of change in volume or movement of the luggage by comparing the acquired voxel data with the voxel data at the reference time. 7. The loading space recognition device according to any one of appendixes 1 to 6. [Appendix 8] The movement amount estimation unit is configured to correct the estimated movement amount to be smaller when the movement direction of the luggage is horizontal, or to correct the estimated movement amount to be larger when the movement direction of the luggage is vertical or diagonal, and to select the longest movement amount from the corrected movement amounts. 5. The loading space recognition device according to claim 3 or 4. [Appendix 9] The determination unit is further configured to determine whether or not there is a possibility of cargo collapse by comparing an amount of change between the voxel data other than the reference time and the previous other voxel data with an amount of change between the voxel data at the reference time and the voxel data other than the reference time. 9. The loading space recognition device according to any one of appendices 1 to 8. [Appendix 10] the determination unit is configured to output warning output instruction information when it determines that there is a possibility of collapse of the luggage; The loading space recognition device further includes a warning output unit configured to output a warning based on the warning output instruction information. 10. The loading space recognition device according to any one of appendices 1 to 9. [Appendix 11] a sensor that senses the surface of the luggage in the loading space and outputs the captured 3D data; A loading space recognition device according to any one of Supplementary Notes 1 to 10; A load space recognition system equipped with: [Appendix 12] A loading space recognition method for recognizing a loading space of a load using hardware resources, comprising: a step of estimating an overall image of the cargo loaded in the loading space based on three-dimensional data obtained by capturing an image of the loading space from a predetermined direction, and outputting the estimated result data; a step of converting the estimation result data into voxel data and outputting the voxel data; a step of estimating the amount of change in volume or movement of the cargo by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and determining whether or not there is a possibility of cargo collapse by comparing the estimated amount of change in volume or movement with a threshold value; A method for recognizing a load space, comprising: [Appendix 13] A program that causes hardware resources to execute a process for recognizing a luggage loading space, a process of estimating an overall image of the cargo loaded in the loading space based on three-dimensional data obtained by capturing an image of the loading space from a predetermined direction, and outputting the result of estimation as data; A process of converting the estimation result data into voxels and outputting it as voxel data; a process of estimating the amount of change in volume or movement of the cargo by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and determining whether or not there is a possibility of cargo collapse by comparing the estimated amount of change in volume or movement with a threshold value; A program that causes the hardware resource to execute the above.

[0100] The disclosures of the above-cited patent documents are incorporated herein by reference and may be used as the basis or part of the present invention, as necessary. Modifications and adjustments of the embodiments and examples are possible within the scope of the entire disclosure of the present invention (including the claims and drawings), and further based on the basic technical concept thereof. Furthermore, various combinations and selections (or non-selections, as necessary) of the various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of the entire disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concept, including the claims and drawings. Furthermore, with regard to the numerical values ​​and numerical ranges described in this application, any intermediate values, lower values, and smaller ranges are deemed to be included, even if not explicitly stated. Furthermore, the disclosures of the above-cited documents, when used in part or in whole in combination with the disclosures herein as part of the disclosure of the present invention, in accordance with the spirit of the present invention, are also deemed to be included in (belong to) the disclosures of this application. [Explanation of symbols]

[0101] 1. Load Space Recognition System 2 Track 3. Container 4. Luggage 5 Loading space 10 sensors 20 Detector 100 shooting data 101 Preprocessing data 102 voxel data 200 Load Space Recognition Device 210 Pretreatment section 211 Format Conversion Unit 212 Noise removal section 220 Packing status grasping unit 221 Area specification section 222 Baggage Overall Image Estimation Unit 223 Voxelization Department 230 Judgment section 231 Standard determination section 232 Difference extraction part 233 Fluctuation volume estimation unit 234 Cluster Extraction Unit 235 Combination Estimation Unit 236 Movement amount estimator 237 Load Collapse Judgment Unit 240 User Interface Section 241 Display section 242 Operation section 243 Warning Output Unit 1000 hardware resources 1001 processor 1002 memory 1003 Network Interface 1004 Internal Bus

Claims

1. a luggage overall image estimation unit configured to estimate an overall image of the luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the luggage loading space from one direction, and output the estimated result data; a voxelization unit configured to voxelize the estimation result data and output it as voxel data; a determination unit configured to estimate a volume change or movement amount of the cargo by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and to determine whether or not there is a possibility of cargo collapse by comparing the estimated volume change or movement amount with a threshold value; A loading space recognition device comprising:

2. The determination unit a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point; a difference extraction unit configured to compare the voxel data at the reference time with the voxel data at a point in time when a predetermined or arbitrary time has elapsed from the reference time, thereby extracting differences between voxels at corresponding positions on a surface of the baggage viewed from a predetermined position, and outputting the difference data; a volume fluctuation estimation unit configured to estimate a volume fluctuation of the entire image of the package based on the difference data and output the volume fluctuation estimation data; a cargo collapse determination unit configured to determine whether or not there is a possibility of cargo collapse by comparing the fluctuation volume amount estimation data with a threshold value for fluctuation volume amount as the threshold value; The load space recognition device according to claim 1 , comprising:

3. The determination unit a cluster extraction unit configured to extract clusters of voxels with no change in volume that exist between a volume-increasing voxel whose volume has increased and a volume-decreasing voxel whose volume has decreased, at the same horizontal position, based on the difference data, and output the cluster data; a combination estimation unit configured to estimate a combination of the volume-increasing voxels and the volume-decreasing voxels based on the difference data and output the combination data as combination data; a movement amount estimation unit configured to estimate at least one movement amount of luggage that has occurred between the reference time and a predetermined or arbitrary time period based on the difference data, the collection data, and the combination data, select the longest movement amount from the estimated movement amounts, estimate the selected movement amount as the longest movement amount, and output the longest movement amount estimated data; Furthermore, 3. The loading space recognition device according to claim 2, wherein the cargo collapse determination unit is configured to determine whether or not there is a possibility of cargo collapse by comparing the fluctuation volume amount estimation data with a threshold value for the fluctuation volume amount as the threshold value, and when it is determined that there is a possibility of cargo collapse, to determine whether or not there is a possibility of cargo collapse by comparing the longest movement amount estimation data with the threshold value for the longest movement amount as the threshold value.

4. The determination unit a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point; a difference extraction unit configured to compare the voxel data at the reference time with the voxel data at a point in time when a predetermined or arbitrary time has elapsed from the reference time, thereby extracting differences between voxels at corresponding positions on a surface of the baggage viewed from a predetermined position, and outputting the difference data; a cluster extraction unit configured to extract clusters of voxels with no change in volume that exist between a volume-increasing voxel whose volume has increased and a volume-decreasing voxel whose volume has decreased, at the same horizontal position, based on the difference data, and output the cluster data; a combination estimation unit configured to estimate a combination of the volume-increasing voxels and the volume-decreasing voxels based on the difference data and output the combination data as combination data; a movement amount estimation unit configured to estimate at least one movement amount of luggage that has occurred between the reference time and a predetermined or arbitrary time period based on the difference data, the collection data, and the combination data, select the longest movement amount from the estimated movement amounts, estimate the selected movement amount as the longest movement amount, and output the longest movement amount estimated data; a cargo collapse determination unit configured to determine whether or not there is a possibility of cargo collapse by comparing the longest movement amount estimation data with a threshold value for the longest movement amount as the threshold value; The load space recognition device according to claim 1 , comprising:

5. an area designation unit configured to designate a cargo loading area in the loading space by a user's operation; the luggage overall image estimation unit is configured to estimate an overall image of luggage loaded in the loading area; The loading space recognition device according to any one of claims 1 to 4.

6. The movement amount estimation unit is configured to correct the estimated movement amount to be smaller when the movement direction of the luggage is horizontal, or to correct the estimated movement amount to be larger when the movement direction of the luggage is vertical or diagonal, and to select the longest movement amount from the corrected movement amounts. The loading space recognition device according to claim 3 or 4.

7. The determination unit is further configured to determine whether or not there is a possibility of cargo collapse by comparing an amount of change between the voxel data other than the reference time and the previous other voxel data with an amount of change between the voxel data at the reference time and the voxel data other than the reference time. The loading space recognition device according to any one of claims 1 to 6.

8. a sensor that senses the surface of the luggage in the loading space and outputs three-dimensional image data; The loading space recognition device according to any one of claims 1 to 7, A load space recognition system equipped with:

9. A loading space recognition method for recognizing a loading space of a load using hardware resources, comprising: a step of estimating an overall image of the cargo loaded in the loading space based on three-dimensional data obtained by capturing an image of the loading space from one direction, and outputting the estimated result data; a step of converting the estimation result data into voxel data and outputting the voxel data; a step of estimating the amount of change in volume or movement of the cargo by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and determining whether or not there is a possibility of cargo collapse by comparing the estimated amount of change in volume or movement with a threshold value; A method for recognizing a load space, comprising:

10. A program that causes hardware resources to execute a process for recognizing a luggage loading space, a process of estimating an overall image of the cargo loaded in the loading space based on three-dimensional data obtained by capturing an image of the loading space from one direction, and outputting the result of estimation as estimation result data; A process of converting the estimation result data into voxels and outputting it as voxel data; a process of estimating the amount of change in volume or movement of the cargo by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and determining whether or not there is a possibility of cargo collapse by comparing the estimated amount of change in volume or movement with a threshold value; A program that causes the hardware resource to execute the above.

Images