Measuring device and measuring method

Abstract

This measurement device comprises an imaging unit for imaging a subject that includes a load and a pallet having a reference object, an image processing unit for extracting a specific region of the reference object and a load region on the basis of image data for the subject captured by the imaging unit, and a calculation unit for acquiring first coordinate information indicating 3D coordinates associated with the image data for the subject and standard load shape structure information indicating the structural dimensions of a standard pallet load shape associated with the coordinates of the specific region and using the specific region, load region, first coordinate information, and standard load shape structure information to calculate the protrusion length of the load protruding from the pallet. The image processing unit extracts the specific region of the reference object by inputting the image data for the subject into a first machine learning model resulting from machine learning using, as training data, pallet image data and specific-region-indicating data.

Description

[Technical Field]

[0001] The present disclosure relates to a measurement device and a measurement method. [Background technology]

[0002] Patent Document 1 discloses a device for detecting the position and orientation of a three-dimensional object.

[0003] The device described in Patent Document 1 uses a single camera to capture an image of a planar object under three types of light sources: red, green, and blue, and detects the three-dimensional position and orientation of the object from the image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-217671 Summary of the Invention

[0005] In recent years, there has been a demand to measure the length of cargo that protrudes from a pallet.

[0006] Therefore, an object of the present disclosure is to solve the above-mentioned problem by providing a measuring device and a measuring method that can measure the protrusion length of cargo placed on a pallet.

[0007] A measurement device according to one aspect of the present disclosure includes: an imaging unit that captures images of objects including a pallet having a reference object and luggage; an image processing unit that extracts a specific area of the reference object and an area of the luggage based on image data of the object captured by the imaging unit; a calculation unit that acquires first coordinate information indicating 3D coordinates associated with the image data and standard package structure information indicating structural dimensions of the standard package of the pallet associated with the coordinates of the specific area, and calculates the protrusion length of the package protruding from the pallet based on the specific area, the area of the package, the first coordinate information, and the standard package structure information; Equipped with The image processing unit obtains output data that extracts the specific area of the reference object by inputting image data of the object captured by the imaging unit as input data into a first machine learning model in which machine learning has been performed using image data of a pallet having the reference object and data indicating the specific area of the reference object as training data.

[0008] A measurement method according to one embodiment of the present disclosure includes: imaging an object including a pallet and a package with a reference object; extracting a specific region of the reference object based on image data of the captured object; extracting an area of the luggage based on image data of the captured object; A step of acquiring first coordinate information indicating 3D coordinates associated with the image data and standard packing structure information indicating structural dimensions of the standard packing of the pallet associated with the coordinates of the specific area; calculating an overhang length of the cargo that overhangs the pallet based on the specific area, the cargo area, the first coordinate information, and the standard packing structure information; Including, The step of extracting the specific area in the reference object includes obtaining output data in which the specific area in the reference object is extracted by inputting image data of the captured object as input data into a first machine learning model in which machine learning has been performed using image data of a pallet having the reference object and data indicating the specific area in the reference object as training data.

[0009] These general and specific aspects may be realized by a system, a method, a computer program, a computer-readable recording medium, and a combination thereof.

[0010] According to the present disclosure, it is possible to provide a measuring device and a measuring method that can measure the protruding length of cargo that protrudes from a pallet. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic block diagram showing an example of the configuration of a measurement device according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram illustrating an example of a pallet. [Figure 3] 1 is a schematic diagram illustrating an example of imaging an object including a pallet and luggage using a measuring device. FIG. [Figure 4] FIG. 2 is a schematic diagram showing an example of 3D coordinate information acquired by the measurement device. [Figure 5] 1 is a schematic block diagram showing an example of a functional configuration of a measurement device according to a first embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram illustrating an example of processing by an image processing unit using a first machine learning model. [Figure 7] FIG. 10 is a schematic diagram illustrating an example of a specific region. [Figure 8] FIG. 10 is a schematic diagram illustrating an example of processing by an image processing unit using a second machine learning model. [Figure 9] FIG. 10 is a schematic diagram showing an example of standard packaging structure information. [Figure 10] FIG. 10 is a schematic plan view of the standard packaging structure information of FIG. 9. [Figure 11] 10 is a schematic diagram illustrating an example of a calculation process of a protruding length of a package by a calculation unit. FIG. [Figure 12] 10 is a schematic diagram illustrating an example of a calculation process of a protruding length of a package by a calculation unit. FIG. [Figure 13] 3 is a flowchart illustrating an example of a measurement method according to the first embodiment of the present disclosure. [Figure 14] 10 is a flowchart illustrating an example of processing by a calculation unit. [Figure 15] FIG. 10 is a schematic diagram illustrating an example of a protruding plane. [Figure 16] 10 is a flowchart illustrating an example of a process for calculating a plane equation of a protruding plane. [Figure 17] FIG. 2 is a schematic diagram illustrating the first plane of a standard packaging style. [Figure 18] 10A and 10B are schematic diagrams illustrating an example of a process for calculating a reference point and a vector in a specific area. [Figure 19] FIG. 10 is a schematic diagram illustrating an example of calculation of a protruding plane. [Figure 20] 10 is a flowchart illustrating an example of a process for calculating an overhang length of a package. DETAILED DESCRIPTION OF THE INVENTION

[0012] (Background to this disclosure) Pallets are used as loading platforms for cargo in logistics and other fields. For example, a post pallet with supports can be used. By aligning the horizontal positions of the pallet supports and stacking them vertically, multiple pallets can be stacked side by side both horizontally and vertically. This allows pallets to be efficiently arranged in limited spaces such as trucks and warehouses.

[0013] Here, if the dimensions of the rectangular parallelepiped that surrounds the outer shape of the target post pallet are taken as the standard package dimensions and the pallets are loaded horizontally and vertically at intervals appropriate to the standard package dimensions, if a pallet is loaded with long cargo that exceeds the standard package dimensions and extends beyond the standard package dimensions, the pallets cannot be placed adjacent to each other due to interference from the cargo, reducing loading efficiency.

[0014] Furthermore, since post pallets, which are loaded one above the other, require the horizontal positions of their supports to be aligned, even if the overall width of the pallet and cargo is the same, the extent to which cargo that protrudes from the standard package dimensions interferes will vary depending on the loading position on the pallet. To improve the efficiency of storing and transporting such pallets, it is necessary to measure the protruding length of cargo for each specific orientation, such as the right side, and a device that can easily measure protruding length is needed.

[0015] Therefore, in order to solve the above problem, the inventors discovered a configuration that easily measures the protruding length of luggage by imaging a pallet having a reference object and an object including luggage, and arrived at the following disclosure.

[0016] A measuring device of a first aspect of the present disclosure comprises an imaging unit that images an object including a pallet having a reference object and luggage; an image processing unit that extracts a specific area of the reference object and an area of the luggage based on image data of the object imaged by the imaging unit; and a calculation unit that acquires first coordinate information indicating 3D coordinates associated with the image data and standard packaging structure information indicating the structural dimensions of the standard packaging of the pallet associated with the coordinates of the specific area, and calculates the protrusion length of the luggage protruding from the pallet based on the specific area, the area of the luggage, the first coordinate information, and the standard packaging structure information.The image processing unit acquires output data in which the specific area of the reference object is extracted by inputting the image data of the object imaged by the imaging unit as input data into a first machine learning model in which machine learning has been performed using image data of the pallet having the reference object and data indicating the specific area of the reference object as training data.

[0017] In the measurement device of the second aspect of the present disclosure, the reference object may have a plurality of vertices that define the specific region, and the first machine learning model may detect the plurality of vertices in the image data.

[0018] In the measurement device according to the third aspect of the present disclosure, the first machine learning model may be KeyPointDetection using Mask R-CNN.

[0019] In the measurement device according to the fourth aspect of the present disclosure, the reference object may have at least one reference plane, and the image processing unit may extract the specific region based on the at least one reference plane.

[0020] In the measurement device according to the fifth aspect of the present disclosure, the at least one reference plane may be a plurality of reference planes, and the plurality of reference planes may be arranged apart from each other on the same plane.

[0021] In the measuring device of the sixth aspect of the present disclosure, the pallet may have a bottom plate and a plurality of pillars provided on the bottom plate, and the reference object may be at least one of the plurality of pillars.

[0022] In the measuring device of the seventh aspect of the present disclosure, the multiple pillars may be arranged along the outer edge of the bottom plate, and at least one of the pillars may have a plane on the outer edge side of the bottom plate that serves as a reference plane, and the image processing unit may extract the specific area based on the plane.

[0023] In the measuring device of the eighth aspect of the present disclosure, the image processing unit may obtain output data that extracts the area of the luggage by inputting image data of the target object as input data into a second machine learning model in which machine learning has been performed using image data of the luggage placed on the pallet and data indicating the area of the luggage as training data.

[0024] In the measurement device of the ninth aspect of the present disclosure, the second machine learning model is an Instance Segmentation may be.

[0025] In the measurement device according to the tenth aspect of the present disclosure, the imaging section may acquire image data of the object and the first coordinate information by capturing an image of the object.

[0026] In the measuring device of the eleventh aspect of the present disclosure, the calculation unit may acquire second coordinate information indicating the 3D coordinates of the specific area based on the specific area and the first coordinate information, acquire third coordinate information indicating the 3D coordinates of the standard packaging defined by the specific area based on the second coordinate information and the standard packaging structure information, acquire fourth coordinate information indicating the 3D coordinates of the luggage area based on the luggage area and the first coordinate information, and calculate the protrusion length of the luggage based on the third coordinate information and the fourth coordinate information.

[0027] In the measuring device of the twelfth aspect of the present disclosure, the calculation unit may acquire the third coordinate information including at least a plane equation of a protrusion plane that serves as a reference for the protrusion of the luggage in the standard packaging style, and calculate the protrusion length of the luggage based on the plane equation of the protrusion plane and the fourth coordinate information.

[0028] In the measuring device of the thirteenth aspect of the present disclosure, the calculation unit may calculate, based on the second coordinate information, a plane equation of a first plane of the standard packaging on the side where the reference object including the specific area is placed, calculate, based on the second coordinate information, a plurality of reference points in the specific area and a vector calculated from the plurality of reference points, and calculate a plane equation of the protruding plane based on the plane equation of the first plane, at least one of the plurality of reference points, the vector, and the standard packaging structure information.

[0029] In the measuring device of the fourteenth aspect of the present disclosure, the calculation unit may acquire the third coordinate information including the 3D coordinates of two opposing planes in the standard packaging, calculate the area sandwiched between the two planes from the 3D coordinates of the two planes, and calculate the protruding length of the luggage in the area sandwiched between the two planes.

[0030] In the measuring device of the fifteenth aspect of the present disclosure, the calculation unit may process, for luggage that protrudes from the pallet, an area in which luggage exists that continues from the area of the standard packaging style as the area of the luggage.

[0031] In the measurement device of the sixteenth aspect of the present disclosure, the calculation unit may calculate a maximum protruding length among the protruding lengths of the luggage.

[0032] The measurement device according to the seventeenth aspect of the present disclosure may further include an output section that outputs the protrusion length.

[0033] A measurement method of an 18th aspect of the present disclosure includes the steps of: capturing an image of an object including a pallet having a reference object and luggage; extracting a specific area of the reference object based on image data of the captured object; extracting the area of the luggage based on the image data of the captured object; acquiring first coordinate information indicating 3D coordinates associated with the image data and standard packaging structure information indicating the structural dimensions of the standard packaging of the pallet associated with the coordinates of the specific area; and calculating the protrusion length of the luggage protruding from the pallet based on the specific area, the area of the luggage, the first coordinate information, and the standard packaging structure information, wherein the step of extracting the specific area of the reference object includes inputting the image data of the captured object as input data into a first machine learning model that has been subjected to machine learning using image data of the pallet having the reference object and data indicating the specific area of the reference object as training data, thereby obtaining output data in which the specific area of the reference object has been extracted.

[0034] A program according to a nineteenth aspect of the present disclosure causes the measurement method according to the aforementioned aspect to be executed.

[0035] An embodiment of the present disclosure will be described below with reference to the accompanying drawings. Note that the following description is merely illustrative in nature and is not intended to limit the present disclosure, its applications, or its uses. Furthermore, the drawings are schematic, and the ratios of the dimensions and the like do not necessarily correspond to reality.

[0036] (Embodiment 1) [Overall configuration] 1 is a schematic block diagram showing an example of the configuration of a measurement device 1 according to a first embodiment of the present disclosure. As shown in FIG. 1, the measurement device 1 includes an imaging unit 10, a storage unit 20, a control unit 30, and an output unit 40. For example, the measurement device 1 is a tablet computer. Note that, in the first embodiment, an example will be described in which the measurement device 1 includes the output unit 40, but the output unit 40 is not an essential component.

[0037] <Image capture unit> The imaging unit 10 captures an image of an object as a subject, thereby acquiring image data of the object and 3D coordinate information associated with the image data. The image data is color image data. The 3D coordinate information associated with the image data is information on 3D coordinates corresponding to each pixel of the image data. In this specification, the "3D coordinate information associated with the image data" may be referred to as "first coordinate information."

[0038] The imaging unit 10 is, for example, a depth camera. The depth camera measures the distance to an object and generates depth information indicating the measured distance as a depth value for each pixel. For example, the depth camera may be an infrared active stereo camera, a LiDAR depth camera, or the like. Note that the imaging unit 10 is not limited to these depth cameras.

[0039] <Storage section> The storage unit 20 is a storage medium that stores programs and data necessary to realize the functions of the measurement device 1. For example, the storage unit 20 can be realized by a hard disk (HDD), a solid state drive (SSD), a random access memory (RAM), a dynamic RAM (DRAM), a ferroelectric memory, a flash memory, a magnetic disk, or a combination of these.

[0040] <Control unit> The control unit 30 can be realized by semiconductor elements or the like. For example, the control unit 30 can be configured by a microcomputer, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit). The functions of the control unit 30 may be configured by hardware alone, or may be realized by combining hardware and software.

[0041] The control unit 30 reads out data and programs stored in the storage unit 20 and performs various arithmetic processing to realize predetermined functions.

[0042] <Output section> The output unit 40 has a display unit that displays the results of the arithmetic processing of the control unit 30. For example, the display unit may be configured as a liquid crystal display or an organic EL display. The output unit 40 may also include a speaker that emits sound.

[0043] Fig. 2 is a schematic diagram illustrating an example of a pallet 2. As shown in Fig. 2, the pallet 2 has a bottom plate 2a and a plurality of support posts 2b provided on the bottom plate 2a. The bottom plate 2a is formed of a rectangular plate member having a longitudinal direction and a lateral direction in a plan view. The plurality of support posts 2b extend upward from the bottom plate 2a and are arranged spaced apart from one another along the outer edge of the bottom plate 2a.

[0044] In the first embodiment, the pallet 2 has four support columns 2b. Specifically, two support columns 2b are arranged at one end of the bottom plate 2a in the short direction, and two support columns 2b are arranged at the other end of the bottom plate 2a in the short direction. The two support columns 2b arranged at one end of the bottom plate 2a in the short direction are arranged along the outer edge of the bottom plate 2a and are spaced apart from each other in the longitudinal direction of the bottom plate 2a. The two support columns 2b arranged at the other end of the bottom plate 2a in the short direction are arranged along the outer edge of the bottom plate 2a and are spaced apart from each other in the longitudinal direction of the bottom plate 2a. The shapes and dimensions of the multiple support columns 2b are approximately identical. In this specification, "approximately" means that the error is, for example, 5% or less. Preferably, "approximately" means that the error is 1% or less.

[0045] In the measuring device 1, two support posts 2b arranged on one end of the bottom plate 2a in the short direction are used as the reference object 3. The "reference object" is a reference object whose positional relationship with the standard packing shape 4 of the pallet 2 is uniquely determined. The "standard packing shape" is the standard appearance of cargo placed on the pallet 2 during transportation. The standard packing shape 4 has a specified size that prevents cargo from protruding from the pallet 2. For example, the standard packing shape 4 is a rectangular parallelepiped and is determined by the width W1, depth D1, and height H1 of the pallet 2. In the first embodiment, the width W1 is the length of the bottom plate 2a in the longitudinal direction, the depth D1 is the length of the bottom plate 2a in the short direction, and the height H1 is the sum of the thickness of the bottom plate 2a and the height of the support posts 2b. The width W1, depth D1, and height H1 are predetermined standard dimensions.

[0046] The reference object 3 has at least one reference plane 3a. Preferably, the reference object 3 has multiple reference planes 3a arranged spaced apart from each other on the same plane. In embodiment 1, the reference object 3 has two reference planes 3a provided on two supports 2b arranged along the outer edge of one end of the bottom plate 2a in the short direction. The two reference planes 3a are arranged spaced apart from each other on the same plane. That is, the two reference planes 3a are arranged on the outer edge side of one end of the bottom plate 2a in the short direction, and are arranged spaced apart from each other in the longitudinal direction of the bottom plate 2a.

[0047] In the first embodiment, the support pillar 2b is a quadrangular pillar, and the reference plane 3a has a rectangular shape with four vertices.

[0048] The reference object 3 is not limited to the two support columns 2b arranged along the outer edge of one end of the bottom plate 2a in the short direction. For example, the reference object 3 may be a part of the upper structure other than the support columns of the pallet, or may be at least one of the multiple support columns 2b or a part thereof.

[0049] 3 is a schematic diagram illustrating an example of imaging an object 6 including a pallet 2 and luggage 5 by the measuring device 1. As shown in FIG. 3, the object 6 includes the pallet 2 and the luggage 5 placed on the pallet 2. The imaging unit 10 generates 3D coordinate information (first coordinate information) by measuring a distance d1 from the imaging unit 10 to the object 6 using the position of the imaging unit 10 as a reference position and generating depth image data.

[0050] Fig. 4 is a schematic diagram showing an example of 3D coordinate information (first coordinate information) acquired by the measuring device 1. As shown in Fig. 4, the 3D coordinate information is depth image data indicating a depth value for each pixel specified by 2D coordinates (X, Y). In the first embodiment, in order to measure the protrusion length of the luggage 5 protruding from the pallet 2, the imaging unit 10 captures an image of the target object 6 so that the entire pallet 2 and luggage 5 are captured, as shown in Figs. 3 and 4.

[0051] Next, the functional configuration of the measurement device 1 will be described in detail with reference to Fig. 5. Fig. 5 is a schematic block diagram showing an example of the functional configuration of the measurement device 1 according to the first embodiment of the present disclosure. As shown in Fig. 5, the control unit 30 has an image processing unit 31 and a calculation unit 32.

[0052] The imaging unit 10 captures an image of an object 6 including a pallet 2 and luggage 5, thereby acquiring image data 11 and first coordinate information 12, which is 3D coordinate information associated with the image data 11. The image data 11 and the first coordinate information 12 are stored in the storage unit 20.

[0053] The image processing unit 31 extracts a specific region in the reference object 3 based on image data 11 of the object 6 captured by the imaging unit 10. Specifically, the image processing unit 31 extracts the specific region in the reference object 3 using the first machine learning model 21.

[0054] FIG. 6 is a schematic diagram illustrating an example of processing by the image processing unit 31 using the first machine learning model 21. As shown in FIG. 6, the image processing unit 31 inputs image data 11 to the first machine learning model 21 as input data. The first machine learning model 21 outputs data in which a specific region R1 is extracted from the image data 11. The image processing unit 31 acquires data in which the specific region R1 is extracted from the first machine learning model 21. The "data in which the specific region R1 is extracted" is, for example, data including coordinates representing pixels occupied by the specific region R1 in the image data 11.

[0055] The specific region R1 is a 2D planar region included in the reference object 3. The image processing unit 31 extracts the specific region R1 based on at least one reference plane 3a. In the first embodiment, the reference object 3 is two support posts 2b arranged along the outer edge of the bottom plate 2a of the pallet 2 at one end of the bottom plate 2a in the short direction. The two support posts 2b have a plane that serves as the reference plane 3a on the outer edge side of the bottom plate 2a. The image processing unit 31 extracts the specific region R1 based on the plane. Specifically, the image processing unit 31 extracts the entire reference plane 3a of each of the two support posts 2b (reference object 3) from the object 6 captured in the image data 11 as the specific region R1. That is, the image processing unit 31 detects two reference planes 3a from the object 6 captured in the image data 11 and extracts the two detected reference planes 3a as two specific regions R1.

[0056] The specific region R1 is not limited to the entire reference plane 3a. For example, the specific region R1 may be a part of the reference plane 3a rather than the entire reference plane 3a. Alternatively, the specific region R1 may be a portion of the reference plane 3a that is defined by a specific color, a specific pattern, and / or specific characters.

[0057] The first machine learning model 21 is stored in the storage unit 20. In the first machine learning model 21, machine learning is performed using, as training data, image data of the pallet 2 having the reference object 3 and data indicating a specific region R1 in the reference object 3. The "data indicating the specific region R1 in the reference object 3" is data in which the specific region R1 is labeled. In the first embodiment, the data indicating the specific region R1 in the reference object 3 is data indicating the entire reference plane 3a of the reference object 3.

[0058] In the first embodiment, the first machine learning model 21 is KeyPointDetection using MaskR-CNN. "KeyPointDetection" is a technology for detecting multiple coordinate points from input image data. In this embodiment, the coordinate points of vertices are detected, and an area defined by the coordinate points is extracted.

[0059] Note that the first machine learning model 21 is not limited to KeyPointDetection using Mask R-CNN, and may be, for example, DeepPose, InstanceSegmentation, or the like.

[0060] The reference object 3 has a reference plane 3a, i.e., a plurality of vertices that define a specific region R1. The first machine learning model 21 detects the plurality of vertices in the image data 11 and extracts a 2D region enclosed by the detected plurality of vertices as the specific region R1.

[0061] FIG. 7 is a schematic diagram illustrating an example of the specific region R1. FIG. 7 shows an example of extracting a rectangular reference plane 3a as the specific region R1. As shown in FIG. 7, the reference object 3 has four vertices P1 to P4 that define the rectangular reference plane 3a. The first machine learning model 21 uses data indicating the four vertices P1 to P4 that define the rectangular reference plane 3a as the "data indicating the specific region R1 in the reference object 3" of the training data. As a result, when image data 11 is input, the first machine learning model 21 detects the four vertices P1 to P4 that define the reference plane 3a and extracts the region surrounded by the detected four vertices P1 to P4 as the specific region R1.

[0062] The first machine learning model 21 may be updated by further machine learning using the image data 11 captured by the imaging unit 10 as training data.

[0063] Furthermore, the image processing unit 31 extracts the area of the luggage 5 based on the image data 11 of the object 6 captured by the imaging unit 10. Specifically, the image processing unit 31 extracts the area of the luggage 5 using the second machine learning model 22.

[0064] FIG. 8 is a schematic diagram illustrating an example of processing by the image processing unit 31 using the second machine learning model 22. As shown in FIG. 8, the image processing unit 31 inputs image data 11 as input data to the second machine learning model 22. The second machine learning model 22 outputs data in which a region R2 of the luggage 5 is extracted from the image data 11. The image processing unit 31 acquires the data in which the region R2 of the luggage 5 is extracted from the second machine learning model 22. The "data in which the region R2 of the luggage 5 is extracted" is, for example, data including coordinates representing pixels occupied by the region R2 of the luggage 5 in the image data 11.

[0065] The second machine learning model 22 is stored in the memory unit 20. In the second machine learning model 22, machine learning is performed using, as training data, image data 11 of the luggage 5 placed on the pallet 2 and data indicating the area R2 of the luggage 5. The "data indicating the area R2 of the luggage 5" is data in which the area R2 of the luggage 5 is labeled.

[0066] In the first embodiment, the second machine learning model 22 is an instance model using the Mask R-CNN model. Segmentation "Instance Segmentation " is a technology that classifies objects and extracts object regions at the pixel level.

[0067] The second machine learning model 22 is an instance model using the Mask R-CNN model. Segmentation The second machine learning model 22 may be, for example, DeepMask, SemanticSegmentation, or the like.

[0068] The second machine learning model 22 may be updated by further machine learning using the image data 11 captured by the imaging unit 10 as training data.

[0069] Information on the specific area R1 extracted by the image processing unit 31 and the area R2 of the baggage 5 is sent to the calculation unit 32.

[0070] 5, the calculation unit 32 acquires first coordinate information 12 indicating 3D coordinates associated with the image data 11, and standard packing structure information 23 indicating the structural dimensions of the standard packing style 4 of the pallet 2 associated with the coordinates of the specific region R1. The calculation unit 32 also calculates the protrusion length L1 of the package 5 protruding from the pallet 2 based on the specific region R1, the region R2 of the package 5, the first coordinate information 12, and the standard packing structure information 23.

[0071] The first coordinate information 12 and the standard packing structure information 23 are stored in the memory unit 20. The calculation unit 32 acquires the first coordinate information 12 and the standard packing structure information 23 from the memory unit 20. The calculation unit 32 also acquires information on a specific region R1 and a region R2 of the package 5 from the image processing unit 31.

[0072] The calculation unit 32 acquires information about the 3D coordinates of the specific region R1 based on the information about the specific region R1 acquired from the image processing unit 31 and the first coordinate information 12. The first coordinate information 12 is information about the 3D coordinates corresponding to the image data 11. Therefore, the calculation unit 32 can easily acquire, from the first coordinate information 12, the 3D coordinates corresponding to the specific region R1 extracted from the image data 11 by the image processing unit 31. In this specification, the information about the 3D coordinates of the specific region R1 acquired based on the information about the specific region R1 acquired from the image processing unit 31 and the first coordinate information 12 may be referred to as "second coordinate information."

[0073] Furthermore, the calculation unit 32 acquires 3D coordinate information of the standard packaging style 4 defined by the specific region R1 based on the 3D coordinates (second coordinate information) of the specific region R1 and the standard packaging style structure information 23. In this specification, the 3D coordinate information of the standard packaging style 4 acquired based on the second coordinate information and the standard packaging style structure information 23 may be referred to as "third coordinate information."

[0074] The standard packaging structure information 23 is information indicating the structural dimensions of the standard packaging style 4 and the positional relationship between the specific region R1 and the standard packaging style 4. FIG. 9 is a schematic diagram showing an example of the standard packaging structure information 23. FIG. 10 is a schematic plan view of the standard packaging structure information 23 of FIG. 9. As shown in FIGS. 9 and 10, the standard packaging structure information 23 includes information on the position (3D coordinates) of the specific region R1 and information on the structural dimensions (width W1, depth D1, and height H1) of the standard packaging style 4 associated with the position (3D coordinates) of the specific region R1. The "information on the structural dimensions of the standard packaging style 4 associated with the position (3D coordinates) of the specific region R1" refers to information on the structural dimensions of the standard packaging style 4 whose positional relationship with the specific region R1 is uniquely determined. Note that the standard packaging structure information 23 may include relative position information of at least one of the six faces constituting the standard packaging style 4, with respect to the face located on the specific region R1 side. Specifically, the standard packaging structure information 23 only needs to include relative position information of the surface for calculating the protruding length L1 of the package 5 relative to the surface located on the specific region R1 side of the standard packaging 4. For example, when calculating the protruding length L1 of the package 5 from the plane on the right side of the standard packaging 4, the standard packaging structure information 23 may include relative position information of the plane on the right side of the standard packaging 4 relative to the surface (front side) located on the specific region R1 side. Alternatively, the standard packaging structure information 23 may include relative position information of the left side, front / back, and / or top / bottom.

[0075] The third coordinate information is 3D coordinate information of the standard packaging style 4 acquired based on the second coordinate information and the standard packaging structure information 23, and may include 3D coordinate information of at least one of the six planes of the standard packaging style 4. Specifically, the third coordinate information may include 3D coordinate information of at least the plane of the standard packaging style 4 for which the protrusion length L1 of the package 5 is to be calculated. For example, when calculating the protrusion length L1 of the package 5 from the plane on the right side of the standard packaging style 4, the third coordinate information may include 3D coordinate information of the plane on the right side of the standard packaging style 4. When calculating the protrusion length L1 of the package 5 from two planes on the right and left sides of the standard packaging style 4, the third coordinate information may include 3D coordinate information of the two planes on the right and left sides of the standard packaging style 4. Alternatively, the third coordinate information may include 3D coordinate information of the six planes of the standard packaging style 4.

[0076] Furthermore, the calculation unit 32 acquires information on the 3D coordinates of the region R2 of the package 5 based on the information on the region R2 of the package 5 acquired from the image processing unit 31 and the first coordinate information 12. The first coordinate information 12 is information on 3D coordinates corresponding to the image data 11. Therefore, the calculation unit 32 can easily acquire the 3D coordinates of the region R2 of the package 5 extracted from the image data 11 by the image processing unit 31 from the first coordinate information 12. In this specification, the information on the 3D coordinates of the region R2 of the package 5 acquired based on the information on the region R2 of the package 5 acquired from the image processing unit 31 and the first coordinate information 12 may be referred to as "fourth coordinate information."

[0077] The calculation unit 32 also calculates the protruding length L1 of the package 5 based on the 3D coordinate information (third coordinate information) of the standard packaging style 4 and the 3D coordinate information (fourth coordinate information) of the region R2 of the package 5.

[0078] The calculation unit 32 calculates the protrusion length L1 for each pixel in the region R2 of the package 5. Figures 11 and 12 are schematic diagrams illustrating an example of the calculation process of the protrusion length L1 of the package 5 by the calculation unit 32. Figure 11 is a 3D image diagram, and Figure 12 is a 2D image diagram. For ease of explanation, the example shown in Figures 11 and 12 focuses on the package 5 protruding to the right side of the pallet 2. Figures 11 and 12 show the protrusion region R3 of the package 5 protruding from the standard packing style 4. As shown in Figures 11 and 12, the calculation unit 32 calculates the length (protrusion length) of the protrusion region R3 of the package 5 protruding from the standard packing style 4 in the region R2 of the package 5.

[0079] The calculation unit 32 also calculates the maximum protrusion length Lmax of the protrusion length L1. That is, the calculation unit 32 calculates the maximum protrusion length Lmax of the portion of the protrusion length L1 of the package 5 that protrudes the most from the standard packing style 4. The maximum protrusion length Lmax and / or the protrusion length L1 calculated by the calculation unit 32 are stored in the memory unit 20. The calculation unit 32 transmits information on the maximum protrusion length Lmax to the output unit 40.

[0080] The output unit 40 acquires information on the maximum overhang length Lmax from the calculation unit 32 and outputs the information on the maximum overhang length Lmax.

[0081] [Operation] Next, the operation of the measurement device 1, i.e., the measurement method performed by the measurement device 1, will be described with reference to Fig. 13. Fig. 13 is a flowchart of an example of the measurement method according to the first embodiment of the present disclosure. Each step in the flowchart of the measurement method shown in Fig. 13 is performed by the measurement device 1. As shown in Fig. 13, the measurement method includes steps ST1 to ST6.

[0082] In step ST1, the imaging unit 10 captures an image of the object 6 including the pallet 2 having the reference object 3 and the luggage 5. By capturing an image of the object 6, the imaging unit 10 acquires image data 11, which is a color image of the object 6, and first coordinate information 12 (see FIG. 4 ) indicating 3D coordinates associated with the image data 11. The acquired image data 11 and first coordinate information 12 are stored in the storage unit 20.

[0083] In step ST2, the image processing unit 31 extracts a specific region R1 in the reference object 3 based on the image data 11 of the captured object 6. The image processing unit 31 inputs the image data 11 of the captured object 6 as input data to the first machine learning model 21, thereby obtaining output data in which the specific region R1 in the reference object 3 is extracted (see FIG. 6).

[0084] Specifically, the image processing unit 31 extracts the reference plane 3a of the reference object 3 as the specific region R1. In the first embodiment, the reference object 3 is two support posts 2b arranged on one end side of the bottom plate 2a in the short direction, and therefore the reference planes 3a are two rectangular planes on the two support posts 2b that are respectively arranged on the outer edge side of one end side of the bottom plate 2a of the pallet 2 in the short direction (see FIG. 2).

[0085] As described above, the first machine learning model 21 is KeyPointDetection using MaskR-CNN. When image data 11 is input, the first machine learning model 21 detects four vertices that define a rectangular reference plane 3a in the reference object 3. The image processing unit 31 extracts a specific region R1 from the four vertices detected by the first machine learning model 21. Information on the extracted specific region R1 is transmitted to the calculation unit 32.

[0086] In step ST3, the image processing unit 31 extracts the region R2 of the luggage 5 based on the image data 11 of the captured object 6. The image processing unit 31 inputs the image data 11 of the captured object 6 as input data to the second machine learning model 22, thereby obtaining output data in which the region R2 of the luggage 5 is extracted (see FIG. 8).

[0087] As described above, the second machine learning model 22 is an instance model using the Mask R-CNN model. Segmentation When the image data 11 is input, the second machine learning model 22 extracts the region R2 of the package 5. Information on the extracted region R2 of the package 5 is transmitted to the calculation unit 32.

[0088] In step ST4, the calculation unit 32 acquires the first coordinate information 12 and the standard packaging style structure information 23 (see Figures 9 and 10). The first coordinate information 12 is information indicating 3D coordinates associated with the image data 11, and the standard packaging style structure information 23 is information indicating the structural dimensions of the standard packaging style 4 of the pallet 2 associated with the position (3D coordinates) of the specific region R1. In the first embodiment, the first coordinate information 12 is acquired by the imaging unit 10 and stored in the memory unit 20. Furthermore, the standard packaging style structure information 23 is stored in advance in the memory unit 20. Therefore, the calculation unit 32 acquires the first coordinate information 12 and the standard packaging style structure information 23 from the memory unit 20.

[0089] In step ST5, the calculation unit 32 calculates the protruding length L1 of the cargo 5 protruding from the pallet 2 based on the specific area R1, the area R2 of the cargo 5, the first coordinate information 12, and the standard packing structure information 23 (see Figures 11 and 12).

[0090] The calculation unit 32 transmits information on the maximum overhang length Lmax of the calculated overhang lengths L1 to the output unit 40.

[0091] In step ST6, the output section 40 outputs the maximum protrusion length Lmax.

[0092] In this way, the measuring device 1 can measure the protruding length of the package 5 from the pallet 2 by performing steps ST1 to ST6. Note that step ST6 is not an essential component of the above-described measuring method. For example, if the measuring device 1 does not have the output unit 40, the measuring method does not need to include step ST6.

[0093] An example of the processing by the calculation unit 32 in step ST5 for calculating the overhang length L1 will be described in detail with reference to Fig. 14. Fig. 14 is a flowchart illustrating an example of the processing by the calculation unit 32. As shown in Fig. 14, the calculation unit 32 performs steps ST11 to ST14 in the processing for calculating the overhang length L1 (step ST5).

[0094] In step ST11, the calculation unit 32 acquires second coordinate information indicating the 3D coordinates of the specific region R1 based on the specific region R1 and the first coordinate information 12. Specifically, the calculation unit 32 acquires the 3D coordinates (second coordinate information) of the specific region R1 based on the information of the specific region R1 extracted in step ST2 and the first coordinate information 12 acquired in step ST4. The first coordinate information 12 is information on 3D coordinates corresponding to the image data 11 (see FIG. 4). The calculation unit 32 acquires the 3D coordinates corresponding to the extracted specific region R1 from the first coordinate information 12.

[0095] In step ST12, the calculation unit 32 acquires third coordinate information indicating the 3D coordinates of the standard packaging style 4 defined by the specific region R1 based on the second coordinate information and the standard packaging style structure information 23. Specifically, the calculation unit 32 acquires the 3D coordinates (third coordinate information) of the standard packaging style 4 defined by the specific region R1 based on the second coordinate information acquired in step ST11 and the standard packaging style structure information 23 acquired in step ST4. The standard packaging style structure information 23 is information indicating the structural dimensions of the standard packaging style 4 and the positional relationship between the specific region R1 and the standard packaging style 4, and includes information on the coordinates of the specific region R1 and information on the structural dimensions of the standard packaging style 4 associated with the coordinates of the specific region R1 (see FIGS. 9 and 10). The calculation unit 32 determines the 3D coordinates of the standard packaging style 4 based on the 3D coordinates of the specific region R1.

[0096] In the first embodiment, the third coordinate information is information on the 3D coordinates of a protrusion plane that serves as a reference for the protrusion of the package 5 in the standard packing style 4. The protrusion plane PL0 is at least one plane determined from the six planes of the standard packing style 4, and is determined in advance. For example, when calculating the protrusion length L1 of the package 5 from the plane on the right side of the standard packing style 4, the plane on the right side of the standard packing style 4 is determined in advance as the protrusion plane PL0. The 3D coordinate information of the protrusion plane PL0 can be calculated, for example, by a plane equation. The plane equation will be described later.

[0097] In step ST13, the calculation unit 32 acquires fourth coordinate information indicating the 3D coordinates of the region R2 of the package 5 based on the region R2 of the package 5 and the first coordinate information 12. Specifically, the calculation unit 32 calculates the 3D coordinates (fourth coordinate information) of the region R2 of the package 5 based on the information of the region R2 of the package 5 acquired in step ST3 and the first coordinate information 12 acquired in step ST4. The first coordinate information 12 is information on 3D coordinates corresponding to the image data 11 (see FIG. 4). The calculation unit 32 acquires the 3D coordinates corresponding to the extracted region R2 of the package 5 from the first coordinate information 12.

[0098] In step ST14, the calculation unit 32 calculates the protrusion length L1 of the package 5 based on the third coordinate information and the fourth coordinate information. Specifically, the calculation unit 32 calculates the protrusion length L1 of the package 5 based on the third coordinate information indicating the 3D coordinates of the standard packing style 4 acquired in step ST12 and the fourth coordinate information indicating the 3D coordinates of the area R2 of the package 5 acquired in step ST13. Specifically, the calculation unit 32 calculates the protrusion length L1 of the package 5 based on the plane equation (third coordinate information) of the protrusion plane PL0, which is the basis for the protrusion of the package 5 in the standard packing style 4, and the fourth coordinate information.

[0099] An example of calculating the plane equation of the protruding plane PL0 will be described. Fig. 15 is a schematic diagram illustrating an example of the protruding plane PL0. Fig. 15 shows an example in which the plane on the right side of the standard packing style 4 is the protruding plane PL0. In the example shown in Fig. 15, the plane on the right side of the standard packing style 4 is predetermined as the protruding plane PL0.

[0100] An example of the calculation process of the plane equation of the protruding plane PL0 will be described with reference to Fig. 16. Fig. 16 is a flowchart of an example of the calculation process of the plane equation of the protruding plane PL0. As shown in Fig. 16, the calculation process of the plane equation of the protruding plane PL0 includes steps ST21 to ST23. Steps ST21 to ST23 are executed by the calculation unit 32.

[0101] In step ST21, the calculation unit 32 calculates a plane equation of a first plane of the standard packing style 4 on the side where the reference object 3 including the specific region R1 is placed, based on the second coordinate information. Fig. 17 is a schematic diagram illustrating the first plane PL1 of the standard packing style 4. In the example of Fig. 17, the first plane PL1 is the plane on the side where the reference object 3 including the specific region R1 is placed in the standard packing style 4. In the example shown in Fig. 17, the plane on the side where the reference object 3 is placed is the plane on the front side of the standard packing style 4.

[0102] In the first embodiment, the second coordinate information is information indicating the 3D coordinates of two specific regions R1 that are arranged apart on the same plane. The calculation unit 32 estimates a plane equation of the first plane PL1 based on the 3D coordinates of the two specific regions R1.

[0103] For example, the plane equation of a specific region R1 can be calculated as a 3D point cloud existing in the 3D coordinates of the two specific regions R1 by the least squares method, the RANSAC method, etc. By calculating the plane equation passing through the two specific regions R1 using the RANSAC method, etc. for the two specific regions R1 that are arranged apart on the same plane, it is possible to reduce the effects of errors and noise in obtaining the 3D coordinates.

[0104] In step ST22, the calculation unit 32 calculates a plurality of reference points in the specific region R1 and a vector calculated from the plurality of reference points based on the second coordinate information. The calculation unit 32 calculates the plurality of reference points and the vector to obtain the position and orientation of the specific region R1. Specifically, the calculation unit 32 calculates a line segment (a plurality of reference points and a vector) that passes through the center of the specific region R1.

[0105] Fig. 18 is a schematic diagram illustrating an example of a process for calculating reference points P5 and P6 and a vector (line segment V1) in a specific region R1. Fig. 18 shows the right-side support pillar 2b, which is the reference object 3 in Fig. 17. As shown in Fig. 18, the calculation unit 32 calculates reference points P5 and P6 at the bottom and top ends of the specific region R1, respectively.

[0106] In the first embodiment, the first machine learning model 21 is KeyPointDetection using Mask R-CNN, and therefore four vertices P1 to P4 that define the specific region R1 are detected. Therefore, the calculation unit 32 can easily calculate the reference points from the 2D coordinates of the four vertices P1 to P4. Specifically, the calculation unit 32 calculates the midpoint of the bottom side of the specific region R1 as the reference point P5, and calculates the midpoint of the top side of the specific region R1 as the reference point P6. The bottom side of the specific region R1 is the line segment connecting the vertices P2 and P3, and the top side of the specific region R1 is the line segment connecting the vertices P1 and P4. The calculation unit 32 calculates a vector from the reference point P5 to the reference point P6, i.e., the line segment V1.

[0107] Although the example has been described in which the line segment V1 is calculated using one of the two pillars 2b that serve as the reference object 3, the present invention is not limited to this. For example, the line segment V1 may be a line segment that passes through the center of a specific region R1 of the two pillars 2b.

[0108] In step ST23, the calculation unit 32 calculates a plane equation (third coordinate information) of the protruding plane PL0 based on the plane equation of the first plane PL1, the reference point P5, the vector (line segment V1), and the standard packing style structure information 23. In steps ST21 to ST22, the information (plane equation) of the first plane PL1 of the standard packing style 4 and the line segment V1 (reference point P5 and vector) of the specific region R1 are obtained. The calculation unit 32 calculates the plane equation (third coordinate information) of the protruding plane PL0 using this information and the positional relationship information included in the standard packing style structure information 23.

[0109] FIG. 19 is a schematic diagram illustrating an example of calculating the protruding plane PL0. In FIG. 19, a case where the plane on the right side of the standard packaging style 4 is the protruding plane PL0 is described. As shown in FIG. 19, the calculation unit 32 calculates a virtual plane VPL1 that intersects the first plane PL1 at right angles and has a line extending from the line segment V1 as its intersecting line. The calculation unit 32 obtains the distance L2 between the line segment V1 and the plane on the right side of the standard packaging style 4 from the structural dimensions included in the standard packaging style structure information 23. The calculation unit 32 calculates the plane equation of the protruding plane PL0 by translating the virtual plane VPL1 to the right by the distance L2. Note that the distance L2 is a distance uniquely determined by the standard packaging style structure information 23.

[0110] In the example shown in FIG. 19, the plane equation of the protruding plane PL0 on the right side of standard packing style 4 is expressed by the following formula.

[0111] [Plane equation] ax+by+cz+d=0 Here, "(a, b, c)" are unit vectors, "d" is a constant, and "(x, y, z)" are coordinates. "(a, b, c)" are unit vectors pointing outward from the standard packaging style 4. In the example shown in Figure 15, a, b, c, and d are calculated so that "(a, b, c)" is a normal unit vector pointing to the right side of Figure 15 on the plane on the right side of the standard packaging style 4 (extension plane PL0).

[0112] The protrusion length L1 of the cargo 5 from the protrusion plane PL0 is calculated by the following formula.

[0113] [Formula for calculating overhang length] f(x,y,z)=ax+by+cz+d If f(x,y,z)<0, the overhang length is set to 0. The above formula for calculating the protrusion length L1 is applied to the 3D coordinates constituting the luggage 5, i.e., the 3D coordinates of the area R2 of the luggage 5. This makes it possible to calculate the protrusion length L1 of the protrusion area R3 of the luggage 5 located outside the protrusion plane PL0. Note that the calculation of the plane equation of the protrusion plane PL0 using the above formula is an example and is not limited to this.

[0114] In the first embodiment, the maximum protrusion length Lmax of the calculated protrusion lengths L1 is output.

[0115] 20 is a flowchart illustrating an example of a calculation process for the maximum protrusion length Lmax of the package 5. As shown in FIG. 20, the calculation unit 32 performs steps ST31 to ST37 to calculate the maximum protrusion length Lmax of the package 5.

[0116] In step ST31, the calculation unit 32 initializes the maximum protrusion length Lmax to "0".

[0117] In step ST32, the calculation unit 32 calculates a calculation formula for the protrusion length L1. Specifically, as described above, the calculation unit 32 calculates the calculation formula for the protrusion length L1, "f(x, y, z)=ax+by+cz+d," based on the plane equation of the protrusion plane PL0, "ax+by+cz+d=0."

[0118] In step ST33, the calculation unit 32 acquires the 3D coordinates (x, y, z) of one pixel in the region R2 of the package 5. Specifically, the calculation unit 32 acquires the 3D coordinates (x, y, z) of one pixel in the region R2 of the package 5 based on the fourth coordinate information indicating the 3D coordinate information of the region R2 of the package 5.

[0119] In step ST34, the calculation unit 32 calculates the protrusion length L1 based on the calculation formula f(x, y, z) for the protrusion length L1 and the 3D coordinates (x, y, z) of one pixel in the region R2 of the package 5.

[0120] In step ST35, the calculation unit 32 determines whether the calculated overhang length L1 is greater than the maximum overhang length Lmax. The maximum overhang length Lmax is stored in the storage unit 20. Note that in the first calculation of the overhang length L1, the maximum overhang length Lmax is initialized in step ST31, and therefore the maximum overhang length Lmax is "0".

[0121] In step ST35, if the calculated overhang length L1 is greater than the maximum overhang length Lmax (Yes in step ST35), the process proceeds to step ST36. If the calculated overhang length L1 is equal to or less than the maximum overhang length Lmax (No in step ST35), the process proceeds to step ST37.

[0122] In step ST36, the calculation unit 32 sets the calculated overhang length L1 as the maximum overhang length Lmax.

[0123] In step ST37, the calculation unit 32 determines whether or not all pixels in the package region R2 have been processed. In step ST37, if all pixels in the package region R2 have been processed (Yes in step ST37), the processing ends. If all pixels in the package region R2 have not been processed (No in step ST37), the processing returns to step ST33.

[0124] 20 is an example, and the calculation process of the maximum protrusion length Lmax of the package 5 is not limited to this. For example, the calculation unit 32 calculates the protrusion length L1 for all pixels in the region R2 of the package 5 and stores the protrusion length L1 for each pixel in the storage unit 20. After completing the calculation of the protrusion length L1 for all pixels in the region R2 of the package 5, the calculation unit 32 may detect the largest value among the calculated protrusion lengths L1 and calculate this value as the maximum protrusion length Lmax.

[0125] Furthermore, when the imaging unit 10 captures an image of the object 6, another piece of luggage not placed on the pallet 2 may be captured, and the luggage not targeted for measurement of the protrusion length L1 may appear in the image data 11. In this case, the image processing unit 31 may extract the luggage not targeted for measurement as part of the region R2 of the luggage 5. To avoid this, the calculation unit 32 may perform processing to identify the region R2 of the luggage 5 placed on the pallet 2.

[0126] The calculation unit 32 may calculate the 3D coordinates of two opposing planes in the standard packing style 4 based on the 3D coordinate information of the first plane PL1 and the standard packing style structure information 23. The calculation unit 32 may calculate the area sandwiched between the two opposing planes from the 3D coordinates of the two opposing planes, and calculate the protruding length L1 of the package 5 in the area sandwiched between the two opposing planes.

[0127] As an example, a case will be described in which the image data 11 captured by the imaging unit 10 shows a package that is not the measurement target behind the object 6. In this case, the calculation unit 32 acquires third coordinate information including the 3D coordinates of the first plane PL1 and the second plane that face each other in the standard packing style 4. The calculation unit 32 also calculates the area sandwiched between the first plane PL1 and the second plane from the 3D coordinates of the first plane PL1 and the second plane, and calculates the protrusion length L1 of the package 5 in the area sandwiched between the first plane PL1 and the second plane. Specifically, the calculation unit 32 calculates a plane equation of the first plane PL1 of the standard packing style 4 and a plane equation of the second plane that faces the first plane PL1. The plane equation of the first plane PL1 may be acquired in a manner similar to step ST21 shown in FIG. 16. The plane equation of the second plane PL2 may be calculated based on the plane equation of the first plane PL1 and the standard packing style structure information 23. The calculation unit 32 calculates the area sandwiched between the first plane PL1 and the second plane based on the plane equation of the first plane PL1 and the plane equation of the second plane, and calculates the protrusion length L1 of the package 5 present in that area. This makes it possible to measure the protrusion length L1 of the package 5 while excluding other packages that are not the measurement target and are located behind the second plane.

[0128] As another example, a case will be described in which the image data 11 captured by the imaging unit 10 shows a package with two pallets 2 stacked one above the other. In this case, the calculation unit 32 acquires third coordinate information including the 3D coordinates of the top and bottom surfaces of the standard packaging style 4, which face each other. The calculation unit 32 also calculates the area sandwiched between the top and bottom surfaces from the 3D coordinates of the top and bottom surfaces, and calculates the protrusion length L1 of the package 5 in the area sandwiched between the top and bottom surfaces. Specifically, the calculation unit 32 calculates a plane equation for the top surface of the standard packaging style 4 and a plane equation for the bottom surface opposite the top surface. The plane equations for the top surface and the bottom surface of the standard packaging style 4 may be calculated based on the plane equation of the first plane PL1, the reference point P5, the line segment V1, and the standard packaging style structure information 23. The calculation unit 32 calculates the area sandwiched between the top and bottom surfaces based on the plane equation for the top surface and the plane equation for the bottom surface, and calculates the protrusion length L1 of the package 5 present in the area. This makes it possible to measure the protruding length L1 of the cargo 5 while excluding cargo on another pallet loaded above the pallet 2.

[0129] As another example, a case will be described in which image data 11 captured by imaging unit 10 shows another piece of luggage that is not the object of measurement in the direction in which luggage 5 protrudes. In this case, calculation unit 32 processes the continuous portion of luggage 5 that protrudes from pallet 2 as region R2 of luggage 5. Specifically, calculation unit 32 identifies protrusion region R3 of luggage 5 that continues from protrusion plane PL0, which is the reference for protrusion of luggage 5 in standard packing style 4, and calculates protrusion length L1 of luggage 5 that exists in that region. This makes it possible to measure protrusion length L1 of luggage 5 while excluding the other piece of luggage that is positioned in the direction in which luggage 5 protrudes.

[0130] [effect] The measuring device 1 according to the first embodiment can achieve the following effects.

[0131] The measuring device 1 according to the first embodiment of the present disclosure includes an imaging unit 10, an image processing unit 31, and a calculation unit 32. The imaging unit 10 captures an image of an object 6 including a pallet 2 having a reference object 3 and luggage 5. The image processing unit 31 extracts a specific region R1 in the reference object 3 and a region R2 of the luggage 5 based on image data 11 of the object 6 captured by the imaging unit 10. The calculation unit 32 acquires first coordinate information 12 indicating 3D coordinates associated with the image data 11 and standard packaging structure information 23 indicating the structural dimensions of the standard packaging style 4 of the pallet 2 associated with the coordinates of the specific region R1. The calculation unit 32 also calculates an overhang length L1 of the luggage 5 overhanging the pallet 2 based on the specific region R1, the region R2 of the luggage 5, the first coordinate information 12, and the standard packaging structure information 23. The image processing unit 31 inputs image data 11 of the object 6 captured by the imaging unit 10 as input data to the first machine learning model 21, thereby obtaining output data that extracts a specific region R1 in the reference object 3. In the first machine learning model 21, machine learning is performed using the image data of the pallet 2 that has the reference object 3 and data that indicates the specific region R1 in the reference object 3 as training data.

[0132] With this configuration, it is possible to measure the protrusion length L1 of the luggage 5 protruding from the pallet 2. Furthermore, in the measuring device 1, the user can easily measure the protrusion length L1 of the luggage 5 by capturing images of the pallet 2 having the reference object 3 and the object 6 including the luggage 5 using the imaging unit 10. This improves the efficiency of the measurement work of the protrusion length L1 of the luggage 5.

[0133] The first machine learning model 21 is KeyPointDetection using Mask R-CNN. With this configuration, it is possible to extract a specific region R1 in the reference object 3 with high accuracy.

[0134] The reference object 3 has a plurality of vertices P1 to P4 that define a specific region R1. The first machine learning model 21 detects the plurality of vertices P1 to P4 in the image data 11. With this configuration, the specific region R1 in the reference object 3 can be extracted with higher accuracy.

[0135] The reference object 3 has at least one reference plane 3a. The image processing unit 31 extracts a specific region R1 based on the at least one reference plane 3a. With this configuration, the specific region R1 in the reference object 3 can be extracted with higher accuracy.

[0136] At least one reference plane 3a is composed of multiple reference planes, and the multiple reference planes 3a are arranged apart from each other on the same plane. This configuration can suppress the effects of errors and noise in obtaining 3D coordinates, allowing the specific region R1 in the reference object 3 to be extracted with higher accuracy.

[0137] The pallet 2 has a bottom plate 2a and a plurality of support posts 2b attached to the bottom plate 2a. The reference object 3 is at least one of the support posts 2b. This configuration makes it easier to measure the protruding length L1 of the cargo 5.

[0138] The multiple support pillars 2b are arranged along the outer edge of the bottom plate 2a. At least one support pillar 2b has a plane that serves as a reference plane 3a on the outer edge side of the bottom plate 2a. The image processing unit 31 extracts a specific region R1 based on the plane. This configuration makes it possible to extract the specific region R1 based on the reference plane 3a with higher accuracy and to more easily measure the protruding length L1 of the luggage 5.

[0139] The image processing unit 31 inputs the image data 11 of the target object 6 as input data to the second machine learning model 22, thereby obtaining output data in which the region R2 of the luggage 5 is extracted. In the second machine learning model 22, machine learning is performed using the image data of the luggage 5 placed on the pallet 2 and data indicating the region R2 of the luggage 5 as training data. With this configuration, the region R2 of the luggage 5 can be extracted with high accuracy.

[0140] The second machine learning model 22 is an instance model using the Mask R-CNN model. Segmentation With this configuration, the region R2 of the baggage 5 can be extracted with higher accuracy.

[0141] The imaging unit 10 captures an image of the object 6 to obtain image data 11 and first coordinate information 12 of the object 6. With this configuration, the image data 11 and the first coordinate information 12 can be easily obtained. Furthermore, in the calculation unit 32, processing such as coordinate conversion processing can be omitted.

[0142] The calculation unit 32 acquires second coordinate information indicating the 3D coordinates of the specific region R1 based on the specific region R1 and the first coordinate information 12. The calculation unit 32 acquires third coordinate information indicating the 3D coordinates of the standard packaging style defined by the specific region R1 based on the second coordinate information and the standard packaging style structure information 23. The calculation unit 32 acquires fourth coordinate information indicating the 3D coordinates of the region R2 of the package 5 based on the region R2 of the package 5 and the first coordinate information 12. The calculation unit 32 calculates the protrusion length L1 of the package 5 based on the third coordinate information and the fourth coordinate information. With this configuration, the specific region R1 and the region R2 of the package 5 can be extracted with high accuracy, while the protrusion length L1 of the package 5 can be measured with high accuracy.

[0143] The calculation unit 32 acquires third coordinate information including at least a plane equation of the protrusion plane PL0, which is the reference for the protrusion of the package 5 in the standard packaging style 4. The calculation unit 32 calculates the protrusion length L1 of the package 5 based on the plane equation of the protrusion plane PL0 and the fourth coordinate information. With this configuration, it is possible to more accurately measure the protrusion length L1 of the package 5 while extracting the specific area R1 and the area R2 of the package 5 with higher accuracy.

[0144] The calculation unit 32 calculates a plane equation of the first plane PL1 of the standard packaging style 4 on the side where the reference object 3 including the specific region R1 is placed, based on the second coordinate information. The calculation unit 32 calculates a plurality of reference points P5, P6 in the specific region R1 and a vector calculated from the plurality of reference points P5, P6, based on the second coordinate information. The calculation unit 32 calculates a plane equation of the protrusion plane PL0 based on the plane equation of the first plane PL1, the reference point P5, the vector, and the standard packaging style structure information 23. With this configuration, the protrusion length L1 of the package 5 can be measured with higher accuracy.

[0145] The calculation unit 32 acquires third coordinate information including the 3D coordinates of two opposing planes in the standard packing style 4, calculates the area sandwiched between the two planes from the 3D coordinates of the two planes, and calculates the protrusion length L1 of the package 5 in the area sandwiched between the two planes. With this configuration, it is possible to measure the protrusion length L1 of the package 5 while excluding other packages that are not placed on the pallet 2, i.e., other packages that are not the target of measurement.

[0146] The calculation unit 32 processes the area of the cargo protruding from the pallet 2 that continues from the area of the standard packing style 4 as the area R2 of the cargo 5. With this configuration, it is possible to measure the protruding length L1 of the cargo 5 while excluding other cargo that is not placed on the pallet 2, i.e., not the object of measurement.

[0147] The calculation unit 32 calculates the largest value (maximum protrusion length Lmax) among the protrusion lengths L1 of the packages 5. With this configuration, information on the maximum protrusion length Lmax of the portion of the package 5 that protrudes the most from the pallet 2 can be obtained.

[0148] The measuring device 1 further includes an output unit 40 that outputs the overhang length L1. With this configuration, the user can easily know the overhang length L1.

[0149] The measurement method according to the first embodiment of the present disclosure also provides the same effects as the measurement device 1 described above.

[0150] In the first embodiment, an example has been described in which the imaging unit 10 captures an image of the object 6 including the pallet 2 and the luggage 5, thereby acquiring the image data 11 and the first coordinate information 12, which is 3D coordinate information associated with the image data 11. However, the present invention is not limited to this. For example, the imaging unit 10 does not need to acquire the first coordinate information 12. In this case, the calculation unit 32 may acquire 3D coordinates corresponding to the image data 11 by a coordinate conversion process.

[0151] In the first embodiment, an example has been described in which the storage unit 20 stores the first machine learning model 21, the second machine learning model 22, and the standard packaging structure information 23, but this is not limiting. For example, the first machine learning model 21, the second machine learning model 22, and the standard packaging structure information 23 may be stored in a server on a network. In this case, the measurement device 1 may include a communication unit that communicates with the server, and acquire the first machine learning model 21, the second machine learning model 22, and the standard packaging structure information 23 from the server via the communication unit. The communication unit includes a circuit that communicates with an external device in accordance with a predetermined communication standard. Examples of the predetermined communication standard include LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), USB, HDMI (registered trademark), CAN (controller area network), and SPI (Serial Peripheral Interface).

[0152] Although the first embodiment has been described with reference to the measuring device 1, the present disclosure is not limited thereto. For example, the present disclosure may be configured in the form of a measurement system or server including a measuring device and a server. For example, the measurement system may include the measuring device 1 including the imaging unit 10, the calculation unit 32, the output unit 40, and a communication unit, and the server including the storage unit 20, the image processing unit 31, and the communication unit. In the measurement system, the measuring device may perform the processes of steps ST1, ST4 to ST6, and the server may perform the processes of steps ST2 and ST3. Note that the configuration of the measurement system is not limited thereto. For example, in the measurement system, the storage unit 20, the image processing unit 31, the calculation unit 32, and the output unit 40 may be included in the measuring device or the server. Furthermore, in the measurement system, the measuring device 1 and the server may each include a storage unit that stores at least one element of the elements stored in the storage unit 20. For example, the first machine learning model 21, the second machine learning model 22, and the standard packaging structure information 23 may be stored in the storage unit of the measuring device or the storage unit of the server. Furthermore, the entity that executes the processes of steps ST1 to ST6 is not limited to the above example. The processes of steps ST2 to ST6 may be executed by the measurement device or the server. Furthermore, the measurement system may include devices other than the measurement device and the server.

[0153] In the first embodiment, an example has been described in which the image processing unit 31 and the calculation unit 32 are separate elements in the control unit 30, but this is not limiting. For example, the image processing unit 31 and the calculation unit 32 may be integrally formed as a single element. Alternatively, the image processing unit 31 and the calculation unit 32 may each be divided into multiple elements. For example, the image processing unit 31 may be divided into a first image processing unit that extracts a specific region R1 and a second image processing unit that extracts the region of the package 5. The calculation unit 32 may be divided into a first calculation unit that acquires the first to fourth coordinate information and a second calculation unit that calculates the protrusion length L1 of the package 5.

[0154] In the first embodiment, an example in which the reference object 3 is a plurality of support posts 2b has been described, but the present invention is not limited to this. The reference object 3 may be any object that serves as a reference for extracting a specific region R1 in the pallet 2.

[0155] In the first embodiment, an example has been described in which the image processing unit 31 extracts the region R2 of the luggage 5 using the second machine learning model 22, but the present invention is not limited to this. For example, the image processing unit 31 may extract the region R2 of the luggage 5 based on information input by a user. In this case, the measuring device 1 may include an input unit that accepts the input information from the user. The input unit may be, for example, an input interface such as a touch panel. In this way, the measuring device 1 may calculate the protrusion length L1 of the luggage 5 based on the information on the region R2 of the luggage 5 input by the user.

[0156] In the first embodiment, the calculation unit 32 calculates the protrusion length L1, but the present invention is not limited to this. For example, the calculation unit 32 may calculate the protrusion width or protrusion height of the package 5.

[0157] In the first embodiment, an example in which the output unit 40 outputs the maximum protrusion length Lmax has been described, but this is not limiting. For example, the output unit 40 may output the protrusion length L1 at a specific position. In this case, the measuring device 1 may include an input unit that accepts input information from a user. Based on information about a position of the package 5 within the region R2 input by the user, the measuring device 1 may output the protrusion length L1 of the package 5 corresponding to that position.

[0158] In the first embodiment, an example has been described in which the third coordinate information is information on the 3D coordinates of the protruding plane PL0, but this is not limiting. The third coordinate information only needs to include 3D coordinate information on the protruding plane PL0. The third coordinate information may include 3D coordinate information on planes that make up the standard packaging style 4 in addition to the protruding plane PL0. For example, the third coordinate information may include 3D coordinate information on all six faces of the standard packaging style 4, or may include 3D coordinates on multiple planes among the six faces of the standard packaging style 4.

[0159] In the first embodiment, an example has been described in which the plane on the right side of the standard packaging style 4 is predetermined as the protruding plane PL0, but this is not limiting. At least one of the six faces of the standard packaging style 4 may be predetermined as the protruding plane PL0. For example, two planes, one on the right side and one on the left side of the standard packaging style 4, may be predetermined as the protruding plane PL0. Alternatively, the plane on the upper side of the standard packaging style 4 may be predetermined as the protruding plane PL0.

[0160] In the first embodiment, an example in which the protrusion plane PL0 is determined in advance has been described, but this is not limiting. For example, the protrusion plane PL0 may be determined based on the fourth coordinate information. For example, the calculation unit 32 may estimate the protrusion direction of the package 5 based on the fourth coordinate information, and determine the plane in the protrusion direction of the package 5 as the protrusion plane PL0. In this case, the third coordinate information may include 3D coordinate information for all six planes of the standard packing style 4, or may include 3D coordinate information for two or more planes of the six planes of the standard packing style 4. The calculation unit 32 may estimate the estimated protrusion direction based on the fourth coordinate information, and determine the plane of the standard packing style 4 located in the protrusion direction as the protrusion plane PL0.

[0161] In the first embodiment, the standard packaging shape 4 is a rectangular parallelepiped having six flat surfaces, but is not limited to this. For example, the standard packaging shape 4 may be a truncated pyramid or other shape.

[0162] In the first embodiment, an example has been described in which the plane equation of the protruding plane PL0 is calculated as the third coordinate information, but the third coordinate information is not limited to this. The third coordinate information is only required to include information on the 3D coordinates of the protruding plane PL0, and is not limited to the plane equation.

[0163] Although the present disclosure has been fully described in connection with the preferred embodiments with reference to the accompanying drawings, various changes and modifications will be apparent to those skilled in the art, and such changes and modifications are to be understood as included within the scope of the present invention as defined by the appended claims unless they depart therefrom. [Industrial Applicability]

[0164] The measuring device and measuring method disclosed herein can easily measure the protruding length of cargo that protrudes from a pallet, and are therefore suitable for use in the field of transportation, such as loading cargo onto trucks or warehouses. [Explanation of symbols]

[0165] 1. Measuring equipment 2 palettes 2a Bottom plate 2b Post 3 Reference Object 3a Reference plane 4 Standard packaging 5. Luggage 6. Objects 10. Imaging unit 11 Image data 12 First coordinate information (3D coordinate information) 20 Memory section 21 First Machine Learning Model 22 Second Machine Learning Model 23 Standard packaging structure information 30 Control Unit 31 Image processing section 32 Calculation section 40 Output section L1 Protrusion length Lmax Maximum protrusion length P1,P2,P3,P4 vertices P5,P6 reference point PL0 Protruding plane PL1 1st plane PL2 2nd plane R1 Specific Area R2 Luggage Area R3 Protrusion area

Claims

1. an imaging unit that captures images of objects including a pallet having a reference object and luggage; an image processing unit that extracts a specific area of the reference object and an area of the baggage based on image data of the object captured by the imaging unit; a calculation unit that acquires first coordinate information indicating 3D coordinates associated with the image data and standard package structure information indicating structural dimensions of the standard package of the pallet associated with the coordinates of the specific area, and calculates the protrusion length of the package protruding from the pallet based on the specific area, the area of the package, the first coordinate information, and the standard package structure information; Equipped with the image processing unit inputs the image data of the object captured by the imaging unit as input data into a first machine learning model in which machine learning has been performed using image data of a pallet including the reference object and data indicating the specific area of the reference object as training data, thereby obtaining output data in which the specific area of the reference object has been extracted. Measuring device.

2. the reference object has a plurality of vertices that define the particular region; the first machine learning model detects the vertices in the image data. The measuring device according to claim 1 .

3. The first machine learning model is KeyPointDetection using MaskR-CNN, The measuring device according to claim 2 .

4. the reference object has at least one reference plane; the image processing unit extracts the specific region based on the at least one reference plane; The measuring device according to any one of claims 1 to 3.

5. the at least one reference plane is a plurality of reference planes; The plurality of reference planes are arranged apart from each other on the same plane.

5. The measuring device according to claim 4.

6. The pallet has a bottom plate and a plurality of support columns provided on the bottom plate, the reference object is at least one support pillar among the plurality of support pillars; The measuring device according to any one of claims 1 to 5.

7. The plurality of support columns are arranged along the outer edge of the bottom plate, the at least one support column has a flat surface serving as a reference plane on an outer edge side of the bottom plate, the image processing unit extracts the specific region based on the plane; The measuring device according to claim 6.

8. the image processing unit inputs the image data of the object as input data into a second machine learning model in which machine learning has been performed using image data of the luggage placed on the pallet and data indicating the area of the luggage as training data, thereby obtaining output data in which the area of the luggage has been extracted. The measuring device according to any one of claims 1 to 7.

9. The second machine learning model is InstanceSegmentation using a MaskR-CNN model. The measuring device according to claim 8.

10. the imaging unit acquires image data of the object and the first coordinate information by imaging the object; The measuring device according to any one of claims 1 to 9.

11. The calculation unit acquiring second coordinate information indicating 3D coordinates of the specific area based on the specific area and the first coordinate information; Based on the second coordinate information and the standard packaging structure information, third coordinate information indicating 3D coordinates of the standard packaging defined by the specific area is acquired; acquiring fourth coordinate information indicating 3D coordinates of the area of the luggage based on the area of the luggage and the first coordinate information; calculating an overhang length of the package based on the third coordinate information and the fourth coordinate information; The measuring device according to any one of claims 1 to 10.

12. The calculation unit The third coordinate information includes at least a plane equation of a protrusion plane that is a reference for the protrusion of the package in the standard packing style, and calculating a protrusion length of the luggage based on a plane equation of the protrusion plane and the fourth coordinate information; The measuring device according to claim 11.

13. The calculation unit calculating a plane equation of a first plane of the standard packaging style on a side on which the reference object including the specific area is placed based on the second coordinate information; calculating a plurality of reference points in the specific region and a vector calculated from the plurality of reference points based on the second coordinate information; calculating a plane equation of the protruding plane based on a plane equation of the first plane, at least one reference point among the plurality of reference points, the vector, and the standard packaging structure information; 13. The measuring device of claim 12.

14. The calculation unit Acquire the third coordinate information including 3D coordinates of two opposing planes in the standard packaging form; Calculating an area sandwiched between the two planes from the 3D coordinates of the two planes; Calculating the protruding length of the luggage in the area sandwiched between the two planes.

14. The measuring device of claim 13.

15. The calculation unit processes an area where the cargo exists, which is continuous with the area of the standard packing style, for the cargo protruding from the pallet, as the cargo area. The measuring device according to any one of claims 1 to 14.

16. The calculation unit calculates a maximum protruding length of the protruding lengths of the luggage. The measuring device according to any one of claims 1 to 15.

17. further comprising an output unit that outputs the overhang length; The measuring device according to any one of claims 1 to 16.

18. imaging an object including a pallet and a package with a reference object; extracting a specific region of the reference object based on image data of the captured object; extracting an area of the luggage based on image data of the captured object; acquiring first coordinate information indicating 3D coordinates associated with the image data and standard packing structure information indicating structural dimensions of the standard packing of the pallet associated with the coordinates of the specific area; calculating an overhang length of the cargo that overhangs the pallet based on the specific area, the cargo area, the first coordinate information, and the standard packing structure information; Including, The step of extracting the specific area in the reference object includes inputting image data of the captured object as input data into a first machine learning model that has been trained using image data of a pallet that includes the reference object and data that indicates the specific area in the reference object as training data, thereby obtaining output data in which the specific area in the reference object has been extracted. Measurement method.

19. A program for causing a computer to execute the method according to claim 18.

Images