Cargo compartment accommodation state detection system, cargo compartment accommodation state detection program, and cargo compartment accommodation state detection method

Abstract

Provided is a cargo compartment storage state detection system capable of accurately detecting the storage state of a cargo compartment. The cargo compartment storage state detection system includes: an image acquisition unit that acquires an image obtained by photographing the entire cargo compartment in a state in which cargo is stored from a predetermined photographing direction; a corner detection unit that detects a plurality of corners of the cargo compartment from the image acquired by the image acquisition unit using a learning completed model for corner detection; a cargo information acquisition unit that acquires cargo information including information related to the position of the cargo in the cargo compartment; and a cargo compartment calculation unit that calculates the range of the entire cargo compartment using position information of two corners of the cargo compartment that are diagonally opposite each other, i.e., diagonal corner position information, and the cargo information acquired by the cargo information acquisition unit.

Description

Technical Field

[0001] This disclosure relates to a cargo compartment containment status detection system, a cargo compartment containment status detection procedure, and a cargo compartment containment status detection method. Background Technology

[0002] The vehicle cargo status detection device described in Patent Document 1 discloses a method of using a camera to take pictures of the cargo box set on the truck bed, detecting the empty space inside the cargo box from the acquired images of the cargo box's interior, and calculating the loading rate inside the cargo box based on the detected empty space.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2001-334864 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, in the case of the aforementioned vehicle cargo status detection device, although the size of the empty space is calculated using camera images, the overall size of the cargo compartment needs to be pre-databaseed for each vehicle model, which is quite cumbersome. Therefore, the inventors of this application have researched a system that uses image analysis based on a machine learning model to determine the overall size of the cargo compartment from camera images, in addition to the size of the cargo, thereby detecting the cargo's storage status within the cargo compartment. Furthermore, the inventors of this application discovered the following problems with such a system: for example, the detection accuracy based on image analysis deteriorates due to differences in the environment surrounding the cargo compartment during machine learning (e.g., the presence or absence of sunlight) and the difference between the environment surrounding the cargo compartment during machine learning and the environment during cargo storage status detection.

[0008] Technical means for solving problems

[0009] This disclosure may be implemented in the following ways.

[0010] (1) According to one aspect of the present disclosure, a cargo compartment containment state detection system is provided. This cargo compartment containment state detection system is for detecting the containment state of goods within a cargo compartment, and includes: an image acquisition unit that acquires an image obtained by photographing the entire cargo compartment in a state where the contained goods are visible along a predetermined shooting direction; a corner detection unit that uses a corner detection learning model to detect multiple corners of the cargo compartment from the image acquired by the image acquisition unit, wherein the corner detection learning model is trained using training image data obtained by marking the positions of at least two diagonally opposite corners of the multiple corners of the cargo compartment, which is rectangular when viewed along the shooting direction, based on the image acquired for machine learning purposes; a cargo information acquisition unit that acquires cargo information including information related to the position of the goods within the cargo compartment; and a cargo compartment calculation unit that uses the position information of the two diagonally opposite corners of the cargo compartment (i.e., diagonal corner position information) and the cargo information acquired by the cargo information acquisition unit to calculate the overall range of the cargo compartment.

[0011] According to this method, the cargo compartment containment status detection system uses the cargo compartment calculation unit to calculate the overall range of the cargo compartment by using the position information of two diagonally opposite corners (i.e., diagonal corner position information) and cargo information, including information related to the position of the cargo. By not only using corner position information but also combining cargo position information as cargo information—for example, since the cargo cannot be located outside the cargo compartment—these positional information can be compared and corrected to make the range closer to the correct one, thus improving the accuracy of the cargo compartment range calculation.

[0012] (2) In the cargo compartment containment state detection system described above, it may also include a cargo compartment overall detection unit. This cargo compartment overall detection unit uses a cargo compartment overall detection learning model to detect the entire cargo compartment from the image acquired by the image acquisition unit. The cargo compartment overall detection learning model is trained using training image data obtained by marking the entire cargo compartment when viewed along the shooting direction for the image acquired for the machine learning purpose. The cargo compartment calculation unit performs the following processing: for the position information of the corners that were not detected by the corner detection unit, the position information of the opposite corners is obtained by using the position information of the entire cargo compartment detected by the cargo compartment overall detection unit.

[0013] According to this cargo compartment containment status detection system, during the cargo compartment calculation performed by the cargo compartment calculation unit, the position information of corners not detected by the corner detection unit is obtained by supplementing the position information of the entire cargo compartment detected by the overall cargo compartment detection unit to obtain the position information of the opposite corners. Therefore, even if two corners that are diagonally opposite each other are not detected by the corner detection unit, the overall cargo compartment range can be calculated using the position information of the entire cargo compartment detected by the overall cargo compartment detection unit. Thus, for example, it is possible to avoid the frequent occurrence of cargo compartment calculation processes being interrupted due to the failure to detect corners and having to repeatedly perform the same process from the beginning.

[0014] (3) In the cargo compartment containment status detection system described above, the cargo information may include the position information of the overall cargo shape, where the overall cargo shape is an imaginary rectangular portion that includes the entire cargo contained in the cargo compartment when viewed along the shooting direction. The cargo compartment calculation unit performs the following processing: if the overall cargo shape exists on the outside compared to the rectangular cargo compartment shape obtained using the diagonal corner position information, the shape that has been expanded to include the portion on the outside is calculated as the range of the entire cargo compartment; if the overall cargo shape does not exist on the outside compared to the rectangular cargo compartment shape obtained using the diagonal corner position information, the rectangular cargo compartment shape obtained using the diagonal corner position information is calculated as the range of the entire cargo compartment.

[0015] According to this cargo compartment containment state detection system, when the overall cargo shape is located on the outside compared to the rectangular cargo compartment shape obtained using diagonal corner position information, the cargo compartment calculation unit calculates an expanded shape that includes the portion located on the outside, and defines this as the overall range of the cargo compartment. Furthermore, when the overall cargo shape is not located on the outside compared to the rectangular cargo compartment shape obtained using diagonal corner position information, the system calculates the rectangular cargo compartment shape obtained using diagonal corner position information, and defines this as the overall range of the cargo compartment.

[0016] Specifically, by referencing easily detectable cargo information based on image analysis, the area of ​​the rectangular cargo compartment, defined by the diagonal corner positions, can be corrected to include the entire cargo outline, thus preventing the calculation of the cargo compartment's area where the cargo is not fully captured. Therefore, the accuracy of calculating the cargo compartment's area can be further improved.

[0017] (4) In the cargo compartment containment status detection system described above, a loading rate calculation unit may also be included. This loading rate calculation unit calculates the loading rate of the goods inside the cargo compartment. The goods information includes information related to the size of the goods. The loading rate calculation unit performs the following processing: it calculates the loading rate by dividing the total area of ​​all the goods inside the cargo compartment when viewed along the shooting direction, as acquired by the goods information acquisition unit, by the area of ​​the cargo compartment detected by the cargo compartment calculation unit. According to this cargo compartment containment status detection system, the loading rate can be calculated more accurately by the loading rate calculation unit using the overall cargo compartment area and goods information calculated with good precision.

[0018] (5) In the cargo compartment containment status detection system described above, the cargo information may also be detected from the image acquired by the image acquisition unit using a cargo detection learning model obtained by machine learning on the features of the cargo. According to this cargo compartment containment status detection system, cargo information for cargo compartment containment status detection can be appropriately detected using the cargo detection learning model.

[0019] (6) In the cargo compartment containment status detection system described above, the cargo compartment may also be a cargo compartment installed on the loading platform of a transport vehicle. According to this type of cargo compartment containment status detection system, the containment status of goods in the cargo compartment installed on the loading platform of a transport vehicle can be detected. Based on the detected containment status, information such as how much more goods can be loaded into the cargo compartment of a transport vehicle such as a truck, or how much goods have been loaded at which location, can be obtained.

[0020] Furthermore, this disclosure can also be implemented in various ways other than a cargo compartment containment status detection system. For example, it can be implemented in the form of a cargo compartment containment status detection method, a cargo compartment containment status detection program, or a non-transitory recording medium storing the program. Attached Figure Description

[0021] Figure 1 This is a block diagram illustrating the hardware configuration of the cargo compartment containment status detection system according to the first embodiment of this disclosure.

[0022] Figure 2 This is a block diagram illustrating the functional structure of the cargo compartment containment status detection system.

[0023] Figure 3 It is a schematic side view of a vehicle with a cargo box.

[0024] Figure 4 This is a flowchart illustrating the processing sequence of the cargo compartment containment status detection method.

[0025] Figure 5This is a diagram used to illustrate image processing in corner detection.

[0026] Figure 6 This is a diagram used to illustrate image processing in cargo inspection.

[0027] Figure 7 This is a flowchart showing the processing sequence in the cargo compartment calculation process.

[0028] Figure 8 This is a diagram illustrating an example of image processing in the cargo box calculation process. Detailed Implementation

[0029] A. First implementation method:

[0030] Reference Figures 1 to 8 The first embodiment of the present disclosure will be described, including the cargo compartment containment status detection system 10 and the cargo compartment containment status detection method.

[0031] A1. Composition of the cargo compartment containment status detection system 10:

[0032] [Hardware configuration of the cargo compartment containment status detection system 10]

[0033] Figure 1 This is a block diagram illustrating the hardware configuration of the cargo compartment holding status detection system 10 according to the first embodiment of this disclosure. Furthermore, the cargo compartment holding status detection system 10 of this embodiment is configured as a portable terminal 12 with a camera function, such as a smartphone or tablet. The cargo compartment holding status detection system 10 of the first embodiment detects the cargo compartment of the vehicle described later and the goods within it, and calculates the proportion of the goods occupying the cargo compartment, i.e., the loading rate. Details of the vehicle configuration and the calculation of the loading rate will be described later. In this embodiment, with a dedicated application pre-installed on the portable terminal 12, the cargo compartment holding status detection system 10 can be utilized by activating the application.

[0034] like Figure 1 As shown, a control unit 14 is provided in the portable terminal 12. The control unit 14 is configured to include a CPU (Central Processing Unit) 16, a ROM (Read Only Memory) 18, a RAM (Random Access Memory) 20, a memory 22 as a storage unit, a communication interface 24, and an input / output interface 26. Each component is connected to each other via a bus 28 in a manner that enables communication.

[0035] CPU 16 is the central processing unit, executing various programs and controlling various components. Specifically, CPU 16 reads programs from ROM 18 or memory 22, using RAM 20 as its working area to execute the programs. CPU 16 performs control and various arithmetic operations according to the programs recorded in ROM 18 or memory 22. Memory 22 stores a program for detecting the cargo compartment's containment status.

[0036] Additionally, memory 22 stores a corner detection learning model for detecting corners of the cargo compartment from an image containing the cargo compartment. This corner detection learning model is generated through machine learning using training image data. This training image data is obtained by labeling the positions of at least two diagonally opposite corners of the cargo compartment when viewed along the shooting direction from an image acquired for machine learning purposes, encompassing the entire cargo compartment. Labeling refers to the machine learning process of assigning labels to a large amount of raw data obtained from the image; in this embodiment, it means establishing a correspondence between the label representing the corner's position and the coordinates corresponding to the corner's position. Using this corner detection learning model, it is possible to detect the positions of multiple corners, including, for example, the upper left and lower right corners.

[0037] Furthermore, memory 22 stores a fully learned cargo detection model for detecting cargo from an image containing a cargo compartment. This fully learned cargo detection model is a machine learning model that performs machine learning on the feature quantities of the cargo for an image containing the entire cargo compartment. The feature quantities of the cargo refer to, for example, the different brightness depending on the material of the cargo, such as iron, plastic, or corrugated paper. Using the fully learned cargo detection model, the shape representing the size of multiple cargoes can be detected based on such cargo feature quantities.

[0038] Furthermore, memory 22 stores a learned model for detecting the entire cargo compartment from an image containing the compartment. This learned model for detecting the entire cargo compartment is generated through machine learning using training image data. This training image data is obtained by marking the position of the entire cargo compartment when viewed along the shooting direction from an image acquired for machine learning purposes that contains the entire cargo compartment. This differs from the learned model for detecting corners, which is obtained by marking the positions of corners. In the learned model for detecting the entire cargo compartment, the positions of, for example, the upper left and lower right corners can be detected as the position of the entire cargo compartment.

[0039] Furthermore, in the learned model for overall cargo compartment detection, the detection accuracy based on image parsing is sometimes worse than that using a learned model for corner detection to detect corner positions. This is because, since the overall position of the cargo compartment is marked during training, the various deviations between the positions of cargo compartments within the images used as training data, as well as the size, shape, and loading order of the goods stacked in the cargo compartments, can become interference. Therefore, in the first embodiment, the learned model for corner detection is basically used to detect corner positions. For corners that are not detected, the learned model for overall cargo compartment detection is used to appropriately complete the positions of the detected corners and crop out the scope of the cargo compartment. Details are described later.

[0040] ROM 18 stores various programs and data. RAM 20 serves as a working area to temporarily store programs or data. Memory 22, consisting of an HDD (Hard Disk Drive) or SSD (Solid State Drive), stores various programs and data, including the operating system. In this embodiment, ROM 18 and memory 22 store various data, including programs for calculating load rates and vehicle-related data.

[0041] Communication interface 24 is an interface for the cargo compartment containment status detection system 10 to communicate with servers and other devices not shown, for example, using standards such as Ethernet (registered trademark), LTE, FDDI, Wi-Fi (registered trademark).

[0042] The input / output interface 26 is connected to a display screen 30 (which serves as a display unit), a microphone 32, a speaker 34, and a camera 36. The display screen 30 is installed on the portable terminal 12 and displays various information to the user. In addition, the display screen 30 displays image data captured by the camera 36, ​​which will be described later. Furthermore, in this embodiment, as an example, the display screen 30 is a touch panel type, configured to accept input by touching the content displayed on the display screen 30.

[0043] Microphone 32 and speaker 34 are respectively provided on the portable terminal 12 and are used when the user is making a call. In addition, microphone 32 can also be used when the user gives instructions with voice, and speaker 34 can also be used to notify the user with voice. Camera 36 is provided on the portable terminal 12 and configured to display the image data captured by camera 36 on display screen 30.

[0044] [Functional Composition of the Cargo Compartment Reception Status Detection System 10]

[0045] The cargo compartment containment status detection system 10 uses the aforementioned hardware resources to implement various functions. For details regarding the functional structure of the cargo compartment containment status detection system 10, please refer to... Figure 2 Please provide an explanation. Figure 2 This is a block diagram showing the functional configuration of the cargo compartment containment status detection system 10. Furthermore, each functional configuration is implemented by the CPU 16 reading a program stored in the memory 22 and executing that program.

[0046] like Figure 2 As shown, the cargo compartment containment status detection system 10 has the following functional configuration: an image acquisition unit 41, a corner detection unit 42, a cargo information acquisition unit 43, a cargo compartment overall detection unit 44, a cargo compartment calculation unit 45, a loading rate calculation unit 46, and a learning unit 47. The image acquisition unit 41 uses a camera 36 to acquire an image of the entire cargo compartment, which is the object of detection. This image is obtained by photographing the entire cargo compartment in a state where the loaded cargo is visible, along a predetermined shooting direction.

[0047] The corner detection unit 42 uses the learned corner detection model stored in the memory 22 to detect the corners of the cargo compartment from the image of the cargo compartment acquired by the image acquisition unit 41. Furthermore, the corner detection unit 42 outputs the position information of the detected corners. Details related to the output of the corner position information will be described later.

[0048] The cargo information acquisition unit 43 uses the cargo detection learning model stored in the memory 22 to detect cargo information inside the cargo compartment from the image of the cargo compartment acquired by the image acquisition unit 41. The cargo information includes information related to the position of the cargo inside the cargo compartment and information related to the size of the cargo inside the cargo compartment. Furthermore, the cargo information includes position information of the overall cargo shape. In this embodiment, "overall cargo shape" means the top-view rectangular portion that completely includes the shape of the overall cargo when viewed from the shooting direction, when considering the cargo as a single cargo (hereinafter, "overall cargo") among the multiple cargoes housed in the cargo compartment.

[0049] The cargo compartment detection unit 44 uses the learned cargo compartment detection model stored in the memory 22 to detect the entire cargo compartment from the image of the cargo compartment acquired by the image acquisition unit 41. Furthermore, the cargo compartment detection unit 44 outputs corner position information based on the detected cargo compartment components.

[0050] The cargo compartment calculation unit 45 uses the corner position information detected by the corner detection unit 42 or the cargo compartment overall detection unit 44 to detect the range of the cargo compartment. Specifically, the cargo compartment calculation unit 45 has the following functions: extracting the position information of two corners that are diagonally opposite each other (hereinafter also referred to as "diagonal corner position information"), using the diagonal corner position information, numerically outputting the range of the cargo compartment, and then cropping the range of the cargo compartment from the image as a partial image.

[0051] The loading rate calculation unit 46 calculates the loading rate of the goods inside the cargo compartment. Specifically, it calculates the loading rate using the range of the cargo compartment detected by the cargo compartment calculation unit 45 and the cargo information inside the cargo compartment acquired by the cargo information acquisition unit 43. The learning unit 47 creates an AI model based on images of corners, goods, etc., taken by camera, and effectively utilizes neural networks. Furthermore, the specific processing of each functional unit will be explained in detail in the cargo compartment containment status detection method described later.

[0052] [Composition of Vehicle 50]

[0053] Figure 3 This is a schematic side view of a transport vehicle 50 (hereinafter referred to as "vehicle 50") equipped with a cargo box 51. Next, the configuration of the vehicle 50, which is equipped with a cargo box 51 that is the object of detection by the cargo box containment status detection system 10 described above, will be described. The vehicle 50 is, for example, a truck that loads and transports a large quantity of goods 52.

[0054] like Figure 3 As shown, the cargo box 51 is located behind the cab 58 of the vehicle 50 and is formed on the cargo platform 57. During the handling of the cargo 52, the cargo box 51 moves from the front-to-back direction of the vehicle 50. Figure 3 Side walls 53 and 54 (shown in the left and right directions), side wall 55 on the right side relative to the direction of travel, lower section 56 on the left side relative to the direction of travel, and upper section (not shown) covering.

[0055] Regarding the upper section, for example, a portion of the left side wall and ceiling forms an integral door with an approximately L-shaped cross-section, which rotates upwards to open the cargo compartment 51. The lower section 56 rotates downwards from the bottom of the platform 57 to open the cargo compartment 51. With both the lower and upper sections open, it is possible to take a picture of the entire cargo compartment 51, including the four corners, from the left side of the vehicle 50. In other words, it is possible to acquire an image of the entire cargo compartment 51 while the loaded cargo 52 is visible. In this embodiment, as... Figure 3 As shown, the predetermined shooting direction is consistent with the direction from which the cargo box 51 is viewed from the left side of the vehicle 50. The cargo box 51 is rectangular in shape when viewed from the shooting direction.

[0056] A2. Cargo compartment containment status detection method based on cargo compartment containment status detection system 10:

[0057] Next, refer to Figures 4-8 The cargo compartment containment status detection method executed by the aforementioned cargo compartment containment status detection system 10 will be described. Figure 4 This is a flowchart illustrating the processing sequence of the cargo compartment containment status detection method. For example... Figure 4 As shown, the cargo compartment containment status detection method includes an image acquisition process (step 100, hereinafter referred to as "step" as "S"), a corner detection process (S200), a cargo information acquisition process (S300), a cargo compartment overall detection process (S400), a cargo compartment calculation process (S500), and a loading rate calculation process (S600), which are executed sequentially.

[0058] In the image acquisition process (S100), the user of the system uses the camera 36 to take a picture of the entire cargo compartment 51, thereby using the function of the image acquisition unit 41 to acquire an image Ia (refer to) containing the entire cargo compartment 51. Figure 3 Hereinafter, it will also be referred to as "the overall image Ia of the cargo compartment 51"). In the corner detection process (S200), the corner detection unit 42 uses the overall image Ia of the cargo compartment 51 acquired in the image acquisition process (S100) to detect the corners. Figure 3 , Figure 5 In the process, the corner detection model is used to detect the four corners of cargo compartment 51.

[0059] Figure 5 This is a diagram used to illustrate image processing in corner detection. For example... Figure 5 As shown, in the corner inspection process (S200), the four corners Ic1, Ic2, Ic3, and Ic4 of the cargo compartment 51 are inspected. Figure 5 In the illustration, each corner Ic1, Ic2, Ic3, and Ic4 is enclosed in a double-dotted line frame. In this embodiment, "corner" means not only the vertices of the four corners of the rectangle representing the cargo compartment 51, but also a portion of a rectangle consisting of two line segments intersecting at approximately right angles to each other with those vertices as endpoints.

[0060] Moreover, specifically, in the corner detection process (S200), the position information of each corner Ic1, Ic2, Ic3, and Ic4 is acquired. Figure 5 In this context, when the horizontal direction is set as the X-axis and the perpendicular direction intersecting the X-axis is set as the Y-axis, for example, "the position information of corner Ic1" refers to... Figure 5The X and Y coordinates of corner point C1 are shown. Here, the X and Y coordinates of the upper left corner point C1 are set to (X1, Y1), the X and Y coordinates of the lower right corner point C2 are set to (X2, Y2), the X and Y coordinates of the lower left corner point C3 are set to (X1, Y2), and the X and Y coordinates of the upper right corner point C4 are set to (X2, Y1). In this embodiment, corner points C1, C2, C3, and C4 are defined as the center coordinates of each corner Ic1, Ic2, Ic3, and Ic4 of the rectangle. The X and Y coordinates of each corner point C1, C2, C3, and C4 are calculated as the "position information of corner Ic1, Ic2, Ic3, and Ic4".

[0061] In the cargo information acquisition process (S300), the cargo information acquisition unit 43 uses the cargo information acquisition unit 43 to detect cargo information inside the cargo compartment 51 from the overall image Ia of the cargo compartment 51 acquired in the image acquisition process (S100) using a cargo detection learning model. Figure 6 This is a diagram used to illustrate image processing in cargo inspection, showing image Ib of detected cargo 52. Figure 6 In the diagram, images Ib of each item 52 are shown enclosed in double-dotted lines. For example... Figure 6 As shown, in the cargo information acquisition process (S300), the shape of each cargo 52 is detected as an image Ib of multiple cargoes 52.

[0062] Furthermore, in the cargo information acquisition process (S300), the overall position information of the multiple cargoes 52 housed within the cargo compartment 51, i.e., the position information of the overall cargo outline, is acquired. The overall cargo outline, if... Figure 6 The image shown in the image, marked with a single-dotted line, is a rectangular image Ic that includes the cargo 52 housed within the cargo compartment when viewed along the shooting direction. Specifically, the coordinates of the four corners 61, 62, 63, and 64 of image Ic are calculated as "positional information of the overall cargo outline." Furthermore, in practice, sometimes loading platform-like wooden blocks are interspersed between multiple cargoes; in such cases, the entire structure including the loading platform can be considered as the "overall cargo outline" and calculated accordingly.

[0063] In the overall cargo compartment inspection process (S400), the overall cargo compartment inspection unit 44 uses the overall image Ia of the cargo compartment 51 acquired in the image acquisition process (S100) and a learned model for overall cargo compartment inspection to inspect the cargo compartment 51. Specifically, as the overall position information of the cargo compartment, similar to the corner detection process (S200) described above, the position information of the four corners Ic1, Ic2, Ic3, and Ic4 of the detected cargo compartment 51, i.e., the X and Y coordinates of corner points C1, C2, C3, and C4, are calculated. Furthermore, the corners and corner points detected by the corner detection unit 42 are strictly different from those detected by the overall cargo compartment inspection unit 44, but for convenience, the same reference numerals are used in the accompanying drawings and this specification.

[0064] In the cargo box calculation process (S500), the cargo box calculation unit 45 extracts the outline of the cargo box 51 from the X and Y coordinates of two diagonally opposite corner points (refer to the image Ir of the cargo box 51). Figure 6 Basically, the image is cropped to resemble a rectangle with its top-left corner point C1(X1,Y1) and bottom-right corner point C2(X2,Y2) as diagonals. However, it is also possible to crop to resemble a rectangle with its bottom-left corner point C3(X1,Y2) and top-right corner point C4(X2,Y1) as diagonals, based on successfully detected corners. See below for reference. Figure 7 The details of the cargo box calculation process (S500) are explained.

[0065] Figure 7 This is a flowchart illustrating the processing sequence executed by the cargo box calculation unit 45 in the cargo box calculation process (S500). For example... Figure 7 As shown, in the cargo box calculation process (S500), firstly in S501, it is determined whether the corner detection unit 42 has successfully detected two corners that are diagonally opposite each other. In S501, if two diagonally opposite corners (upper left corner Ic1 and lower right corner Ic2, or lower left corner Ic3 and upper right corner Ic4) are successfully detected (S501: Yes), the process proceeds to S502 to obtain the position information of the diagonal corners.

[0066] On the other hand, in S501, if the corner detection unit 42 fails to detect two corners that are diagonally opposite each other due to interference such as sunlight during image acquisition (S501: No), the process proceeds to S503 to determine whether a corner has been successfully detected. In S503, if a corner has been successfully detected (S503: Yes), the process proceeds to S504, where the diagonal corner position information is obtained using the position information of one or two corners detected by the corner detection unit 42 and the position information of the corners detected by the overall cargo compartment inspection unit 44 in the overall cargo compartment inspection process (S400).

[0067] If a specific example is shown, for instance, if the corner detection unit 42 successfully detects only the upper left corner Ic1, the position information of the upper left corner Ic1 (upper left corner point C1) is obtained by the corner detection unit 42, and the position information of the lower right corner Ic2 (lower right corner point C2) which is diagonally opposite to the upper left corner Ic1 is obtained by the position information of the corner detected by the overall cargo compartment detection unit 44 in the overall cargo compartment inspection process (S400).

[0068] Furthermore, as another example, when the corner detection unit 42 detects both the upper left corner Ic1 and the lower left corner Ic3, for instance, the position information of the upper left corner Ic1 is obtained from the corner detection unit 42, and the position information of the lower right corner Ic2, which is diagonally opposite to the upper left corner Ic1, is obtained from the corner position information detected by the overall cargo compartment inspection unit 44 in the overall cargo compartment inspection process (S400). Alternatively, the position information of the lower left corner Ic3 detected by the corner detection unit 42 and the position information of the upper right corner Ic4 detected by the overall cargo compartment inspection unit 44 can also be used.

[0069] That is, in S504, the position information of corners detected by the corner detection unit 42 is generally prioritized, and the position information of corners detected by the cargo box overall detection unit 44 is only used for corners that cannot be detected by the corner detection unit 42. This is because, compared with the detection performed by the cargo box overall detection unit 44, which uses the learned model for corner detection, the detection range of corners is smaller than that of the entire cargo box 51, and is less affected by interference such as sunlight during image acquisition or the conditions inside the cargo box, resulting in better accuracy in image parsing.

[0070] In step S503, if no corner is detected (S503: No), proceed to step S505 to obtain the diagonal corner position information from the position information detected by the overall cargo compartment detection unit 44. Essentially, since the overall cargo compartment detection unit 44 has obtained the position information of the four corners, the X and Y coordinates of, for example, the upper left corner point C1 and the lower right corner point C2 are obtained from them. Thus, in steps S502, S504, and S505, the corner detection unit 42 or the overall cargo compartment detection unit 44 obtains the diagonal corner position information.

[0071] After obtaining the diagonal corner position information in each of the processes S502, S504, and S505, in S506, it is determined whether the cargo 52 is detected outside the cargo compartment 51. This determination is based on whether the X and Y coordinates of the four corners 61, 62, 63, and 64 of the overall cargo shape image Ic obtained in the cargo information acquisition process (S300) are outside the image of a rectangle formed by taking the two diagonally opposite points as two vertices, based on the diagonal corner position information obtained in each of the processes S502, S504, and S505.

[0072] In S506, if cargo 52 is detected outside cargo compartment 51 (S506: Yes), proceed to S507, and cut out cargo compartment based on the coordinates of cargo 52. Specifically, calculate the overall extent of cargo compartment as an expanded shape that includes the coordinates of the cargo present on the outside. Further details are described later.

[0073] On the other hand, in S506, if no cargo 52 is detected outside the cargo compartment 51 (S506: No), proceed to S508, and cut out the cargo compartment based on the diagonal corner position information obtained in S502, 504, and 505. That is, calculate the rectangular cargo compartment obtained using the diagonal corner position information, and use it as the range of the entire cargo compartment.

[0074] Goods 52 should have been inside cargo compartment 51, such as Figure 6 As shown, the rectangular image Ic based on the position information of the overall cargo shape should converge to the inside of the image Ir of the cargo compartment 51. Figure 8 This is a diagram used to illustrate an example of image processing in S506. Figure 8 The image Ic, which shows the overall cargo shape as a single-dotted line, has a lateral width that is present on the outside compared to the shape of the rectangular cargo compartment 51, which is shown as a dashed line and obtained using diagonal corner position information.

[0075] Compared to the detection of corners and cargo compartment 51, the feature values ​​of cargo 52 are less susceptible to interference during image capture, making cargo 52 detection easier and reducing the likelihood of detection errors based on image analysis. Therefore, the detected overall cargo shape image Ic is generally correct. So, if the overall cargo shape image Ic is located outside the cargo compartment 51 image Ir obtained using diagonal corner position information, the cargo compartment 51 image Ir can be inferred to be a state where cargo 52 is not fully captured. In S506, it is checked whether such incomplete capture of cargo 52 has occurred.

[0076] exist Figure 8 In the case of the detection results shown, the lateral width of the cargo compartment 51 can be considered to be the correct lateral width in the overall cargo shape image Ic. Regarding the smaller vertical width of the overall cargo shape image Ic, it can also be considered that the cargo 52 is not present at the top, so the vertical width of the cargo compartment 51 image Ir is used. As described above, when the cargo 52 is detected outside the cargo compartment 51 image Ir, a rectangle corrected to include the overall cargo shape image Ic is cropped as the final range of the cargo compartment 51.

[0077] exist Figure 8 In the example, the final cut-out area of ​​the cargo compartment 51 is a rectangular area If, with the lower right corner 62 in the overall cargo shape image Ic and the upper left corner 65 obtained by setting the X coordinate of the upper left corner point C1 in the image Ir of the cargo compartment 51 to the X coordinate of the upper left corner 61 (or lower left corner 63) in the overall cargo shape image Ic as diagonal.

[0078] Refer again Figure 4 In the loading rate calculation process (S600), the loading rate calculation unit 46 calculates the loading rate based on the image Ir of the cargo compartment 51 detected in the cargo compartment calculation process (S500) described in detail above and the images Ib of multiple goods 52 detected in the goods information acquisition process (S300). The loading rate is obtained by dividing the total area of ​​all goods 52 inside the cargo compartment 51 when viewed in the shooting direction by the area of ​​the range of the cargo compartment 51 detected by the cargo compartment calculation unit 45. As a specific image processing step, it can be calculated by blackening the image Ib of all the multiple goods 52 on the vehicle and dividing the number of blackened pixels by the number of pixels in the image Ir of the cargo compartment 51.

[0079] A3. Effect:

[0080] (1) In the cargo compartment storage state detection system 10 and cargo compartment storage state detection method of the first embodiment described above, in the cargo compartment calculation process (S500), if the image Ic of the overall cargo shape exists on the outside compared to the image of the rectangle formed by the diagonal corner position information, an expanded shape that includes the overall cargo shape is calculated as the overall range of the cargo compartment. That is, using the diagonal corner position information and cargo information, the range of the finally calculated cargo compartment 51 is corrected using the cargo information to avoid calculating the range of the cargo compartment 51 such that the cargo 52 is not fully captured. Therefore, the accuracy of calculating the range of the cargo compartment 51 can be improved.

[0081] (2) In addition, since the loading rate of cargo 52 is calculated based on the range of cargo compartment 51 that is detected with good accuracy, the loading rate can be calculated more accurately.

[0082] (3) In the cargo compartment containment state detection system 10 and cargo compartment containment state detection method of the first embodiment described above, as shown in the processes S501 to S505 of the cargo compartment calculation process (S500), if the corner detection unit 42 fails to detect a diagonal corner and thus fails to obtain diagonal corner position information, the diagonal corner position information is obtained from the position information detected by the overall cargo compartment detection unit 44. In this way, the position information of the corners that could not be detected by the corner detection unit 42 is supplemented using the position information detected by the overall cargo compartment detection unit 44. Therefore, even if the corner detection unit 42 cannot detect two corners that are diagonally opposite each other, it is not necessary to repeatedly process until the corners can be accurately detected or the calculation process is interrupted, and the cargo compartment 51 can be appropriately calculated.

[0083] (4) According to the cargo compartment containment status detection system 10 and cargo compartment containment status detection method of the first embodiment described above, based on the calculated loading rate as containment status, information such as the remaining loadable space in the cargo compartment 51 or how much cargo 52 is loaded at which location can be obtained.

[0084] B. Other implementation methods:

[0085] (B1) In the cargo compartment containment status detection system 10 and cargo compartment containment status detection method of the first embodiment described above, it is configured such that, with a dedicated application pre-installed on the portable terminal 12, the system can be utilized by launching the application. Alternatively, it can be configured such that only images of the cargo compartment 51 are captured on the user's smartphone or tablet, and the images are sent to a conventional computer with a processor and storage device as a system management terminal. At the destination computer, various programs are executed to perform corner detection, cargo information acquisition, overall cargo compartment detection, cargo compartment calculation, and loading rate calculation, among other processes. Furthermore, in this case, the processing results can also be returned to the user's smartphone or tablet.

[0086] (B2) In the cargo compartment storage state detection system 10 and cargo compartment storage state detection method of the first embodiment described above, the cargo compartment 51 of the vehicle 50 is set as the detection object, but the cargo compartment 51 is not limited to being possessed by the vehicle 50. For example, the cargo compartment 51 may also be a chute that appropriately stores and supplies parts in a manufacturing plant for vehicle 50 parts.

[0087] (B3) In the cargo compartment containment status detection system 10 and cargo compartment containment status detection method of the first embodiment described above, a loading rate calculation unit 46 is provided to perform the calculation until the loading rate is calculated. However, the loading rate calculation unit 46 may not be provided and the loading rate may not be calculated. For example, it may be a system that only detects which position of the cargo 52 exists in the cargo compartment 51 as the cargo compartment containment status.

[0088] (B4) In the cargo compartment containment status detection system 10 and cargo compartment containment status detection method of the first embodiment described above, the image acquisition unit 41 acquires an image including the entire cargo compartment containing the object to be detected by the camera 36 provided in the cargo compartment containment status detection system 10, but it can also acquire an image obtained by taking pictures using other cameras via the communication interface 24.

[0089] (B5) In the cargo compartment containment status detection system 10 of the first embodiment described above, the overall cargo compartment detection unit 44 may not be included. Even in this case, by using the diagonal corner position information and cargo information obtained by the corner detection unit 42 to correct the range of the cargo compartment 51, the accuracy of calculating the range of the cargo compartment 51 can be improved to avoid calculating the range of the cargo compartment 51 in a way that the cargo 52 is not fully captured. In addition, regarding the "position of the cargo" as cargo information, for example, the coordinates of a corner of the overall cargo shape can be used.

[0090] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in each embodiment corresponding to the technical features described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. In addition, any technical feature that is not described as an essential technical feature in this specification can be appropriately deleted.

[0091] Label Explanation

[0092] 10…Cargo compartment containment status detection system, 12…Portable terminal, 14…Control unit, 16…CPU, 18…ROM, 20…RAM, 22…Memory, 24…Communication interface, 26…Input / output interface, 28…Bus, 30…Display screen, 32…Microphone, 34…Speaker, 36…Camera, 41…Image acquisition unit, 42…Corner detection unit, 43…Cargo information acquisition unit, 44…Cargo compartment overall detection unit, 45…Cargo compartment calculation unit, 46…Loading rate calculation unit, 47…Learning unit, 50…Transport vehicle Vehicle, 51…cargo box, 52…cargo, 53, 54, 55…side wall, 56…lower section, 57…platform, 58…cab, 61, 62, 63, 64…corners, 65…top left corner, C1…top left corner, C2…bottom right corner, C3…bottom left corner, C4…top right corner, Ia…overall image, Ib…cargo image, Ic…overall cargo outline image, Ic1…top left corner, Ic2…bottom right corner, Ic3…bottom left corner, Ic4…top right corner, Ir…cargo box image

Claims

1. A cargo compartment accommodation state detection system that detects an accommodation state of a cargo in a cargo compartment, comprising: an image acquisition unit that acquires an image obtained by photographing an entirety of the cargo compartment in a state in which the cargo accommodated can be seen, along a predetermined photographing direction; a corner detection unit that detects a plurality of corners of the cargo compartment from the image acquired by the image acquisition unit, using a learning completed model for corner detection that has been learned using training image data obtained by labeling positions of at least two corners that are diagonally opposite each other among the plurality of corners of the cargo compartment that are rectangular when viewed along the photographing direction, from images acquired for machine learning; a cargo information acquisition unit that acquires cargo information including information related to a position of the cargo in the cargo compartment; and a cargo compartment calculation unit that calculates a range of the entirety of the cargo compartment, using position information of the two corners that are diagonally opposite each other in the cargo compartment, that is, diagonal corner position information, and the cargo information acquired by the cargo information acquisition unit, the cargo information being detected from the image acquired by the image acquisition unit, using a learning completed model for cargo detection that has been learned using a feature amount of the cargo.

2. The cargo compartment accommodation state detection system according to claim 1, further comprising a cargo compartment entirety detection unit that detects the entirety of the cargo compartment from the image acquired by the image acquisition unit, using a learning completed model for cargo compartment entirety detection that has been learned using training image data obtained by labeling the entirety of the cargo compartment when viewed along the photographing direction, from the images acquired for the machine learning, wherein the cargo compartment calculation unit performs the following processing: the diagonal corner position information is acquired by complementing position information of the corner that is not detected by the corner detection unit, among the two corners, using position information of the entirety of the cargo compartment detected by the cargo compartment entirety detection unit.

3. The cargo compartment accommodation state detection system according to claim 2, wherein the cargo information includes position information of an entirety cargo outline portion that is a portion of an imaginary rectangle that contains the entirety of the cargo accommodated in the cargo compartment when viewed along the photographing direction, and the cargo compartment calculation unit performs the following processing: in a case in which the entirety cargo outline portion exists outside compared to an outline of the cargo compartment that is rectangular and is acquired using the diagonal corner position information, the outline that has been expanded so as to contain the portion that exists outside is calculated as the range of the entirety of the cargo compartment, and in a case in which the entirety cargo outline portion does not exist outside compared to the outline of the cargo compartment that is rectangular and is acquired using the diagonal corner position information, the outline of the cargo compartment that is rectangular and is acquired using the diagonal corner position information is calculated as the range of the entirety of the cargo compartment. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 4. The cargo compartment accommodation state detection system according to any one of claims 1 to 3, wherein a loading rate calculation section that calculates a loading rate of the cargo in the cargo compartment is further provided, the cargo information includes information related to a size of the cargo, the loading rate calculation section performs the following processing: the loading rate is calculated by dividing a total area of all the cargo in the cargo compartment when viewed in the photographing direction, which is obtained by the cargo information acquisition section, by an area of the range of the cargo compartment detected by the cargo compartment calculation section.

5. The cargo compartment accommodation state detection system according to any one of claims 1 to 3, wherein the cargo compartment is a cargo compartment provided on a cargo bed of a transport vehicle.

6. A recording medium storing a cargo compartment accommodation state detection program for detecting an accommodation state of cargo in a cargo compartment, the cargo compartment accommodation state detection program causing a computer to realize the following functions: a function of acquiring an image obtained by photographing an entirety of the cargo compartment in a state in which the loaded cargo can be seen in a predetermined photographing direction; a function of detecting a plurality of corners of the cargo compartment from the acquired image using a learned model for corner detection that has been learned using training image data obtained by labeling positions of at least two corners that are diagonally opposite each other among the plurality of corners of the cargo compartment that are rectangular when viewed in the photographing direction, from images acquired for machine learning; a function of acquiring cargo information including information related to a position of the cargo in the cargo compartment; and a function of calculating a range of the entirety of the cargo compartment using position information of the two corners that are diagonally opposite each other in the cargo compartment, i.e., diagonal corner position information, and the cargo information. the cargo information is detected from the image acquired by the function of acquiring an image using a learned model for cargo detection that has been learned using features of the cargo.

7. A cargo compartment accommodation state detection method of detecting an accommodation state of cargo in a cargo compartment using a cargo compartment accommodation state detection system, wherein the cargo compartment accommodation state detection system includes an image acquisition section, a corner detection section, a cargo information acquisition section, and a cargo compartment calculation section, the cargo compartment accommodation state detection method includes: an image acquisition process of acquiring, by the image acquisition section, an image obtained by photographing an entirety of the cargo compartment in a state in which the loaded cargo can be seen in a predetermined photographing direction; a corner detection process of detecting, by the corner detection section, a plurality of corners of the cargo compartment from the image acquired by the image acquisition section using a learned model for corner detection that has been learned using training image data obtained by labeling positions of at least two corners that are diagonally opposite each other among the plurality of corners of the cargo compartment that are rectangular when viewed in the photographing direction, from images acquired for machine learning. The cargo information acquisition step acquires cargo information including information about the position of the cargo in the cargo compartment, using the cargo information acquisition unit. And The cargo compartment calculation step calculates the range of the entire cargo compartment, using the position information of the two corners diagonally opposite each other in the cargo compartment, i.e., diagonal corner position information, and the cargo information acquired by the cargo information acquisition unit. The cargo information is detected from the image acquired by the image acquisition unit, using a completed model for cargo detection that has been machine-learned on a feature quantity of the cargo.

Images