Information processing system, information processing method and program

Abstract

To efficiently and properly count the number of cargoes.SOLUTION: When a load-carrying platform is imaged from a first direction, an information processing system performs notification that urges imaging from a second direction and detects a detection object included in a photographed image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] In the transportation of goods, goods in a logistics warehouse are loaded onto trucks, etc., and then unloaded from trucks, etc., and carried into a warehouse. In managing goods, it is necessary to accurately grasp and manage how many goods have been loaded onto which trucks, and how many goods have been unloaded. Furthermore, efficient transportation is achieved by placing the goods on pallets and transporting them with a forklift, etc. These pallets are often loaned out by rental companies, so it is necessary to accurately count the number of pallets loaded onto and unloaded from trucks, etc. However, when counting is done manually, it is sometimes difficult to accurately count the number of pallets, which is one of the reasons why pallets are lost. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-197500 A DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0004] Patent Document 1 describes a technology that uses images to measure the quantity and weight of cargo loaded in the hold of a cargo ship. However, in the case of transport trucks, trucks with double doors called wing trucks are used, and cargo (including pallets, etc.) must be counted from both sides. Therefore, in the current situation where the number is counted manually, there are cases where counting is missed, such as only counting one side of the truck, and the number of pallets cannot be measured accurately.

[0005] Therefore, an object of the present invention is to count the number of packages efficiently and appropriately. [Means for solving the problem]

[0006] In order to achieve the above object, the information processing system of the present invention comprises: An imaging means for imaging the loading platform of the vehicle; a notification means for notifying the user to take an image from a second direction when the image capturing means captures an image of the loading platform from a first direction; A detection means for detecting a detection target included in an image captured by the imaging means; The present invention is characterized by the following. Effect of the Invention

[0007] According to the present invention, it is possible to count the number of packages efficiently and appropriately. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of an overall system configuration according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to an embodiment of the present invention. [Diagram 3] 10 is a flowchart illustrating an example of photography for object detection processing in an embodiment of the present invention. [Figure 4] 10 is a flowchart illustrating an example of photography for object detection processing in an embodiment of the present invention. [Diagram 5] 11 is a flowchart illustrating an example of an object detection process in the embodiment of the present invention. [Figure 6] FIG. 11 is a diagram showing an example of a basic information input screen in the embodiment of the present invention. [Figure 7] FIG. 4 is a diagram showing an example of a front image capturing screen in the embodiment of the present invention. [Figure 8] FIG. 13 is a diagram showing an example of a front image shooting result screen in the embodiment of the present invention. [Figure 9] FIG. 13 is a diagram showing an example of a back image photographing screen in the embodiment of the present invention. [Figure 10] FIG. 13 is a diagram showing an example of a back image capture result screen in the embodiment of the present invention. [Figure 11] FIG. 13 is a diagram showing an example of object detection before correction in the embodiment of the present invention. [Figure 12] FIG. 13 is a diagram showing an example of object detection after correction in the embodiment of the present invention. [Figure 13] FIG. 13 is a diagram showing an example of a surface object number correction screen in the embodiment of the present invention. [Figure 14] FIG. 13 is a diagram showing an example of a back-surface object number correction screen in the embodiment of the present invention. [Figure 15] FIG. 13 is a diagram showing an example of a front surface (second sheet) object number correction screen in the embodiment of the present invention. [Figure 16] FIG. 13 is a diagram showing an example of a back side (second sheet) object number correction screen in the embodiment of the present invention. [Figure 17] FIG. 13 is a diagram showing an example of a screen displaying a list of registered information such as the number of detected objects in the embodiment of the present invention. [Figure 18] FIG. 13 is a diagram showing an example of a detailed display screen of the number of detected objects in the embodiment of the present invention. [Figure 19] 11 is a diagram showing an example of a detected palette rectangle list in the embodiment of the present invention. FIG. [Figure 20] 4 is a data table showing an example of information registration such as the number of detected objects in the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0010] FIG. 1 is a diagram showing an example of the overall configuration of a counting system according to an embodiment of the present invention.

[0011] The counting system of the present invention is a system in which a server device 101 and a client terminal 102 are connected to be able to communicate with each other. The client terminal 102 may be used as an edge device in a logistics center where pallets are loaded and unloaded, or may be used by a manager in an office. The client terminal 102 used in a logistics center is preferably a terminal (information processing device such as a smartphone) that has an imaging function by itself, but may be a terminal that can be connected to another imaging device. The client terminal 102 used in an office is, for example, an information processing device such as a personal computer, and is a terminal that acquires and manages captured images and the number of pallets from the client terminal 102 in each logistics center. In FIG. 1, only one client terminal 102 is shown in each of the office and the logistics center, but there may be multiple client terminals in each.

[0012] Server device 101 is connected to client terminal 102 so as to be able to communicate with it, and has the function of detecting palettes from a captured image, counting the number of palettes, and storing and managing data such as the captured image and the number of palettes. Also, server device 101 transmits various data to client terminal 102 in response to a request from client terminal 102. Note that, although server device 101 is shown in Fig. 1 as a single server device, it may be a system made up of multiple server devices, or may be a so-called cloud system.

[0013] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device (server device 101, client terminal 102) according to the present invention.

[0014] As shown in FIG. 2, the information processing device has a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a storage device 204, an input controller 205, an audio controller 206, a video controller 207, a memory controller 208, and a communication I / F controller 209 connected via a system bus 200.

[0015] The CPU 201 comprehensively controls each device and controller connected to the system bus 200 .

[0016] ROM 202 or external memory 213 holds the BIOS (Basic Input / Output System) and OS (Operating System), which are control programs executed by CPU 201, computer-readable and executable programs for realizing this information processing method, and various necessary data (including data tables).

[0017] The RAM 203 functions as a main memory, a work area, etc., for the CPU 201. The CPU 201 loads programs and the like required for executing processes from the ROM 202 or the external memory 213 into the RAM 203, and executes the loaded programs to realize various operations.

[0018] The input controller 205 controls input from an input device such as a keyboard 210 or a pointing device such as a mouse (not shown). If the input device is a touch panel, the user can give various instructions by pressing (touching with a finger or the like) icons, cursors, or buttons displayed on the touch panel.

[0019] The touch panel may be a touch panel capable of detecting positions touched by multiple fingers, such as a multi-touch screen.

[0020] The video controller 207 controls display on an external output device such as a display 212. The display also includes the display of a notebook computer integrated with the main body. Note that the external output device is not limited to a display, and may be, for example, a projector. In addition, for the device capable of receiving the above-mentioned touch operation, an input device is also provided.

[0021] The video controller 207 is capable of controlling a video memory (VRAM) for display control, and can use part of the RAM 203 as the video memory area, or can provide a separate dedicated video memory.

[0022] The memory controller 208 controls access to the external memory 213. As the external memory, an external storage device (hard disk) for storing a boot program, various applications, font data, user files, edit files, and various data, a flexible disk (FD), or a compact flash (registered trademark) memory connected to a PCMCIA card slot via an adapter, etc., can be used.

[0023] The communication I / F controller 209 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, communication using TCP / IP, telephone lines such as ISDN, and 4G and 5G lines for mobile phones are possible.

[0024] The CPU 201 enables display on the display 212 by, for example, executing a process of expanding (rasterizing) an outline font in a display information area in the RAM 203. The CPU 201 also enables a user to give instructions using a mouse cursor (not shown) or the like on the display 212.

[0025] Next, an overview of the task of counting the number of pallets, which is an example of an application of the present invention, will be described. Note that in this embodiment, pallets used for transporting luggage will be used as a specific example of the detection object, but the detection object is not limited to pallets and can be anything that is transported by a so-called wing vehicle.

[0026] First, a worker at the logistics center registers the details of the work via a basic information input screen, an example of which is shown in Fig. 6, displayed on the client terminal 102. Then, in accordance with the registered details, a photograph is taken of the luggage loaded on a vehicle such as a truck (e.g., a wing truck), and the number of pallets is counted by detecting the pallets from the photographed image.

[0027] A pallet is a platform on which cargo is placed when transporting cargo using a forklift or hand truck. By inserting the claws of a forklift or other vehicle into the pallet, the cargo on the pallet can be transported all at once.

[0028] In this embodiment, the pallets to be counted are those loaded on a truck called a wing truck, which has openings on both sides of the loading platform and allows loading and unloading of cargo from either side.

[0029] The pallets to be counted are those entering the logistics center and those leaving the logistics center, so in the case of entering the center (i.e., when the cargo is being unloaded), a photograph is taken of the pallet before the cargo is unloaded, and in the case of leaving the center (i.e., when the cargo is being loaded), a photograph is taken of the pallet after the cargo is loaded and counted. Also, when additional cargo is loaded onto a vehicle that already has cargo, or when only some of the cargo loaded on a vehicle is unloaded, photographs are taken before and after loading and unloading and the difference is counted.

[0030] Next, an example of the pallet counting process will be described with reference to the flowcharts of FIGS.

[0031] The processing shown in the flowcharts of FIGS. 3 to 5 is processing in which the CPU 201 of the client terminal 102 reads out and executes a predetermined control program.

[0032] In step S101, information related to the entry / exit category (1: entry, 2: exit) 602, vehicle number 603, notes 604, front side photography (photographed from a first direction) judgment (1: all, 2: part) 605, and back side photography (photographed from a second direction) judgment (1: all, 2: part, 3: none) 606 input by the user via the basic information input screen 601 shown in Figure 6 is obtained.

[0033] Here, the basic information input screen, an example of which is shown in Fig. 6, will be described. The basic information input screen is a screen that accepts instructions such as work contents by user operation. As shown in Fig. 6, it includes buttons (602-1, 602-2) for selecting whether the work is related to warehousing or warehousing, an input field 603 for accepting input of a vehicle number that is information identifying the vehicle to be loaded and unloaded, a remarks input field 604 for accepting input of other necessary information, a button 605 for selecting work contents for the front side (right side or left side) of the vehicle, and a button 606 for selecting work contents for the back side of the vehicle (left side if the front side is the right side, right side if the front side is the left side). Then, the input information is registered by pressing a basic information registration button 607.

[0034] In step S102, it is determined whether the photographing of the front surface is "all" or "part" based on the information acquired in step S101.

[0035] If it is "all", the process proceeds to step S103. If it is "part", the process proceeds to the process shown in the flowchart of FIG.

[0036] "All" means that all pallets on a vehicle will be removed, or that pallets will be loaded onto a vehicle that does not already have pallets. "Part" means that some of the pallets on a vehicle will be removed, or that additional pallets will be loaded onto a vehicle that already has some pallets.

[0037] In step S103, the front image shooting screen shown in Fig. 7 is displayed, and shooting is performed by accepting a user's press of the shooting button (701). Then, the shot image is displayed in the shot image area (801) of the front image shooting result screen shown in Fig. 8.

[0038] The user checks the photographed image displayed in the photographed image area 801, and if he / she determines that there is no problem, presses the next button (802). When the next button is pressed, the process proceeds to step S104.

[0039] To retake the photograph, the user presses the camera button 803. When the camera button is pressed, the front image photographing screen (FIG. 7) is displayed again and the screen waits for the user to press the photographing button.

[0040] Here, the front image photographing screen shown in FIG. 7 will be described.

[0041] The front image shooting screen displays a shooting button 701 and a shooting image area 702. The shooting image area is an area where a shooting subject is displayed in live view. The shooting button is a button that accepts an instruction from a user to shoot an image displayed in the shooting image area, and pressing the shooting button executes an image capture process.

[0042] Also, in the photographed image area, a guideline (line 703 shown by a dashed line) is displayed so that the angle of view is such that an image suitable for counting the number of pallets can be photographed. Specifically, a guideline (a figure showing the outline of the loading platform, for example) is displayed to guide the user to photograph the loading platform of the truck from the diagonal front or diagonal rear. When photographing from the diagonal front or diagonal rear, the display method of the guideline is switched according to the photographing direction. For example, when a switching button (704 or 902) is pressed, the guideline is controlled so that the left and right sides are reversed. By photographing according to this guideline, the loading platform is photographed from an oblique direction, so that the pallets loaded on the back side of the loading platform are less likely to be included in the angle of view, and it is easier to distinguish between the pallets loaded on the back side and the pallets loaded on the front side. As a result, there is an effect of reducing erroneous detection, in which the pallets on the back side are detected as the pallets loaded on the front side. In addition, by photographing from an oblique direction, there is no need to move to the front of the front and back sides to photograph, so there is also an effect of shortening the distance the worker must travel when photographing.

[0043] In step S104, the number of pallets is counted from the image captured in step S103. The process of step S104 will be described later with reference to the flowchart of FIG.

[0044] In step S105, it is determined whether the back side is photographed as "all", "part", or "none" based on the information acquired in step S101.

[0045] If the answer is "all", the process proceeds to step S106. If the answer is "some", the process proceeds to step S109. If the answer is "none", the process proceeds to step S114.

[0046] "All" and "some" are as explained in step S102, and "none" means that no pallets will be loaded or unloaded from the back side of the vehicle (the side that is not the front, such as the left side if the right side of the vehicle is considered the front).

[0047] In this embodiment, the necessity of photographing the back side is determined based on the information inputted in the basic information input screen, but the determination may be made using the photographed image of the front side. Specifically, since it is not usual to load a large amount of luggage on only one side (only the front side) in order to ensure the left-right balance of the loading platform, when a predetermined number of luggage or more is / has been loaded on the front side (when a predetermined condition is met, such as when a predetermined number of pallets or more are detected from the photographed image of the front side), it is determined that luggage is / has been loaded on the back side as well, and control is exercised to transition to photographing processing of the back side.

[0048] First, a case where it is determined in step S105 that the result is "all" will be described.

[0049] In step S106, the back image capture screen shown in Fig. 9 is displayed, and capture is performed by accepting a user's press of the capture button (901). The captured image is then displayed in the captured image area (1001) of the back image capture result screen shown in Fig. 10.

[0050] The user checks the captured image displayed in the captured image area, and if he or she determines that there is no problem, presses the next button (1002). When the next button is pressed, the process proceeds to step S107.

[0051] To retake the image, the user presses the camera button 1003. When the camera button is pressed, the back image capture screen (FIG. 9) is displayed again and the screen waits for the user to press the capture button.

[0052] The back image photographing screen shown in FIG. 9 has the same configuration as the front image photographing screen in FIG. 7, so a description thereof will be omitted.

[0053] In step S107, the number of pallets is counted from the image captured in step S106. The process of step S107 will be described later with reference to the flowchart of FIG.

[0054] In step S108, the current date and time, the entry / exit classification (1: entry, 2: exit) obtained in step S101, the vehicle number, notes, the number of pallets detected in step S104 (number of front sides detected 1), the corrected number of pallets (number of front sides corrected 1), the number of pallets detected in step S107 (number of back sides detected 1), the corrected number of pallets (number of back sides corrected 1), an image of the pallets enclosed in a rectangle in the image taken in step S103 (front side detected image 1), and an image of the pallets enclosed in a rectangle in the image taken in step S106 (back side detected image 1) are registered in the data table shown in FIG. 20.

[0055] Then, the process of this flowchart ends.

[0056] Next, a case where it is determined in step S105 that the number of pieces is "partial" will be described.

[0057] The processes in steps S109 and S110 are similar to those in steps S106 and S107, and involve taking images of the cargo before it is loaded onto or unloaded from a vehicle, and detecting the number of pallets.

[0058] The processing of steps S111 and S112 is similar to that of steps S106 and S107, but unlike S109 and S110, steps S111 and S112 are processes for taking images of the cargo after it has been loaded onto a vehicle or after it has been unloaded (part of) from a vehicle, and detecting the number of pallets.

[0059] In step S113, the following information is registered in the data table of Figure 20 according to the entry / exit classification obtained in step S101, along with the current date and time, the vehicle number obtained in step S101, remarks, an image of the pallet surrounded by a rectangle in the image taken in step S103 (front detection image 1), an image of the pallet surrounded by a rectangle in the image taken in step S109, and an image of the pallet surrounded by a rectangle in the image taken in S111.

[0060] If the entry / exit category is "1: Entry", the number of pallets detected in step S110 (back detection number 1) minus the number of pallets detected in step S112 (back detection number 2) (i.e., the number of pallets removed from the vehicle) and the number of pallets obtained by subtracting the number of pallets after correcting back detection number 1 (back correction number 1) minus the number after correcting back detection number 2 (back correction number 2) are registered.

[0061] If the entry / exit category is "2: Exit", the number of pallets detected in step S112 (back detection number 2) minus the number of pallets detected in step S110 (back detection number 1) (i.e., the number of pallets loaded onto the vehicle) and the number of pallets obtained by subtracting the number of pallets after correcting back detection number 2 (back correction number 2) from the number after correcting back detection number 1 (back correction number 1) are registered.

[0062] Next, a case where it is determined in step S105 that "3: None" (backside photography is not performed) will be described.

[0063] In step S114, the current date and time, the entry / exit category (1: entry, 2: exit) obtained in step S110, the vehicle number, notes, the number of pallets detected in step S104 (number of front detections 1), the corrected number of pallets (number of front corrections 1), and an image of the pallets in the image photographed in step S103 surrounded by a rectangle (number of front detections 1) are registered in the data table shown in FIG. 20.

[0064] Next, using the flowchart in Figure 4, we will explain the processing when the result of step S102 is determined to be "partial" (i.e., when some of the luggage loaded on the surface of the vehicle is unloaded, or when there is luggage already loaded on the surface of the vehicle and additional luggage is to be loaded).

[0065] The processing in steps S201 to S203 is similar to the processing in steps S103 to S105 in the flowchart of FIG. 3, and therefore a description thereof will be omitted.

[0066] First, the process when the result of the determination in step S203 is "1: All" will be described.

[0067] The processes in steps S204 and S205 are similar to those in steps S106 and S107 in FIG. 3, and therefore will not be described.

[0068] The processing of steps S206 and S207 is similar to that of steps S103 and S104, but unlike S103 and S104, steps S206 and S207 are processes for taking images of the cargo after it has been loaded onto or unloaded from a vehicle, and detecting the number of pallets.

[0069] In step S208, the following information is registered in the data table of Figure 20 according to the entry / exit classification obtained in step S101, along with the current date and time, the vehicle number obtained in step S101, remarks, an image of the pallet surrounded by a rectangle in the image taken in step S204 (back detection image 1), an image of the pallet surrounded by a rectangle in the image taken in step S201, and an image of the pallet surrounded by a rectangle in the image taken in S206.

[0070] If the inbound / outbound classification is "1: Inbound", the number of pallets detected in step S202 (front detection number 1) minus the number of pallets detected in step S207 (front detection number 2) (i.e., the number of pallets unloaded from the vehicle) and the number of pallets obtained by subtracting the number of pallets after correcting front detection number 1 (front correction number 1) minus the number after correcting front detection number 2 (front correction number 2) are registered.

[0071] If the inbound / outbound category is "2: Outbound", the number of pallets detected in step S207 (front detection number 2) minus the number of pallets detected in step S202 (front detection number 1) (i.e., the number of pallets loaded onto the vehicle) and the number of pallets after correcting front detection number 2 (front correction number 2) minus the number after correcting front detection number 1 (front correction number 1) are registered.

[0072] Next, the process when the result of the determination in step S203 is "2: Part" will be described.

[0073] The processing in steps S209 and S210 is similar to that in steps S106 and S107 in FIG. 3, and therefore a description thereof will be omitted here.

[0074] The processing in steps S211 and S212 is similar to that in steps S206 and S207, and therefore the description thereof will be omitted here.

[0075] The processing in steps S213 and S214 is similar to that in steps S111 and S112 in FIG. 3, and therefore a description thereof will be omitted here.

[0076] In step S215, the following information is registered in the data table of Figure 20 according to the entry / exit classification obtained in step S101, along with the current date and time, the vehicle number obtained in step S101, remarks, an image of the pallet surrounded by a rectangle in the image taken in step S201, an image of the pallet surrounded by a rectangle in the image taken in step S211, an image of the pallet surrounded by a rectangle in the image taken in step S209, and an image of the pallet surrounded by a rectangle in the image taken in S213.

[0077] If the entry / exit category is "1: entry", the number of pallets detected in step S202 (front detection number 1) minus the number of pallets detected in step S212 (front detection number 2) (i.e., the number of pallets unloaded from the vehicle), and the number after front detection number 1 is corrected (front correction number 1) minus the number after front detection number 2 is corrected (front correction number 2) are registered. The number of pallets detected in step S210 (reverse detection number 1) minus the number of pallets detected in step S214 (reverse detection number 2) (i.e., the number of pallets unloaded from the vehicle), and the number after back detection number 1 is corrected (reverse correction number 1) minus the number after back detection number 2 is corrected (reverse correction number 2) are registered.

[0078] If the entry / exit category is "2: Exit", the number of pallets detected in step S212 (front detection number 2) minus the number of pallets detected in step S202 (front detection number 1) (i.e., the number of pallets loaded onto the vehicle), and the number after front detection number 2 is corrected (front correction number 2) minus the number after front detection number 1 is corrected (front correction number 1) are registered. The number of pallets detected in step S214 (reverse detection number 2) minus the number of pallets detected in step S210 (reverse detection number 1) (i.e., the number of pallets loaded onto the vehicle), and the number after back detection number 2 is corrected (reverse correction number 2) minus the number after back detection number 1 is corrected (reverse correction number 1) are registered.

[0079] Next, a case where it is determined in step S203 that "3: None" (back image is not captured) will be described.

[0080] The processes in steps S216 and S217 are similar to those in steps S206 and S207, and therefore will not be described here.

[0081] In step S218, the following information is registered in the data table of Figure 20 according to the entry / exit classification obtained in step S101, along with the current date and time, the vehicle number obtained in step S101, remarks, an image of the pallet surrounded by a rectangle in the image taken in step S201, and an image of the pallet surrounded by a rectangle in the image taken in step S216.

[0082] If the inbound / outbound classification is "1: Inbound", the number of pallets detected in step S202 (front detection number 1) minus the number of pallets detected in step S217 (front detection number 2) (i.e., the number of pallets unloaded from the vehicle) and the number of pallets after correcting front detection number 1 (front correction number 1) minus the number after correcting front detection number 2 (front correction number 2) are registered.

[0083] If the inbound / outbound classification is "2: Outbound", the number of pallets detected in step S217 (front detection number 2) minus the number of pallets detected in step S202 (front detection number 1) (i.e., the number of pallets loaded onto the vehicle) and the number of pallets after correcting front detection number 2 (front correction number 2) minus the number after correcting front detection number 1 (front correction number 1) are registered.

[0084] Next, the palette detection process in steps S104, S107, S110, S112, S202, S205, S207, S210, S212, S214, and S217 will be described with reference to the flowchart of FIG.

[0085] In step S301, the client terminal 102 transmits the captured image acquired in step S103 or the like to the server device 101.

[0086] In step S302, the server device 101 receives the image transmitted from the client terminal 102 in step S301.

[0087] In step S303, the server device 101 detects a palette from the received image, and acquires a palette detection image and a detected palette rectangle list.

[0088] The palette detection process is performed using a known technique. For example, there is a method of detecting the palette using a trained model that has been trained using a palette image as learning data. In addition, the method is not limited to the method using a trained model, and any technique that detects a specific object from an image may be used.

[0089] FIG. 11 shows an image in which a palette is detected and the detected palette is enclosed in a rectangle.

[0090] An example of the detected palette rectangle list is shown in Fig. 19. As shown in Fig. 19, the detected palette rectangle list includes information such as the starting point of the detected palette rectangle (XY coordinates that serve as the reference for identifying the position of the palette rectangle in the image) and the height and width of the palette rectangle, and the list is generated for each captured image. Note that the starting point in Fig. 19 is coordinate information (X is the vertical direction, Y is the horizontal direction) with the top left coordinate of the captured image being (0,0).

[0091] In step S304, the server device 101 performs a correction process for the pallet detection image. This correction process makes it possible to exclude pallets loaded on the back side of the vehicle from detection targets when pallets loaded on the front side of the vehicle are detected even though it is intended to detect pallets loaded on the front side of the vehicle.

[0092] A specific correction method will be described below.

[0093] First, a pallet in the front row (the row closest to the driver's seat) and at the bottom of the vehicle is identified (hereinafter, the identified pallet is referred to as pallet A).

[0094] Specifically, if the right side of the vehicle is the surface, the rectangle with the largest X and Y coordinates of the origin is identified from the information registered in the rectangle list in Fig. 19 (i.e., the pallet at the bottom right in the captured image is identified). If the left side of the vehicle is the surface, the rectangle with the largest X coordinate and the smallest Y coordinate of the origin is identified (i.e., the pallet at the bottom left in the captured image is identified). The pallet 1101 in the example in Fig. 11 is pallet A.

[0095] In this embodiment, a process is described for identifying a pallet in the front row (the row closest to the driver's seat) and at the bottom of the vehicle. However, this process is based on the assumption that the luggage is loaded close to the driver's seat. If the luggage is loaded close to the rear, the pallet in the last row and at the bottom will be identified.

[0096] Then, the following process is performed to sequentially identify the bottommost pallet among the pallets loaded on the surface (pallets 1101, 1102, 1103, etc. in the example of FIG. 11).

[0097] First, create a palette lower list, store palette A in it, and set the pointer to 0.

[0098] Next, from the detected palette rectangle list, a palette (hereinafter, palette B) whose origin Y is smaller than the origin Y of the palette currently pointed to by the pointer and whose Euclidean distance is the closest is obtained.

[0099] Then, if the X coordinate of the bottom left of palette B (i.e., the X coordinate of the origin + height) is greater than the X coordinate of the origin of the palette pointed to by the pointer, palette B is registered in the palette bottom list and the pointer is incremented.

[0100] The reason why pallets that overlap in height are identified in this way is that when the front side is photographed obliquely, the pallets loaded on the front side will be in a positional relationship where they overlap in height with the pallets in the adjacent row, whereas the pallets loaded on the back side (e.g. 1103) will not be in a positional relationship where they overlap in height. Therefore, this processing makes it possible to identify only the pallets on the front side.

[0101] By repeating this process, it is possible to identify a pallet that overlaps in height with the bottommost pallet in the front row, and then to identify other pallets that overlap in height with the identified pallet, thereby making it possible to sequentially identify the bottommost pallets among the pallets loaded on the surface of the vehicle.

[0102] Next, the pallets stacked on top of the pallets in the lower pallet list are detected. Specifically, this is done by the following process.

[0103] First, the midpoint Z of each detected palette rectangle stored in the palette lower list is calculated.

[0104] Then, among the rectangles included in the detected palette rectangle list, a rectangle whose midpoint Z Y coordinate is between the Y coordinate of the starting point and the Y coordinate of the upper right corner of the rectangle is identified, and the identified rectangle is stored in the palette surface list. The number of palettes stored in the palette surface list is set as the corrected number of palettes.

[0105] The above process is the process for identifying and counting the pallets loaded on the surface.

[0106] Of the palettes detected from the captured image, those that are not stored in the palette rectangle list may be displayed in a different way so that they can be identified in the detection result image (for example, by changing the color of the rectangle, drawing a red diagonal line through the rectangle, etc.). An example of an image with a different display mode is shown in FIG. 12.

[0107] Furthermore, if it is the second image, the display format is controlled so that the differences from the first image (areas where luggage was removed from the loading platform and newly loaded luggage) can be recognized.

[0108] It is also possible to display pallets in an identifiable manner by changing the display format of the rectangles depending on the type of pallet, or to count the number of pallets by type. Regarding pallets, since different types of pallets (which need to be managed separately), such as "pallets owned by the company and rented pallets" or "pallets of different sizes," may be loaded onto the same vehicle, it is possible to manage them separately by displaying them in an identifiable manner. When counting each type separately, the number can also be modified for each type, and the number for each type can be registered in the data table (Figure 20).

[0109] In step S305, the detection result image after correction in step S304 and the number of detected palettes after correction are obtained.

[0110] In step S306, the corrected detection result image and the number of detected palettes are transmitted to the client terminal.

[0111] In step S307, the detection result image and the number of detected palettes transmitted in step S306 are received.

[0112] In step S308, the number of detected palettes received in step S307 is modified by the user via the screens, examples of which are shown in FIGS.

[0113] 13 is a screen displayed on the client terminal 102 in the detailed processing of steps S104 and S202, and displays a palette detection result image 1301, a palette detection number 1302, a palette number correction acceptance section 1303, and the like. The palette detection result image is an image showing the result of executing the palette detection process on the photographed image as described above, and the detected palettes are surrounded by rectangles. The palette detection number 1302 is the number of palettes detected by the detection process. The correction acceptance section 1303 is a button that accepts correction of the number of palettes detected by the detection process, and when + is selected, the palette number is increased by 1, and when - is selected, the palette number is decreased by 1. When the registration button 1804 is pressed, data such as the detection number and the corrected number is recorded in the memory of the client terminal 102, and is registered in the data table shown in FIG. 20 in the palette registration process of step S108 of FIG. 3 as described above. Note that the above data may be transmitted to the server device 101 by accepting the pressing of the registration button 1804, and may be registered in the data table shown in FIG. 20.

[0114] Then, the process transitions to back side photographing processing by pressing a back side photographing button 1305. When back side photographing is not necessary (when "None" is selected in steps S105 and S203, etc.), the button is controlled not to be displayed (when the button is not "None", the button is controlled to be displayed).

[0115] Fig. 14 shows a screen displayed on the client terminal in detailed processing of steps S107, S110, S205, and S210, Fig. 15 shows a screen displayed on the client terminal in detailed processing of steps S207, S212, and S217, and Fig. 16 shows a screen displayed on the client terminal in detailed processing of steps S112 and S214. In all cases, as in Fig. 13, the display of the transition button (a button corresponding to 1305 in Fig. 13) is controlled depending on whether the next shooting process is required.

[0116] In this way, when it is necessary to photograph the back side or the second image (when it is determined that it is necessary based on the information entered in the basic information input screen, or when it is determined that it is necessary to photograph the back side based on the photographed image of the front side, etc.), the screen display is controlled to prompt the user to photograph the back side or the second image (prompting notification is given), thereby making it possible to prevent the user from forgetting to photograph the back side or the second image. Also, when it is not necessary to photograph the back side or the second image, the button for transitioning to photographing the back side or the second image is controlled not to be displayed, making it possible to efficiently proceed with work without incurring unnecessary work.

[0117] Once the above processing has been performed, the flow returns to the processing in the flowcharts of FIGS.

[0118] Next, a description will be given of the data list screen shown in Fig. 17. The data list screen is a screen generated using data registered by executing the processes in the flowcharts of Fig. 3 and Fig. 4 (data registered in the data table shown in Fig. 20), and is a screen displayed on the client terminal 102.

[0119] In a date input field 1701 and a time input field 1702, input of the date and time related to the data that the user wishes to confirm is accepted, and by pressing a query button 1703, data for the entered date and time is searched for, and the data is displayed in a list display area 1704.

[0120] The list display area displays the date and time when the data was registered (1705), a category (1706) indicating whether it is in-stock or out-stock that was entered on the basic information input screen, the vehicle number (1707), the number of pallets detected from the front image before unloading (1708), the number of pallets detected from the front image after unloading (1709), the number of pallets detected from the back image before unloading (1710), the number of pallets detected from the back image after unloading (1711), the number of in-stock and out-stock pallets calculated based on the number of pallets detected from each image (1712), a button / icon (1713) for receiving an instruction to display detailed data, etc. As for the number of pallets, the value after correction by the user is displayed, but the value before correction may be displayed, or the values ​​before and after correction may be displayed separately.

[0121] In addition, an icon indicating that an image has been registered is displayed in the item of the number of detected palettes, and by selecting the icon, the image related to the number of detected palettes is displayed.

[0122] The screen displayed on the client terminal 102 when the details button is pressed will be described with reference to FIG.

[0123] When the details button is pressed, a screen including the first front image is displayed as shown in 1801 in Fig. 18. A button for accepting an instruction to display the second front image is provided on the display screen for the first front image, and when this button is pressed, the screen transitions to a screen including the second front image as shown in 1802. A button for accepting an instruction to display the first back image is provided on the display screen for the second front image, and when this button is pressed, the screen transitions to a screen including the first back image as shown in 1803. A button for accepting an instruction to display the second back image is provided on the display screen for the first back image, and when this button is pressed, the screen transitions to a screen including the second back image as shown in 1804.

[0124] In this way, by transitioning from the first page to the second page on the front and from the first page to the second page on the back, it is possible to easily compare the situation before and after loading and before and after unloading. Also, by clearly displaying the difference between the first and second pages, it is possible to easily grasp the cargo that has been unloaded and the newly loaded cargo.

[0125] Also, in each of the screens 1801 to 1804, a correction acceptance section for accepting corrections to the number of palettes and a button for returning to the previous screen are provided.

[0126] Next, the data table shown in FIG. 20 will be described.

[0127] The data table shown in Figure 20 is a data table in which the number of pallets and images calculated by the processing of the flowcharts of Figures 3 to 5 are registered, and is composed of the date and time the data was registered (2001), the category indicating whether the data is entering or leaving the warehouse which was entered on the basic information input screen (2002), the vehicle number (2003), the number of pallets detected from the image on the front side before unloading work (2004), the number of pallets detected from the image on the front side after unloading work (2005), the number of pallets detected from the image on the back side before unloading work (2006), the number of pallets detected from the image on the back side after unloading work (2007), the number of entering and leaving pallets calculated based on the number of pallets detected from each image (2008), the image on the front side before unloading work (2009), the image on the front side after unloading work (2010), the image on the back side before unloading work (2011), and the image on the back side after unloading work (2012).

[0128] Regarding the number of pallets (2004 to 2007), the values ​​after correction by the user are shown in FIG. 20, but the values ​​before correction by the user may also be registered.

[0129] The present invention can be embodied, for example, as a system, an apparatus, a method, a program, or a recording medium, etc. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

[0130] Moreover, the program of the present invention is a program that enables a computer to execute the processing method of the flowcharts shown in Figures 3, 4 and 5, and the storage medium of the present invention stores a program that enables a computer to execute the processing method of Figures 3, 4 and 5. The program of the present invention may be a program for each processing method of each device of Figures 3, 4 and 5.

[0131] As described above, it goes without saying that the object of the present invention can be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0132] In this case, the program itself read out from the recording medium will realize the novel functions of the present invention, and the recording medium on which the program is recorded will constitute the present invention.

[0133] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, etc.

[0134] Furthermore, it goes without saying that the functions of the above-mentioned embodiments are not only realized by the computer executing a program it has read, but also includes cases where an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0135] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0136] The present invention may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device. Needless to say, the present invention can also be applied to a case where the object is achieved by supplying a program to a system or apparatus. In this case, the effect of the present invention can be enjoyed by the system or apparatus by reading a recording medium storing a program for achieving the present invention into the system or apparatus.

[0137] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples. [Explanation of symbols]

[0138] 101 Server device 102 Client terminals 110 Network

Claims

1. A detection means for detecting a detection object included in an image relating to a vehicle bed; a display control means for controlling the detection objects detected by the detection means so as to be displayed in a manner that distinguishes the detection objects from one another according to their types; An information processing system comprising:

2. A determination means for determining the number of detection targets detected by the detection means, 2. The information processing system according to claim 1, wherein the display control means controls so as to display the number of the detection objects for each type of the detection objects.

3. The information processing system according to claim 2, further comprising a correction receiving means for receiving corrections to the number of detection objects for each type of detection object.

4. An information processing system as described in any one of claims 1 to 3, characterized in that the type of the object to be detected is distinguished according to the owner of the object to be detected.

5. An information processing system as described in any one of claims 1 to 3, characterized in that the type of the object to be detected is distinguished according to the size of the object to be detected.

6. An information processing system as described in any one of claims 1 to 5, characterized in that the display control means controls the display of an image in which the detection object detected by the detection means is surrounded by a rectangle.

7. The information processing system according to claim 6, characterized in that the display control means controls the display of the rectangles to identify each type of detected object by changing the display form of the rectangles.

8. Further comprising an imaging means for imaging the loading platform of the vehicle, 8. The information processing system according to claim 1, wherein the detection means detects a detection target included in an image captured by the imaging means.

9. The information processing system described in Claim 8, characterized in that the display control means controls the display of a guide to encourage the user to photograph the vehicle from an oblique angle when photographing the vehicle's cargo bed using the photographing means.

10. An information processing system as described in any one of claims 1 to 9, characterized in that it is provided with a notification means for issuing a notification to encourage photographing from a second direction when the photographing means photographs the loading platform from a first direction.

11. An information processing system according to any one of claims 1 to 10, characterized in that the display control means controls to display a button for transitioning to taking a second photograph.

12. An information processing system as described in claim 10 or 11, characterized in that the first direction is a direction in which the surface of the vehicle's cargo bed can be photographed, and the second direction is a direction in which the underside of the vehicle's cargo bed can be photographed.

13. An information processing system described in any one of claims 10 to 12, characterized in that the first direction is a direction in which the surface of the loading platform of a wing vehicle capable of carrying luggage can be photographed from both the front and back sides of the loading platform.

14. An information processing system described in any one of claims 10 to 13, characterized in that the second direction is a direction in which the underside of the loading platform of a wing vehicle capable of loading luggage from both the front and back sides of the loading platform can be photographed.

15. An information processing system as described in any one of claims 1 to 14, characterized in that the object to be detected is a pallet on which cargo can be loaded.

16. A detection step in which a detection means of the information processing system detects a detection object included in an image relating to a vehicle bed; a display control step of controlling a display control means of the information processing system to distinguish and display the detection objects detected by the detection step according to the types of the detection objects; An information processing method comprising:

17. A program for causing at least one computer to function as each of the means of an information processing system described in any one of claims 1 to 15.

Images