Information processing systems, information processing methods, programs

Abstract

This invention provides an information processing system, information processing method, and program for efficiently and appropriately counting the number of packages. [Solution] In a counting system in which a server device 101 and a client terminal 102 are connected in a communicative manner, the server device 101 is connected in a communicative manner to the client terminal 102 and has the function of detecting pallets from captured images and counting the number of pallets, and storing and managing data such as captured images and the number of pallets. It also transmits various data to the client terminal 102 in response to requests from the client terminal 102. The client terminal 102 is an information processing device such as a personal computer and acquires and manages captured images and the number of pallets from the client terminal 102 at each logistics center.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] In the transportation of goods, goods in a logistics warehouse are loaded onto a truck or the like, or goods loaded on a truck or the like are unloaded and carried into the warehouse. In the management of goods, it is necessary to accurately grasp and manage how many goods are loaded onto which truck and how many goods are unloaded. In addition, when transporting goods, the goods are placed on a pallet and transported by a forklift or the like to achieve efficient transportation. Since this pallet is often rented from a rental company, it is necessary to accurately count the number loaded on a truck or the like and the number unloaded. However, when counting manually, the number of pallets may not be accurately counted, which is one of the causes of pallet loss.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 describes a technique for measuring the quantity and weight of loaded goods in the cargo hold of a cargo ship using an image. However, in the case of a transport truck, a double-opening truck called a wing truck is used, and goods (including pallets, etc.) must be counted from both sides. Therefore, in the current situation where counting is done manually, counting omissions such as only counting one side of the truck occur, and the accurate number of pallets cannot be measured.

[0005] Therefore, the present invention aims to efficiently and appropriately count the number of packages. [Means for solving the problem]

[0006] To achieve the above objective, the information processing system of the present invention is A means of photographing the cargo bed of a vehicle, When the aforementioned photographing means photographs the cargo bed from a first direction, a notification means provides a notification prompting the user to photograph from a second direction. A detection means for detecting an object to be detected included in an image captured by the aforementioned imaging means, It is characterized by having the following features. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to count the number of packages efficiently and appropriately. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the overall system configuration in an embodiment of the present invention. [Figure 2] This is a block diagram showing an example of the hardware configuration of an information processing device in an embodiment of the present invention. [Figure 3] This flowchart shows an example of a photograph taken for object detection processing in an embodiment of the present invention. [Figure 4] This flowchart shows an example of a photograph taken for object detection processing in an embodiment of the present invention. [Figure 5] This flowchart shows an example of object detection processing in an embodiment of the present invention. [Figure 6] This figure shows an example of a basic information input screen in an embodiment of the present invention. [Figure 7] This figure shows an example of a table image capture screen in an embodiment of the present invention. [Figure 8] This figure shows an example of a table image capture result screen in an embodiment of the present invention. [Figure 9] It is a diagram showing an example of a back image capture screen in an embodiment of the present invention. [Figure 10] It is a diagram showing an example of a back image capture result screen in an embodiment of the present invention. [Figure 11] It is a diagram showing an example before correction of object detection in an embodiment of the present invention. [Figure 12] It is a diagram showing an example after correction of object detection in an embodiment of the present invention. [Figure 13] It is a diagram showing an example of a front surface object number correction screen in an embodiment of the present invention. [Figure 14] It is a diagram showing an example of a back surface object number correction screen in an embodiment of the present invention. [Figure 15] It is a diagram showing an example of a front surface (second sheet) object number correction screen in an embodiment of the present invention. [Figure 16] It is a diagram showing an example of a back surface (second sheet) object number correction screen in an embodiment of the present invention. [Figure 17] It is a diagram showing an example of a screen for listing registered information such as the number of detected objects in an embodiment of the present invention. [Figure 18] It is a diagram showing an example of a detailed display screen for the number of object detections in an embodiment of the present invention. [Figure 19] It is a diagram showing an example of a detection palette rectangle list in an embodiment of the present invention. [Figure 20] It is a data table showing an example of information registration such as the number of detected objects in an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0010] FIG. 1 is a diagram showing an example of the overall configuration of a counting system in an embodiment of the present invention.

[0011] The counting system in this invention is a system in which a server device 101 and a client terminal 102 are communicated with each other. The client terminal 102 is used as an edge device at a logistics center where pallets are loaded and unloaded, and is used by administrators, etc., in an office. The client terminal 102 used at the logistics center is preferably a terminal equipped with an imaging function (information processing device such as a smartphone), but it may also be a terminal that can connect to other imaging devices. The client terminal 102 used at the office is, for example, an information processing device such as a personal computer, and is a terminal that acquires and manages captured images and pallet counts from the client terminals 102 at each logistics center. In Figure 1, only one client terminal 102 each is shown in the office and the logistics center, but there may be multiple terminals in each location.

[0012] The server device 101 is connected to the client terminal 102 in a communicative manner and has the function of detecting palettes from captured images, counting the number of palettes, and storing and managing data such as captured images and the number of palettes. It also transmits various data to the client terminal 102 in response to requests from the client terminal 102. In Figure 1, the server device 101 is shown as a single server device, but it may be a system composed of multiple server devices, or it may be a so-called cloud system.

[0013] Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device (server device 101, client terminal 102) of the present invention.

[0014] As shown in Figure 2, the information processing device is connected via a system bus 200 to a CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, storage device 204, input controller 205, audio controller 206, video controller 207, memory controller 208, and communication I / F controller 209.

[0015] CPU201 provides comprehensive control over all devices and controllers connected to system bus 200.

[0016] ROM202 or external memory213 holds the BIOS (Basic Input / Output System) and OS (Operating System), which are control programs executed by the CPU201, as well as computer-readable and executable programs and various necessary data (including data tables) for realizing this information processing method.

[0017] RAM203 functions as the main memory, work area, etc., of the CPU201. The CPU201 loads the necessary programs, etc., from ROM202 or external memory213 into RAM203, and then executes the loaded programs to perform various operations.

[0018] The input controller 205 controls input from input devices such as a keyboard 210 or a pointing device such as a mouse (not shown). If the input device is a touch panel, the user can give various instructions by pressing (touching with a finger, etc.) icons, cursors, or buttons displayed on the touch panel.

[0019] Furthermore, the touch panel may be a multi-touch screen or other touch panel capable of detecting the positions of multiple fingers touching it.

[0020] The video controller 207 controls the display to an external output device such as the display 212. The display may include the display of a notebook computer integrated with the main unit. The external output device is not limited to a display; for example, it may be a projector. Furthermore, for the aforementioned touch-enabled device, an input device is also provided.

[0021] The video controller 207 can control the video memory (VRAM) used for display control. It can utilize a portion of the RAM 203 as the video memory area, or it can provide a separate, dedicated video memory.

[0022] The memory controller 208 controls access to the external memory 213. The external memory can include an external storage device (hard disk), a flexible disk (FD), or a CompactFlash® memory connected to a PCMCIA card slot via an adapter, which stores boot programs, various applications, font data, user files, editing files, and other data.

[0023] The communication interface controller 209 connects to and communicates with external devices via a network and performs communication control processing over the network. For example, it can handle communication using TCP / IP, telephone lines such as ISDN, and mobile phone 4G and 5G lines.

[0024] Furthermore, the CPU 201 enables display on the display 212 by, for example, performing the process of expanding (rasterizing) outline fonts into the display information area in RAM 203. The CPU 201 also enables user input via a mouse cursor (not shown) on the display 212.

[0025] Next, we will describe the overall picture of the operation of counting the number of pallets, which is an example of an application of the present invention. In this embodiment, we will use pallets used for transporting goods as a specific example of the object to be detected, but the object to be detected is not limited to pallets; any object transported by a so-called wing truck may be used.

[0026] First, workers at the logistics center register their work details via a basic information input screen, as shown in Figure 6, displayed on the client terminal 102. Then, according to the registered details, they take photos of the cargo loaded onto vehicles such as trucks (wing vans, etc.), and count the number of pallets by detecting them from the captured images.

[0027] A pallet is a platform used to carry goods when transporting them with a forklift or hand lift. By inserting the forks of a forklift or similar equipment into the pallet, it becomes possible to transport all the goods on the pallet at once.

[0028] Furthermore, in this embodiment, we will explain the count using pallets loaded onto a truck known as a "wing truck," which has openings on both the left and right sides of the cargo bed, allowing loading and unloading of goods from either side.

[0029] The pallets to be counted include pallets entering the logistics center and pallets leaving the logistics center. Therefore, in the case of entering (i.e., when unloading goods), a photograph of the pallet before unloading is taken, and in the case of leaving (i.e., when loading goods), a photograph of the pallet after loading is taken and counted. In addition, when loading goods onto a vehicle that is already loaded, or when only a portion of the goods loaded on a vehicle are unloaded, photographs are taken before and after loading and unloading, and the difference is counted.

[0030] Next, an example of palette counting will be explained using the flowcharts in Figures 3 to 5.

[0031] The flowcharts in Figures 3 to 5 show the process in which the CPU 201 of the client terminal 102 reads and executes a predetermined control program.

[0032] In step S101, information related to the entry / exit classification (1: entry, 2: exit) 602, vehicle number 603, remarks 604, judgment on front surface photography (photographed from the first direction) (1: all, 2: part) 605, and judgment on back surface photography (photographed from the second direction) (1: all, 2: part, 3: none) 606, which are entered by the user via the basic information input screen 601 shown in Figure 6, is acquired.

[0033] Here, we will explain the basic information input screen, which is shown as an example in Figure 6. The basic information input screen is a screen that accepts instructions such as the content of work from the user. As shown in Figure 6, it includes buttons (602-1, 602-2) to select whether the work is related to receiving goods or to shipping goods, an input field 603 to accept input of the vehicle number, which is information that identifies the vehicle to be loaded or unloaded, a remarks input field 604 to accept input of other necessary information, a button 605 to select the work content for the front of the vehicle (right side or left side), and a button 606 to select the work content for the back of the vehicle (left side if the front is the right side, and right side if the front is the left side). When the basic information registration button 607 is pressed, the entered information is registered.

[0034] In step S102, based on the information obtained in step S101, it is determined whether the surface should be photographed "all" or "partially".

[0035] If the answer is "all," proceed to step S103. If the answer is "partial," proceed to the process shown in the flowchart in Figure 4.

[0036] Note that "all" means that all pallets loaded on the vehicle will be unloaded, or that pallets will be loaded onto a vehicle that does not currently have any pallets. "Some" means that some of the pallets loaded on the vehicle will be unloaded, or that additional pallets will be loaded onto a vehicle that already has some pallets.

[0037] In step S103, the table image capture screen shown in Figure 7 is displayed, and the capture is performed when the user presses the capture button (701). The captured image is then displayed in the captured image area (801) of the table image capture result screen shown in Figure 8.

[0038] The user checks the captured image displayed in the captured image area 801, and if they determine that there are no problems, they press the Next button (802). When the Next button is pressed, the process moves to step S104.

[0039] To take the picture again, press the camera button (803). When the camera button is pressed, the front image capture screen (Figure 7) is displayed again, and the system waits for the user to press the capture button.

[0040] Here, we will explain the image capture screen shown in Figure 7.

[0041] The front image capture screen displays a capture button 701 and a capture image area 702. The capture image area is the area where the subject to be captured is displayed in live view. The capture button is a button that receives a command from the user to capture the image displayed in the capture image area, and the image capture process is executed when the capture button is pressed.

[0042] Furthermore, guidelines (lines 703, indicated by dashed lines) are displayed in the image capture area to ensure that the field of view is suitable for capturing images that are appropriate for counting the number of pallets. Specifically, guidelines (a figure representing the outline of the truck bed, for example) are displayed to guide the operator to photograph the truck bed from the front or rear at an angle. When photographing from the front or rear at an angle, the display method of the guidelines is switched according to the shooting direction. For example, when the switch button (704 or 902) is pressed, the left and right of the guidelines are reversed. By photographing according to these guidelines, the truck bed is photographed from an angle, making it less likely for pallets loaded on the underside of the truck bed to be included in the field of view, and making it easier to distinguish between pallets loaded on the underside and those loaded on the front. As a result, false detections, such as detecting pallets on the underside as pallets loaded on the front, are reduced. In addition, by photographing from an angle, it is not necessary to move to the front of both the front and back surfaces to take photographs, thus reducing the distance the operator has to move when taking photographs.

[0043] In step S104, the number of palettes is counted from the image captured in step S103. The process in step S104 will be described later using the flowchart in Figure 5.

[0044] In step S105, based on the information obtained in step S101, it is determined whether the back side should be photographed "all", "partially", or "none".

[0045] If the answer is "all", proceed to step S106. If the answer is "some", proceed to step S109. If the answer is "none", proceed to step S114.

[0046] The terms "all" and "partial" are as explained in step S102, and "none" means that no work is performed to load pallets onto the underside of the vehicle (the left side if the right side of the vehicle is considered the front, i.e., the side that is not the front) or to unload pallets from the underside.

[0047] In this embodiment, the determination of whether or not to photograph the back side is made based on the information entered on the basic information input screen, but the determination may also be made using the image of the front side. Specifically, since it is not common practice to load a large amount of cargo on only one side (the front side only) in order to ensure left-right balance of the cargo bed, if a predetermined amount of cargo is loaded on the front side or if it has been loaded (for example, if a predetermined number of pallets or more are detected from the image of the front side, or if a predetermined condition is met), it is determined that cargo is also loaded on the back side or has been loaded, and the system is controlled to proceed to the back side photography process.

[0048] First, let's explain the case where "all" is determined in step S105.

[0049] In step S106, the back image capture screen shown in Figure 9 is displayed, and the capture is performed when the user presses the capture button (901). The captured image is then displayed in the captured image area (1001) of the back image capture result screen shown in Figure 10.

[0050] The user checks the captured image displayed in the captured image area, and if they determine that there are no problems, they press the Next button (1002). When the Next button is pressed, the process moves to step S107.

[0051] To take a picture again, press the camera button (1003). When the camera button is pressed, the rear image capture screen (Figure 9) is displayed again, and the system waits for the user to press the capture button.

[0052] The back image capture screen shown in Figure 9 has the same configuration as the front image capture screen shown in Figure 7, so no explanation is provided.

[0053] In step S107, the number of palettes is counted from the image captured in step S106. The process in step S107 will be described later using the flowchart in Figure 5.

[0054] In step S108, the current date and time, the entry / exit classification obtained in step S101 (1: entry, 2: exit), vehicle number, remarks, the number of pallets detected in step S104 (front detection count 1), the corrected number of pallets (front correction count 1), the number of pallets detected in step S107 (back detection count 1), the corrected number of pallets (back correction count 1), the image of the pallets enclosed in a rectangle in the image taken in step S103 (front detection image 1), and the image of the pallets enclosed in a rectangle in the image taken in step S106 (back detection image 1) are registered in the data table shown in Figure 20.

[0055] Then, the process of this flowchart is terminated.

[0056] Next, we will explain the case where "partial" is determined in step S105.

[0057] The processes in steps S109 and S110 are the same as those in steps S106 and S107, and involve taking images of the cargo before loading it onto the vehicle or before unloading it from the vehicle, and detecting the number of pallets.

[0058] The processes in steps S111 and S112 are similar to those in steps S106 and S107, but unlike S109 and S110, steps S111 and S112 involve taking images of the cargo after it has been loaded onto the vehicle or after some of the cargo has been unloaded from the vehicle, and then detecting the number of pallets.

[0059] In step S113, the following information is registered in the data table in Figure 20, along with the current date and time, the vehicle number obtained in step S101, the remarks, the image of the palette of the image taken in step S103 enclosed in a rectangle (table detection image 1), the image of the palette of the image taken in step S109 enclosed in a rectangle, and the image of the palette of the image taken in S111 enclosed in a rectangle, according to the entry / exit classification obtained in step S101.

[0060] If the inbound / outbound classification is "1: Inbound", the number of pallets obtained by subtracting the number of pallets detected in step S112 (number of pallets detected in step S112) from the number of pallets detected in step S110 (number of pallets detected in step 1) (i.e., the number of pallets unloaded from the vehicle), and the number of pallets obtained by subtracting the number of pallets obtained by correcting the number of pallets detected in step 2 (number of pallets corrected in step 2) from the number of pallets obtained by correcting the number of pallets detected in step 1 (number of pallets corrected in step 1) are registered.

[0061] If the inbound / outbound classification is "2: Outbound", the number of pallets obtained by subtracting the number of pallets detected in step S110 (number of pallets detected in the back) from the number of pallets detected in step S112 (number of pallets detected in the back 2) (i.e., the number of pallets loaded onto the vehicle), and the number of pallets obtained by subtracting the number of pallets obtained by correcting the number of pallets detected in the back 1 (number of pallets corrected in the back) from detected in the back 2 (number of pallets corrected in the back 2) are registered.

[0062] Next, we will explain the case where step S105 determines "3: None" (the back side is not photographed).

[0063] In step S114, the current date and time, the entry / exit classification (1: entry, 2: exit) obtained in step S110, the vehicle number, remarks, the number of pallets detected in step S104 (Table detection count 1), the corrected number of pallets (Table correction count 1), and the image of the pallets enclosed in a rectangle in the image taken in step S103 (Table detection image 1) are registered in the data table shown in Figure 20.

[0064] Next, using the flowchart in Figure 4, we will explain the process when it is determined that "part" is being unloaded in step S102 (i.e., when a portion of the cargo loaded on the surface of the vehicle is being unloaded, or when there is already cargo loaded on the surface of the vehicle and additional cargo is being loaded).

[0065] The processes in steps S201 to S203 are the same as those in steps S103 to S105 of the flowchart in Figure 3, so their explanation will be omitted.

[0066] First, let's explain the process when "1: All" is determined in step S203.

[0067] The processes in steps S204 and S205 are the same as those in steps S106 and S107 in Figure 3, so their explanation is omitted.

[0068] The processing in steps S206 and S207 is the same as in steps S103 and S104, but unlike S103 and S104, steps S206 and S207 involve taking images of the cargo after it has been loaded onto the vehicle or unloaded from the vehicle, and detecting the number of pallets.

[0069] In step S208, the following information is registered in the data table in Figure 20, along with the current date and time, the vehicle number obtained in step S101, the remarks, the image of the palette of the image taken in step S204 enclosed in a rectangle (back detection image 1), the image of the palette of the image taken in step S201 enclosed in a rectangle, and the image of the palette of the image taken in S206 enclosed in a rectangle, according to the entry / exit classification obtained in step S101.

[0070] If the inbound / outbound classification is "1: Inbound", the number of pallets obtained by subtracting the number of pallets detected in step S207 (table detection number 2) from the number of pallets detected in step S202 (table detection number 1) (i.e., the number of pallets unloaded from the vehicle), and the number of pallets obtained by subtracting the number of table detection number 2 after correction (table correction number 2) from the number after correction (table correction number 1) of table detection number 1 are registered.

[0071] If the inbound / outbound classification is "2: Outbound", the number of pallets obtained by subtracting the number of pallets detected in step S202 (table detection number 1) from the number of pallets detected in step S207 (table detection number 2) (i.e., the number of pallets loaded onto the vehicle), and the number of pallets obtained by subtracting the number of table detection number 1 (table correction number 1) from the number after correcting table detection number 2 (table correction number 2) are registered.

[0072] Next, we will explain the process when "2: Partial" is determined in step S203.

[0073] The processes in steps S209 and S210 are the same as those in S106 and S107 in Figure 3, so their explanation is omitted here.

[0074] The processes in steps S211 and S212 are the same as those in S206 and S207, so their explanation is omitted here.

[0075] The processes in steps S213 and S214 are the same as those in S111 and S112 in Figure 3, so their explanation is omitted here.

[0076] In step S215, along with the current date and time, the vehicle number obtained in step S101, the remarks, the image of the palette of the image taken in step S201 enclosed in a rectangle, the image of the palette of the image taken in S211 enclosed in a rectangle, the image of the palette of the image taken in step S209 enclosed in a rectangle, and the image of the palette of the image taken in S213 enclosed in a rectangle, the following information is registered in the data table in Figure 20, according to the entry / exit classification obtained in step S101.

[0077] If the inbound / outbound classification is "1: Inbound", the following numbers are registered: the number of pallets obtained by subtracting the number of pallets obtained by step S212 (number of pallets obtained by step S212) from the number of pallets obtained by step S202 (number of pallets obtained by step S212) (i.e., the number of pallets unloaded from the vehicle); the number of pallets obtained by subtracting the number of pallets obtained by correcting number 2 (number of pallets obtained by step S212) from the number of pallets obtained by correcting number 1 (number of pallets obtained by step S214) (i.e., the number of pallets unloaded from the vehicle); and the number of pallets obtained by subtracting the number of pallets obtained by correcting number 2 (number of pallets obtained by step S214) from the number of pallets obtained by correcting number 1 (number of pallets obtained by step S210) (i.e., the number of pallets unloaded from the vehicle); and the number of pallets obtained by subtracting the number of pallets obtained by correcting number 2 (number of pallets obtained by step S214) from the number of pallets obtained by correcting number 1 (number of pallets obtained by step S214).

[0078] If the inbound / outbound classification is "2: Outbound", the number of pallets is registered as follows: the number of pallets detected in step S212 (number of pallets detected in front 2) minus the number of pallets detected in step S202 (number of pallets detected in front 1) (i.e., the number of pallets loaded onto the vehicle); the number of pallets is registered as follows: the number of pallets after correcting number of front detections 2 (number of corrected front 2) minus the number of pallets after correcting number of front detections 1 (number of corrected front 1). The number of pallets is registered as follows: the number of pallets detected in step S214 (number of pallets detected in back 2) minus the number of pallets detected in step S210 (number of pallets detected in back 1) (i.e., the number of pallets loaded onto the vehicle); the number of pallets is registered as follows: the number of pallets after correcting number of back detections 2 (number of corrected back 2) minus the number of corrected back detections 1 (number of corrected back 1).

[0079] Next, we will explain the case where step S203 determines "3: None" (no back image is taken).

[0080] The processes in steps S216 and S217 are the same as those in S206 and S207, so their explanation is omitted here.

[0081] In step S218, the following information is registered in the data table in Figure 20, along with the current date and time, the vehicle number obtained in step S101, the remarks, the image of the palette of the image taken in step S201 enclosed in a rectangle, and the image of the palette of the image taken in step S216 enclosed in a rectangle, according to the entry / exit classification obtained in step S101.

[0082] If the inbound / outbound classification is "1: Inbound", the number of pallets (i.e., the number of pallets unloaded from the vehicle) obtained by subtracting the number of pallets detected in step S217 (table detection number 2) from the number of pallets detected in step S202 (table detection number 1) is registered, and the number of pallets obtained by subtracting the number of pallets after correcting table detection number 2 (table correction number 2) from the number after correcting table detection number 1 (table correction number 1) is registered.

[0083] If the inbound / outbound classification is "2: Outbound", the number of pallets obtained by subtracting the number of pallets obtained by step S202 (table detection number 1) from the number of pallets detected in step S217 (table detection number 2) (i.e., the number of pallets loaded onto the vehicle), and the number of pallets obtained by subtracting the number of detected number 1 (table correction number 1) from the number after correcting table detection number 2 (table correction number 2) are registered.

[0084] Next, the pallet detection process in steps S104, S107, S110, S112, S202, S205, S207, S210, S212, S214, and S217 will be explained using the flowchart in Figure 5.

[0085] In step S301, the client terminal 102 transmits the captured image acquired in step S103, etc., to the server device 101.

[0086] In step S302, the server device 101 receives the image sent from the client terminal 102 in step S301.

[0087] In step S303, the server device 101 detects a palette from the received image and obtains a palette detection image and a list of detected palette rectangles.

[0088] For palette detection processing, known techniques shall be used. For example, one method is to use a pre-trained model that has been trained using palette images as training data. However, any technique for detecting a specific object from an image may be used, not limited to the method using a pre-trained model.

[0089] Figure 11 shows an image in which palettes have been detected from the image and the detected palettes are enclosed in rectangles.

[0090] An example of a detected palette rectangle list is shown in Figure 19. As shown in Figure 19, the detected palette rectangle list includes information such as the origin of the detected palette rectangle (the XY coordinates that serve as a reference for identifying the position of the palette rectangle in the image), the height of the palette rectangle, and its width, and the list is generated for each captured image. In Figure 19, the origin is the coordinate information with the top-left coordinate of the captured image as (0,0) (X is the vertical direction, Y is the horizontal direction).

[0091] In step S304, the server device 101 performs correction processing on the pallet detection image. This correction processing makes it possible to exclude pallets loaded on the underside of the vehicle from detection, even if the system is intended to detect pallets loaded on the surface of the vehicle but also detects pallets loaded on the underside of the vehicle.

[0092] The specific correction method is explained below.

[0093] First, identify the pallet located in the front row (the row closest to the driver's seat) and on the bottom level of the vehicle (hereinafter, this identified pallet will be referred to as Pallet A).

[0094] Specifically, if the right side of the vehicle is the front, the system identifies the rectangle with the largest X and Y coordinates of its starting point among the information registered in the rectangle list in Figure 19 (i.e., the palette at the bottom right of the captured image). If the left side of the vehicle is the front, the system identifies the rectangle with the largest X coordinate and the smallest Y coordinate of its starting point (i.e., the palette at the bottom left of the captured image). In the example in Figure 11, palette 1101 is palette A.

[0095] In this embodiment, the process for identifying the pallet in the front row (closest to the driver's seat) and on the bottom level of the vehicle is described. However, this process assumes that the cargo is loaded towards the driver's seat. If the cargo is loaded towards the rear, the process should identify the pallet in the last row and on the bottom level.

[0096] Then, the following process sequentially identifies the bottommost pallet among the pallets stacked on the surface (pallets 1101, 1102, 1103, etc. in the example in Figure 11).

[0097] First, create the lower palette list, store palette A in it, and set the pointer to 0.

[0098] Next, from the list of detected palette rectangles, we obtain the palette (let's call it Palette B) whose starting point Y is smaller than the starting point Y of the palette currently pointed to by the pointer, and which has the closest Euclidean distance.

[0099] Then, if the X-coordinate of the bottom left of palette B (i.e., the X-coordinate of the starting point + height) is greater than the X-coordinate of the starting point of the palette pointed to by the pointer, palette B is registered in the lower palette list and the pointer is incremented.

[0100] In this way, the reason why pallets with overlapping heights can be identified is that, as shown in Figure 11, when the surface is photographed from an oblique angle, pallets stacked on the surface will have a height relationship that overlaps with pallets in adjacent rows, while pallets stacked on the back (e.g., 1103) will not have a height relationship that overlaps. Therefore, this process makes it possible to identify only the pallets on the surface.

[0101] By repeating this process, it becomes possible to identify pallets whose height overlaps with the bottom pallet in the front row, and then identify pallets whose height overlaps with the identified pallets, thereby sequentially identifying the bottom pallet among the pallets loaded on the surface of the vehicle.

[0102] Next, we detect pallets stacked on top of the pallets in the lower pallet list. Specifically, this is done using the following process.

[0103] First, calculate the midpoint Z of each detection palette rectangle stored in the lower palette list.

[0104] Then, from among the rectangles included in the detected palette rectangle list, the system identifies the rectangle whose midpoint Z Y coordinate is between the starting point Y coordinate and the upper right Y coordinate of the rectangle, and stores the identified rectangle in the palette surface list. The number of palettes stored in the palette surface list is then considered the corrected number of palettes.

[0105] The above process identifies and counts the pallets stacked on the surface.

[0106] For palettes detected from the captured image that were not stored in the palette rectangle list, their display format may be changed so that they can be identified in the detection result image (for example, by changing the color of the rectangle or drawing red diagonal lines on the rectangle). An example of an image with a changed display format is shown in Figure 12.

[0107] Furthermore, if it is the second image, the display format is controlled so that the differences from the first image (such as the area where cargo was unloaded from the truck bed or newly loaded cargo) can be recognized.

[0108] Furthermore, the display can be made identifiable by changing the rectangular display format depending on the type of pallet, or by counting the number of pallets for each type. Regarding pallets, different types of pallets (which need to be managed separately), such as "pallets owned by the company and rented pallets" or "pallets of different sizes," may be loaded onto the same vehicle. By displaying them identifiablely and counting them separately, it becomes possible to manage them separately. If each type is counted separately, corrections to the number should also be accepted for each type, and the number for each type should be registered in the data table (Figure 20).

[0109] In step S305, the corrected detection result image and the corrected number of palette detections from step S304 are obtained.

[0110] In step S306, the corrected detection result image and the number of palettes detected are sent to the client terminal.

[0111] In step S307, the detection result image and the number of palettes detected, which were transmitted in step S306, are received.

[0112] In step S308, the number of pallets detected, received in step S307, is modified by the user via a screen shown as an example in Figures 13 to 16.

[0113] Figure 13 shows the screen displayed on the client terminal 102 during the detailed processing of steps S104 and S202, displaying the pallet detection result image 1301, the number of detected pallets 1302, and the pallet count correction acceptance unit 1303. The pallet detection result image is an image showing the result of performing the pallet detection process on the captured image as described above, and the detected pallets are enclosed in rectangles. The number of detected pallets 1302 is the number of pallets detected by the detection process. The correction acceptance unit 1303 is a button that accepts corrections to the number of pallets detected by the detection process; selecting "+" increases the number of pallets by 1, and selecting "-" decreases the number of pallets by 1. When the registration button 1804 is pressed, data such as the number of detected and the number after correction are recorded in the memory of the client terminal 102, and as described above, in the pallet registration process such as step S108 in Figure 3, it is registered in the data table shown in Figure 20. Alternatively, when the press of the registration button 1804 is accepted, the above data may be sent to the server device 101 and registered in the data table shown in Figure 20.

[0114] Then, when the back-side capture button 1305 is pressed, the process transitions to back-side capture. If back-side capture is not required (for example, if "none" is selected in steps S105 and S203), the button is controlled not to be displayed (it is controlled to be displayed only if "none" is selected).

[0115] Figure 14 shows the screen displayed on the client terminal during the detailed processing of steps S107, S110, S205, and S210; Figure 15 shows the screen displayed on the client terminal during the detailed processing of steps S207, S212, and S217; and Figure 16 shows the screen displayed on the client terminal during the detailed processing of steps S112 and S214. In all cases, as in Figure 13, the display of the transition button (the button corresponding to 1305 in Figure 13) is controlled according to whether the next shooting process is necessary.

[0116] In this way, when it is necessary to photograph the back side or take a second photograph (for example, when it is determined that it is necessary based on the information entered on the basic information input screen, or when it is determined that it is necessary to photograph the back side based on the image of the front side), the screen display can be controlled to prompt the user to take a photograph of the back side or a second photograph (a notification will be displayed), which can prevent users from forgetting to take a photograph of the back side or a second photograph. Also, when it is not necessary to photograph the back side or a second photograph, the button to proceed to taking a photograph of the back side or a second photograph will not be displayed, allowing the work to proceed efficiently without creating unnecessary work.

[0117] Once the above processes are executed, the process returns to the flowcharts shown in Figures 3 and 4.

[0118] Next, we will explain the data list screen shown in Figure 17. The data list screen is a screen generated using the data registered by executing the flowcharts in Figures 3 and 4 (the data registered in the data table shown in Figure 20), and is displayed on the client terminal 102.

[0119] The date input field 1701 and the time input field 1702 accept the date and time of the data the user wants to check. When the inquiry button 1703 is pressed, the system searches for data for the entered date and time, and the data is displayed in the list display area 1704.

[0120] The list display area shows the date and time the data was registered (1705), the classification indicating whether it is an incoming or outgoing shipment as entered on the basic information input screen (1706), the vehicle number (1707), the number of pallets detected from the image of the front side before loading / unloading (1708), the number of pallets detected from the image of the front side after loading / unloading (1709), the number of pallets detected from the image of the back side before loading / unloading (1710), the number of pallets detected from the image of the back side after loading / unloading (1711), the number of incoming and outgoing pallets calculated based on the number of pallets detected from each image (1712), and a button / icon to receive instructions to display detailed data (1713). The number of pallets will be displayed as the value after correction by the user, but the value before correction may also be displayed, or the values ​​before and after correction may be displayed separately.

[0121] Additionally, an icon indicating that an image is registered will be displayed in the "Number of Palette Detected" field, and selecting this icon will display the image associated with that palette detection count.

[0122] Figure 18 illustrates the screen displayed on client terminal 102 when the "Details" button is pressed.

[0123] When the Details button is pressed, a screen containing the first image of the front side, as shown in Figure 18, 1801, is displayed. The screen displaying the first image of the front side has a button that accepts the instruction to display the second image of the front side, and when this button is pressed, the screen transitions to the screen containing the second image of the front side, as shown in 1802. The screen displaying the second image of the front side has a button that accepts the instruction to display the first image of the back side, and when this button is pressed, the screen transitions to the screen containing the first image of the back side, as shown in 1803. The screen displaying the first image of the back side has a button that accepts the instruction to display the second image of the back side, and when this button is pressed, the screen transitions to the screen containing the second image of the back side, as shown in 1804.

[0124] In this way, by transitioning from the first to the second image on the front side, and from the first to the second image on the back side, it becomes easier to compare the conditions before and after loading and before and after unloading. Furthermore, by clearly displaying the differences between the first and second images, it becomes easy to identify unloaded cargo and newly loaded cargo.

[0125] Furthermore, each of the screens from 1801 to 1804 includes a correction request section for changing the number of palettes, as well as a button to return to the previous screen.

[0126] Next, we will explain the data table shown in Figure 20.

[0127] The data table shown in Figure 20 is a data table in which the number of pallets and images calculated by the flowcharts in Figures 3 to 5 are registered. It consists of the date and time the data was registered (2001), the classification indicating whether it is an incoming or outgoing shipment entered on the basic information input screen (2002), the vehicle number (2003), the number of pallets detected from the image of the front before loading / unloading (2004), the number of pallets detected from the image of the front after loading / unloading (2005), the number of pallets detected from the image of the back before loading / unloading (2006), the number of pallets detected from the image of the back after loading / unloading (2007), the number of incoming and outgoing pallets calculated based on the number of pallets detected from each image (2008), the image of the front before loading / unloading (2009), the image of the front after loading / unloading (2010), the image of the back before loading / unloading (2011), the image of the back after loading / unloading (2012), etc.

[0128] Regarding the number of palettes (2004-2007), Figure 20 shows the values ​​after user modification, but the values ​​before user modification may also be registered.

[0129] The present invention can take the form of, for example, a system, apparatus, method, program, or recording medium. Specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0130] Furthermore, the program in this invention is a program that a computer can execute using the processing methods shown in the flowcharts in Figures 3, 4, and 5, and the storage medium of this invention stores a program that a computer can execute using the processing methods shown in Figures 3, 4, and 5. Note that the program in this invention may also be a separate program for each processing method of the devices shown in Figures 3, 4, and 5.

[0131] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium containing a program that realizes the functions of the embodiments described above to a system or device, and by having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0132] In this case, the program read from the recording medium itself realizes the novel function of the present invention, and the recording medium on which that program is recorded constitutes the present invention.

[0133] For recording media used to supply programs, examples include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, and the like.

[0134] Furthermore, it goes without saying that the functions of the aforementioned embodiments are realized not only by the computer executing the program it has read, but also by the operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of that program, thereby realizing the functions of the aforementioned embodiments.

[0135] Furthermore, it goes without saying that this also includes cases where, after a program read from a recording medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer, the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the functions of the aforementioned embodiments are realized through that processing.

[0136] Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device. It goes without saying that the present invention can also be applied when the results are achieved by supplying a program to a system or device. In this case, by reading a recording medium containing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

[0137] Furthermore, by downloading and reading the program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. It should be noted that configurations combining the above-described embodiments and their variations are all included in the present invention. [Explanation of symbols]

[0138] 101 Server device 102 Client terminals 110 Network

Claims

1. A means of photographing the cargo bed of a vehicle, When the aforementioned photographing means photographs the cargo bed from a first direction, a notification means provides a notification prompting the user to photograph from a second direction. A detection means for detecting an object to be detected included in an image captured by the aforementioned imaging means, An information processing system characterized by having the following features.

2. The information processing system according to claim 1, further characterized in that the notification means notifies the number of objects detected by the detection means.

3. The information processing system according to claim 1 or 2, further comprising control means for controlling whether or not to issue a notification prompting the taking of images from a second direction using the notification means, depending on whether or not the number of detected objects detected from an image taken from a first direction satisfies predetermined conditions.

4. The information processing system according to any one of claims 1 to 3, characterized in that the notification means provides a notification prompting the user to take a picture from a second direction when the number of objects detected from an image taken from a first direction satisfies a predetermined condition.

5. The information processing system according to any one of claims 1 to 4, characterized in that the notification means does not issue a notification prompting the user to take a picture from a second direction if the number of objects detected from an image taken from a first direction does not meet predetermined conditions.

6. The information processing system according to any one of claims 1 to 5, further characterized in that the notification means prompts the user to take an image after the loading / unloading operation when a portion of the object to be detected loaded on the vehicle's cargo bed is unloaded or when additional objects to be detected are loaded onto the vehicle's cargo bed.

7. The information processing system according to any one of claims 1 to 6, further comprising a display control means for controlling the display of an image captured by the aforementioned shooting means.

8. The information processing system according to claim 7, referencing claim 6, characterized in that the display control means controls the display to display an image that makes it possible to identify the difference between the images before and after the loading and unloading work when images before and after the loading and unloading work are captured by the shooting means.

9. The information processing system according to claim 7, characterized in that the display control means controls the display of an image in which the detected object detected by the detection means is enclosed in a rectangle.

10. The information processing system according to any one of claims 1 to 9, characterized in that the object to be detected is cargo loaded on a wing truck capable of loading cargo from both the front and back surfaces of the cargo bed.

11. The information processing system according to claim 10, characterized in that the first direction is a direction in which the surface of the cargo bed of the wing truck can be photographed.

12. The information processing system according to claim 10 or 11, characterized in that the second direction is a direction in which the underside of the cargo bed of the wing truck can be photographed.

13. The information processing system according to any one of claims 10 to 12, characterized in that the notification means notifies the number of detection targets detected from the image taken from the first direction, excluding the detection targets stacked on the back surface.

14. The information processing system according to any one of claims 10 to 13, characterized in that the display control means displays an image in which, among the detection targets detected from an image taken from the first direction, the detection targets stacked on the back surface are identifiable.

15. The information processing system according to any one of claims 1 to 14, further comprising a receiving means for receiving corrections to the number of objects to be detected by the detection means.

16. The information processing system's imaging means includes an imaging process for photographing the cargo bed of a vehicle, The notification means of the information processing system includes a notification step that, when the cargo bed is photographed from a first direction by the photography step, prompts the photographer to take a photograph from a second direction, The detection means of the information processing system includes a detection step of detecting an object to be detected included in the image captured by the shooting step, An information processing method characterized by having the following features.

17. A program for causing a computer to function as one of the means of an information processing system described in any one of claims 1 to 15.

Images