Information processing system and information processing method

Abstract

By simply scanning the AR marker, the delivery status of packages can be updated, improving user convenience. [Solution] The information processing system (10) includes a management server (12) and a plurality of user terminals 16. Each user has a role and uses their user terminal to read the AR marker attached to the package. The identification information associated with the AR marker is then identified, and the identification information and the user's role information on the user terminal are sent to the management server. The management server updates the delivery status of the package indicated by the identification information and sends the updated delivery status to the user terminal that sent the identification information and role information. On the user terminal, presentation information 80 based on the updated delivery status is displayed superimposed on the captured image 70 in relation to the image 72 of the AR marker included in the captured image 70.

Description

Technical Field

[0004]

[0001] This invention relates to an information processing system and an information processing method, and particularly relates to an information processing system and an information processing method that provide a new usage method for an AR (Augmented Reality) marker.

Background Art

[0002] An example of the background art is disclosed in Patent Document 1. In the Internet-based home delivery system of Patent Document 1, a parcel tracking information management server collects confirmation information indicating the status of each stage from when a parcel is received until it is delivered to the consignee, such as shipment confirmation (store departure confirmation), arrival at store confirmation, delivery completion, and further information such as pick-up information, damage information, mis-delivery information, transfer information, return information, etc. (these are referred to as "parcel tracking information") from a logistics base server and a delivery person's terminal regularly or in real time, and accumulates and manages them as one record unit in association with delivery identification information such as a bill number and a collection request acceptance number for each shipping case. For example, when a driver (the delivery person) picks up a parcel (goods), the driver transmits "collected information", "delivery completed information", "not-at-home information", etc. from the delivery person's terminal to the parcel tracking information management server on the spot. The parcel tracking information management server that receives this changes the status of the "parcel tracking information record" of that shipping case to "shipment confirmed", "delivery completed", or "not-at-home", etc.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] <000002In the background technology of the internet-based delivery system, it is disclosed that the assigned driver registers information about the package from the driver's terminal to the package tracking information management server, and the package tracking information management server appropriately changes the status of the package tracking information record accordingly. In such an internet-based delivery system, it is desirable to improve or enhance the convenience of users such as the assigned driver, but the above-mentioned Patent Document 1 does not disclose any specific method for the assigned driver to register information about the package or a method for confirming that the information about the package has been registered.

[0005] Therefore, the main objective of this invention is to provide a novel information processing system and information processing method that utilize AR markers.

[0006] Another object of this invention is to provide an information processing system and an information processing method that can improve user convenience. [Means for solving the problem]

[0007] The first invention is an information processing system comprising a user terminal used by a user and a management device for managing setting information set for each of the AR markers attached to each of one or more delivery items, wherein the user terminal comprises a camera, identification means for detecting AR markers and identifying identification information associated with the AR markers when the image captured by the camera includes AR markers, and a first transmission means for transmitting the identified identification information and role information about a pre-set user role to the management device, the management device comprises a first receiving means for receiving the identification information and role information, an updating means for updating related information related to the delivery items according to the identification information and role information based on the setting information, and a second transmission means for transmitting the updated related information to the user terminal that sent the identification information and role information, the user terminal further comprises a second receiving means for receiving related information, a generation means for generating presentation information based on the related information and role information, and a display means for displaying the presentation information superimposed on an image in a display manner associated with the AR markers included in the image.

[0008] The second invention is subordinate to the first invention, wherein the updating means further updates the relevant information according to information other than identification information and role information, and the display means, upon receiving the relevant information updated according to the other information, displays the presented information on the display by superimposing it onto an image in a display manner different from when it was updated according to identification information and role information.

[0009] The third invention relates to a situation where an image captured by a camera contains multiple AR markers, wherein the identification means identifies each of the identification information associated with each of the multiple AR markers, the first transmitting means transmits each of the identification information and role information to a management device, the first receiving means receives each of the identification information and role information, the updating means updates related information to each of the deliveries to which some or all of the multiple AR markers are attached, based on the setting information and according to each of the identification information and role information, and the display means, when the related information to each of the deliveries to which some or all of the multiple AR markers are attached is updated, overlays presentation information for some or all of the multiple AR markers onto the image and displays it on a display.

[0010] The fourth invention is an information processing method for an information processing system, comprising a user terminal equipped with a camera and used by a user, and a management device that manages setting information set for each of the AR markers attached to each of the one or more delivered items, wherein the user terminal, when an image captured by the camera contains an AR marker, detects the AR marker, identifies the identification information associated with the AR marker, transmits the identified identification information and role information about the user's pre-set role to the management device, the management device receives the identification information and role information, updates related information related to the delivered items according to the identification information and role information based on the setting information, transmits the updated related information to the user terminal that transmitted the identification information and role information, the user terminal further receives the related information, generates presentation information based on the related information and role information, and displays the presentation information superimposed on the image on a display in a display manner associated with the AR markers contained in the image. [Effects of the Invention]

[0011] According to this invention, simply by taking a picture of an AR marker with the user's terminal camera, relevant information about the delivered item with the AR marker attached is updated, and suggested information based on that relevant information is displayed, thereby improving user convenience.

[0012] The above-mentioned objectives, other objectives, features, and advantages of this invention will become even clearer from the following detailed description of embodiments with reference to the drawings. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows an example of an information processing system, not limited to this embodiment of the present invention. [Figure 2] Figure 1 is a block diagram showing an example of the electrical configuration of a management server, without any limitations. [Figure 3] Figure 1 is a block diagram showing an example of the electrical configuration of a user terminal, without limitations. [Figure 4] (A) is a diagram showing an example of a package with a certain AR marker attached, (B) is a diagram showing an example of an application screen displayed when the person in charge of shipping takes a picture of an image containing the AR marker shown in (A) with a user terminal, and (B) is a diagram showing an example of an application screen displayed when the person in charge of shipping takes a picture of an image containing a different AR marker than the one shown in (A) with a user terminal. [Figure 5] (A) is a diagram showing an example of an app screen that may appear when a delivery person takes a picture of the package containing the AR marker shown in Figure 4(A) upon receiving the package; (B) is a diagram showing another example of an app screen that may appear when a delivery person takes a picture of the package containing the AR marker shown in Figure 4(A) upon receiving the package; and (C) is a diagram showing an example of an app screen that may appear when a delivery person takes a picture of the package containing the AR marker shown in Figure 4(A) upon taking it out. [Figure 6] Figure 4(A) shows an example of an app screen that may appear when a recipient who has received a package takes a picture of the package containing the AR marker. [Figure 7](A) is a diagram showing a non-limiting example of a delivery box with an AR marker attached, (B) is a diagram showing a non-limiting example of an app screen displayed when a delivery person who has put the luggage shown in Fig. 4(A) into the delivery box shown in (A) takes a picture of the image including the AR marker attached to the delivery box, and (C) is a diagram showing a non-limiting example of an app screen displayed when a delivery person who brings back the luggage shown in Fig. 4(A) takes a picture of the image including the AR marker attached to the luggage. [Figure 8] (A) is a diagram showing a non-limiting example of marker data managed by a management server, and (B) is a diagram showing a non-limiting example of luggage data managed by the management server. [Figure 9] It is a diagram showing an example of the memory map of the RAM of the management server shown in Fig. 2. [Figure 10] It is a diagram showing an example of the memory map of the RAM of the user terminal shown in Fig. 3. [Figure 11] It is a flowchart showing a part of an example of the information processing of the CPU of the management server shown in Fig. 2. [Figure 12] It is another part of an example of the information processing of the CPU of the management server shown in Fig. 2, and is a flowchart following Fig. 11. [Figure 13] It is another part of an example of the information processing of the CPU of the management server shown in Fig. 2, and is a flowchart following Fig. 12. [Figure 14] It is a flowchart showing an example of the tracking app processing of the CPU of the user terminal shown in Fig. 3. [Figure 15] (A) is a diagram showing a non-limiting example of a state where a plurality of luggage are placed on the floor surface, and (B) is a diagram showing a non-limiting example of an app screen displayed when a picture is taken of each of the AR markers attached to the plurality of luggage shown in (A).

Embodiments for Carrying Out the Invention

[0014] Referring to FIG. 1, the information processing system 10 of this embodiment includes a management server 12, and the management server 12 is communicably connected to a plurality of user terminals 16 via a network 14. In this embodiment, the plurality of user terminals 16 are each used by a different user.

[0015] The management server 12 is an information processing device that manages information about markers (hereinafter referred to as "marker data 304a"), and refers to this marker data 304a to update information about the delivery item (hereinafter referred to as "cargo") according to the information obtained from the user terminal 16, and transmits the updated information (hereinafter referred to as related information) to the user terminal 16. Further, the management server 12 updates the cargo data 304b according to information other than the information obtained from the user terminal 16, and when obtaining information from the user terminal 16, transmits the updated information (hereinafter referred to as delivery cancellation information) to the user terminal 16. Further, the management server 12 does not update the cargo data 304b according to the information obtained from the user terminal 16, and transmits information about the fact that it has not been updated (hereinafter referred to as error information) to the user terminal 16.

[0016] The network 14 is composed of an IP network (or IP network) including the Internet and an access network (or access network) for accessing this IP network. As the access network, a public telephone network, a mobile phone network, a wired LAN, a wireless LAN, CATV (Cable Television), etc. can be used.

[0017] The user terminal 16 is another information processing device, and as an example, it is a smartphone or a tablet terminal. In some cases, a notebook PC with a built-in or connected camera can also be used as the user terminal 16.

[0018] As shown in Figure 2, the management server 12 includes a CPU 30, which is connected via an internal bus to RAM 32, a communication interface (hereinafter referred to as "communication I / F") 34, and an input / output interface (hereinafter referred to as "input / output I / F") 36.

[0019] The CPU 30 is responsible for the overall control of the management server 12. However, instead of the CPU 30, a System-on-a-chip (SoC) that includes multiple functions such as CPU functionality and GPU (Graphics Processing Unit) functionality may be provided.

[0020] RAM32 is the main memory and is used as the work area or buffer area for the CPU30. Although not shown in the diagram, the management server 12 is equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD. Various software such as the management server 12's operating system, middleware, and various application programs are stored in this auxiliary storage device.

[0021] Communication I / F34 is a wired interface for sending and receiving control signals and data to and from an external computer, such as a user terminal 16, via the network 14 under the control of the CPU 30. However, a wireless interface for connecting to a wireless LAN can also be used as the communication I / F34.

[0022] Input / output interface 36 is connected to input devices such as a keyboard and computer mouse, and display devices such as an LCD display as appropriate. The input / output interface 36 outputs operation data (or operation information) received from the input devices to the CPU 30. It also outputs image data generated by the CPU 30 to the display, and displays the screen or image corresponding to the image data on the display.

[0023] Note that the electrical configuration of the management server 12 shown in Figure 2 is just one example, and the configuration is not limited to this example.

[0024] As shown in Figure 3, the user terminal 16 includes a CPU 50, which is connected to RAM 52, communication I / F 54, and input / output I / F 56 via an internal bus.

[0025] The CPU 50 is responsible for the overall control of the user terminal 16. However, instead of the CPU 50, an SoC (System on a Chip) that includes multiple functions such as CPU and GPU functions may be provided.

[0026] RAM 52 is the main memory and is used as the work area or buffer area for the CPU 50. Although not shown in the diagram, the user terminal 16 is equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD. Various software such as the operating system, middleware, and various application programs of the user terminal 16 are stored in this auxiliary storage device.

[0027] Communication I / F 54 is a wireless interface for sending and receiving control signals and data to and from an external computer, such as a management server 12, via the network 14 under the control of the CPU 50. However, a wired interface for connecting to a wired LAN can also be used as communication I / F 54.

[0028] The input / output interface 56 is connected to an input unit 58, a display 60, and an audio output unit 62. The input unit 58 can be a keyboard, a computer mouse, or a camera. A touch panel may also be provided. The display 60 is, for example, a liquid crystal display. The camera is, for example, a CCD camera.

[0029] Furthermore, the input / output interface 56 outputs operation data received from the input unit 58 to the CPU 50, and outputs image data generated by the CPU 50 to the display 60, displaying a screen or image corresponding to the image data on the display 60. In addition, the input / output interface 56 outputs shooting instructions from the CPU 50 to the camera and outputs the captured image data received from the camera to the CPU 50.

[0030] Furthermore, the input / output interface 56 converts the audio data output from the CPU 50 into analog audio and outputs it to the sound output unit 62.

[0031] Note that the electrical configuration of the user terminal 16 shown in Figure 3 is just one example, and the invention is not limited to this configuration. The user terminal 16 is typically a smartphone and therefore has a calling function, but since calling is unrelated to the present invention, it is not shown or described.

[0032] In this information processing system 10, the user terminal 16 stores application software for tracking the delivery status of packages (hereinafter simply referred to as the "tracking app"). Using this tracking app, each user, such as the person responsible for shipping, the delivery person, and the recipient, updates the delivery status of packages managed by the management server 12. The following describes the process of tracking the delivery status of a single package, but if you wish to track the delivery status of multiple packages, you can track them individually.

[0033] As shown in Figure 4(A), a predetermined image is attached to the package. However, the package is a product or item placed in packaging material (for example, a cardboard box or bag). The predetermined image is attached to the package by either attaching a sticker with the predetermined image printed on it to the packaging material, or by printing the image directly onto the packaging material.

[0034] The specified image is an AR (Augmented Reality) marker such as ArUco Marker. However, the AR marker is not limited to ArUco Marker; other AR markers or AR tags such as AprilTag, ARTag, ARToolkit, ARToolkitPlus, RUNE-Tag, and reacTIVison can also be used. Furthermore, the specified image can also be a 2D code such as QR (Quick Response) code (registered trademark), DataMatrix, VeriCode, and Chameleon Code (registered trademark), or a 1D code such as a barcode, which can be used as an AR marker.

[0035] Each AR marker is associated with at least identification information. In this embodiment, when the tracking application is executed on the user terminal 16, the camera is activated, the shooting process is performed, and the captured image (i.e., the through image) is displayed on the display 60. Furthermore, when the user terminal 16 detects or recognizes (hereinafter simply referred to as "detection") an AR marker contained in the captured image taken by the camera using the tracking application, it identifies the identification information associated with the detected AR marker. In other words, when the AR marker is read by the user terminal 16, the identification information is identified.

[0036] When the user terminal 16 identifies the identification information, it transmits the identified identification information and information about the user's role (hereinafter, "role information") to the management server 12. The user's role information includes information about the user's role, namely, whether they are the shipping manager, delivery person, or recipient, and information about the area (hereinafter, "responsible area") for which the user (i.e., shipping manager, delivery person, or recipient) is responsible for delivering packages. Hereafter in this specification, when it is not necessary to distinguish between the shipping manager, delivery person, and recipient, they will simply be referred to as "user".

[0037] The management server 12 updates the information related to the package (hereinafter referred to as "related information") based on the identification information and role information received from the user terminal 16, and sends the updated related information to the user terminal 16 that sent the identification information and role information. However, if the delivery area of ​​the package corresponding to the identification information differs from the area of ​​responsibility indicated by the role information, the management server 12 does not update the related information of the package, but instead sends error information indicating that it is not a package that the management server 12 is responsible for to the user terminal 16 that sent the identification information and role information.

[0038] The relevant information includes, at a minimum, the delivery status of the package. The shipping manager, the delivery person, and the recipient each read the AR marker attached to the package using their respective user terminals 16, and the management server 12 updates the delivery status of the package, tracking it from shipment to handover. The delivery status of the package changes, such as received, in transit, out for transport, and handed over (or completed delivery), and the management server 12 updates the delivery status of the package according to the information pre-set for the AR marker (marker data 304a, described later) (see Figure 8(A)). For the delivery person, the delivery status is updated at multiple stages from the time they receive the package until they hand it over to the recipient, so when the delivery person reads the AR marker using the user terminal 16, the management server 12 updates the delivery status in stages according to the current (i.e., pre-update) delivery status.

[0039] Furthermore, if the shipping source or destination instructs the cancellation of a shipment, the person in charge of receiving the shipment instruction updates the shipment status to "cancelled." For example, a person in charge of receiving orders accepts shipment orders and registers the accepted shipments in the shipment data (see Figure 8(B)). The registration of shipment data is done using the person in charge's own terminal (PC or tablet), which is different from the user terminal 16. The person in charge also registers the cancellation of a shipment using their own terminal. Therefore, if the management server 12 finds that a shipment cancellation is indicated in the shipment data corresponding to the identification information, it sends the shipment cancellation information to the user terminal 16 that sent the identification information and role information.

[0040] Note that the update (or registration) of the delivery cancellation may be performed by the operator or administrator of the management server 12 using the management server 12 or their own terminal.

[0041] When the user terminal 16 receives relevant information, delivery cancellation information, or error information from the management server 12 in response to transmitting identification information and role information to the management server 12, it generates information to present to the user (hereinafter referred to as "presented information") based on the received relevant information, delivery cancellation information, or error information, and displays the generated presented information on the display 60 by superimposing it on the image captured by the camera and associating it with the image of the AR marker. The presented information is information presented to the user regarding the package when the user reads the AR marker using their user terminal 16.

[0042] Therefore, when the AR marker on the package, as shown in Figure 4(A), is read by the shipping manager's user terminal 16, the presentation information 80 is displayed superimposed on the captured image 70 in relation to the AR marker image 72, as shown in Figure 4(B). In other words, the presentation information 80 is displayed in a balloon so that it is in front of the captured image 70 and related to the AR marker contained in the captured image 70. Hereafter, unless otherwise specified, the presentation information 80 will be displayed in a balloon.

[0043] The information displayed 80 shown in Figure 4(B) is the delivery destination region and the shipping registration date. This information is displayed when shipping registration is successfully performed, and is generated in the management server 12 when the delivery status of the package is updated from before shipping registration (i.e., unregistered) to shipping registration, based on the relevant information received from the management server 12. The same applies to cases where the delivery status is successfully updated (or registered).

[0044] The delivery destination is the area assigned to the shipping officer. The assigned area registered in the user terminal 16 is retrieved. The shipping registration date is obtained from the date and time information measured by a clock circuit such as an RTC in the user terminal 16. The same applies below when the date is displayed.

[0045] In this way, the display of the delivery region allows you to confirm that the package belongs to the shipping manager's area of ​​responsibility. Additionally, the display of the shipping registration date indicates that the shipping registration has been completed and that the date (year, month, and day) was confirmed.

[0046] Once shipment registration is complete, the package is received (or collected) by the delivery person. When the AR marker of the package is read by the delivery person's user terminal 16, the management server 12 updates the delivery status of the package based on the identification information and role information, if the delivery area of ​​the package corresponding to the identification information is the same as the area of ​​responsibility indicated by the role information. Therefore, when the AR marker of a package that has been registered for shipment, such as the package shown in Figure 4(A), is read by the user terminal 16 of the responsible delivery person, the presentation information 80 is displayed superimposed on the captured image 70 in relation to the AR marker image 72, as shown in Figure 5(A).

[0047] The information 80 shown in Figure 5(A) is the delivery destination region and the date of receipt of the package. This information 80 is displayed when the package has been received and its delivery status has been successfully updated to "in transit". The display of the delivery destination region allows the user to confirm that the package is within the delivery person's service area. The display of the receipt date allows the user to know that the registration of receipt has been completed by reading the AR marker with the user terminal 16, and to know the date (year, month, day) of that registration.

[0048] However, even if the delivery area of ​​the package corresponding to the identification information and the area of ​​responsibility indicated by the role information are the same, as described above, if the delivery status of the package is canceled, the management server 12 sends information to notify the user terminal 16 that sent the identification information and role information, i.e., delivery cancellation information, to inform the user that the delivery has been canceled. In this case, as shown in Figure 5(B), the presentation information 80 based on the delivery cancellation information is superimposed on the captured image 70 in relation to the AR marker image 72. The presentation information 80 shown in Figure 5(B) displays the string "delivery canceled" and the date (year, month, day) on which the delivery cancellation was confirmed. The presentation information 80 based on the delivery cancellation information is the same even if the delivery status is "in transit" or "out for delivery".

[0049] When a delivery person receives a package, the delivery status of the package is updated during delivery, and the package is transported by the delivery person. When the delivery person has transported the package to the delivery destination or its vicinity, they prepare the package for handover, take it out, and hand it over to the recipient. At this time, when the AR marker of the package is read by the delivery person's user terminal 16, the management server 12 updates the delivery status of the package based on the identification information and role information, if the delivery area of ​​the package corresponding to the identification information is the same as the area of ​​responsibility indicated by the role information. Therefore, when the AR marker of a package that has been prepared for handover, such as the package shown in Figure 4(A), is read by the delivery person's user terminal 16, the presentation information 80 is displayed superimposed on the captured image 70 in relation to the AR marker image 72, as shown in Figure 5(C).

[0050] The information 80 shown in Figure 5(C) is the destination region and the date of dispatch of the package. This information 80 is displayed when the package is ready for delivery and its delivery status is successfully updated to "dispatch in progress". The display of the destination region allows the user to confirm that the package belongs to the delivery person's area. The display of the dispatch date allows the user to know that the registration for dispatching the package has been completed by reading the AR marker with the user terminal 16, and to know the date (year, month, day) of that registration.

[0051] The delivery person hands over the package to the recipient. In other words, delivery is completed upon handover. When the AR marker of the package is read by the recipient's user terminal 16, the management server 12 updates the delivery status of the package based on the identification information and role information, if the delivery area of ​​the package corresponding to the identification information is the same as the area of ​​responsibility indicated by the role information. Therefore, when the AR marker of a package that has been handed over, such as the package shown in Figure 4(A), is read by the recipient's user terminal 16, the presentation information 80 is displayed superimposed on the captured image 70 in relation to the AR marker image 72, as shown in Figure 6.

[0052] The information displayed 80 in Figure 6 consists of the string "Received" and the delivery completion date (i.e., the date of receipt). This information displayed 80 is shown when the package has been handed over and the delivery status of the package has been successfully updated to "Delivered". The display of the string "Received" allows the user to confirm that the package has been delivered. In addition, the display of the delivery completion date allows the user to know that the delivery completion of the package has been registered by reading the AR marker with the user terminal 16, and to know the date (year, month, day) of that registration.

[0053] If the recipient is unable to receive the package for reasons such as absence, the delivery person may consider the delivery complete by placing the package in a delivery box, or by taking the package back and delivering it at a different time or on a different day.

[0054] As shown in Figure 7(A), the delivery box is equipped with an AR marker. In this case, the AR marker is attached to the outside surface of the delivery box door, but it may also be attached to the inside surface of the delivery box door. However, the AR marker attached to the delivery box is associated not only with identification information but also with information indicating that it is a delivery box.

[0055] In this embodiment, when a package is delivered and placed in a delivery box to complete the delivery, the delivery person reads the AR marker attached to the package before placing it in the delivery box, or after placing the package in the delivery box, reads the AR marker attached to the package, and then reads the AR marker attached to the delivery box with their user terminal 16.

[0056] Then, after the delivery person's user terminal 16 sends identification information and role information to the management server 12, information indicating that it is a delivery box is sent to the management server 12. The management server 12 receives the information that it is a delivery box, updates the delivery status of the package corresponding to the previously received identification information to "delivered," and sends related information to the user terminal 16 indicating that the delivery has been completed because the package has been placed in the delivery box.

[0057] Therefore, in the case of a package like the one shown in Figure 4(A), if the package is placed in a delivery box and the AR marker attached to the package is read by the delivery person's user terminal 16, and then the AR marker attached to the delivery box is read, the presentation information 80 will be superimposed on the captured image 70 in relation to the image 72 of the AR marker, as shown in Figure 7(B).

[0058] The information displayed in Figure 7(B) is the text "delivery box" and the delivery completion date. This information is displayed when the package is delivered by placing it in the delivery box and the delivery status of the package is successfully updated to "delivery completed". The display of the text "delivery box" confirms that the package has been placed in the delivery box and the delivery has been completed. The display of the delivery completion date indicates that the delivery of the package has been registered by reading the AR markers of both the package and the delivery box with the user terminal 16, and that the date (year, month, day) of that registration has been determined.

[0059] Furthermore, if the delivery person decides to take the package back, they read the AR marker on the package and wait for a predetermined time (for example, about 1 minute). Then, after the delivery person's user terminal 16 sends identification information and role information to the management server 12, the information that it is a delivery box is not sent to the management server 12 for the predetermined time. Therefore, after the predetermined time has elapsed, the delivery status of the package indicated by the identification information is updated to "take back" and related information regarding the return of the package is sent to the user terminal 16. In other words, the predetermined time is the time (hereinafter referred to as the "decision time") for the delivery person to decide whether to place the package in the delivery box and complete the delivery, or to take the package back.

[0060] Therefore, in the case of a package like the one shown in Figure 4(A), after the AR marker attached to the package is read by the delivery person's user terminal 16 and a judgment time has elapsed, the presentation information 80 is displayed superimposed on the captured image 70 in relation to the image 72 of the AR marker, as shown in Figure 7(C).

[0061] The information 80 shown in Figure 7(C) consists of the string "Takeaway" and the date of return. This information 80 is displayed when it is determined that the package should be taken away and the delivery status of the package is successfully updated to "Takeaway". The display of the string "Takeaway" allows the user to confirm that the package is being taken away. In addition, the display of the date of return allows the user to know that the package has been registered for return by reading the AR marker attached to the package with the user terminal 16, and to know the date (year, month, day) of that registration.

[0062] As shown in Figure 8(A), the marker data is data from a table that contains information on roles, delivery status, and update details, corresponding to the identification information.

[0063] The identification information is the identification information of the AR marker, which in this embodiment is indicated by a number using digits. Note that the role and / or delivery status may differ depending on the assigned region and / or delivery region. To flexibly respond without changing the information processing program on the management server 12, the update content is described for each AR marker according to its role and delivery status.

[0064] The roles listed here refer to the user's roles, excluding their assigned region. The roles listed include "Shipping Manager," "Delivery Person," and "Recipient."

[0065] The delivery status refers to the delivery status of the package corresponding to the identification information, and is the current (i.e., pre-update) delivery status used to determine the update content according to its role. In the marker data, the delivery status is recorded as "Unregistered," "Before Receipt," "In Transit," or "Out for Delivery."

[0066] The update content refers to the actions taken when updating the delivery status of a package, based on identification information, role, and delivery status.

[0067] The delivery status of the package data is updated according to this marker data. However, the CPU 30 of the management server 12 decides whether to register the package as delivered and placed in the delivery box, or to register it as returned (see Figures 11-13).

[0068] Specifically, before shipment registration, the delivery status of a package is "unregistered." Once shipment registration is performed, the delivery status of the package is updated to "before receipt." Next, once receipt registration is performed, the delivery status of the package is updated to "in transit." Furthermore, if departure registration is performed when the package is being prepared for handover, the delivery status of the package is updated to "out for delivery." Finally, once delivery completion registration is performed, such as when the package is handed over to the recipient or placed in a delivery box, the delivery status of the package is updated to "delivered." However, if the delivery person takes the package back and departure registration is performed, the delivery status of the package is updated to "taken back." Also, if the delivery of a package is canceled after shipment registration, the delivery status of the package is updated to "delivery canceled."

[0069] As shown in Figure 8(B), the package data is a table that contains identification information, delivery destination, delivery region, and delivery status, corresponding to the package ID.

[0070] The package ID is an ID used to identify each package. In the example shown in Figure 8, the packages are indicated by adding an alphabet letter to the package name, such as Package A, Package B, ...

[0071] The identification information is the information associated with the AR marker attached to the package indicated by the package ID. In other words, the marker data and package data are linked by the identification information.

[0072] The delivery address is information about the delivery destination, including the delivery address, telephone number, email address, and name (or full name). In Figure 8, for simplicity, the delivery destinations are indicated with letters of the alphabet, such as Delivery Destination X, Delivery Destination Y, ...

[0073] The delivery area is the delivery area of ​​the package indicated by the package ID, and is not limited to administrative divisions such as cities, towns, or villages. It is an area or range predetermined by the company to which the delivery person belongs or the administrator of the information processing system 10. In Figure 8, the delivery areas are classified in an identifiable manner by adding numbers or the like before the word "area," such as Area 1, Area 2, etc., but they may also be classified in an identifiable manner using symbols such as letters and / or numbers.

[0074] The delivery status is the status of the package indicated by the package ID, and is registered (updated) by the user scanning the AR marker attached to the package as needed, with statuses such as Unregistered, In Transit, Out for Delivery, Returned, and Delivered. Also, as mentioned above, if the sender or recipient instructs the cancellation of a package delivery, the person in charge of delivery who receives the instruction will update the delivery status of that package to Cancelled.

[0075] As shown in Figure 9, the RAM 32 includes a program storage area 302 and a data storage area 304. The program storage area 302 stores the information processing program that is executed by the management server 12.

[0076] The information processing programs executed on the management server 12 include a main processing program 302a, a communication program 302b, an update content determination program 302c, and a delivery status update program 302d, among others.

[0077] The main processing program 302a is a program for executing the main routine of information processing on the management server 12.

[0078] The communication program 302b is a program for communicating with external devices, in this embodiment, each user terminal 16 and a predetermined database.

[0079] The update content determination program 302c is a program that determines the content of the delivery status update based on the current delivery status and the identification information and role information received from the user terminal 16, by referring to the marker data 304a.

[0080] The delivery status update program 302d is a program that executes the update content determined according to the update content determination program 302c and sends the update content, i.e., the updated delivery status, to the user terminal 16 that sent the identification information and role information.

[0081] Although not shown in the diagram, the program storage area 302 stores application programs necessary to execute the information processing program of the management server 12.

[0082] The data storage area 304 stores marker data 304a, package data 304b, identification information data 304c, role information data 304d, related information data 304e, delivery cancellation information data 304f, and error information data 304g, among others.

[0083] Marker data 304a, as shown in Figure 9(A), is table data for update content set according to the role and delivery status for each package ID to which an AR marker is attached.

[0084] As shown in Figure 9(B), package data 304b is data for each package, and corresponds to the package ID, it is table data that lists the delivery destination, delivery region, and delivery status.

[0085] The identification information data 304c is data about the identification information received from the user terminal 16, and the identification information is the identification information set on the AR marker read by the user terminal 16.

[0086] Role information data 304d is data about role information received from user terminal 16.

[0087] Related information data 304e is data about the updated delivery status of the package in accordance with the delivery status update program 302d. If the package was placed in a delivery box, the delivery status also includes information that the package was placed in a delivery box. However, the package is the one corresponding to the identification information indicated in the identification information data 304c. This is also true for the delivery cancellation information data 304f and the error information data 304g.

[0088] Delivery cancellation information data 304f contains data about the cancellation of a package delivery. Error information data 304g contains data about errors that occur when the delivery area of ​​a package differs from the user's assigned area.

[0089] Although not shown in the diagram, the data storage area 304 stores other data necessary for performing information processing, and also contains timers (counters) and flags necessary for performing information processing.

[0090] Although the marker data 304a and package data 304b are stored in the memory of the management server 12, they may also be stored in external memory accessible by the management server 12. The external memory may be located on the network 14.

[0091] As shown in Figure 10, the RAM 52 includes a program storage area 402 and a data storage area 404. The program storage area 402 stores the tracking application of this embodiment, which is an example of an information processing program executed on the user terminal 16.

[0092] The information processing program executed on the user terminal 16 includes a main processing program 402a, a communication program 402b, an operation detection program 402c, an image generation program 402d, an image output program 402e, an image capture program 402f, and an AR marker detection program 402g, among others.

[0093] The main processing program 402a is a program that executes the main routine for processing the tracking application on the user terminal 16.

[0094] The communication program 402b is a program for communicating with an external device, in this embodiment, the management server 12.

[0095] The operation detection program 402c is a program for detecting user actions.

[0096] The image generation program 402d is a program for generating display image data using the image generation data 404b. However, when generating display image data for an application screen, the presentation information data 404h is also used.

[0097] The image output program 402e is a program for outputting display image data generated according to the image generation program 402d to the display 60. Therefore, various screens such as application screens are displayed.

[0098] The shooting program 402f is a program that causes the camera on the user terminal 16 to perform the shooting process.

[0099] The AR marker detection program 402g is a program for detecting AR markers contained in a captured image 70 taken according to the shooting program 402f, and for identifying identification information associated with the detected AR markers, and is an image processing library such as OpenCV.

[0100] Although not shown in the diagram, the program storage area 402 stores the application programs necessary to execute the information processing program of the user terminal 16.

[0101] The data storage area 404 stores operation data 404a, image generation data 404b, role information data 404c, identification information data 404d, related information data 404e, delivery cancellation information data 404f, error information data 404g, and presentation information data 404h, among others.

[0102] Operation data 404a is data about user operations detected according to the operation detection program 402c.

[0103] Image generation data 404b is data for displaying display image data corresponding to various screens.

[0104] Role information data 404c is data about the role information of a user registered on the device, and the role information is information about the user's role and assigned region. Role information data 404c may be registered in the user terminal 16 in advance, or it may be set by the user when the tracking application is run.

[0105] Identification data 404d is data of identification information associated with AR markers detected according to the AR marker detection program 402g from captured images 70 taken by the camera while running the tracking app.

[0106] Related information data 404e is related information received from the management server 12, namely data about the updated delivery status of the package. However, the package is the one corresponding to the identification information indicated in the identification information data 404d. The same applies to the delivery cancellation information data 404f and the error information data 404g.

[0107] Delivery cancellation information data 404f is data received from the management server 12 regarding the cancellation of a package delivery.

[0108] Error information data 404g is data received from the management server 12 regarding an error that occurred when the delivery area of ​​a package differs from the user's assigned area.

[0109] The presentation information data 404h is data about the presentation information generated based on the related information data 404e, the delivery cancellation information data 404f, or the error information data 404g.

[0110] Although not shown in the diagram, the data storage area 404 stores other data necessary for performing information processing, and also contains timers (counters) and flags necessary for performing information processing.

[0111] Using the flowcharts shown in Figures 11-13, we will explain an example of information processing by the CPU 30 of the management server 12 shown in Figure 2. The codes starting with "S" shown in these figures are codes assigned to identify each step (process), and in the following explanation, each step (process) will be represented by a code. The same applies to the flowchart shown in Figure 14.

[0112] As shown in Figure 11, the CPU 30 of the management server 12 starts processing information according to instructions from the administrator, etc., and in S1, it determines whether it has received identification information and role information from the user terminal 16. If the answer in S1 is "NO", that is, if it has not received identification information and role information from the user terminal 16, it proceeds to S21.

[0113] On the other hand, if the answer in S1 is "YES," that is, if identification information and role information have been received from the user terminal 16, then in S3, it is determined whether the package is in the area that the user of the user terminal 16 is responsible for. Here, the CPU 30 determines whether the area of ​​responsibility included in the user's role information is the same as the delivery area of ​​the package indicated by the identification information.

[0114] If the result in S3 is "NO," that is, if the package is not in the area handled by the user terminal 16, then in S5, error information is sent to the user terminal 16 that sent the identification information and role information, and the process proceeds to S41 as shown in Figure 13.

[0115] On the other hand, if the answer in S3 is "YES," that is, if the package is in the area managed by the user of user terminal 16, then in S7, the delivery status of the package indicated by the identification information is obtained from the package data 304b, and the process proceeds to S9 as shown in Figure 12.

[0116] As shown in Figure 12, in S9, it is determined whether the judgment time is being counted. If the answer in S9 is "NO", that is, if the judgment time is not being counted, the process proceeds to S19. On the other hand, if the answer in S9 is "YES", that is, if the judgment time is being counted, then in S11, it is determined whether the judgment time has elapsed.

[0117] If the answer in S11 is "NO," that is, if the judgment time has not elapsed, the process proceeds to S23. On the other hand, if the answer in S11 is "YES," that is, if the judgment time has elapsed, the process proceeds to S13, where it is decided to register the update as a takeaway; in S15, the delivery status of the package corresponding to the identification information is updated to takeaway in the package data 304b; and in S17, the relevant information about the takeaway is sent to the user terminal 16 that sent the identification information and role information, and the process proceeds to S41.

[0118] In S19, it is determined whether the delivery status obtained in S7 indicates that the package is out for delivery. If the answer in S19 is "NO," that is, if the delivery status obtained in S7 does not indicate that the package is out for delivery, the process proceeds to S31 as shown in Figure 13. On the other hand, if the answer in S19 is "YES," that is, if the delivery status obtained in S7 indicates that the package is out for delivery, the process starts counting the determination time in S21 and proceeds to S23.

[0119] In S23, it is determined whether information about the delivery box has been received from the user terminal 16. If the answer in S23 is "NO," that is, if information about the delivery box has not been received from the user terminal 16, the process returns to S11. On the other hand, if the answer in S23 is "YES," that is, if information about the delivery box has been received from the user terminal 16, then in S25, the update content is determined to be registered as delivery completed, in S27, the delivery status of the package corresponding to the identification information is updated to delivery completed, and in S29, relevant information about the package being placed in the delivery box is sent to the user terminal 16 that sent the identification information and role information, and the process proceeds to S41.

[0120] As shown in Figure 13, in S31, it is determined whether or not to cancel the delivery. If the answer in S31 is "YES", that is, if the delivery is canceled, in S33 the delivery cancellation information is sent to the user terminal 16 that sent the identification information and role information, and the process proceeds to S41.

[0121] On the other hand, if the answer in S31 is "NO," that is, if the delivery is not canceled, the action is decided in S35. Here, the CPU 30 refers to the marker data 304a and decides on the update content according to the delivery status obtained in S7, and the package indicated by the identification information and the role indicated by the role information. In the next step, S37, the update content decided in S35 is executed to update the delivery status of the package indicated by the identification information, and in S39, related information about the updated delivery status is sent to the user terminal 16 that sent the identification information and role information, and the process proceeds to S41.

[0122] In S41, it is determined whether there is an instruction to terminate the information processing. If the answer in S41 is "NO," that is, if there is no instruction to terminate the information processing, the process returns to S1. On the other hand, if the answer in S41 is "YES," that is, if there is an instruction to terminate the information processing, the information processing is terminated.

[0123] Figure 14 illustrates an example of tracking application processing by the CPU 50 of the user terminal 16 shown in Figure 3. When the tracking application is executed on the user terminal 16, as shown in Figure 14, the CPU 50 starts tracking application processing, activates the camera in S101, displays the AR marker reading screen on the display 60 in S103, and then displays the camera's captured image 70 on the display 60 in S105.

[0124] In the next step, S107, it is determined whether or not an AR marker has been detected. If the answer in S105 is "NO," that is, if no AR marker has been detected, the process proceeds to S111. On the other hand, if the answer in S107 is "YES," that is, if an AR marker has been detected, the process proceeds to S109 to identify the identification information and then to S111. In S109, the CPU 50 identifies the identification information associated with the detected AR marker and stores the identification information data 404d corresponding to the identified identification information in the RAM 52.

[0125] In S111, identification information and role information are sent to the management server 12. As described above, the role information may be registered in advance on the user terminal 16, or it may be set by the user when the tracking application process is started.

[0126] In the next step, S113, it is determined whether information, namely related information, delivery cancellation information, or error information, has been received from the management server 12. If the answer in S113 is "NO," that is, if no information has been received, the process returns to S113. On the other hand, if the answer in S113 is "YES," that is, if information has been received, in S115, the presentation information 80 based on the received information is superimposed on the captured image 70 and displayed on the display 60, and the process proceeds to S117.

[0127] In S115, if relevant information is received, the CPU 50 displays presentation information 80, which includes a normal message, in front of the captured image 70 in relation to the image of the corresponding AR marker. If delivery cancellation information is received, the CPU 50 displays presentation information 80, which includes a delivery cancellation message, in front of the captured image 70 in relation to the image of the corresponding AR marker. If error information is received, the CPU 50 displays presentation information 80, which includes an error message, in front of the captured image 70 in relation to the image of the corresponding AR marker.

[0128] In S117, it is determined whether there is an instruction to terminate the tracking app. If the answer in S117 is "NO," that is, if there is no instruction to terminate the tracking app, the process returns to S105. On the other hand, if the answer in S117 is "YES," that is, if there is an instruction to terminate the tracking app, the camera is stopped and the tracking app process is terminated in S119.

[0129] According to this embodiment, when a user reads an AR marker using a user terminal, the identification information associated with the AR marker and the user's role information are sent to the management server. The management server updates the delivery status of the package according to the identification information and role information, and sends the updated delivery status, i.e., related information, to the user terminal. Therefore, simply by reading the AR marker with the user terminal, the delivery status of the package with the AR marker is updated, and the user can easily find out that the delivery status has been updated and to know the updated delivery status. In other words, user convenience can be improved.

[0130] In this embodiment, the delivery status is updated for one AR marker according to three different roles, but it is also possible to update the delivery status for one AR marker according to two or four or more different roles. For example, the person responsible for shipping and the person delivering may be the same user, while the recipient (collector) and the person delivering may be different users. Furthermore, the delivery status may be further subdivided, with different users assigned to each stage.

[0131] Furthermore, in this embodiment, the information presented is displayed as text next to the AR marker. However, since it is sufficient for the user to be notified that the delivery status has been updated, that delivery has been canceled, or that an error has occurred, each piece of information may be associated with a visible graphic and displayed superimposed on the captured image. For example, a rectangular frame image of a color corresponding to each piece of information may be displayed so as to overlap or surround the AR marker. Instead of rectangular frame images of different colors, images of rectangular frames with different line types or images of different shapes or symbols may be displayed.

[0132] Furthermore, while this embodiment updates the delivery status of a single package by reading one AR marker attached to that package, it is not limited to this. The delivery status of multiple packages could be updated simultaneously by reading each of the AR markers attached to each of multiple packages at once.

[0133] As shown in Figure 15(A), when multiple packages are placed on the floor with the sides bearing AR markers facing the same direction, and the user launches the tracking application on the user terminal 16 and takes an image including the AR markers attached to each of the packages, multiple AR markers included in the captured image 70 are detected. In this case, the identification information for each of the detected AR markers is identified, and the identified identification information and the user's role information are sent to the management server 12.

[0134] In Figure 15(A) (and the same applies to Figure 15(B)), AR markers are indicated by double-lined squares, and each AR marker is distinct from the others.

[0135] The management server 12 individually determines whether the delivery area for each package corresponding to each identification piece of information is the same as the assigned area included in the role information. For packages where the delivery area and assigned area are the same, the management server 12 updates the delivery status and sends related information to the user terminal 16. For packages where the delivery area and assigned area are different, the management server 12 sends error information to the user terminal 16. However, even if the delivery area and assigned area are the same, if the delivery status is canceled, the management server 12 sends cancellation information to the user terminal 16. In other words, the CPU 30 of the management server 12 executes the S3-S39 processes for each package corresponding to multiple identification pieces of information in the information processing shown in Figures 11-13. Furthermore, since the user terminal 16 needs to identify the related information, cancellation information, or error information sent for each package, when processing multiple packages at once, the identification information corresponding to each package is also added to the related information, cancellation information, and error information.

[0136] Therefore, as shown in Figure 15(B), presentation information 80 based on related information, delivery cancellation information, or error information is displayed superimposed on the captured image 70. As can be seen from Figure 15(B), for packages whose delivery status has been successfully updated and related information has been received, a solid-lined rectangular frame image is displayed as presentation information 80, surrounding the AR marker image 72. For packages that have received error information, a dotted-lined rectangular frame image is displayed as presentation information 80, surrounding the AR marker. Although not shown in the illustration, for packages that have received delivery cancellation information, a rectangular frame image with lines other than solid and dotted lines is displayed as presentation information 80, surrounding the AR marker image 72. In other words, in the information processing shown in Figure 14, when the CPU 50 of the user terminal 16 detects multiple AR markers in S107, it executes the processing S109-S115 for each AR marker, i.e., each package.

[0137] However, instead of using different line types for each piece of information in the image of the rectangular frame, the rectangular frame may be made a different color, a different shape, or a different symbol may be used instead of the frame.

[0138] Additionally, for packages that have received error information, nothing may be displayed in relation to the AR marker image to distinguish them from packages that have received related information or delivery cancellation information.

[0139] Furthermore, the order in which each step in the flowchart shown in this embodiment is processed can be changed if the same result can be obtained.

[0140] Furthermore, the various screens and specific numerical values ​​shown in this specification and drawings are merely examples and can be modified as needed. [Explanation of Symbols]

[0141] 10. Information Processing Systems 12 ... Management Server 14 ... Network 16 ... User terminal 30, 50 ...CPU 32, 52... RAM 34, 54 ... Communication I / F 36, 56 ... Input / Output Interface 58 ... Input section 60…Display 62 ... Output section

Claims

1. An information processing system comprising a user terminal used by the user and a management device that manages the setting information set for each of the AR markers attached to each of the one or more delivered items, The aforementioned user terminal is Camera and, If the image captured by the camera contains the AR marker, the identification means detects the AR marker and identifies the identification information associated with the AR marker. The system includes a first transmission means that transmits the identified identification information and the pre-configured role information of the user to the management device, The aforementioned control device is A first receiving means for receiving the aforementioned identification information and the aforementioned role information, An update means that updates related information relating to the delivery item according to the identification information and role information based on the setting information, The system includes a second transmission means for transmitting the updated related information to the user terminal that is the source of the identification information and the role information, The aforementioned user terminal further, A second receiving means for receiving the aforementioned related information, A generation means for generating presentation information based on the aforementioned related information and the aforementioned role information, An information processing system comprising: display means for displaying the presented information on a display by superimposing it onto the image in a display manner corresponding to the AR marker contained in the image.

2. The update means further updates the related information in accordance with other information besides the identification information and the role information, The information processing system according to claim 1, wherein the display means, upon receiving the related information updated according to the other information, displays the presented information on the display by superimposing it onto the image in a display manner different from when it was updated according to the identification information and the role information.

3. When the image captured by the camera includes a plurality of AR markers, The identification means identifies each of the identification information associated with each of the plurality of AR markers, The first transmission means transmits each of the identification information and the role information to the management device. The first receiving means receives each of the identification information and the role information, The update means updates the related information relating to each of the deliveries to which some or all of the plurality of AR markers are attached, based on the setting information, according to each of the identification information and the role information. The information processing system according to claim 1 or 2, wherein when the display means updates the related information relating to each of the deliverables to which some or all of the plurality of AR markers are attached, it superimposes the presentation information for some or all of the plurality of AR markers onto the image and displays it on the display.

4. An information processing method for an information processing system comprising a user terminal equipped with a camera and used by the user, and a management device that manages setting information set for each of the AR markers attached to each of the one or more delivered items, The aforementioned user terminal is If the image captured by the camera contains the AR marker, the AR marker is detected, and the identification information associated with the AR marker is identified. The identified identification information and the pre-configured role information for the user's role are transmitted to the management device. The aforementioned control device is Upon receiving the aforementioned identification information and the aforementioned role information, Based on the aforementioned setting information, the relevant information related to the delivery item is updated according to the identification information and the role information. The updated related information is transmitted to the user terminal that sent the identification information and the role information. The aforementioned user terminal further, Upon receiving the aforementioned related information, Based on the aforementioned related information and the aforementioned role information, the presentation information is generated. An information processing method for displaying the presented information on a display screen by superimposing it onto the image in a display manner that corresponds to the AR marker contained in the image.

Images