Information processing system, information processing device, and program

Abstract

To efficiently perform support for store operation including work procedures and placement of assistants in store operation, and support for teaching the assistants, based on states of inside and outside of the store.SOLUTION: In an information processing system, an imaging control unit 201 of a camera 2 captures a subject included in the surrounding 360 degrees omnidirectional range. A generation unit 202 generates data of the captured image of the subject. An acquisition unit 101 of a server 1 acquires the data of the captured image including at least a part of inside and outside of a store R, which is captured and generated by each of the plurality of cameras 2. An operation support unit 102 presents operation support information of a store staff member T of the store R based on the data of the captured image. A teaching support unit 103 presents teaching support information of the store staff member T of the store R based on the data of the captured image. This solves the above problem.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, and a program.

Background Art

[0002] In recent years, the following problems (1) to (3) exist in Japan. That is, there is a problem that "local creation" aimed at correcting the state where social capital such as politics, economy, and population is concentrated in urban areas and raising the vitality of the whole country, "regional activation" aimed at improving the economic power of regions and increasing the population, has become a mere formality. As shown in Fig. 1(A), this is said to be caused by the limitations of administratively led scattered budgets, empty strategic discussions on paper, and the commercialization of so-called "events". In addition, there is a problem that measures for foreign tourists visiting Japan (inbound) are urgently needed. As shown in Fig. 1(A), this is said to be caused by the response to the tourism promotion strategy based on the Tourism Promotion Basic Law, the responsibilities as the host country of international conferences such as the Olympics, and the increase and diversification of foreign tourists (inbound). In addition, there is a problem that social demands are becoming more sophisticated. As shown in Fig. 1(A), this is said to be caused by the response to systems such as HACCP (Hazard Analysis and Critical Control Point), the trend towards cashless, and the demands for so-called "de-black" (companies) due to the reform of work styles.

[0003] One method for solving the problems listed in (1) through (3) above is to give the "town" the "power to earn." As shown in Figure 1(B), this is a method based on the "win-win-win" concept, in which three parties—managers, professional personnel, and finance—work together in a balanced and fully effective manner to create a "profitable town." Specifically, this concept aims to revitalize those who provide goods or services by having the three parties "collaborate" to support them. In other words, if the three parties mentioned above actively support restaurants and other establishments that prepare and serve food using ingredients sourced from the local area, the "town" will gain the "ability to generate income," and as a result, the issues listed in (1) through (3) above can be resolved. This concept is also known as "digital hospitality HACCP."

[0004] When supporting restaurants and other similar establishments, there are, for example, the following conventional technologies that can be applied. Specifically, Patent Document 1 describes a device that analyzes the customer service situation in a store based on image data captured by a camera that takes images inside the store, in order to enable store managers to grasp the actual situation of customer service. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Patent No. 5314200 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] However, in conventional technologies, including the technology described in Patent Document 1, the analysis is limited to image data captured by cameras installed "inside the store," and therefore, "outside the store" situations that could have a significant impact on the store's operations are not included in the analysis. External factors include, for example, the status of various events being held in the vicinity of the store, the number of people coming and going in the vicinity of the store, and the weather conditions at the store's location. All of these can influence the number of customers visiting the store.

[0007] This invention has been made in view of the above circumstances, and aims to provide a method that can efficiently support store operations, including the work procedures and placement of assistants in store operations, and support the training of assistants, based on the conditions inside and outside the store. [Means for solving the problem]

[0008] To achieve the above objective, an information processing system according to one aspect of the present invention is: In an information processing system that supports a provider who provides goods or services to a customer at a designated location, and an assistant to such provider, Equipped with multiple imaging devices and an information processing device, The plurality of imaging devices, It has a generation means that captures images of a subject included in a 360-degree range in all directions and generates data of the captured images of the subject. The aforementioned information processing device is An acquisition means for acquiring data of an image captured and generated by each of the plurality of imaging devices, which includes at least a portion of the field and its surroundings. Operation support means that, based on the data acquired by the acquisition means, performs a first process to support operations performed by the assistant in the field and at least a portion of the surrounding area, An educational support means that performs a second process to support the education of the assistant based on the data acquired by the acquisition means, It is equipped with.

[0009] Furthermore, the system may include a suggestion means that proposes the locations and number of imaging devices to be placed in the field and its surroundings, based on the data of the captured images that would be obtained if each of the multiple imaging devices were placed at each of the multiple predetermined locations in and around the field.

[0010] Furthermore, an information processing apparatus according to one embodiment of the present invention is In an information processing device included in an information processing system that supports a provider who provides goods or services to a customer at a designated location, and an assistant of said provider, An acquisition means for acquiring data of an image captured and generated by each of the plurality of imaging devices, which includes at least a portion of the field and its surroundings. Operation support means that, based on the data acquired by the acquisition means, performs a first process to support operations performed by the assistant in the field and at least a portion of the surrounding area, An educational support means that performs a second process to support the education of the assistant based on the data acquired by the acquisition means, It is equipped with.

[0011] Furthermore, the system may include a suggestion means that proposes the locations and number of imaging devices to be placed in the field and its surroundings, based on the data of the captured images that would be obtained if each of the multiple imaging devices were placed at each of the multiple predetermined locations in and around the field.

[0012] An information processing method and program according to one aspect of the present invention are a processing method and program corresponding to the information processing system according to one aspect of the present invention described above. [Effects of the Invention]

[0013] According to the present invention, it is possible to efficiently provide support for store operations, including the work procedures and placement of assistants in store operations, and support for the training of assistants, based on the conditions inside and outside the store. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram showing the social background in recent years. [Figure 2] This is a diagram showing the outline of the store operation support service that can be realized by the information processing system of the present invention. [Figure 3] This is a diagram showing the types of businesses targeted by this service and the effects of providing this service. [Figure 4] This is an image diagram showing a specific example of this service shown in FIG. 1. [Figure 5] This is a diagram showing specific examples of cameras installed in multiple locations inside and outside the store. [Figure 6] This is a flowchart showing the flow of the proposed service among the services of this service. [Figure 7] This is an image diagram showing imaging result information. [Figure 8] This is a diagram showing the outline of the configuration of the information processing system of the present invention. [Figure 9] This is a block diagram showing the hardware configuration of the server in FIG. 8. [Figure 10] This is a functional block diagram showing an example of the functional configuration required for various processes executed by the server in FIG. 9. [Figure 11] This is a diagram showing an example of a common infrastructure operation model. [Embodiments for Carrying Out the Invention]

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0016] [Service Outline] FIG. 2 is a diagram showing the outline of a store operation support service (hereinafter referred to as "this service") that can be realized by the information processing system of the present invention.

[0017] This service is a service that supports providers who provide goods or services to customers at a predetermined location and their assistants to improve productivity. For example, Figure 2 shows an example of how this service can be applied, illustrating how it supports store manager P, who provides food and beverages and customer service to customer C at store R (a designated location), and store staff T, who assists P.

[0018] Restaurant R is a restaurant operated by store manager P. Restaurant R serves customers C dishes that are made with locally sourced ingredients, a practice known as "local production for local consumption." Business partner M is a related business partner of store R, and provides support for store R's business. As mentioned above, customer C is a user of store R.

[0019] This service can be realized through the collaboration of a server 1 (cloud), a camera 2, a store terminal 3, a trading partner terminal 4, and a customer terminal 5, which constitute the information processing system according to the present invention.

[0020] Server 1 performs various processes necessary to realize this service while communicating with Camera 2, Store Terminal 3, Business Partner Terminal 4, and Customer Terminal 5 as needed. Server 1 also functions as a cloud managed by AI (Artificial Intelligence). Server 1 generates information to support the operations of store manager P at store R (hereinafter referred to as "operation support information") and information to support the training of store staff T (hereinafter referred to as "training support information") based on the image data transmitted from camera 2, for example. The operation support information and training support information generated by Server 1 are presented to at least one of the store manager P and the store staff T. The presentation method is not particularly limited, and the information may be presented by displaying operation support information and training support information on an administrator terminal (not shown) operated by the store manager P, or on the store terminal 3 described later. Other methods (for example, presentation by voice or presentation on paper) may also be used.

[0021] Furthermore, Server 1 generates captured image data (hereinafter referred to as "simulation data") that would have been obtained if multiple cameras 2 had been placed at each of the multiple predetermined locations in and around Store R. Then, based on the generated simulation data, Server 1 generates information (hereinafter referred to as "proposed information") that shows the optimal combination of placement locations and number of cameras 2 in and around store R. The specific method for generating proposed information based on simulation data will be described later, with reference to Figure 6.

[0022] Camera 2 is a camera installed in multiple locations both inside and outside store R. Camera 2 captures images of subjects included in the surrounding area of ​​its installation location and generates image data of those subjects. The specific configuration of Camera 2 will be described later with reference to Figure 5. Camera 2 is an IoT (Internet of Things) device connected to a predetermined network N (Figure 8), such as the Internet, and therefore uploads the generated captured image data to the cloud (server 1) as needed.

[0023] Store terminal 3 is an information processing device connected to a predetermined network N (Figure 8), such as the internet, and, like camera 2, is connected to the cloud (server 1). Store terminal 3 acquires operation support information, training support information, etc., from the cloud (server 1) and displays this information. This improves the productivity of store R. Store terminal 3 has a dedicated application program (hereinafter referred to as the "staff-only app") installed for receiving the service that supports the work of store staff T (hereinafter referred to as the "staff support service"). The staff support service includes a service that displays the aforementioned operation support information and training support information on store terminal 3. Furthermore, unless otherwise specified, when we refer to "store staff member T operating store terminal 3," it means that store staff member T is launching the staff-only application installed on store terminal 3 and performing various operations.

[0024] The trading partner terminal 4 is an information processing device operated by trading partner M. The customer terminal 4 has a dedicated application program (hereinafter referred to as the "customer-specific app") installed for receiving the service (hereinafter referred to as the "customer support service") that supports customer M, who intends to provide business support (for example, various management tasks) to store manager P. The business partner support service includes a service that displays information on the business partner terminal 4 to support business strategies, including information on business creation (e.g., product development), information to support compliance with laws and regulations, and information on customer acquisition. By providing customer support services, the productivity of store R can be further improved through operational support from customer M, and customer M itself can expand its business opportunities. This can contribute to supporting the "optimization" described later, as shown in Figure 2. Unless otherwise specified, the phrase "customer M operates customer terminal 4" below means that customer M launches the customer-specific application installed on customer terminal 4 and performs various operations.

[0025] Customer terminal 5 is an information processing device operated by customer C. Customer terminal 5 has a dedicated application program (hereinafter referred to as the "customer-only app") installed for receiving services from this service that promote the use of store R and support customers C who actually use store R (hereinafter referred to as the "customer support service"). The customer support service includes a service that displays information related to creating customer satisfaction on customer terminal 5. The provision of customer support services facilitates customer communication between the store R and the customer C, including media and CRM (Customer Relationship Management). This allows customer C to utilize services that meet diverse consumer needs, thus contributing to the "growth" described later, as shown in Figure 3. Unless otherwise specified, the phrase "Customer C operates Customer Terminal 5" below means that Customer C launches the customer-specific application installed on Customer Terminal 5 and performs various operations.

[0026] In this service, at least some of the various types of information handled by the store terminal 3, the trading partner terminal 4, and the customer terminal 5 will be provided to regional information agencies and regional financial institutions via a cloud managed by server 1. Here, "regional information agency" refers to an organization that specializes in acquiring and providing key data related to a particular region. Regional information agencies include, for example, regional information banks. Furthermore, "regional financial institution" refers to a financial institution whose main business base is a particular region, and specifically refers to local banks, second-tier regional banks, credit unions, agricultural cooperatives, fisheries cooperatives, labor banks, etc.

[0027] Regional information agencies and financial institutions that receive various types of information through this service can use the provided information to provide business support to businesses in other regions and to communities in general. As a result, it is possible to prevent regional revitalization and development from becoming mere formalities.

[0028] Figure 3 shows the types of businesses targeted by this service and the effects of providing this service.

[0029] Figure 3(A) shows a diagram that visualizes the types of businesses that should be targeted as the recipients of this service. Specifically, in Figure 3(A), the business is classified using "Brand Level" (vertical axis), which represents the brand value of a business evaluated based on acquired standards, etc., and "Business Stage" (horizontal axis), which represents the business evaluated over time.

[0030] Here, the "hospitality standards certification" shown in Figure 3(A) refers to the standards that are certified for each business that provides the service, based on a standards certification system established to "visualize" the quality of the service. Among the various certifications that make up the "Hospitality Standards Certification," the "Red Certification" is a standard that businesses that are enthusiastic about improving their services can receive, and there are "Red Certification," "Gold Certification," "Navy Certification," and "Purple Certification." Of these, the "Red Certification" shown in Figure 3(A) is a standard that is certified by the business operator declaring that it meets the conditions for certification (conforms to the standards). Among the various certifications that make up the "Hospitality Standards Certification," the "Gold Certification" is a standard awarded to businesses that have been certified by a third party as providing services that exceed customer expectations, although this is not illustrated in the diagram. Businesses that have received the "Gold Certification" are considered to have a higher level of service than businesses that have received the "Red Certification." Among the various certifications that make up the "Hospitality Standards Certification," the "Navy Certification" shown in Figure 3(A) is a standard awarded to businesses that have been certified by a third party as providing services that incorporate unique creativity and ingenuity. Businesses that have received the "Navy Certification" are considered to have an even higher level of service than businesses that have received the "Gold Certification." Among the various certifications that make up the "Hospitality Standards Certification," the "Purple Certification" is a standard granted to businesses that have been certified by a third party as providing services (i.e., "hospitality") that far exceed customer expectations, although this is not illustrated in the diagram. Businesses that have been granted the "Purple Certification" are considered to have an even higher level of service than businesses that have been granted the "Navy Certification."

[0031] Furthermore, in Figure 3(A), "compliant with Ministry of Health, Labour and Welfare HACCP B" refers to businesses that focus on general hygiene management and, based on the HACCP concept, set and manage critical control points according to the type of business and its conditions to the extent possible. Furthermore, in Figure 3(A), "HACCP certification" for businesses in the "transformation phase" or later, and with a brand level of "upper" or higher, refers to businesses whose internal hygiene management system is evaluated by a third party (HACCP certification body) as functioning correctly with HACCP. Examples of HACCP certification bodies include the industry association to which the business belongs and local governments. Furthermore, in Figure 3(A), "ISO22000," which is in the "transformation phase" or later and belongs to the "high-end" brand level, refers to a business that has received certification for the international standard "Food safety management systems - Requirements for organizations in the food chain." Furthermore, in Figure 3(A), "FSSC22000," which refers to a business whose stage is "growth / maturity" or later and whose brand level is "high-end," refers to a business that has received accreditation for the international standard for food safety management systems, which is an enhanced version of ISO22000 with additional requirements.

[0032] Of these, the businesses that should be targeted for this service are those belonging to the shaded area in Figure 3(A). In other words, these are businesses that meet both the "Navy Blue Certification" of the Hospitality Standards Certification and "Compliance with the Ministry of Health, Labour and Welfare's HACCP B."

[0033] Figure 3(B) shows a growth model for businesses that utilize this service. As shown in Figure 3(B), "strengthening management," "securing human resources," and "improving operational efficiency" in business operations require "optimization," and achieving this "optimization" generates "surplus funds." On the other hand, "growth" is required for "branding," "increasing customer attraction," and "increasing the unit price" of products and services in business, so the aforementioned "surplus funds" are invested to realize this "growth." This "investment" can "grow" the business (for example, the retail business), and if that "growth" spreads to surrounding businesses, it can ultimately contribute to the revitalization of the local economy.

[0034] Figure 4 is an illustrative diagram showing a specific example of the service described in Figure 1.

[0035] Figure 4 depicts a store R equipped with cameras 2-1 to 2-7, multiple store staff T engaged in providing services within the store R, and multiple customers C who are using or could use the store R.

[0036] Cameras 2-1 and 2-2 are installed on the exterior wall of store R and each captures a predetermined area outside store R. Specifically, camera 2-1 captures at least subjects included in the imaging areas A1-1 and A1-2 indicated by dashed lines. Camera 2-2 captures at least subjects included in the imaging areas A2-1 and A2-2 indicated by dashed lines. Cameras 2-3 are installed outside the entrance of store R and capture images of subjects included in the imaging area A3 indicated by the dashed line. Cameras 2-4 and 2-5 are installed near the ceiling of the floor of store R and each captures a predetermined area of ​​the floor of store R. Specifically, camera 2-4 captures at least subjects included in the imaging area A4 indicated by the dashed line. Camera 2-5 captures at least subjects included in the imaging area A5 indicated by the dashed line. Camera 2-6 is installed in the kitchen of store R and captures images of subjects included in the imaging area A6 indicated by the dashed line. Camera 2-7 is a camera installed in the warehouse of store R and captures images of subjects included in the imaging area A7 indicated by the dashed line.

[0037] As described above, seven cameras, 2-1 through 2-7, are installed in every location both inside and outside store R, enabling a wide range of analyses based on the image data captured by each camera. For example, AI (artificial intelligence) analysis can be used to generate comprehensive operational support information and training support information.

[0038] Here, the specific installation configurations of cameras 2-1 to 2-7 are not particularly limited, but it is preferable that the installation configurations take into consideration that the cameras are positioned at a human eye level. In other words, it is preferable that cameras 2-1 to 2-7 be installed so as to function as a substitute for the store manager P's line of sight. Specifically, the height from the ground (floor) where cameras 2-1 to 2-7 are installed should preferably be around 1.8m to 2.5m. This will allow for more accurate observation of the facial expressions of store staff T and customer C. Furthermore, it is preferable to set the imaging areas of cameras 2-1 to 2-7 so that at least a portion of them overlap. This allows the image data from cameras 2-1 to 2-7 to be seamlessly connected.

[0039] Furthermore, the following services can be provided to the store staff T of store R shown in Figure 4. In other words, by using cameras 2-1 to 2-3 installed on the exterior wall or entrance of store R to monitor pedestrian traffic around store R, it is possible to generate and present operational support information that is optimal for store staff T. Specifically, based on the number and appearance of multiple customers C present as subjects in imaging areas A1-1 and A2-1, the weather present as a subject in imaging areas A1-2 and A2-2, and events being held near the store R, it is possible to generate and present operational support information related to, for example, preparing seating areas, assigning store staff T (whether or not they can take breaks), food preparation in the kitchen (e.g., cooking), and setting up lighting and air conditioning on the floor.

[0040] Furthermore, hospitality can be improved based on customer C's visit patterns to store R. In other words, by using cameras 2-3 installed at the entrance and exit of store R to monitor customer C's visit status, it is possible to generate and present optimal operational support information and educational support information for store staff T. Specifically, based on information such as the number and appearance of customer C present in the imaging area A3, their movement from outside to the store, or information obtained from facial recognition using cameras 2-3, store staff T can be guided to coincide with the timing of customer C's arrival. This helps to avoid the loss of opportunities for customer C to visit the store. Furthermore, based on the information about customer C obtained from the captured image data, it becomes possible to provide individualized service to each customer C. In other words, it becomes possible to improve hospitality. That is, it becomes possible to provide service based on the spirit of "omotenashi," the kind of service that people provide to other people. Furthermore, it enables analytics based on information about customer C. In other words, by utilizing various analytical methods and algorithms, it becomes possible to uncover specific patterns and correlations hidden within information about customer C. This allows for adjustments such as lighting, sound, and air conditioning tailored to customer C's visit. Furthermore, the interaction (customer service) between store staff member T and customer C can be monitored and stored as a work record. This enables real-time evaluation of customer service quality by comparing it with the customer service manual at store R, and the distribution of e-learning content based on the real-time evaluation.

[0041] Furthermore, by monitoring table service provided by store staff member T, it is possible to generate and present operational support information and training support information that is optimal for store staff member T. Specifically, cameras 2-4 and 2-5 installed on the floor of store R are used to monitor the availability of seats at tables on the floor, the status of customers C who have come to the store, and the status of food and drink service. This allows staff to guide customer C to a table appropriate to the number of people in their party, instruct on table setting, anticipate orders and any problems from customer C upon arrival, and provide appropriate assistance, including clearing empty dishes. Furthermore, it can be used to increase sales by recommending additional orders based on customer C's food and drink consumption and preferences. Furthermore, the service provided by store staff T to customer C can be monitored and stored as a work record. This enables real-time evaluation of service quality by comparing it with the customer service manual at store R, suggestions for revising the customer service manual based on improved service quality, and distribution of e-learning content based on real-time evaluations.

[0042] Furthermore, by monitoring the accounting process for customer C by store staff member T, and the situation regarding seeing customer C off, it is possible to generate and present operational support information and educational support information that is optimal for store staff member T. Specifically, by using cameras 2-4 and 2-5 installed on the floor of store R to monitor customer C leaving the store and store staff T attending to customer C, it is possible to realize services such as the following. In other words, it becomes possible to record customer C's stay time and actions during their stay, implement seamless payment with multiple pricing options using facial recognition, automatically notify customers if they dine and dash, and have store staff T see customer C off and provide additional services. Furthermore, by monitoring the causal relationship between customer C's behavior and the customer service provided by store staff T, it becomes possible to identify service patterns that contribute to store R's sales and propose effective promotional measures.

[0043] Furthermore, by monitoring the status of various tasks performed by store staff member T, it is possible to generate and present operational support information and educational support information that is optimal for store staff member T. Specifically, by using cameras 2-4 and 2-5 installed on the floor of store R, and camera 2-6 installed in the kitchen, it is possible to monitor store staff T's cooking, cleaning, and restocking activities, as well as the usage of ingredients and the staff's behavior, thereby enabling services such as the following: In other words, it enables recording who performed what task and what errors occurred, verifying whether the work was performed correctly according to the work manual, issuing warnings (alerts, etc.) for tasks that may cause errors (e.g., use of incorrect equipment or materials), calculating necessary inventory by accurately understanding the amount of work and usage, issuing work instructions for cleaning and various replenishments, creating shift records based on the timing when store staff member T started work, and checking and supporting the health status of store staff member T, which can be judged from their appearance. Furthermore, it's possible to evaluate the quality of store staff member T's work in real time and suggest manuals to improve work efficiency.

[0044] Furthermore, by monitoring product inventory and losses, it is possible to generate and present operational support information and training support information that is optimal for store staff member T. Specifically, cameras 2-7 installed in the warehouse of store R can be used to optimize delivery operations (e.g., labeling, delivery location) by suppliers (e.g., business partner M). Furthermore, waste disposal methods can be optimized to ensure thorough hygiene management. Furthermore, by monitoring delivery status, inventory status, waste loss status, etc., the following effects can be expected, for example: In other words, it becomes possible to thoroughly implement inventory management using the first-in, first-out method, enable the use of consigned inventory (i.e., charge only for what is used), prevent losses by optimizing inventory management for semi-prepared items, prevent theft of ingredients by store staff T, automate additional orders according to inventory levels, optimize inventory storage locations upon delivery, and clarify responsibility by managing the history of information regarding who takes inventory.

[0045] Figure 5 shows a specific example of multiple cameras 2 installed inside and outside store R.

[0046] As mentioned above, the specific configuration of camera 2 is not particularly limited, but by adopting camera 2 as shown in Figure 5, for example, this service can be provided in a more suitable environment. In other words, it becomes possible to generate operation support information and educational support information with more precise content. Specifically, the camera 2 in Figure 5 is configured to include a main body 211 and a lens 212, and is an ultra-high-resolution camera capable of capturing images of subjects within a 360-degree omnidirectional range and generating image data of those subjects. Here, "ultra-high resolution" refers to a particularly high resolution value, for example, a camera with a resolution of 4K (8 million pixels) or higher.

[0047] Figure 6 is a flowchart showing the flow of the proposed service within this service.

[0048] As described above, Server 1 generates information (suggestion information) that indicates the optimal combination of placement locations and number of cameras 2 in and around store R, based on the generated simulation data. The generated suggestion information is presented to store manager P. Hereafter, this type of service will be referred to as the "suggestion service". This allows store manager P to efficiently receive the service while maximizing cost-effectiveness.

[0049] Specifically, as shown in Figure 6, in the construction of placing multiple cameras 2 in store R, before construction (step SS1), server 1 generates simulation data, and based on the generated simulation data, generates proposal information. The proposal information generated by server 1 is presented to store manager P.

[0050] Here, the simulation data can be generated, for example, using design and simulation software that employs 3D rendering technology. The simulation data generation can refer to accumulated historical data, inputs to various parameters, and facial data. Furthermore, the AI ​​(artificial intelligence) used for generating the simulation data can be plugged in with various data (sensors, etc.). Camera 2 can be embedded as part of the interior of store R. Camera 2 can also be installed in various ways, such as on a light bulb (pendant light), a table candle, or artificial flowers. The communication environment (Wi-Fi, etc.) and power supply environment of the installed camera 2 can also be simulated. Finally, the system can simulate the camera being installed at the face height of store staff T and customer C. The specific method for presenting the proposal information to the store manager P is not limited. The proposal information may be presented by displaying it on an administrator terminal (not shown) operated by the store manager P. Other methods (for example, presentation by voice or presentation on paper) may also be used. Furthermore, the presentation of proposed information can be an opt-in service, provided based on a request from the store manager P.

[0051] During construction (Step SS2), store manager P performs construction based on the proposed information presented by server 1. That is, store manager P performs construction in accordance with the simulation data and design information included in the presented proposed information. Here, it is preferable that the proposed information presented is information that can be commercially proposed. For example, information regarding the interior design and layout of store R is information that architects and furniture stores can commercially propose.

[0052] After construction (Step SS3), store manager P begins operations at store R, where each of the multiple cameras 2 is installed in its designated location. By using the proposed service, store manager P can smoothly proceed with all preparations from the design of store R to the actual start of operations, and can also maximize the benefits obtained from using this service even after operations begin.

[0053] Figure 7 is an illustrative diagram showing the imaging results.

[0054] The image data captured by multiple cameras 2 is linked to information that uniquely identifies the image data (hereinafter referred to as "image ID"), information about the time the image was captured (hereinafter referred to as "timestamp information"), and information about the location where the image was captured (hereinafter referred to as "location information"), and is stored and managed in the image result DB 181 as image result information. The imaging results, combined with AI (artificial intelligence), form the framework of the digital infrastructure necessary for providing this service.

[0055] As shown in Figure 7, the imaging results include a history of communication between store staff T and customer C, between store staff T members themselves, and between customer C members themselves (hereinafter referred to as the "communication log"). From the communication log, information regarding the sensibilities of store staff T and customer C, who are the subjects of the communication (hereinafter referred to as "sensibility data"). The extracted sensibility data is used as one of the basis data for generating operation support information and educational support information.

[0056] Furthermore, the imaging results include sensing data (hereinafter referred to as "sensing logs") that covers not only store staff T and customer C, but also subjects within a 360-degree omnidirectional range around each of the multiple cameras 2 installed inside and outside the store R. From the sensing logs, physical information (hereinafter referred to as "physical data") about each subject that was the object of sensing can be extracted. The extracted physical data is used as one of the basis data for generating operation support information and educational support information.

[0057] Furthermore, since the imaging results include all records related to the provision of goods or services to customer C at store R, they can be used for traceability or as marketing material.

[0058] [System Configuration] Figure 8 is a diagram showing an overview of the configuration of the information processing system of the present invention.

[0059] The information processing system shown in Figure 1 (hereinafter referred to as "this system") consists of a server 1, cameras 2-1 to 2-n (where n is an integer value of 1 or more), a store terminal 3, a trading partner terminal 4, and a customer terminal 5, all of which are interconnected via a predetermined network N such as the Internet. This system is necessary for providing the aforementioned service. Hereafter, when it is not necessary to distinguish between cameras 2-1 through 2-n individually, they will be collectively referred to as "camera 2".

[0060] Server 1 is an information processing device managed by service provider G. Server 1 performs various processes to manage the operation of cameras 2-1 to 2-n, store terminal 3, trading partner terminal 4, and customer terminal 5. Cameras 2-1 through 2-n are cameras installed at multiple locations inside and outside store R. Each of cameras 2-1 through 2-n captures images of subjects included in the surrounding area of ​​its installation location and generates image data of those subjects. The image data captured by each of the cameras 2-1 through 2-n is linked and associated with a common key image ID, timestamp information, and location information, and is stored and managed in the image result DB 181 as image result information.

[0061] (Store terminal) As described above, store terminal 3 is an information processing device operated by store staff T. Store terminal 3 consists of a personal computer, smartphone, tablet, etc. Store terminal 3 receives input operations necessary for store staff T to use the staff support service and transmits information indicating the input content to management server 1. Store terminal 3 also displays various information transmitted from server 1, either by itself or under the control of server 1.

[0062] (Client terminal) As described above, the customer terminal 4 is an information processing device operated by customer M. The customer terminal 4 consists of a personal computer, smartphone, tablet, etc. The customer terminal 4 receives input operations necessary for customer M to use the customer support service and transmits information indicating the input content to the management server 1. In addition, the customer terminal 4 displays various information transmitted from server 1, either by itself or under the control of server 1.

[0063] (Customer terminal) As described above, customer terminal 5 is an information processing device operated by customer C. Customer terminal 5 consists of a personal computer, smartphone, tablet, etc. The customer terminal 5 receives input operations necessary for customer C to use the customer support service and transmits information indicating the content of those inputs to the management server 1. The customer terminal 5 also displays various information transmitted from server 1, either by itself or under the control of server 1.

[0064] [Hardware configuration] Next, we will describe the hardware configuration of Server 1, which performs various processes in this system. Figure 9 is a block diagram showing the hardware configuration of Server 1 in Figure 8.

[0065] Server 1 comprises a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0066] The CPU 11 executes various processes according to the program recorded in the ROM 12 or the program loaded from the storage unit 18 into the RAM 13. RAM13 also stores data and other information necessary for the CPU11 to perform various processes.

[0067] The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14. An output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input / output interface 15.

[0068] The output unit 16 consists of various liquid crystal displays and outputs various types of information. The input unit 17 is composed of various hardware components and other elements, and inputs various types of information. The memory unit 18 is composed of DRAM (Dynamic Random Access Memory) and stores various types of data. The communication unit 19 controls communication with other devices (for example, cameras 2-1 to 2-n in Figure 8, store terminal 3, trading partner terminal 4, and customer terminal 5) via a network N including the Internet.

[0069] A drive 20 is provided as needed. A removable media 30, consisting of a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately mounted on the drive 20. Programs read from the removable media 30 by the drive 20 are installed in the storage unit 18 as needed. The removable media 30 can also store various data stored in the storage unit 18, just like the storage unit 18.

[0070] Although not shown in the diagram, the store terminal 3, the trading terminal 4, and the customer terminal 5 also have the hardware configuration described above.

[0071] [Functional configuration] Next, the functional configuration of Server 1, which has the hardware configuration shown in Figure 9, will be described. Figure 10 is a functional block diagram showing an example of the functional configuration required for the various processes performed by Server 1 in Figure 9.

[0072] As shown in Figure 10, the CPU 11 of server 1 functions as follows: acquisition unit 101, operation support unit 102, education support unit 103, and proposal unit 104.

[0073] The acquisition unit 101 acquires data of an image captured and generated by each of the multiple imaging devices, which includes the field and at least a portion of its surroundings. Specifically, the acquisition unit 101 acquires data from captured images generated by cameras 2-1 to 2-n, which include at least a portion of the store R and its surroundings. As described above, the acquired image data is linked to a key image ID, timestamp information, and location information, and is stored and managed in the imaging result DB181 as imaging result information.

[0074] The operation support unit 102 performs a first process to support operations carried out by staff in the field and at least a portion of its surroundings, based on the image data acquired by the acquisition unit 101. Specifically, the operation support unit 102 generates operation support information based on the imaging result information stored in the imaging result DB 181. The operation support unit 102 also presents the generated operation support information to at least one of the administrator terminal (not shown) and the store terminal 3.

[0075] The Education Support Unit 103 performs a second process to support staff training based on the image data acquired by the Acquisition Unit 101. Specifically, the Education Support Unit 103 generates educational support information based on the imaging result information stored in the imaging result DB 181. The Education Support Unit 103 also presents the generated educational support information to at least one of the administrator terminal (not shown) and the store terminal 3.

[0076] The proposal unit 104 proposes the locations and number of imaging devices to be placed in the field and around the field, based on the data of the captured images that would be obtained if each of the multiple imaging devices were placed at each of the multiple predetermined locations in and around the field. Specifically, the proposal unit 104 generates simulation data and proposal information based on it, and presents this to at least one of the administrator terminal (not shown) and the store terminal 3.

[0077] Furthermore, as shown in Figure 10, the CPU of camera 2 (not shown) includes an imaging control unit 201 and a generation unit 202.

[0078] The imaging control unit 201 executes control to image subjects included in the surrounding area where the unit is installed. The generation unit 202 generates image data of the captured subject as a result of control by the image capture control unit 201. The generated image data is transmitted to server 1 at a predetermined time. The timing of transmission of the generated image data to server 1 is not particularly limited. It may be transmitted in real time along with the generation of the image data, or it may be transmitted every few seconds or every few minutes.

[0079] Through the cooperation of Server 1 and Camera 2, which have the functional configuration described above, each process related to the provision of this service can be executed. This enables service provider G to provide the service to each of the following: store manager P, store staff T, business partner M, and customer C.

[0080] (Common Platform Operation Model) Figure 11 shows an example of a common infrastructure operation model.

[0081] The various types of information accumulated through this service can be positioned as a resource or common infrastructure for the local community, enabling operation and development through a consortium model. Hereafter, this model will be referred to as the "common infrastructure operation model."

[0082] As shown in Figure 11, the common platform operation model is a consortium-type operation model in which the "market," "target organizations," "operating body," "collaboration between regional financial institutions and VCs (venture capital)," "collaborating businesses (investors)," "operating council," and "solution providers" cooperate with each other.

[0083] Specifically, in the common platform operation model, the "market" becomes the venue for business activities by the "target organizations, etc." This allows the "target organizations, etc." to generate revenue (sales).

[0084] In the common platform operation model, a "solution provider" is a party that provides services related to "planning, development, and maintenance" to the "target organization, etc." This allows the "solution provider" to generate revenue (development costs, etc.).

[0085] In the common platform operation model, the "operating body" provides maintenance services for standard data resources held by the "target organizations, etc." The "operating body" also creates a fund for unique data resources held by the "target organizations, etc." Furthermore, the "operating body" manages the "operating council." Through this, the "operating body" can generate revenue (returns, operating expenses, etc.).

[0086] In the common platform operation model, the "Operating Council" provides various forms of "certification" and "support" to "solution providers." This allows the "Operating Council" to generate revenue (membership fees and certification fees).

[0087] In the common platform operation model, "collaborating businesses (investors)" support the "operating body." This allows "collaborating businesses (investors)" to receive revenue (revenue share) from the "operating body (investors)." Furthermore, they can obtain revenue (fees, data usage charges, etc.) from "target organizations, etc." through "collaboration between regional financial institutions and VCs (venture capital)."

[0088] Although one embodiment of the present invention has been described above, the present invention is not limited to the embodiments described above, and any modifications, improvements, etc. that can achieve the objectives of the present invention are included in the present invention.

[0089] For example, with this service, in addition to the benefits mentioned above, you can enjoy the following benefits: In other words, it can perform spatial rendering and capture emotional information from store staff T and customer C. Furthermore, by using an infrared or night vision camera for camera 2, it can support even in dark imaging areas. Additionally, by equipping camera 2 with a temperature sensor, it becomes possible to determine temperature using, for example, far-infrared radiation. It can also determine discharge states using ultraviolet light. Moreover, by linking with AR (augmented reality) lenses, it can present proposal information using lasers or maps. Furthermore, digital support becomes possible even for those with low job proficiency.

[0090] Furthermore, in the above embodiment, for example, a restaurant, Store R, was used as an example of the target of application of this service, but this is merely an example, and the target of application of this service is not limited to restaurants or other food establishments. For example, it can be applied to any place or field, such as warehouses, factories, schools, welfare, medical, and childcare facilities, sports, entertainment, and craftsmanship fields, events, offices, civil engineering and construction sites, etc.

[0091] When this service is applied to a warehouse, by installing multiple cameras 2 at designated locations to monitor the warehouse's operational status, the movements of loading and unloading personnel, inventory status, etc., services such as the following can be realized. In other words, it enables the following: providing instructions on the location of inventory items, placing additional orders according to inventory levels, properly managing inventory losses such as expiration dates using the first-in, first-out method, optimizing the allocation of personnel and equipment, and identifying the scope of impact and issuing response instructions in the event of warehouse accidents or other problems.

[0092] When this service is applied to a factory, by installing multiple cameras 2 at designated locations and monitoring the factory's operating status, staff movements, inventory status, production status, etc., it is possible to realize services such as the following. In other words, it enables features such as checking staff work status and placement (e.g., error checking), early warnings (alerts, etc.) for processes where mistakes are likely to occur, instructions to minimize losses when errors occur, automation linked to sensors, material transfer according to operating conditions, and additional orders based on inventory levels.

[0093] When this service is applied to schools, by installing multiple cameras 2 in designated locations and monitoring the learning status of students, the content of teachers' instruction, the locations of students and teachers, etc., it becomes possible to realize services such as the following. In other words, it becomes possible to develop an educational environment that is safe for the community and parents, to efficiently procure necessary teaching materials through community cooperation (e.g., sharing, ecological methods, etc.), to support the learning of children and students (e.g., a service that notifies them if their hand movements stop), to verify whether teachers' instruction is appropriate, to develop more appropriate teaching methods, to keep track of who is where, and to prevent missing persons by guiding them with personnel.

[0094] When this service is applied to welfare, medical, and childcare facilities, by installing multiple cameras 2 at designated locations and monitoring the status of users, staff, and the locations of users and staff, it becomes possible to realize services such as the following. In other words, it can enable measures such as preventing elderly people from wandering and ensuring their safety, reducing the burden on staff, ensuring appropriate responses such as dispatching staff quickly in the event of trouble, responding immediately to requests from users (e.g., toilets, meals), recording accidents and analyzing their causes, and issuing instructions for accident prevention and warning signs.

[0095] When this service is applied to fields such as sports, performing arts, and crafts, by installing multiple cameras 2 at designated locations and monitoring the situation of learners (those being instructed), the situation of instructors, and the process leading to results, it is possible to realize services such as the following. In other words, it enables the proposal of improvement methods based on the evaluation of implementation results, analysis of the gap between instruction content and effectiveness, proposal of necessary instruction, analysis and proposal of effective learning methods (instruction methods) tailored to learners (instructors), and analysis of tacit knowledge that has become a craft, as well as proposal of teaching methods. Furthermore, it can prevent the loss of technology and create training content for technology transfer.

[0096] When this service is applied to an event, by installing multiple cameras 2 at designated locations, or by using a drone or other aircraft equipped with cameras 2 to monitor seating availability, visitor status, staff status, content status, etc., services such as the following can be realized. In other words, it enables the optimization and verification of event operation manuals, evaluation of the effectiveness of content such as halftime shows, analysis and prediction of congestion and guidance of available seats, navigation to seats tailored to attendees, guidance of service providers such as drink vendors, recording of incidents and prevention by predicting signs of trouble, analysis of congestion and securing traffic flow at the end of the event, analysis of evacuation routes in the event of a disaster, and management of customer behavior according to the level of excitement of the content (for example, when to go to the sales area).

[0097] When this service is applied to an office, by installing multiple cameras 2 in designated locations and monitoring the status of staff, installed equipment, visitors, and the space within a designated area including the office, it is possible to realize services such as the following. In other words, it can enable the analysis of work efficiency and communication, the proposal of measures to improve productivity (e.g., promoting breaks), the putting of unnecessary equipment into sleep mode, contributions to energy saving, automatic calling of staff when visitors arrive, security management measures (e.g., entry and exit control), space optimization, and proposals for methods to effectively utilize dead space.

[0098] When this service is applied to civil engineering and construction sites, for example, by erecting poles within the space and installing multiple cameras 2 at designated locations to monitor the work situation, accident prevention can be achieved.

[0099] In other words, this service can be realized by creating a network that digitizes each aspect of research and development, production, distribution, service, and community. Specifically, this service can be realized by integrating elements such as e-learning, AI speakers, multilingual menus, personal money transfers, check-in / check-out systems, ordering systems, traceability systems, high-resolution monitoring systems, communication media, and social capital credit rating systems.

[0100] Furthermore, the hardware configuration shown in Figure 9 is merely an example for achieving the objectives of the present invention and is not particularly limited.

[0101] Furthermore, the functional block diagram shown in Figure 10 is merely illustrative and not particularly limiting. In other words, it is sufficient for the information processing system to have the functionality to execute the series of processes described above as a whole, and the functional blocks used to realize this functionality are not particularly limited to the example in Figure 10.

[0102] Furthermore, the location of the functional block is not limited to Figure 10, but can be any location. Furthermore, a single functional block may consist of hardware alone, software alone, or a combination of both.

[0103] When the processing of each functional block is to be executed by software, the programs that make up that software are installed on a computer or other device from a network or storage medium. A computer may be a computer built into dedicated hardware. Alternatively, a computer may be a computer capable of performing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0104] Recording media containing such programs consist not only of removable media distributed separately from the main unit to provide programs to each user, but also of recording media provided to each user in a state where they are pre-installed in the main unit.

[0105] In this specification, the step of describing a program to be recorded on a recording medium includes not only processes that are performed chronologically in that order, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually.

[0106] Furthermore, in this specification, the term "system" refers to an overall system composed of multiple devices, means, etc.

[0107] In summary, the information processing system to which the present invention is applied only needs to have the following configuration, and various embodiments can be adopted. In other words, the information processing system to which the present invention is applied is: In an information processing system that supports a provider (e.g., store manager P) who provides goods or services to a customer (e.g., customer C) at a designated location, and an assistant to the provider (e.g., store staff T), The system comprises multiple imaging devices (e.g., camera 2) and an information processing device (e.g., server 1), The plurality of imaging devices, It has a generation means (for example, the generation unit 202 in Figure 10) that captures an image of a subject included in a 360-degree range in all directions and generates data of the captured image of the subject. The aforementioned information processing device is An acquisition means (for example, the acquisition unit 101 in Figure 10) acquires data of an image captured and generated by each of the plurality of imaging devices, which includes the field and at least a portion of the area surrounding the field. Based on the data acquired by the acquisition means, an operation support means (for example, the operation support unit 102 in Figure 10) performs a first process to support the operation performed by the assistant in the field and at least a part of the surrounding area, Based on the data acquired by the acquisition means, an educational support means (for example, the educational support unit 103 in Figure 10) performs a second process to support the education of the assistant, It is equipped with.

[0108] This allows for efficient support of assistant operations and training of assistants based on the conditions of the location (e.g., store R) and the surrounding environment.

[0109] Furthermore, based on the data of captured images (e.g., simulation data) that would be obtained if each of the multiple imaging devices were placed at each of the multiple predetermined positions in and around the field, a proposal means (e.g., proposal information) proposes the placement locations and number of the multiple imaging devices in and around the field. It can be equipped with.

[0110] This allows venue managers to efficiently receive the service while maximizing cost-effectiveness.

[0111] Furthermore, the information processing apparatus to which the present invention is applied is: In an information processing device included in an information processing system that supports a provider who provides goods or services to a customer at a designated location, and an assistant of said provider, An acquisition means for acquiring data of an image captured and generated by each of the plurality of imaging devices, which includes at least a portion of the field and its surroundings. Operation support means that, based on the data acquired by the acquisition means, performs a first process to support operations performed by the assistant in the field and at least a portion of the surrounding area, An educational support means that performs a second process to support the education of the assistant based on the data acquired by the acquisition means, It is equipped with.

[0112] This allows for efficient support of assistant operations and training of assistants, based on the situation at the site and the surrounding environment.

[0113] Furthermore, based on the data of the captured images that would be obtained if each of the multiple imaging devices were placed at each of the multiple predetermined positions in and around the said field, a proposal means proposes the placement locations and number of the multiple imaging devices in and around the said field. You can prepare for it.

[0114] This allows venue managers to efficiently receive the service while maximizing cost-effectiveness. [Explanation of Symbols]

[0115] 1: Server, 2, 2-1 to 2-n: Camera, 3: Store terminal, 4: Business partner terminal, 5: Customer terminal, 11: CPU, 12: ROM, 13: RAM, 14: Bus, 15: Input / Output interface, 16: Output unit, 17: Input unit, 18: Storage unit, 19: Communication unit, 20: Drive, 30: Removable media, 101: Acquisition unit, 102: Operation support unit, 103: Education support unit, 104: Proposal unit, 201: Imaging control unit, 202: Generation unit, 212: Lens unit, 211: Main unit, 181: Imaging result DB, G: Service provider, T: Store staff, R: Store, P: Store manager, M: Business partner, C: Customer, A1-1, A1-2, A2-1, A2-2, A3 to A7: Imaging area, SS: Each step in this service

Claims

1. In an information processing system that supports a natural person provider who provides goods or services to a customer at a designated location, and an assistant of such provider, The system comprises one or more imaging devices and an information processing device. The first or more imaging devices are It has a generation means that captures images of a subject included in a 360-degree range in all directions and generates data of the captured images of the subject. They are each installed so that they are 1.8m to 2.5m above the ground or floor. The aforementioned information processing device is An acquisition means for acquiring the data of the captured images generated by each of the one or more imaging devices as image data of the field and at least a portion of its surroundings as seen from the provider's perspective, A management means for managing each of the acquired image data (one or more) as one or more pieces of imaging result information, linked to data relating to the subject contained in the image, The system includes a support means that, among the one or more managed imaging result information, selects the information to be processed as processing target information, and sets at least one of the following as execution target processing: a first processing for supporting operations performed by the assistant in the field and at least a part of the surrounding area, and a second processing for supporting the training of the assistant, according to a pattern based on the subject identified based on the processing target information, and executes the execution target processing based on the processing target information. An information processing system equipped with the following features.

2. Based on the data of captured images that would be obtained if the one or more imaging devices were placed at one or more predetermined locations in the field and its surroundings, a proposal means proposes the locations and number of the one or more imaging devices to be placed in the field and its surroundings. The information processing system according to claim 1, further comprising:

3. In an information processing system comprising one or more imaging devices and information processing devices, which supports a natural person provider who provides goods or services to a customer at a designated location, and an assistant of said provider, The one or more imaging devices described above It has a generation means that captures images of a subject included in a 360-degree range in all directions and generates data of the captured images of the subject. In cases where each of the one or more imaging devices is installed at a height of 1.8 m to 2.5 m from the ground or floor, an acquisition means is provided to acquire the data of the captured images generated by each of the one or more imaging devices as image data of the area and at least a portion of its surroundings as seen from the provider's perspective. A management means for managing each of the acquired image data (one or more) as one or more pieces of imaging result information, linked to data relating to the subject contained in the image, The system includes a support means that, among the one or more managed imaging result information, selects the information to be processed as processing target information, and sets at least one of the following as execution target processing: a first processing for supporting operations performed by the assistant in the field and at least a part of the surrounding area, and a second processing for supporting the training of the assistant, according to a pattern based on the subject identified based on the processing target information, and executes the execution target processing based on the processing target information. An information processing device equipped with the following features.

4. Based on the data of captured images that would be obtained if the one or more imaging devices were placed at one or more predetermined locations in the field and its surroundings, a proposal means proposes the locations and number of the one or more imaging devices to be placed in the field and its surroundings. The information processing apparatus according to claim 3, comprising:

5. As a control process to be executed by a computer that controls an information processing device included in an information processing system comprising one or more imaging devices and information processing devices, which supports a natural person provider who provides goods or services to a customer at a designated location, and an assistant of said provider, The one or more imaging devices described above It has a generation means that captures images of a subject included in a 360-degree range in all directions and generates data of the captured images of the subject. In cases where each of the one or more imaging devices is installed at a height of 1.8 m to 2.5 m from the ground or floor, the acquisition step is to acquire the data of the captured images generated by each of the one or more imaging devices as image data of the area and at least a part of its surroundings as seen from the provider's perspective, A management step involves managing each of the acquired image data (one or more) as one or more pieces of imaging result information, linked to data relating to the subject contained in the image. A support step in which, among the one or more managed imaging result information, those to be processed are designated as processing target information, and according to the pattern based on the subject identified based on the processing target information, at least one of the following is set as the execution target process: a first process for supporting operations performed by the assistant in the field and at least a part of the surrounding area; and a second process for supporting the training of the assistant; and the execution target process is executed based on the processing target information. A program that executes control processes, including those mentioned above.

Images