Information processing system, and information processing method

Abstract

To give instructions for proper work in response to various situations.SOLUTION: The present invention is directed to an information processing system for presenting at least one additional work to be done next to a worker who is working in a specific space. The system has: a database for storing, in association with each other, combination of a plurality of items respectively indicating a plurality of evens which can happen in the specific space and works to be done in the specific space; an environment recognition unit for, through an image recognition processing on an image obtained by capturing the specific space, determining whether or not an event corresponding to each of the plurality of items in the specific space occurs; an additional work determination unit for determining that a work corresponding to the determined result among works stored in the database is determined as an additional work; and a display unit for presenting information relating to a schedule including the additional work.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system and an information processing method. [Background technology]

[0002] In stores such as convenience stores, the tasks performed by store clerks are becoming more complex and diverse. To enable store clerks to appropriately perform these increasingly complex and diverse tasks, methods for presenting information about tasks to store clerks have been studied (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2014 / 033979 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is room for further consideration regarding methods for instructing appropriate work depending on various situations.

[0005] Non-limiting examples of the present disclosure contribute to providing an information processing system and an information processing method that can instruct appropriate work depending on various situations. [Means for solving the problem]

[0006] An information processing system according to one embodiment of the present disclosure is an information processing system that presents at least one additional task to be performed next to a worker working in a specific space, and includes: a database that stores a correspondence between a combination of multiple items indicating each of multiple events that may occur in the specific space and the tasks to be performed in the specific space; an environment recognition unit that determines whether an event corresponding to each of the multiple items has occurred in the specific space by image recognition processing of an image taken of the specific space; an additional task determination unit that determines that a task stored in the database that corresponds to the result of the determination is the additional task; and a display unit that presents information regarding a schedule including the additional task.

[0007] An information processing method according to one embodiment of the present disclosure is an information processing method that presents at least one additional task to be performed next to a worker working in a specific space, by storing a combination of multiple items indicating each of multiple events that may occur in the specific space in correspondence with the task to be performed in the specific space, determining whether an event corresponding to each of the multiple items has occurred in the specific space through image recognition processing of an image taken of the specific space, determining that the task stored in the database that corresponds to the result of the determination is the additional task, and presenting information regarding a schedule including the additional task.

[0008] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0009] According to an embodiment of the present disclosure, it is possible to instruct appropriate work depending on various situations.

[0010] Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings. Such advantages and / or benefits may be provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of them may be provided to obtain one or more identical features. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a first example of a configuration of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating a second example of the configuration of an information processing system according to an embodiment of the present disclosure. [Figure 3] FIG. 10 is a diagram showing an example of information stored in the environment and work DB. [Figure 4] A diagram showing the case where a new environment item is added using Image Captioning [Figure 5] FIG. 10 is a diagram showing an example of a determination made by an additional work determination unit; [Figure 6] A diagram showing an example of how to handle environmental items with low occurrence frequency [Figure 7] An example of removing environmental items [Figure 8] 1 is a flowchart illustrating an example of a processing flow of an information processing system according to an embodiment of the present disclosure. [Figure 9] A diagram showing an example of a display transition [Figure 10] A diagram showing a first example of a task display. [Figure 11] Figure showing a second example of task display [Figure 12] Figure showing a third example of task display [Figure 13] A diagram showing the hardware configuration of a computer that implements the functions of each device through a program. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or redundant explanation of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0013] The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0014] (One embodiment) In convenience stores and other stores, the tasks performed by store clerks are becoming more complex and diverse. Furthermore, a shortage of store clerks is becoming a problem. To resolve this labor shortage, measures such as automating tasks using robots and unmanning stores using self-checkouts are being considered.

[0015] However, automating tasks and unmanned stores requires high implementation costs. Furthermore, the robots and self-checkout systems used to automate tasks and unmanned stores require advanced technology to handle increasingly complex and diverse tasks.

[0016] On the other hand, the increasing complexity and diversity of work makes it difficult for store clerks to decide which tasks to perform in what situations, which places a heavy burden on inexperienced store clerks in particular.

[0017] Therefore, in this embodiment, a system will be described that suggests the next necessary task to a store clerk so that even an inexperienced store clerk can appropriately determine which task to perform in what situation.

[0018] <System configuration example> FIG. 1 is a diagram showing a first example of the configuration of an information processing system 1 according to this embodiment. FIG. 2 is a diagram showing a second example of the configuration of an information processing system 1 according to this embodiment. The information processing system 1 according to this embodiment is a system that suggests the next task to be performed to a store clerk or the like. FIG. 1 shows a case where the information processing system 1 performs a process of suggesting a task, and FIG. 2 shows a case where the information processing system 1 learns a model that determines the task to be suggested.

[0019] The information processing system 1 includes an imaging unit 11, an activity suggestion unit 12, an environment and activity database (DB) 13, and a display unit .

[0020] The imaging unit 11 is installed on the ceiling or the like inside the store and captures video or still images inside the store. The imaging unit 11 is also installed outside the store and captures video or still images outside the store. The imaging unit 11 may be attached to the uniform or the like of a store clerk, or may be mounted on the clerk's glasses or the like. The imaging unit 11 attached to the uniform or the like of a store clerk captures video or still images in the range in which the clerk is facing. The imaging unit 11 mounted on the clerk's glasses or the like captures video or still images in the range of the clerk's field of vision.

[0021] Note that, although the present embodiment illustrates an example in which video or still images captured by the imaging unit 11 are input to the work suggestion unit 12, the information input to the work suggestion unit 12 is not limited to this. For example, audio collected by a microphone may be input to the work suggestion unit 12. In this case, the microphone may be installed inside or outside the store, or may be worn by a store employee. Furthermore, information acquired by a sensor other than a microphone (for example, a sensor that detects people, animals, etc., a sensor that measures temperature, or a sensor that measures weight) may also be input to the work suggestion unit 12.

[0022] The work suggestion unit 12 includes an environment recognition unit 121 , a current work recognition unit 122 , an additional work determination unit 123 , an additional work determination model 124 , a schedule creation unit 125 , and an additional work learning unit 126 .

[0023] First, referring to FIG. 1, a case where a next task is suggested to a store clerk or the like will be described.

[0024] The environment recognition unit 121 acquires images from the imaging unit 11 and recognizes the environment. For example, the environment recognition unit 121 outputs sensing results for environmental items based on the acquired images. Note that "recognition" may be replaced with other terms such as "determination," "estimation," or "decision." The environment recognition unit 121 outputs sensing results for environmental items by performing image recognition processing on the acquired images (for example, image recognition processing using natural language).

[0025] Here, environmental items refer to events that may occur in the environment and are used to determine the next task to be performed. Environmental items are expressed in natural language as events that may occur in the environment. They are stored in the environment and task DB 13. For example, when the task suggestion unit 12 suggests tasks for store clerks, the environmental items include items related to the environment inside and outside the store. For example, environmental items include items such as "Delivery truck arrives," "People waiting at the register," "Cashier is present," "Product is out of stock," and "Product displays are disorganized." These environmental items may include predetermined items or items added by the environment recognition unit 121.

[0026] The sensing results also include a determination result for an environmental item, i.e., a determination result as to whether or not an event corresponding to the environmental item has occurred. For example, if the environmental item is "delivery truck arrival," it is determined whether or not the delivery truck has arrived based on the acquired image of the outside of the store. If the delivery truck has arrived, a determination result of "YES" is output for the environmental item "delivery truck arrival." If the delivery truck has not arrived, a determination result of "NO" is output for the environmental item "delivery truck arrival." The sensing results include a determination result for each environmental item. Note that the environmental item and the sensing results may be associated with a question and an answer, respectively. Note that the determination result for the environmental item is not limited to a result of determining either "YES" or "NO" (e.g., an answer to a closed-ended question). The determination result for the environmental item may also include, for example, a result indicating the number of customers in the store, the temperature in the store, etc. (e.g., an answer to an open-ended question).

[0027] The sensing results may also include sensing results relating to new environmental items that are not included in the existing environmental items. For example, an item that is not included in the existing environmental items may be sensed and included in the sensing results.

[0028] The environment recognition unit 121 may use a pre-trained classifier (object recognition, action recognition), may have a Visual Question Answering model answer a pre-prepared question list, or may analyze explanatory text generated by an Image Captioning model. An example of the processing of the environment recognition unit 121 will be described later.

[0029] The current task recognition unit 122 acquires an image from the imaging unit 11, determines the task (action) actually performed by the store clerk, and outputs information indicating the determined task (for example, task recognition result). For example, the current task recognition unit 122 determines the task (action) actually performed by the store clerk using a machine learning model for task recognition that has been trained in advance.

[0030] The additional work determination unit 123 acquires the sensing results from the environment recognition unit 121, estimates the next required work (action), and outputs information indicating the estimated work (for example, additional work information). Information about the work to be performed in the store, which is stored in the environment and work DB 13, may be used to estimate the next required work (action). The additional work determination unit 123 may estimate that the work corresponding to the sensing results, among the works stored in the environment and work DB 13, is the next required work (action). The additional work determination unit 123 may be determined in advance based on rules, or may be trained in advance using a machine learning model. The pre-trained machine learning model is stored in the additional work determination model 124. The additional work determination unit 123 estimates the next required work for the store clerk using the machine learning model read from the additional work determination model 124.

[0031] Based on the output from the current task recognition unit 122, the schedule creation unit 125 deletes completed tasks from the task list currently being handled by the store clerk. The schedule creation unit 125 also adds the next task that is estimated to be required by the additional task determination unit 123 to the task list. Note that in the task list, each task may be sorted based on the task priority. The priority of each task may be defined separately.

[0032] Next, with reference to FIG. 2, a case where a model for determining a proposed task is learned will be described.

[0033] The information sensed by the environment recognition unit 121 may include environmental information that has not previously been registered in the environment and task DB 13. In such cases, the sensed new environmental information and the task performed by the store clerk at that time are added to the environment and task DB 13 as correct values ​​(for example, correct tasks). The environment information includes environmental items and sensing results for the environmental items. The environment and task DB 13 also stores combinations of multiple items that indicate multiple events that may occur in the store, in association with tasks to be performed in the store.

[0034] The additional task learning unit 126 generates the additional task determination model 124 by machine learning (class identification, etc.) based on the information in the environment and task DB 13. For example, the additional task learning unit 126 may update the additional task determination model 124 by machine learning (class identification, etc.) when a new environmental item is registered in the environment and task DB 13, or may update the additional task determination model 124 by machine learning (class identification, etc.) when a new task (for example, a task with a limited time period) is registered in the environment and task DB 13. Note that the additional task learning unit 126 may update the additional task determination model 124 immediately when a new environmental item or task is registered, or may update the additional task determination model 124 at a predetermined time period, such as the closing time of a store.

[0035] The task suggestion unit 12 may be configured by one or more information processing devices. For example, when the task suggestion unit 12 is configured by one information processing device, the environment recognition unit 121, current task recognition unit 122, additional task determination unit 123, additional task determination model 124, schedule creation unit 125, and additional task learning unit 126 may be included in a processing unit (e.g., a processor) of the information processing device.

[0036] The display unit 14 is provided, for example, on a terminal (e.g., a smartphone or a smartwatch) owned by the store clerk. Alternatively, the display unit 14 may be a head-mounted display worn by the store clerk. The display unit 14 displays a list of one or more tasks to be performed by the store clerk, as well as supplementary information for the tasks. The supplementary information includes the priority of each task, the basis for determining the task, etc.

[0037] The display unit 14 may be integrated with the imaging unit 11. For example, the imaging unit 11 that captures the field of view of the store clerk may be mounted on the display unit 14, such as a head-mounted display worn by the store clerk. The display unit 14 and the imaging unit 11 may be connected wirelessly or by wire, and the image captured by the imaging unit 11 may be displayed on the display unit 14.

[0038] In the present embodiment, an example is shown in which task information is notified to the store clerk by displaying it on the display unit 14, but the present disclosure is not limited to this. Task information may also be notified to the store clerk by a method other than displaying it (for example, a method using audio).

[0039] <Example of environment recognition part> Next, we will explain an example of environment recognition in the environment recognition unit 121. The environment recognition unit 121 acquires an image and outputs the sensing result.

[0040] The sensing result indicates, for example, the situation inside and outside the store. The sensing result may also be referred to as the environment. The sensing method is not particularly limited. For example, a pre-trained classifier may be used, a VQA (Visual Question Answering) model may be used, or an image captioning model may be used.

[0041] The pre-trained classifier is, for example, a classifier generated by machine learning an image captured of an event to be sensed. When a classifier is used, multiple classifiers (or detectors) that perform object identification and / or action identification may be used. For example, a truck detector that detects a truck parked outside a store from an image captured of a parking lot outside the store, a person detector that detects a person inside a store from an image captured inside the store, and a cash register detector that detects a cash register inside the store from an image captured inside the store may be prepared. Classifiers may be created individually, and the environment may be recognized by combining multiple classifiers.

[0042] A VQA model is a machine learning model that accepts input of an image and a question in natural language and outputs an answer to the question. A VQA model can be constructed, for example, by learning pairs of images and questions using machine learning. When a VQA model is used, a question for each environmental item is created in natural language, and a captured image and the created question are input to the model. For example, to check for trucks parked in a parking lot outside a store, a question such as "Has the delivery truck arrived?" is created. Similarly, to check for people waiting at the register inside a store, a question such as "Is anyone waiting at the register?" is created. The captured image and the created question are then input to the VQA model, and an answer to the question asked about the image is output.

[0043] An Image Captioning model is a machine learning model that accepts an image and generates a description of that image. For example, an Image Captioning model can be constructed by learning pairs of images and their descriptions through machine learning. When an Image Captioning model is used, a captured image is input into the Image Captioning model, and a description for the image is generated by the Image Captioning model and output from the Image Captioning model. In this case, the output description is analyzed.

[0044] For example, the description is analyzed using natural language processing, and new environmental items are extracted. For example, if words are extracted using morphological analysis and a word that has not been included in the previous environmental items is found, an environmental item such as "XX exists" can be added.

[0045] Note that because the Image Captioning model can generate a description based only on the input image, questions do not need to be used when the Image Captioning model is used. For example, if the description "A delivery truck for delivery arrived at the convenience store parking lot" is obtained through Image captioning for an image, the description and environmental items are input into the language model. The language model determines whether the input description corresponds to a YES or NO environmental item. For example, if the description "A delivery truck for delivery arrived at the convenience store parking lot" and the environmental item "Delivery truck arrived" are input, the language model determines that "Delivery truck arrived" corresponds to a YES. Note that the description output from the Image Captioning model may be a description that is weakly related to the environmental items. Therefore, to make it easier to output a description that is closely related to the environmental items, a prompt (prior knowledge in natural language) focusing on the environmental items may be used to guide the Image Captioning model to output the description that is desired.

[0046] Fig. 3 is a diagram showing an example of information stored in the environment and work DB. Fig. 3 shows, as some examples of environmental items, the following environmental items: "Delivery truck arrives," "People waiting at the register," "Cashier is present," "Product is out of stock," and "Product displays are disorganized." The determination result of YES or NO for each environmental item is associated with the work.

[0047] The environment recognition unit 121 outputs the sensing result for each environmental item. In addition, in the learning stage, the environment recognition unit 121 outputs to the environment and task DB 13 the work actually performed by the store clerk in response to the determination result for each environmental item, in association with the determination result for each environmental item.

[0048] In this way, when environmental items are determined in advance, the environment recognition unit 121 may perform classification using a classifier that has been trained in advance for each environmental item. Alternatively, the environment recognition unit 121 may use a VQA model to input an image and a question into the VQA model, thereby acquiring an answer to the question. Alternatively, the environment recognition unit 121 may fill in the environmental items using an image captioning model and natural language processing. Alternatively, the environment recognition unit 121 may create new environmental items using the image captioning model and natural language processing.

[0049] The classifier, VQA model, and image captioning model may be combined depending on the frequency and / or importance of the environmental items. For example, the question "Is there a person present?" is expected to be very common and frequently used, so pre-training a classifier may enable more accurate sensing. On the other hand, the VQA model allows for flexible responses in natural language, so it may be used to determine ambiguous numbers (e.g., numerical expressions such as "a lot") or relative positions. Image captioning may be used to automatically add items that humans tend to overlook or a large number of items.

[0050] Here, when adding a new environmental item (for example, a new question) using Image Captioning, environmental items with binary answers and environmental items with multi-valued answers can be created by processing the description with a large-scale language model. For example, if a large-scale language model is used to generate a question with a binary answer based on the description "There are two people" output by Image Captioning, the question "Are there any people?" is created, and if a question with a multi-valued answer is generated, the question "How many people are there?" is created.

[0051] Figure 4 shows a case where a new environmental item is added using Image Captioning. Figure 4 shows an example of a captured image and a description obtained for the image using Image Captioning. In the example in Figure 4, the description obtained is "People are queuing in front of the cash register inside the store, and a truck is parked behind them."

[0052] In the example of Figure 4, the environment recognition unit 121 processes the obtained description using a large-scale language model, extracts terms such as "store," "cash register," "person," and "truck" from the description, and creates environment items related to the extracted terms.

[0053] In the example of Figure 4, environmental items such as "Is there a person?" and "Is there a truck?" may be created based on the extracted terms and the instruction "Create a question to confirm whether XX exists."

[0054] The environment and task DB is assumed to be a DB in which vectors of environmental information and corresponding tasks are paired, as shown in Figure 3. For example, the environmental information is expressed in the form of giving a value of YES or NO to each environmental item. For example, the corresponding task is defined as a predetermined action to be taken by the store clerk.

[0055] The environmental information to be stored may be of other patterns. For example, information that is not defined as a binary value, such as weather and time, may be set. Also, for example, when sensing the environment using Image Captioning, not all environmental items may be mentioned. Furthermore, when new environmental items are added, there may be gaps in past data. In such cases, one or more of the following approaches may be used.

[0056] When setting information that is not defined by two values, a classifier that can handle continuous values ​​and class numbers may be adopted in the additional operation determination unit 123. For example, a classifier that can handle continuous values ​​and class numbers is a classifier such as a neural network or multiple regression analysis.

[0057] If not all environmental items are mentioned, that is, if sensing results cannot be obtained for all environmental items, sensing of at least one of the missing environmental items may be redone using VQA or an individual classifier. Alternatively, if not all environmental items are mentioned, the unmentioned items may be determined as missing by the additional work determination unit 123. Alternatively, if not all environmental items are mentioned, the unmentioned items may be determined as missing by the additional work determination unit 123, and if an additional work cannot be uniquely determined, sensing may be redone using VQA or an individual classifier.

[0058] Note that, instead of redoing sensing when there is a missing value, environmental information including the missing value may be stored. For example, environmental information may be handled as information defined by three values ​​including YES, NO, and missing value. More specifically, in a neural network, YES may be defined as 1, NO as -1, and missing value as 0.

[0059] The environmental items used when the additional work was estimated may be displayed to the user, and if the estimated additional work contains an error, the user may correct the error. An example of this will be described later.

[0060] When a new environmental item is added, the added environmental item may be treated as missing in the environmental information vector before the environmental item was added. Alternatively, when a new environmental item is added, if an image used to generate the environmental information vector before the environmental item was added is saved, new sensing may be performed.

[0061] <Variations for adding and deleting questions (environmental items)> Here, we will explain variations in adding and deleting questions (environmental items).

[0062] There are no limitations on the number and content of questions. For example, multiple questions may be created for the same item. For example, if there are people lined up in front of a cash register, a question that can be answered with two values, "Are there people lined up?", such as "Are there people lined up?", and a question that can be answered with multiple values, such as "How many people are lined up?", may be created. If there are people lined up, only one store clerk needs to handle the cash register, and there is no need to increase the number of cashiers. However, if there are many people lined up, it is possible to increase the number of cashiers and contact a store clerk to free up additional cash registers.

[0063] Additionally, multiple questions with the same content may be created for important items. By generating answers to multiple questions, the reliability of the answers can be increased. For example, if the answer to both the question "Is there a product out of stock?" and the question "Is there empty space on the shelves?" is YES, it is possible to double-check that there is a product out of stock.

[0064] The processing speed of the environment recognition unit 121 may decrease if the number of environment items becomes enormous, so the environment items may be consolidated or eliminated in post-processing.

[0065] For example, environmental items (eg, questions) that always result in the same answer or always in the opposite answer may be merged.

[0066] For example, the questions "Are there any out-of-stock items?" and "Are there any empty shelves?" have almost the same meaning, so it is expected that there will be no problem merging them into one of them. The answers to questions with almost the same meaning will almost always be the same unless there is a misidentification. Since these overlapping environmental items will almost always produce the same results, there will be no problem merging them into one of them by keeping one of the questions "Are there any out-of-stock items?" and "Are there any empty shelves?" and deleting the other. Also, for example, the questions "Are there any out-of-stock items?" and "Are there any out-of-stock items?" have opposite meanings, and unless there is a misidentification, the answer to one question will always be YES and the answer to the other will always be NO. In this way, the answers to questions with opposite meanings will always be opposite. Therefore, it is possible to merge the two questions "Are there any out-of-stock items?" and "Are there any out-of-stock items?" into one of them by keeping one of the questions and deleting the other.

[0067] Whether answers to multiple questions are a set of the same or opposite answers may be determined based on the track record of questions and answers obtained in the past. However, in this case, there is a risk that the track record itself may contain erroneous judgments. Therefore, if the track record is set as a condition for merging environmental items, that is, if all answers to multiple questions are the same or opposite, environmental items that would have been merged without erroneous judgments may be excluded from the integration due to the influence of the erroneous judgment. Therefore, the track record may be set as a condition for selecting environmental items to be merged, such as the number of times that answers to multiple questions are the same or opposite to each other (T%) or the number of times that answers to multiple questions (e.g., two questions) are opposite to each other (T%) or more (T=threshold). Note that even if seemingly unrelated questions have a correlation in their answer trends, the answers to questions may be the same or opposite even if the meanings of the questions themselves are not the same or opposite. Therefore, questions to be merged may be selected based solely on the answers, without considering the meanings of the questions.

[0068] Furthermore, by adding a determination "Does the customer have a product?" in which "bean bread" in the question sentence is replaced with the higher-level concept "product," based on the question "Does the customer have anpan?", all product determination patterns may be deleted if the customer does not have a product. Furthermore, if the questions "Does the customer have anpan?" and "Does the customer have a curry bun?" are recorded, these may be combined into a question "Does the customer have bread?" or "Does the customer have a product?", which are generalized into higher-level concepts. Note that determining the hierarchical relationship of words included in questions and selecting words that indicate higher-level concepts can be achieved by using ontology data provided externally. Since ontology data is constructed, for example, using a graph database or the like and is publicly available, a detailed description will be omitted.

[0069] <Example of additional work determination section> FIG. 5 is a diagram showing an example of a determination made by the additional work determination unit. As shown in FIG. 5, a list of certain sensing results is input to the additional work determination unit 123. The additional work determination unit 123 estimates the next work (or the next action) based on the input list of sensing results and outputs information indicating the next work. The additional work determination unit 123 may estimate the next work in advance based on a rule, or may have a machine learning model trained in advance. The additional work determination unit 123 may also use an existing classifier such as a neural network, a support vector machine, or a random forest.

[0070] <Relationship between work frequency and environmental items> Among the tasks performed by store clerks, there are tasks that are performed relatively frequently and tasks that are performed infrequently. For example, the task of "closing the register because there was no one waiting in line" is a task that is performed relatively frequently, while the task of "closing the register because (by chance) there was no one waiting in line and liquid from a product spilled" is a task that does not occur frequently.

[0071] When the conditions for closing a register include the normal case of "no one waiting in line" and the rare case of "someone spilling a liquid product," the additional work determination unit 123, which has learned by machine learning that the normal case and the rare case occur simultaneously, evaluates that "the rare case did not affect the result." In this case, the information that "the register was closed because the liquid product was spilled" is likely to be ignored as information that does not affect the determination result. Also, in this case, the relationship between "someone spilling a liquid product" and "closing the register" cannot be learned until the event occurs in which "the register was closed because there was someone waiting in line but someone had spilled a liquid product."

[0072] Therefore, when a new environmental item is observed, the work suggestion unit 12 (for example, the additional work determination unit 123 and / or the schedule creation unit 125) may issue a warning or present past cases until sufficient data is collected. When issuing a warning, the content presented to the store clerk may be different from when issuing instructions. For example, in the rare case described above, a warning such as "You may need to close the cash register because liquid from the product has spilled" may be issued. In this case, by including an explanation of the reason for issuing the warning, "Because liquid from the product has spilled," and a phrase indicating the possibility that the content of the warning is uncertain, such as "You may need to close the cash register," the store clerk is asked to make the final decision on whether or not to actually perform the work.

[0073] For the question "Is the liquid spilling from the product?", it can be determined whether the distribution of other items is the same for the YES sample and the NO sample by quantifying it using a method such as Kullback-Leibler divergence.

[0074] Fig. 6 is a diagram showing an example of how to handle environmental items with low occurrence frequencies. Fig. 6 shows examples of environmental items that exist in the feature space of the additional work determination unit 123 at each of the first to third stages. One point represents an environmental item that has been sensed once.

[0075] In the first stage of Figure 6, the environmental items for which the additional work is determined to be "closing the register," the environmental items for which the additional work is determined to be "stock replenishing goods," and the environmental items for which the additional work is determined to be "receiving trucks" are all present together in the feature space.

[0076] The second stage in Figure 6 shows the state in which a new environmental item, a liquid spill, appears. Focusing on the horizontal and vertical directions of the feature space, it appears that the additional task of "closing the register" occurs when a feature similar to the existing "closing the register" appears. On the other hand, focusing on the vertical direction of the feature space, although "spilling liquid" may be the cause of this "closing the register," the small number of samples makes it difficult to say with certainty that "spilling liquid" is indeed the cause of this "closing the register." In other words, this "closing the register" occurred because the same feature as the existing "closing the register" appeared in the environment, and the "spilling liquid" event may have simply occurred by chance at the timing of the "closing the register." It is unclear whether the new environmental item, a liquid spill, is related to the decision to determine the additional task as "closing the register." In this case, a warning such as "You may need to close the register because liquid is spilling from a product" may be issued.

[0077] The third stage in Figure 6 shows a state in which the number of instances (e.g., samples) of the environmental item "liquid spilling" has increased. Focusing on the horizontal and vertical directions of the feature space reveals that, despite the presence of features similar to "truck reception" and "product replenishment," there are instances in which "cash register closing" has occurred. On the other hand, focusing on the vertical direction of the feature space reveals that "cash register closing" is always performed whenever the environmental item "liquid spilling" occurs. This indicates that the environmental item "liquid spilling" is closely related to the additional task of "cash register closing." Therefore, when the environmental item "liquid spilling" is subsequently detected, the instruction "Please close the cash register" is issued, rather than a warning such as "There is liquid in the product, so you may need to close the cash register." Note that the content of the warning and instruction is not limited to this. Other output may be used as long as the output format for the warning and the display is at least different.

[0078] In the example of FIG. 6, as described above, if the environmental item "liquid is spilling" occurs infrequently (for example, if the number of occurrences is less than a predetermined number), a warning such as "You may need to close the register because liquid from a product is spilling" is output. Also, as described above, if the environmental item "liquid is spilling" occurs frequently (for example, if the number of occurrences is greater than or equal to a predetermined number), the instruction "Please close the register" is issued. In this way, the output mode is changed based on the number of occurrences of the environmental item "liquid is spilling" that is recognized almost simultaneously with the task of "closing the register."

[0079] <Example of master-subordinate relationship of environment items> When environmental items are automatically added, there is a possibility that meaningless and / or redundant environmental items may be added. For example, if an environmental item that is considered to have no relation to the store clerk's work, such as a customer's car license plate number, appears, that environmental item may be deleted. Also, if environmental items that refer to the same content, such as "There are people waiting to check out" and "There are people waiting to pay," appear, at least one of those environmental items may be deleted.

[0080] Furthermore, treating environmental items equally can be inefficient. For example, in a case where environmental item A indicates "out of stock" and environmental item B indicates "products have been delivered," if environmental item A is YES, a decision is made to replenish the products regardless of the result of environmental item B. On the other hand, if environmental item A is NO, a decision is made to replenish the products depending on whether environmental item B is YES or NO. For example, even if there is no shortage, if the product is running low, a decision is made to replenish the products. In such a case, a master-slave relationship exists between environmental items A and B. Of these two environmental items, environmental item A is the master and environmental item B is the slave. When such a master-slave relationship exists, it is inefficient to constantly check questions related to all environmental items. Therefore, it is possible to check questions related to the master environmental item and check the slave environmental items depending on the results of the master environmental item.

[0081] For example, environmental items that always occur in the same combination may be reduced. By reducing environmental items that always occur in the same combination, meaningless environmental items can be reduced.

[0082] Alternatively, environmental items may be reduced by a statistical method, for example, by performing principal component analysis on the environmental items and reducing dimensions that have a small contribution to any principal component.

[0083] By applying the above method, when a specific environmental item X is fixed to YES (or NO), an environmental item Y that always has the same YES (or NO) can be estimated. In this case, environmental item Y is an environmental item with no information content. Such environmental items with no information content can be estimated and used to identify superior-subordinate relationships.

[0084] The information processing system 1 in this embodiment is assumed to continuously receive images from store clerks, etc. Furthermore, there is a possibility that a store clerk may perform multiple tasks at the same time.

[0085] For example, when a truck arrives, loading work is required, and when a product runs out on the shelves, product replenishment is required. The need for loading work and the need for product replenishment may occur simultaneously. Furthermore, even after the loading work is completed, the truck may continue to be sensed as an environmental item until it leaves the store parking lot. In this way, the sensing results of environmental items at a given moment may include environmental items from past tasks that occurred at the same time. By appropriately removing the influence of environmental items from past tasks, the next task can be more appropriately determined.

[0086] For example, when the first task is completed, the environmental factors that affected the first task may be removed to determine the next additional task. For example, the environmental factors may be removed by using a method such as random forest that quantifies the extent to which each environmental factor was taken into consideration in determining the task content. Note that the completion of the task may be reported manually by the store clerk, or may be reported automatically by recognizing the task performed by the store clerk through action recognition of a first-person perspective image of the store clerk.

[0087] FIG. 7 is a diagram showing an example of removing an environmental item. FIG. 7 shows an example of an environmental information vector and an operation corresponding to the environmental information vector. In the example of FIG. 7, two operations, "receiving merchandise" and "operating the cash register," are associated with each other. In the example of FIG. 7, weights are set for each environmental item for each of these two operations.

[0088] In the example of Figure 7, the weights of the two environmental items "There are people waiting in line at the register" and "There is a cashier" for "Cashier service" are larger than that for "Accepting goods." In the example of Figure 7, when "Cashier service" is completed, the two environmental items that had a large weight for "Cashier service" are removed.

[0089] In this way, if an environmental item has a different weight between multiple tasks, it will affect the determination of whether the task with the larger weight is an additional task, but will not affect the determination of whether the task with the smaller weight is an additional task. Therefore, when the task with the larger weight is completed, the corresponding environmental item may be deleted.

[0090] <Processing flow> Fig. 8 is a flowchart showing an example of the processing flow of the information processing system 1 in this embodiment. In the flow shown in Fig. 8, the imaging unit 11 captures a still image or a video, and sends the captured still image or video to subsequent processing (step 01 (ST01)). For example, the captured still image or video is sent to the current task recognition unit 122 and / or the environment recognition unit 121.

[0091] The current task recognition unit 122 processes the image using a task recognition model that has been trained in advance, and recognizes the task that the store clerk is currently performing (ST02).

[0092] The environment recognition unit 121 performs image recognition processing on the current image to recognize the environment (ST03). For example, the environment recognition unit 121 outputs sensing results for existing environment items. Also, for example, the environment recognition unit 121 extracts environment items that are different from the existing environment items as new environment items.

[0093] The environment recognition unit 121 determines whether or not there is a new environment item (ST04).

[0094] If a new environmental item exists (YES in ST04), the environment recognition unit 121 registers the new environmental item in the environment and task DB 13 (ST07). In this case, the current task recognition unit 122 may register the result of recognizing the current task as a task performed by the store clerk.

[0095] If there are no new environmental items (NO in ST04), or after registering new environmental items in the environment and work DB 13 (after ST07), the additional work determination unit 123 determines the additional work using an additional work determination model using the environment recognized in ST03 (ST05).

[0096] The schedule creation unit 125 creates a schedule by determining priorities for the work performed up to that point and the estimated additional work (ST06). The created schedule is notified to the store clerk. For example, if the highest priority task changes, the display may be changed to alert the store clerk. Various procedures for determining priorities are possible. For example, prioritizing work that has occurred frequently in the past or work that has a large impact on store sales may be prioritized. Priorities may also be determined based on the attributes of the people or equipment involved in the work. For example, prioritizing work involving people or equipment other than store clerks, such as customers or trucks, may be prioritized. Priorities may also be manually set or changed at the discretion of the store clerk or manager.

[0097] <Display example> Next, a display example of the display unit 14 as a user interface (UI) will be described. The display unit 14 displays a schedule output from the work suggestion unit 12. Furthermore, when an image corresponding to the schedule is acquired from the work suggestion unit 12, the image may be displayed on the display unit 14. Furthermore, a message regarding the environment obtained by image recognition processing of the acquired image may be displayed. Note that when a terminal (e.g., a smartphone) having an imaging unit 11 and a display unit 14 is used, the image displayed on the display unit 14 may be an image captured by the imaging unit 11.

[0098] Fig. 9 is a diagram showing an example of a transition of the display, in which four display examples (a) to (d) are shown.

[0099] Example (a) of FIG. 9 shows an example of a top screen. As shown in this example, the display of the display unit 14 includes an area where an image is displayed, an area where a message is displayed, and an area where a task is displayed. In the case of example (a) of FIG. 9, since no image has been selected, text information prompting the user to select an image is displayed in the area where the image is displayed. In the example shown below, information is displayed in each of the three areas, similar to example (a) of FIG. 9.

[0100] Example (b) in FIG. 9 shows an example of a state in which a certain image is displayed. In example (b) in FIG. 9, an image of a delivery truck parked in a parking lot outside a store is displayed. Here, the displayed image may be an image input to the work suggestion unit 12. The work suggestion unit 12 recognizes environmental items in this input image and suggests the next work that needs to be done.

[0101] Example (c) in Figure 9 shows the display after example (b) in Figure 9. In example (c) in Figure 9, image recognition processing of the input image recognizes that a delivery truck is parked in the parking lot, and a message stating "Accept truck from parking lot" is displayed. In addition, a new task, "Accept truck," is displayed as an additional task.

[0102] Example (d) in FIG. 9 shows a display subsequent to example (c) in FIG. 9. For example, example (d) switches from an image captured of a delivery truck parked in a parking lot outside the store to an image captured of a person waiting in front of a cash register inside the store. Here, the displayed image may be an image input to the work suggestion unit 12. The work suggestion unit 12 recognizes environmental items from this input image and suggests the next work that needs to be done. In the case of example (d), it is recognized that there is a person waiting in front of a cash register inside the store, and a message saying "There is someone waiting in front of the cash register" is displayed. In addition, a new task called "servicing the cash register" is displayed as an additional work.

[0103] As shown in Fig. 9, tasks are added according to various environments. Note that when a store clerk performs an operation, the task may be deleted. An example of deleting a task will be described with reference to Fig. 10.

[0104] FIG. 10 is a diagram showing a first example of a display of tasks.

[0105] The example (c) in Fig. 10 is similar to the example (c) in Fig. 9. That is, a message is displayed to the effect that a truck is to be accepted from the parking lot, and a task of "accept truck" is displayed.

[0106] Example (d) in FIG. 10 is similar to example (d) in FIG. 9. That is, a message is displayed indicating that there are people waiting at the register, and a task called "Accept truck" and a task called "Serve register" are displayed. In this example, the task called "Serve register" has a higher priority than the task called "Accept truck," so the task called "Serve register" is displayed above the task called "Accept truck."

[0107] In example (d), the store clerk performs the task "operate the cash register" because the task "operate the cash register" is displayed above the task "accept the truck."

[0108] Example (e) in Figure 10 shows a display example when the store clerk completes the task "operating the cash register" after example (d) is displayed. In example (e), an image is displayed in which there are no more people waiting at the cash register in the store. In example (e), it is detected that the store clerk has completed the task "operating the cash register," and the task "operating the cash register" is removed from the display.

[0109] Fig. 11 is a diagram showing a second example of a task display, in which the store clerk performs an action to confirm the reason for generating the task in a display similar to that of example (d) of Fig. 10.

[0110] Example (d) in Fig. 11 is similar to example (d) in Fig. 10, and displays an image of people waiting at the register inside a store. A message is also displayed indicating that there are people waiting at the register, and the tasks "Accept truck" and "Handle register" are displayed.

[0111] In the display of example (d), the store clerk operates the operation unit (for example, a touch panel) to select the task "operate the register" in order to confirm the reason why the task "operate the register" was generated.

[0112] Example (f) in Fig. 11 shows a display example when the task "Cashier" is selected from the display of example (d). Example (f) displays that the reason (basis) for generating the task "Cashier" is that "There is someone at the register," "There is no store clerk at the register," or "There is someone carrying a product."

[0113] In the example of Fig. 11, there are two tasks, but the reasons for generating each of the two tasks may be different. For example, if two tasks, a first task and a second task, are generated based on the sensing results of 10 environmental items, there may be six reasons for generating the first task and four reasons for generating the second task among the 10 environmental items. For example, by using a classifier that calculates relatively more important environmental items as weights, only some of the environmental items may be displayed.

[0114] If the estimated additional work contains an error, the error may be manually corrected by a store clerk. An example of correcting an error by a store clerk will be described with reference to FIG.

[0115] Fig. 12 is a diagram showing a third example of the display of tasks. Fig. 12 shows four display examples, example (g) to example (j).

[0116] In the example (g) of Figure 12, an image of a truck transporting merchandise for the store parked in the store's parking lot and an image of people waiting at the register inside the store are displayed. A message is displayed indicating that there are people waiting at the register, and three tasks are displayed: "Accept the truck," "Handle the register," and "Replenish merchandise."

[0117] As explained in the example of Figure 11, in the display of example (g), the store clerk operates the operation unit (e.g., a touch panel) to select the task "stock inventory" to confirm the reason why the task "stock inventory" was created.

[0118] Example (h) in Fig. 12 shows a display example when the task "stock replenishment" is selected in the display of example (g). Example (g) displays the reasons (basis) for generating the task "stock replenishment": "There is a shortage of products on the shelves," "There is someone at the register," "There is a person holding a product," and "There is no store clerk at the register."

[0119] When the store clerk visually confirms that there are no missing items on the shelves, he / she determines that the information displayed in example (h) that indicates "there are missing items on the shelves" is incorrect. In this case, the store clerk can report that the information displayed is incorrect. For example, the store clerk operates the operation unit to select the information displayed as "there are missing items on the shelves."

[0120] Example (i) in Fig. 12 shows an example of a display when the reason "there is a shortage of product on the shelf" is selected in the display of example (h). Example (i) displays an option to "report as an error" to report that the selected reason (for example, the reason "there is a shortage of product on the shelf") is an error. If the store clerk operates the operation unit to select the option to "report as an error," it is reported that the reason "there is a shortage of product on the shelf" is incorrect information.

[0121] Upon receiving the report, the additional task determination unit 125 re-estimates the tasks and creates a new task list. The created task list is then again displayed on the display unit 14. The corrected information is stored in the environment and task DB 13.

[0122] Example (j) in Figure 12 shows an example of how the task list is displayed after correction. In example (g), the task "stock replenishment" was displayed, but in example (j), the task "stock replenishment" has been deleted by updating the task list.

[0123] As described above, according to this embodiment, by recognizing the environment based on an image and determining additional work depending on the environment, it is possible to instruct appropriate work depending on various situations. For example, even if a situation occurs that the store clerk has not experienced, it is possible to instruct appropriate work depending on the situation. This not only improves the work efficiency of the relevant store clerk, but also improves the productivity of the entire store.

[0124] Furthermore, for example, in this embodiment, existing environmental items are obtained as sensing results (recognition results) using a query-type method such as a VQA model, while new environmental items not included in the existing environmental items are extracted using a Vision Language Model. For example, as described above, a description may be created for an image using Image Captioning, and new environmental items may be extracted by comparing the created description with existing environmental items. Furthermore, the comparison between existing environmental items and the description may be performed using a model such as a Large Language Model (LLM) or simple morphological analysis. In this way, by extracting new environmental items from existing environmental items, even if there are omissions in the existing environmental items, the appearance of new environmental items can be detected using a general-purpose language model, thereby preventing omissions.

[0125] (Variation) In the above-described embodiment, the tasks performed by the store clerk are adopted as the correct answer value. However, the tasks adopted as the correct answer value may be limited to tasks performed by a limited number of store clerks, such as the store manager or experienced store clerks. These store clerks are more likely to perform appropriate tasks than less experienced store clerks, even in the same environment. Therefore, by limiting the tasks to be added to the environment and task DB 13 to more appropriate tasks, it is possible to reduce the possibility of suggesting incorrect tasks, inappropriate tasks, tasks that should not be prioritized, and inefficient tasks.

[0126] In the above-described embodiment, work in a store such as a convenience store has been described as an example. However, the concept of this embodiment can be applied to any work performed in a space other than a store. For example, an office, a theme park, a factory, a distribution center, etc. can be considered as an example of a space.

[0127] The notation "... part" in the above-described embodiments may be replaced with other notations such as "... circuitry", "... assembly", "... device", "... unit", or "... module".

[0128] The embodiments of the present disclosure have been described above in detail with reference to the drawings, but the functions of the above-described product management system 1 can be realized by a computer program.

[0129] 13 is a diagram showing the hardware configuration of a computer that realizes the functions of each device by a program. This computer 1100 includes an input device 1101 such as a keyboard, a mouse, or a touchpad, an output device 1102 such as a display or a speaker, a central processing unit (CPU) 1103, a graphics processing unit (GPU) 1104, a read-only memory (ROM) 1105, a random access memory (RAM) 1106, a storage device 1107 such as a hard disk drive or a solid-state drive (SSD), a reading device 1108 that reads information from a recording medium such as a digital versatile disk read-only memory (DVD-ROM) or a universal serial bus (USB) memory, and a transmitting / receiving device 1109 that communicates via a network, and each unit is connected by a bus 1110.

[0130] The reading device 1108 then reads the program for realizing the functions of each of the above-mentioned devices from a recording medium on which the program is recorded, and stores the program in the storage device 1107. Alternatively, the transmitting / receiving device 1109 communicates with a server device connected to the network, and stores the program for realizing the functions of each of the above-mentioned devices downloaded from the server device in the storage device 1107.

[0131] The CPU 1103 then copies the program stored in the storage device 1107 to the RAM 1106, and sequentially reads out and executes instructions contained in the program from the RAM 1106, thereby realizing the functions of the above-mentioned devices.

[0132] The present disclosure can be realized in software, hardware, or software in conjunction with hardware.

[0133] Each functional block used in the description of the above embodiments may be partially or entirely realized as an LSI, which is an integrated circuit, and each process described in the above embodiments may be partially or entirely controlled by a single LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of a single chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may be called an IC, system LSI, super LSI, or ultra LSI.

[0134] The integrated circuit method is not limited to LSI, but may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Also, a field programmable gate array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI, may be used. The present disclosure may be realized as digital processing or analog processing.

[0135] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology may be used to integrate functional blocks. The application of biotechnology, etc. is also a possibility.

[0136] The present disclosure may be implemented in any type of apparatus, device, or system (collectively referred to as a communications apparatus) that has a communications function. The communications apparatus may include a wireless transceiver and processing / control circuitry. The wireless transceiver may include a receiver and a transmitter, or both functions. The wireless transceiver (transmitter and receiver) may include a radio frequency (RF) module and one or more antennas. The RF module may include an amplifier, an RF modulator / demodulator, or the like. Non-limiting examples of communication devices include telephones (e.g., cell phones, smartphones), tablets, personal computers (PCs) (e.g., laptops, desktops, notebooks), cameras (e.g., digital still / video cameras), digital players (e.g., digital audio / video players), wearable devices (e.g., wearable cameras, smartwatches, tracking devices), game consoles, digital book readers, telehealth / telemedicine devices, communication-enabled vehicles or mobile transportation (e.g., cars, airplanes, ships), and combinations of the above devices.

[0137] Communications equipment is not limited to portable or mobile equipment, but also includes non-portable or fixed equipment, devices, and systems of any kind, such as smart home devices (such as appliances, lighting equipment, smart meters or metering devices, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0138] Furthermore, in recent years, in the field of IoT (Internet of Things) technology, CPS (Cyber ​​Physical Systems) has been attracting attention as a new concept that creates new added value by linking information between physical space and cyberspace. This CPS concept can also be adopted in the above-mentioned embodiments.

[0139] That is, as a basic configuration of a CPS, for example, an edge server located in physical space and a cloud server located in cyberspace can be connected via a network, and processing can be distributed and performed by processors installed on both servers. Here, it is preferable that each piece of processing data generated on the edge server or cloud server is generated on a standardized platform, and the use of such a standardized platform can improve the efficiency of building a system that includes a variety of sensor groups and IoT application software.

[0140] In the above-described embodiments, for example, the edge server may be located in a store and perform product recognition processing and product misrecognition risk assessment processing. The cloud server may perform model learning using data received from the edge server via a network. Alternatively, for example, the edge server may be located in a store and perform product recognition processing, and the cloud server may perform product misrecognition risk assessment processing using data received from the edge server via a network.

[0141] Communications include data communications via cellular systems, wireless LAN systems, communications satellite systems, etc., as well as data communications via combinations of these.

[0142] A communications apparatus also includes devices such as controllers and sensors connected or coupled to a communications device that performs the communications functions described in this disclosure, such as controllers and sensors that generate control and data signals used by the communications device to perform the communications functions of the communications apparatus.

[0143] The communication apparatus also includes infrastructure facilities, such as base stations, access points, and any other apparatus, device, or system that communicates with or controls the various apparatuses listed above, but are not limited to these.

[0144] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components of the above-described embodiments may be combined in any manner without departing from the spirit of the disclosure.

[0145] Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and alterations of the specific examples exemplified above.

[0146] An information processing system according to one embodiment of the present disclosure is an information processing system that presents at least one additional task to be performed next to a worker working in a specific space, and includes: a database that stores a correspondence between a combination of multiple items indicating each of multiple events that may occur in the specific space and the tasks to be performed in the specific space; an environment recognition unit that determines whether an event corresponding to each of the multiple items has occurred in the specific space by image recognition processing of an image taken of the specific space; an additional task determination unit that determines that a task stored in the database that corresponds to the result of the determination is the additional task; and a display unit that presents information regarding a schedule including the additional task.

[0147] In one embodiment of the present disclosure, the item is information indicating the event in natural language, and the environment recognition unit determines whether the event has occurred by performing image recognition processing using the natural language indicating the item.

[0148] In one embodiment of the present disclosure, the environment recognition unit generates a question using natural language contained in the item, and obtains an answer to the question for the image using a pre-trained model, thereby determining whether an event corresponding to each of the multiple items has occurred.

[0149] In one embodiment of the present disclosure, the information processing system consolidates items in the database that correspond to multiple questions to which the answers are substantially identical or items that correspond to multiple questions to which the answers are substantially opposite.

[0150] In one embodiment of the present disclosure, the environment recognition unit acquires an explanation of the event contained in the image using a pre-trained model, and determines whether an event corresponding to each of the multiple items has occurred by natural language processing using the question text and the natural language contained in the items.

[0151] In one embodiment of the present disclosure, if the description includes natural language indicating an event that is not stored in the database, the environment recognition unit adds an item including the natural language indicating the event to the database.

[0152] In one embodiment of the present disclosure, the information processing system integrates a plurality of items in the database based on the natural language meanings contained in the plurality of items.

[0153] In one embodiment of the present disclosure, when the natural language meanings contained in multiple items are substantially the same, the information processing system merges the multiple items into one of the items having substantially the same natural language meaning.

[0154] In one embodiment of the present disclosure, when the natural language included in multiple items is included in approximately the same superordinate concept, the information processing system integrates the multiple items into an item using the natural language that indicates the superordinate concept.

[0155] In one embodiment of the present disclosure, the environment recognition unit further detects work being performed in the specific space, and when the work is designated as the correct work, the information processing system associates the work with a combination of items indicating multiple events that were recognized approximately simultaneously with the work and adds the associated work to the database.

[0156] In an embodiment of the present disclosure, the display unit changes a manner in which information regarding a schedule including the additional work is presented based on the number of occurrences of an event that was recognized to occur substantially simultaneously with the work.

[0157] In one embodiment of the present disclosure, when there are multiple additional tasks depending on the environment and the first additional task among the multiple additional tasks is performed, the additional task determination unit again determines the additional task based on items excluding the item corresponding to the first additional task.

[0158] An information processing method according to one embodiment of the present disclosure is an information processing method that presents at least one additional task to be performed next to a worker working in a specific space, by storing a combination of multiple items indicating each of multiple events that may occur in the specific space in correspondence with the task to be performed in the specific space, determining whether an event corresponding to each of the multiple items has occurred in the specific space through image recognition processing of an image taken of the specific space, determining that the task stored in the database that corresponds to the result of the determination is the additional task, and presenting information regarding a schedule including the additional task. [Industrial Applicability]

[0159] An embodiment of the present disclosure is useful for a system that suggests tasks to store staff. [Explanation of symbols]

[0160] 1. Information Processing Systems 11 Imaging unit 12 Work Proposal Department 13 Environment and Work DB 14 Display section 121 Environmental Awareness Department 122 Current Work Recognition Unit 123 Additional Work Judgment Department 124 Additional Work Judgment Model 125 Schedule Creation Department 126 Additional Work Learning Department

Claims

1. An information processing system that presents at least one additional task to be performed next to a worker performing work in a specific space, a database that stores a combination of a plurality of items indicating each of a plurality of events that may occur in the specific space and an operation to be performed in the specific space in association with each other; an environment recognition unit that determines whether or not an event corresponding to each of the plurality of items has occurred in the specific space by performing image recognition processing on an image captured of the specific space; an additional work determination unit that determines, among the works stored in the database, a work corresponding to the result of the determination as the additional work; a display unit that displays information about a schedule including the additional work; An information processing system comprising:

2. the item is information indicating the event in natural language, the environment recognition unit determines whether the event has occurred by performing image recognition processing using natural language indicating the item; The information processing system according to claim 1 .

3. the environment recognition unit generates a question using a natural language included in the item, and acquires an answer to the question for the image using a pre-trained model, thereby determining whether or not an event corresponding to each of the plurality of items has occurred; The information processing system according to claim 2 .

4. The information processing system integrates, in the database, items corresponding to a plurality of questions to which the answers are substantially identical or items corresponding to a plurality of questions to which the answers are substantially opposite. The information processing system according to claim 3 .

5. the environment recognition unit acquires an explanation of an event included in the image using a pre-trained model, and determines whether or not an event corresponding to each of the plurality of items has occurred by natural language processing using the question sentence and natural language included in the items. The information processing system according to claim 2 .

6. when the description includes a natural language indicating an event not stored in the database, the environment recognition unit adds an item including the natural language indicating the event to the database. The information processing system according to claim 5 .

7. the information processing system integrates the plurality of items in the database based on the natural language meanings contained in the plurality of items; The information processing system according to claim 2 .

8. When the natural language meanings contained in a plurality of items are substantially the same, the information processing system integrates the plurality of items into one of the items having substantially the same natural language meaning. The information processing system according to claim 5 .

9. When natural language included in a plurality of items is included in substantially the same superordinate concept, the information processing system integrates the plurality of items into an item using the natural language that indicates the superordinate concept. The information processing system according to claim 5 .

10. The environment recognition unit further detects an operation being performed in the specific space, When the task is designated as a correct task, the information processing system associates the task with a combination of items indicating a plurality of events that were recognized approximately simultaneously with the task, and adds the associated task to the database. The information processing system according to claim 2 .

11. the display unit changes a manner in which information about the schedule including the additional work is presented based on the number of occurrences of an event that was recognized to occur substantially simultaneously with the work. The information processing system according to claim 8 .

12. When there are a plurality of additional tasks corresponding to the environment and a first additional task among the plurality of additional tasks has been executed, the additional task determination unit determines the additional task again according to items excluding the item corresponding to the first additional task. The information processing system according to claim 2 .

13. An information processing method for presenting at least one additional task to be performed next to a worker performing work in a specific space, comprising: a combination of a plurality of items indicating each of a plurality of events that may occur in the specific space and an operation to be performed in the specific space are stored in association with each other; determining whether or not an event corresponding to each of the plurality of items has occurred in the specific space by performing image recognition processing on an image captured of the specific space; Among the tasks stored in the database, a task corresponding to the result of the determination is determined to be the additional task; providing information regarding a schedule including said additional work; Information processing methods.

Images