Information processing system and method for controlling the same, and program
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-02-07
- Publication Date
- 2026-03-25
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing system for detecting a detection target from an image, and a control method and program thereof. [Background technology]
[0002] 2. Description of the Related Art Conventionally, there is known a technique for detecting a detection object from a captured image by image recognition and displaying the detection result.
[0003] Patent Document 1 proposes displaying slip data and an inspection target image, and judging whether or not the inspection target product included in the inspection target image is included in the slip data. If the product is not included in the slip data, the word "WRONG" is displayed in the position of the product in the inspection target image to indicate that the product is different from the product included in the slip data. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2017-88345 A DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]
[0005] When capturing images for detecting objects from captured images, it is necessary to reset the detection results of the previous set so as not to mistake the detection results of the previous set for the detection results of the next set. When the imaging means is provided on a wearable device such as a head-mounted device for a worker, the worker may have both hands occupied while holding other devices or performing other tasks, and performing a reset operation using his / her hands may hinder the work. In addition, a method of resetting the detection results of the previous set is conceivable when the detection results of the previous set are different from those of the next set. However, if the images before and after are the same, the division of the images cannot be determined and accurate resetting cannot be performed. In addition, a method of intentionally inserting a division into the image is conceivable. For example, it is conceivable to reset the detection results when a state in which the detection object is not reflected continues for a certain period of time by having the user intentionally look at a place other than the detection object. However, if the detection object cannot be detected with high accuracy, it is conceivable that the detection results will be reset even if the detection object is actually reflected in the image but cannot be reliably detected and this state continues for a certain period of time (i.e., when the user is not intentionally looking at a place other than the detection object). This can cause the device to be reset even if the same object is captured, making it difficult to compare the results detected through a wearable device such as a head-mounted device with the actual object. Patent Document 1 does not take the above problem into consideration.
[0006] In view of the above problem, the present invention aims to provide a mechanism that can more appropriately update display information related to detection results while preventing interruptions to work. [Means for solving the problem]
[0007] a first acquisition means for continuously acquiring images captured by the imaging means; A second acquisition means for acquiring result information of a detection process for detecting a detection target from the image acquired by the first acquisition means; a first piece of information based on the type of the detection target detected in the detection process, the first piece of information being based on the result information acquired by the second acquisition means at a first time point, is continuously displayed until a specific condition is satisfied, including that an article including the detection target is not detected for a predetermined period of time, and thereafter, the first piece of information is hidden in response to the specific condition being satisfied; Displaying second information based on the type of the detection object detected in the detection process, the detection object being based on result information acquired by the second acquisition means at a second time point after the specific condition is satisfied. A display control means for controlling the display in such a manner as described above; having The specific condition is a condition based on the presence or absence of detection of the article, and is not based on the type of the article. Effect of the Invention
[0008] It is possible to provide a mechanism that enables more optimal updating of display information related to detection results while preventing interruptions to work. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 illustrates an example of a configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of various devices. [Diagram 3] 13 is a flowchart showing an outline of processing by the client terminal 100. [Figure 4] 13 is a flowchart showing a screen display on the client terminal 100. [Diagram 5] FIG. 2 illustrates an example of the configuration of a server 110. [Figure 6] FIG. 13 is a diagram illustrating an example of a screen displayed after an application is started. [Figure 7] 13A and 13B are diagrams illustrating an example of a screen transition when a menu determination result is obtained after a list is obtained. [Figure 8]13A and 13B are diagrams illustrating an example of a screen transition when a list is acquired after a menu determination result is acquired. [Figure 9] FIG. 13 is a diagram showing an example of a screen displayed on the wearable terminal 120. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.
[0011] First, an example of the configuration of an information processing system according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 shows a system configuration for detecting a detection target from an image and displaying the result on a client terminal 100 or a wearable terminal 120.
[0012] This embodiment can be utilized, for example, for a user to check whether the meal on the tray is correct. For example, in a hospital, meals are set for each patient, and great care is taken to prevent mistakes such as incorrect meals being served to patients. A menu list is prepared in advance for each patient, and the meal that is actually served is compared with the list to manually determine whether there are any problems.
[0013] The information processing system of the present invention is configured by connecting a client terminal 100, a server 110, a wearable terminal 120, and a menu DB 130 via a network 101 so as to be able to communicate with each other. do.
[0014] The serving tray 140 holds at least one or more meals, which are to be the set to be detected, and a paper on which a QR code (registered trademark) having information on a menu list is printed. An image including the meals on the serving tray 140 and the QR code is captured by the imaging unit 121 of the wearable terminal 120 worn by the user. The imaging unit 121 is a camera unit that captures an image in the direction (the direction the face is facing, the field of view) of the user wearing the wearable terminal 120. The display 122 is a display unit that is disposed in the eyeglass lens portion in the line of sight of the user wearing the wearable terminal 120, and the contents displayed on the display 122 are visually recognized by the user together with visible light (optical image) from a real object that passes through the display 122. In other words, various display objects described later are controlled to be displayed on the display 122, which transmits the optical image of the subject captured by the imaging unit 121, so that the optical image can be visually recognized at the same time. The contents to be displayed may be projected and displayed on the display 122 (the lens portion of the eyeglasses). The wearable device 120 is a device that can be attached to a part of the body, such as smart glasses. Smart glasses are glasses-type devices with a display that can superimpose an image onto the real field of view.
[0015] The client terminal 100 is an information processing device such as a personal computer or a smartphone, and is communicably connected to the wearable terminal 120. An application program (hereinafter, referred to as an app) for controlling the wearable terminal 120 is installed in the client terminal 100. The client terminal 100 is communicably connected to the wearable terminal 120, the server 110, and the menu DB 130. The client terminal 100 transmits an image (image captured by the imaging unit 121) transmitted from the wearable terminal 120 to the server 110, receives a detection result of image recognition by the server 110 or a menu list read from a QR code, and displays the detection result or the menu list on the display 205. The client terminal 100 also mirrors the content displayed on the client terminal 100, thereby displaying the same content on the display 122 of the wearable terminal 120. That is, the process of controlling to display on the display 205 is also the process of controlling to display on the display 122 of the wearable terminal 120. The wearable terminal 120 is a wearable (wearable on the human body) glasses-type device. The display 122 is a type of head-mounted display that can be worn on the human head.
[0016] FIG. 9 shows an example of display on the display 122 of the wearable terminal 120. On the smart glasses, information 900 indicating the detection result and an image 901 captured by the imaging unit 121 are displayed. The image 901 is a live view image, and the information 900 indicating the detection result is displayed so as to be superimposed on the image 901. When displaying on the smart glasses, if data is displayed on both sides, the quality will decrease and it will be difficult to see, so the information 900 is displayed in the center of the left and right of the screen. The information 900 is a display object corresponding to the result display area 600 described later. In addition, the display screen does not display information such as a frame indicating the position of the detected object, and only displays the menu list, the detection result of image recognition, and the image 901. In the case of smart glasses, since they are worn on the head of a person, the content of the image 901 captured by the imaging unit 121 changes significantly depending on the movement of the head. Therefore, it is difficult to detect the position of the object in real time due to problems of processing load and detection accuracy, and there is a possibility that the position cannot be displayed correctly. In addition, if a frame showing the position of the target object is included, the display becomes cluttered when the display screen size is small, such as with smart glasses. Therefore, by displaying only the minimum necessary information, it is possible to display a screen that is easy to check visually.
[0017] The menu DB 130 stores daily menus. By creating a candidate list including the menu of the day, false positives caused by meals that look similar are reduced. For example, assume that the dinner menu of that day includes Hayashi rice. It is difficult to distinguish between Hayashi rice and curry using image recognition, and curry may be mistakenly detected. Therefore, a candidate list including Hayashi rice is created in advance and transmitted to the server 110, changing the determination conditions so that the probability of determining that it is Hayashi rice is increased. In this way, accurate detection is possible even if the meals look similar.
[0018] The server 110 is at least one information processing device on a network, and performs image recognition processing on an image captured by an imaging unit 121. The server 110 has at least one processor, a CPU 111, a RAM 112 serving as a work memory, and a ROM 113 for storing programs. Each of the CPU 111, RAM 112, and ROM 113 does not need to be hardware included in a single server device, and may be at least one hardware resource available in a network environment.
[0019] Next, referring to FIG. 2, an example of the configuration of a client terminal 100 as an example of an apparatus to which the present invention can be applied is shown.
[0020] 2, a CPU 201, a memory 202, a non-volatile memory 203, an image processing unit 204, a display 205, an operation unit 206, a recording medium I / F 207, an external I / F 209, and a communication I / F 210 are connected to an internal bus 250. The units connected to the internal bus 250 are configured to be able to exchange data with each other via the internal bus 250.
[0021] The memory 202 is, for example, a RAM (a volatile memory using a semiconductor element, etc.). The CPU 201 uses the memory 202 as a work memory in accordance with a program stored in, for example, the non-volatile memory 203 to control each part of the client terminal 100. The non-volatile memory 203 stores image data, audio data, other data, various programs for the operation of the CPU 201, etc. The non-volatile memory 203 is, for example, a hard disk (HD) or a ROM.
[0022] The image processing unit 204 performs various image processing on image data stored in the non-volatile memory 203 or the recording medium 208, a video signal acquired via the external I / F 209, image data acquired via the communication I / F 210, captured images, etc., based on the control of the CPU 201. The image processing performed by the image processing unit 204 includes A / D conversion processing, D / A conversion processing, image data encoding processing, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, color conversion processing, etc. The image processing unit 204 may be configured with a circuit block dedicated to performing specific image processing. Depending on the type of image processing, the CPU 201 can also perform image processing according to a program without using the image processing unit 204. The processing of recognizing an object to be recognized from an image is performed by the CPU 201 in cooperation with the image processing unit 204.
[0023] The display 205 displays images, GUI screens constituting a GUI (Graphical User Interface), and the like under the control of the CPU 201. The CPU 201 generates a display control signal according to a program, and controls each unit of the client terminal 100 to generate a video signal for display on the display 205 and output it to the display 205. The display 205 displays video based on the output video signal. Note that the configuration of the client terminal 100 itself is limited to an interface for outputting a video signal for display on the display 205, and the display 205 may be configured as an external monitor (such as a television).
[0024] The operation unit 206 is an input device for receiving user operations, including a character information input device such as a keyboard, a pointing device such as a mouse or a touch panel, a button, a dial, a joystick, a touch sensor, a touch pad, etc. The touch panel is an input device that is configured as a plane overlaid on the display 205 and outputs coordinate information according to the touched position.
[0025] The recording medium I / F 207 allows a recording medium 208 such as a memory card, CD, or DVD to be mounted, and reads data from the mounted recording medium 208 and writes data to the recording medium 208 under the control of the CPU 201. The external I / F 209 is an interface for connecting to an external device via a wired cable or wirelessly, and inputting and outputting video signals and audio signals. The communication I / F 210 is an interface for communicating with an external device, the Internet 211, etc., and transmitting and receiving various data such as files and commands.
[0026] The camera unit 212 is a camera unit including an image sensor (image sensor) including a CCD or CMOS element that converts an optical image into an electrical signal.
[0027] Next, the basic processing of the client terminal 100 in the embodiment of the present invention will be described with reference to Fig. 3. The processing of each step is realized by the CPU 201 of the client terminal 100 expanding a program recorded in the non-volatile memory 203 into the memory 202 and executing it. When the user starts an app in the client terminal 100 as a process before checking whether the meal on the serving tray 140 is appropriate, the processing of Fig. 3 starts.
[0028] In S301, when the CPU 201 receives an instruction to start an application for controlling the wearable terminal 120, the CPU 201 starts the application. For example, the application is started by touching and clicking an icon of the application installed in the client terminal 100. Immediately after the start-up, a result display area 600 is displayed on the UI as shown in FIG. 6. The result display area 600 includes a reliability display area 601, a menu list display area 602, and an other detection result display area 603. At this point, the reliability display area 601, the menu list display area 602, and the other detection result display area 603 are displayed as blank spaces. Note that the dotted frames shown in FIGS. 6 to 9 are auxiliary lines on the drawings to illustrate the areas, and are not displayed.
[0029] In S302, the CPU 201 acquires an image captured by the imaging unit 121 of the wearable terminal 120. For example, the wearable terminal 120 captures an image of a meal on a serving tray 140 or a QR code having information on a menu list. The CPU 201 of the client terminal 100 acquires the captured image. Note that hereinafter, when referring to an image captured by the wearable terminal 120, it is assumed that the image is an image captured by the imaging unit 121.
[0030] In S303, the CPU 201 displays the video acquired in S302 as a background image on the display 205. A display example of the basic screen of the UI is shown in Fig. 7. Note that the display destination when displaying in the steps described below is the display 205 unless otherwise specified.
[0031] 7, an image acquired from the wearable terminal 120 is displayed as a background image 610. The background image 610 is a live view image. If the user wearing the wearable terminal 120 is facing the serving tray, the background image includes a food image 803 on the serving tray and a QR code image 804. In addition, a result display area 600 is displayed, which is an area for displaying the processing results (image recognition results) by the server 110.
[0032] In S304, the CPU 201 transmits the image acquired in S302 to the server 110. The server 110 infers the food menu on the tray from the acquired image and decodes the QR code. The processing on the server side will be described in detail with reference to the flowchart in FIG.
[0033] In S305, the CPU 201 determines whether or not a processing result has been received from the server 110. If a processing result has been received, the process proceeds to S306, and if not, the process proceeds to S307. The processing result received here is any one of the following: a menu list that is a decoded result of the QR code transmitted in S517 of FIG. 5, which will be described later, the number of detected dishes transmitted in S524, and the menu inference result transmitted in S527.
[0034] In S306, CPU 201 executes display control processing based on the processing result received from server 110. This processing will be described in detail with reference to the flowchart in Fig. 4. When the processing in Fig. 4 (processing of S306) ends, the process returns to S305, and it is determined whether or not the next processing result has been received from server 110. This processing is repeated until the app is terminated.
[0035] In S307, the CPU 201 judges whether or not the time for one frame (for example, 1 / 30th of a second) has elapsed. If the time has elapsed, the process returns to S302, and the image from the smart glasses is acquired again. That is, the images captured by the imaging unit 121 are acquired continuously for each frame. If it is judged in S307 that the time for one frame has not elapsed, the process proceeds to S305.
[0036] Fig. 4 shows a flowchart of the display control process. This process is a detailed explanation of S306 in Fig. 3 described above. The process of each step is realized by the CPU 201 of the client terminal 100 expanding a program recorded in the non-volatile memory 203 into the memory 202 and executing the program. When the client terminal 100 receives a processing result from the server 110, the process of Fig. 4 is started.
[0037] In S401, the CPU 201 judges whether or not the processing result received from the server 110 is a correct menu list. The correct menu list is information transmitted from the server 110 in S517 of FIG. 5, which will be described later. The correct menu list is comparison information that includes a menu item group to be compared with the menu inference result. If it is a correct menu list, the process proceeds to S402, and if not, the process proceeds to S406. Specifically, the server 110 performs inference (detection by image recognition) of the meal on the serving tray 140 and decoding of the QR code. It is judged whether or not the client terminal 100 has received the result of decoding the QR code, i.e., the menu list, before the meal inference result (hereinafter, menu inference result).
[0038] In S402, the CPU 201 displays the menu list in the menu list display area 602 of the result display area 600. An example of the menu list display is shown in FIG. 7(a). The received menu list is displayed in the menu list display area 602 within the dotted frame in FIG. 7(a). In the display example, the result of obtaining a menu list of rice, gratin, vegetable salad, fruit (orange), and cabbage soup is displayed. Each of the multiple items displayed in the menu list display area 602 represents the name of a meal (dish) for one dish included in the menu list.
[0039] In S403, the CPU 201 determines whether or not a menu inference result (a detection result by processing in S525 in FIG. 5, which is image recognition processing from an image captured by the imaging unit 121, to be described later) has been received from the server 110. If it has been received, the process proceeds to S404, and if not, the process ends.
[0040] In S404, the CPU 201 displays a judgment result symbol in the reliability display area 601 of the result display area 600. More specifically, the correct answer menu list received in S401 is compared with the menu inference result determined to have been received in S403, and it is determined whether or not each of the dish items (comparison objects) in the correct answer menu list is included as a detected object in the menu inference result. If the dish items (comparison objects) are included as detected objects in the menu inference result, the detection reliability in the menu inference result is obtained, and a judgment result symbol is displayed based on the detection reliability. That is, detection information regarding whether or not the dish items (comparison objects) in the correct answer menu list were detected in the menu inference process is displayed in the reliability display area 601 in association with each of the dish items (comparison objects) in the correct answer menu list. Note that, for other items displayed in the detection result display area 603, the reliability (judgment result symbol) is not displayed in association with them.
[0041] An example of the display at this time is shown in FIG. 7(b). The reliability of the detection result is displayed in the reliability display area 601 with symbols, for example, ◎, 〇, △, and ×, and the closer to ◎, the higher the reliability. In the display example of FIG. 7(b), rice in the menu list is displayed with ◎, indicating that it was detected with a very high score value in the menu inference result. Gratin and fruit (orange) are detected with a high score value in the menu inference result. Cabbage soup has a low score value, indicating that it was detected but with a low reliability (there is a certain possibility that it is not cabbage soup). Vegetable salad has a very low score value, indicating that it has a low reliability (it is highly likely that it is not vegetable salad) or has not been detected (an inference score could not be obtained). By displaying the detection results of image recognition with three or more levels of reliability instead of a binary choice of "correct" or "incorrect," the user can focus on visually checking objects with low reliability. In other words, when detection results are displayed as a two-choice option, they are assigned to one of two options, and even if the result is not displayed as "incorrect," there is a possibility that the result may be incorrect. Therefore, by allowing a range for the detection results, more accurate results can be obtained, and the user can efficiently identify the target object.
[0042] Note that the reliability display object displayed in the reliability display area 601 does not have to be a symbol, as long as the user can recognize the reliability, such as a numerical value (e.g., 80% or 60%) or a letter (high, medium, low, etc.). Note that when the reliability of the detection result is displayed as △ or × (i.e., when the reliability is lower than a predetermined standard), only the △ or × may be highlighted in red or bold, or a separate message may be displayed to notify the user to check.
[0043] In addition, if a meal (dish) not included in the correct menu list acquired in S401 is detected, the name of the detected meal is displayed in the other detection result display area 603 in S409 and S410 described later. Each item displayed in the other detection result display area 603 is the name of a dish (meal name) for one item (one dish). The display example of FIG. 7(b) shows that an Italian salad that does not match the contents shown in the menu list has been detected. By displaying the name of the meal not included in the list in this way, it is possible to check whether the correct answer data of the image recognition and the image recognition result really do not match. In other words, the user can visually check whether the meal detected as not being included in the list does not match the information in the list. Note that information other than the name may be used, such as an identification number, as long as it can be used to identify the meal as not matching the contents shown in the menu list. Even if the image recognition result shows other detection results, there may be cases where there is no problem when actually visually confirmed. For example, if the correct menu list includes "vegetable salad" and the other detection results include "Italian salad," there is no problem if "Italian salad" is actually a type of "vegetable salad" and is included in the list, and the food may be served as is. On the other hand, if the correct menu list includes "vegetable salad" and the other detection results include "hamburger steak," it is necessary to add "vegetable salad" and exclude "hamburger steak" before serving the food. Such a judgment is made by the worker, who is the user wearing the wearable terminal 120, by comparing the display contents of the reliability display area 601, the menu list display area 602, and the other detection result display area 603 with the actual object. In this way, when an object not included in the correct menu list is detected, the identification information (name, type) of the object is displayed in the other detection result display area 603, making it easier for the worker to confirm whether the detected object not included in the correct menu list is really inappropriate. As a further example, if “fruit and vegetables” is included in the correct menu list and “banana” is detected in the menu inference process, since the names do not match, “fruit and vegetables” is displayed in the menu list display area 602 and “banana” is displayed in the other detection results display area 603.However, since the worker can see that "fruit and vegetables" displayed in the menu list display area 602 is a higher-level word that includes "banana" displayed in the other detection result display area 603, the worker can recognize that it is not an inappropriate result ("error"), but rather a problem-free result (i.e., the dish can be served as is). This provides a motivation to make a more accurate judgment by looking at the actual product.
[0044] In S405, the CPU 201 deletes any menu items that have been displayed in the menu list display area 602 and that have a judgment result symbol displayed in the reliability display area 601, among the dish names (menus, items) already displayed in the other detection result display area 603. A specific description will be given later, but if a menu inference result is received first in S406 and S407, the contents are displayed in the other detection result display area 603. After that, when a menu list is received, the CPU 201 deletes the menu items that have been judged (the same dishes as those displayed in the menu list display area 602) among the contents displayed in the other detection result display area 603.
[0045] In S406, the CPU 201 judges whether the processing result received from the server 110 is a menu inference result. If it is a menu inference result, the process proceeds to S407, otherwise, the process proceeds to S411. If the processing result received in S305 is the menu inference result transmitted from the server 110 in S527 of FIG. 5 described later, the result of the judgment in S406 is Yes.
[0046] In S407, the CPU 201 determines whether or not the menu list has been received from the server 110. If the menu list has been received, the process proceeds to S408, and if not, the process proceeds to S410. If the menu list has been received, this refers to the case where the dish names included in the menu list have been displayed in the menu list display area 602.
[0047] In S408, the CPU 201 displays in the reliability display area 601 the determination result symbols corresponding to the dish names already displayed in the menu list display area 602, and which are based on the menu inference result determined to have been received in S406.
[0048] In S409, the CPU 201 displays in the other detection result display area 603 the dish names included in the menu inference result determined to have been received in S403, but which are not included in the dish names already displayed in the menu list display area 602.
[0049] In S410, the CPU 201 displays the menu inference result in the other detection result display area 603. An example of the display at this time is shown in FIG. 8(a). The display example of FIG. 8(a) is a case where the menu inference result is received before the menu list, and the dish name is not displayed in the menu list display area 602, and the dish name indicated by the menu inference result is displayed in the other detection result display area 603. FIG. 8(b) is an example of the display after the menu list is received. The dish name indicated by the received menu list is displayed in the menu list display area 602, and a symbol indicating the reliability is displayed in the reliability display area 601. At this time, meals that exist in the menu list and have already been determined to be reliable are deleted from the other detection result display area 603. In the example of the transition from FIG. 8(a) to FIG. 8(b), rice, gratin, fruit (orange), and cabbage soup are deleted from the other detection result display area 603. Also, meals that are not in the menu list (Italian salad shown in the display example of FIG. 8(b)) are not deleted from the other detection result display area 603, but are displayed as they are. If nothing is displayed when the menu inference result is received first, the user may be concerned about a system failure or malfunction, or may feel that the system has poor response (low responsiveness). Therefore, when the menu inference result is received, even if the menu list has not been received, by displaying the menu inference result in the other detection result display area 603, the user can recognize that the system is processing normally. Or, the user can feel that the system has favorable responsiveness.
[0050] In S411, the CPU 201 determines whether the number of tableware detections is 0 for a specified number of times or more. That is, it determines whether a state in which no detection object is detected from an image captured by the wearable terminal 120 is repeated for a specified number of times or more. If the number of detections is 0 for a specified number of times or more, the process proceeds to S410, and if not, the process ends. The specified number of times is, for example, 60 times. This corresponds to 2 seconds if the frame rate of the image capture is 30 fps and the image transmission to the server 110 and the image recognition process in the server 110 can be performed at the same frame rate. Note that the number of detections does not have to be the number of times as long as it is possible to determine the state in which the number of detections is 0. For example, a method of determining whether the number of detections continues to be 0 for a certain period of time may be used.
[0051] In S412, the CPU 201 initializes the result display area 600 and the background image 610. After the initialization, as shown in FIG. 6, the reliability display area 601, the menu list display area 602, and the other detection result display area 603 are displayed as blanks, and the system waits until the next video is transmitted from the wearable device 120. By seeing this display, the user can recognize that a reset has been performed. In other words, the user can recognize that the contents displayed next in the reliability display area 601, the menu list display area 602, and the other detection result display area 603 relate to the set of dishes to be photographed (the next group of dishes on the tray), and are different from the contents related to the set of dishes photographed in the past.
[0052] The judgment in S411 is not based on the menu inference result, but on the detection result of the presence or absence of tableware. The tableware presence or absence detection process (the process of S523 in FIG. 5 described later) does not detect the type of the object to be detected, compared with the menu inference process (the process of S535 described later), and therefore can be performed at high speed with a small processing load and can obtain a reliable processing result (highly reliable processing result). Therefore, by making the judgment in S411 based on the tableware presence or absence detection result that does not detect the type of the object to be detected, the display contents of the result display area 600 can be reset at a more suitable timing according to the user's intention. For example, when a worker (user) wearing the wearable terminal 120 checks the serving menus on multiple trays in succession, the reset can be performed reliably by simply looking at a place unrelated to the trays for 2 seconds (turning the face, changing the shooting range, and waiting for 2 seconds). Alternatively, if the operation is such that the worker naturally looks at something other than the trays for 2 seconds when replacing the trays, the reset can be performed without the worker making a conscious action to reset. Since the operator does not use his / her hands to reset, it does not impede other tasks that require hands. In addition, since the judgment condition is more reliable than that of the menu inference process, it is less likely that the operator will continue to look at (take pictures of) the dishes on the tray, but the dishes will not be detected due to the uncertainty of the image recognition results, resulting in an unintentional reset. In other words, unintentional resets are also suppressed. If the judgment condition in S411 were to be that the type of food was not detected for a certain period of time (certain number of times) as a result of menu inference, rather than the presence or absence of dishes, it would be relatively easy for the operator to unintentionally reset even though the operator is actually continuing to look at (take pictures of) the dishes on the tray.
[0053] Next, the processing of server 110 in the embodiment of the present invention will be described with reference to Fig. 5(a) to Fig. 5(c). Note that the processing of each step is realized by CPU 111 of server 110 expanding a program recorded in ROM 113 into RAM 112 and executing it. When an image captured by wearable terminal 120 is received from client terminal 100, the processing of Fig. 5(a) is started.
[0054] In S501, the CPU 111 of the server 110 starts up the server program.
[0055] In S502, the CPU 111 of the server 110 acquires a candidate menu corresponding to the current date and time from the menu DB 130. For example, a setting may be made to acquire the next lunch menu at 10:00, and at the set time, a candidate list including the lunch menu is acquired from the menu DB 130.
[0056] In S503, the CPU 111 of the server 110 receives the video from the smart glasses (the image captured by the smart glasses 120) from the client terminal 100. The image received here is the image transmitted in S304 of FIG. 3 described above.
[0057] In S504, the CPU 111 of the server 110 queues the frame image (one frame's worth of image) captured by the smart glasses 120 received in S503 in a queue of the QR code reader.
[0058] In S505, the CPU 111 of the server 110 queues the frame image (one frame's worth of image) captured by the smart glasses 120 received in S503 in an image recognition queue.
[0059] In S506, the CPU 111 of the server 110 determines whether the scheduled restart time has passed. If the scheduled restart time has passed, the process proceeds to S507, and if not, the process returns to S503. The scheduled restart time refers to the time set for acquiring a candidate list from the menu DB. If the scheduled restart time has passed, the server program is restarted to acquire the candidate menu corresponding to the next time.
[0060] In S507, the CPU 111 of the server 110 ends the server program, and the process proceeds to S501 for rebooting.
[0061] Next, with reference to Fig. 5(b), a description will be given of the QR code reading process by the server 110. When data is queued in the queue of the QR code reader in S504, the process of Fig. 5(b) starts.
[0062] In S511, the CPU 111 of the server 110 starts acquiring data from the queue of the QR code reader. The data acquired here is the image queued in S504 (the frame image captured by the smart glasses 120).
[0063] In S512, the CPU 111 of the server 110 determines whether or not there is data in the queue. If there is data, the process proceeds to S513, and if there is no data, the process returns to S511.
[0064] In S513, the CPU 111 of the server 110 detects the position of the QR code from the frame image included in the data acquired in S511. For example, if a QR code placed on the serving tray 140 is captured in the image, the CPU 111 detects its position.
[0065] In S514, the CPU 111 of the server 110 determines whether or not the detection of the position of the QR code has been successful. If the detection of the position of the QR code has been successful, the process proceeds to S515, and if it has failed (if the image does not include a QR code), the process returns to S511.
[0066] In S515, the CPU 111 of the server 110 starts decoding the QR code detected in S514 (the QR code shown in the frame image included in the data acquired in S511).
[0067] In S516, the CPU 111 of the server 110 determines whether or not the QR code has been successfully decoded. If the decoding is successful, the process proceeds to S517, and if not, the process returns to S511.
[0068] In S517, the CPU 111 of the server 110 transmits the decoded result to the client terminal 100. That is, the menu list obtained as a result of the decoding is transmitted to the client terminal 100. Thereafter, the process returns to S511, and the next data is obtained from the queue. If the menu data transmitted here is received in 305 in Fig. 3 described above, the determination in S401 in Fig. 4 described above is Yes.
[0069] Next, with reference to Fig. 5(c), a description will be given of the image recognition process by the server 110. When data is queued in the image recognition queue in S505, the process of Fig. 5(c) starts.
[0070] In S521, the CPU 111 of the server 110 starts acquiring data from the image recognition queue. The data acquired here is the image queued in S505 (the frame image captured by the smart glasses 120).
[0071] In S522, the CPU 111 of the server 110 judges whether or not there is data in the queue. If there is data, the process proceeds to S523, and if there is no data, the process returns to S521.
[0072] In S523, the CPU 111 of the server 110 detects the position of tableware from the frame image included in the data acquired in S521. This process is a process for detecting the presence or absence of tableware as the detection target, and does not detect the type of tableware as the detection target (for example, detection of whether it is a large plate or a small plate). Since the type of the detection target is not detected, the process can be performed at high speed with a smaller processing load than the detection of the type of food by the menu inference process of S525 described later, and a more reliable processing result (a processing result with a higher reliability) can be output than the detection of the type of food by the menu inference process of S525 described later. Tableware is an object (container) that contains the food that is the detection target of the menu inference process of S525 described later. In other words, if the tableware is not detected in S523, it can be assumed that there is no food to be detected in the process of S525. Also, if the tableware is detected in S523, it can be assumed that the food to be detected in the process of S525 is in the area inside the tableware in the image.
[0073] In S524, the CPU 111 of the server 110 transmits the number of detected dishes detected in S523 to the client terminal 100. Based on the number of detected dishes transmitted here, the determination in S411 described above is made.
[0074] In S525, the CPU 111 of the server 110 infers the menu from the frame image included in the data acquired in S521. More specifically, the CPU 111 performs inference processing by inputting the image acquired in S521 into previously prepared learned data, and as a result, obtains the type of food (food name, item) included in the image and its inference score (reliability). The inference processing performed here is a type of image recognition processing that detects a detection target from an image, and is a processing that detects not only the presence or absence of the food that is the detection target, but also the type of food. For example, it is a processing that is necessary to detect whether the type of the food that is the detection target is hamburger steak or curry. Since it goes as far as detecting the type of the detection target, a higher processing load is required and a longer processing time is required than the above-mentioned detection processing of the presence or absence of tableware in S523. In addition, the accuracy of the detection result is not high (the reliability is relatively low) compared to the detection processing of the presence or absence of tableware in S524.
[0075] In S526, the CPU 111 of the server 110 determines whether the inference score is equal to or greater than a threshold for each of the dishes (items) detected in the process of S525. Then, for dishes whose inference scores are equal to or greater than the threshold, the dish name (type of dish, item) and its score are included in the menu inference result transmitted in S527. For dishes whose inference scores are less than the threshold, the dish name and score are not included in the menu inference result transmitted in S527. If detection results for dishes with extremely low inference scores are transmitted to the client terminal 100, it becomes impossible to accurately compare the menu inference results with the menu list. For this reason, a threshold is set and only results that are relatively reliable are transmitted to the client terminal 100.
[0076] In S527, the CPU 111 of the server 110 transmits the menu inference result to the client terminal 100. That is, the result of inferring the type of meal on the serving tray 140 is transmitted to the client terminal 100. After that, the process returns to S521, and the next data is obtained from the queue. If the menu inference result transmitted in S527 is received in S305 in FIG. 3, the determination in S406 in FIG. 4 is Yes.
[0077] According to the display of the reliability display area 601 of this embodiment, it is possible to provide a mechanism that allows the user to efficiently check the reliability of the result of the detection process in image recognition.
[0078] Assume that the present embodiment is not applied and the result of the image recognition is displayed in the reliability display area 601 as a binary choice of "correct" or "incorrect". In this case, even if it is difficult to make a judgment, it will be assigned to one of them, so even if "incorrect" is displayed, the judgment result may not be correct and the image may actually contain the correct dish (the dish in the menu list). In contrast, according to the present embodiment, the reliability is displayed with symbols such as ◯△×. Therefore, when it is difficult to make a judgment by image recognition, it is assigned to △ or ×, so the user can focus on visually confirming the correctness of the image recognition for that part. For example, even if "incorrect" is displayed when displaying the binary choice of "correct" or "incorrect", in the present embodiment, there are cases where "△" is displayed, and the user can visually confirm the actual dish without judging the dish of that item only based on the result of image recognition. Furthermore, even in cases where a choice between "correct" and "incorrect" is displayed and "correct" is displayed, in this embodiment it may be displayed as "o" or "△", allowing the user to visually check the actual dish without making a judgment based solely on the image recognition results.
[0079] In addition, if the judgment results of image recognition were to be checked one by one in the above hypothetical scenario, it would take a lot of time and effort. In contrast, according to this embodiment, it is possible for a person to check only the parts where the reliability of the detection results is low. Therefore, by having a person check only the parts that need checking, it is possible to shorten the work time and reduce the effort required for checking.
[0080] Moreover, according to this embodiment, when the result display area 600 is displayed on the display 122, the position of the detected object detected in the menu inference process or the tableware detection process is not displayed on the live view or with respect to the position of the subject light transmitted through the display 122. This makes it possible to display a screen that is easy to confirm visually. Suppose that this embodiment is not applied and information including a frame or the like indicating the position of the detected object is to be displayed on the display 122. In that case, if the movement of the position of the object on the screen of the display 122 is large, it is difficult for the display indicating the position to follow the position of the object in real time, and there is a possibility that the correct position cannot be displayed. Even if the correct position can be displayed, the movement is too large and the display becomes complicated. In contrast, according to this embodiment, information such as a frame indicating the position of the detected object is not displayed, and symbols and characters are displayed in fixed areas (601 to 603) in the result display area 600 whose display position is fixed, making it easy to confirm visually. Furthermore, when using a terminal device with a small display screen size, such as a glasses-type device, if information indicating the location of the detected item is displayed, the screen becomes cluttered and the information becomes difficult to read. In contrast, according to this embodiment, the list and the detection results are displayed as symbols and characters, and information indicating the location of the detected item is not displayed. Therefore, it is possible to provide a screen that is easy for the user to see.
[0081] Furthermore, according to this embodiment, when an object not included in the information (menu list) to be compared is detected by image recognition (menu inference process), a mechanism for displaying more useful information can be provided. Specifically, when an object not included in the information (menu list) to be compared is detected by image recognition (menu inference process), the type of the detected object (food item name) is displayed in the other detection result display area 603.
[0082] Furthermore, according to this embodiment, even in a situation where the detection target cannot be detected reliably, a mechanism can be provided that can prevent unintended updates to display information while preventing interruptions to work. If this embodiment is not applied and the detection target is a meal menu and the result is reset due to a continued low reliability of the detection result, the result will be reset frequently even if the user continues to look at the tray. In contrast, according to this embodiment, the edge of the tableware is detected. This makes it possible to respond to the detection result more quickly than when the detection target is a meal menu, and to prevent frequent resets.
[0083] Moreover, according to this embodiment, a suitable display can be performed regardless of the order of acquisition of the comparison object information (correct menu list) and the image recognition result (the result of the menu inference process). It is possible that the QR code (registered trademark) having the menu list information is hidden by tableware or has fallen off the tray, and the QR code cannot be read. However, if a user quickly checks many trays, he or she may not notice that the QR code has not been accurately recognized. In contrast, according to this embodiment, the result of the menu inference process is first displayed in the other detection result display area 603, so that the user can recognize that the menu inference process has already ended and that the list is waiting to be acquired, that is, the QR code is waiting to be read. Therefore, the user can understand that the code has not been acquired (photographed) from the printed paper, and can realize that the whereabouts of the paper are unknown. In other words, even if the result is not displayed in the reliability display area 601 and the menu list display area 602, it is possible to quickly recognize the cause and solve the problem, and to prevent the work from being interrupted for a long time.
[0084] In the above embodiment, an example has been described in which a screen is displayed on the wearable terminal 120 through the client terminal 100, but the present invention is not limited to this. The wearable terminal 120 and the server 110 may be directly connected via the network 101 without the client terminal 100 being interposed therebetween. In this case, the wearable terminal 120 mainly operates in place of the client terminal 100 described in the above embodiment.
[0085] Also, an example has been described in which an image captured by the wearable terminal 120 is transmitted from the client terminal 100 to the server 110, the server 110 performs image recognition processing, and the client terminal 100 displays the image according to the results of the image recognition processing, but the present invention is not limited to this. The image recognition processing of Figs. 5(a) to 5(c) described as being performed by the server 110 may be performed by the client terminal 100. Also, the image recognition processing of Figs. 5(a) to 5(c) described as being performed by the server 110 and the processing including the display control processing described in Fig. 4 may be performed by the wearable terminal 120.
[0086] The present invention can be embodied, for example, as a system, an apparatus, a method, a program, or a recording medium, etc. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.
[0087] The various controls described above as being performed by CPU201 and CPU111 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.
[0088] In addition, although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.
[0089] In addition, the above-mentioned embodiment has been described with an example in which the client terminal 100 is applied to a mobile phone terminal (smartphone), but this is not limited to this example, and the present invention can be applied to any device that can display the results of comparing an object detected by image processing with a list of items to be detected. In other words, the present invention can be applied to personal computers, PDAs, portable image viewers, tablet terminals, game consoles, electronic book readers, head-mounted displays, and the like.
[0090] Furthermore, in the above-mentioned embodiment, an example was described in which the item (type) of food contained in an image of a meal (cooking) on a tray is detected by image recognition from an image obtained by photographing the meal (cooking), and the result of comparing the item (type) of food contained in the image is displayed by comparing it with a list of correct menu items obtained from a QR code on the same tray. However, the present invention is not limited to this, and can be applied to any display of a result of comparing items detected by image recognition from an image obtained by photographing a subject with a list of items that should be there, which is information obtained separately. For example, the present invention can be applied to inspection work to check whether the type and quantity of products are correct when receiving or shipping products to a logistics warehouse. It can also be applied to a factory, etc., when an assembly part is picked up from a storage shelf, etc., and the result is checked to see whether it matches the part that should be picked up. It can also be applied to a case where a result of picking up medicine based on a prescription is photographed and the result is checked to see whether it matches the prescription. The method of acquiring the list of items that should be there is not limited to a method of acquiring the list by photographing a QR code, and it may be acquired by other methods, such as by communicating with a server.
[0091] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]
[0092] 100 client terminals 101 Network 110 Server 120 Wearable Devices 130 Menu DB 140 Serving Tray
Claims
1. Based on the results information of the detection process, first information relating to the detected object is displayed, A display control means controls the display of the first information to be suppressed when certain conditions are met, including the fact that no article containing the object to be detected is detected. An information processing system characterized by comprising the following features.
2. The aforementioned specific conditions are conditions that are not based on the type of article, The information processing system according to claim 1, characterized in that the display control means controls the display of the first information to be suppressed when certain conditions are met, including the fact that no article containing the object to be detected is detected, regardless of the type of article.
3. The suppression of the display of the first information means the deletion of the display of the first information, The display control means includes a display control means that controls the deletion of the first information when certain conditions are met, including the fact that no article containing the object to be detected is detected. The information processing system according to claim 1, characterized by comprising the following:
4. The information processing system according to claim 1, characterized in that the display control means controls the display of the first information to be suppressed in response to the fulfillment of the specific conditions, and then to display the second information relating to the detected object.
5. Acquisition means for acquiring detection process result information relating to the object to be detected from an image. Furthermore, The display control means displays first information relating to the detected object in the image based on the result information of the detection process acquired from the image by the acquisition means. The information processing system according to claim 1, characterized in that it controls the display of the first information to be suppressed when certain conditions are met, including the fact that no article containing the object to be detected is detected in the image.
6. The information processing system according to claim 5, characterized in that the specific condition is a condition based on the result of the second detection process, which is one of the results of a first detection process that detects an object to be detected and information relating to the object to be detected in the image, and the result of a second detection process that detects the presence or absence of the article in the image.
7. The first detection process is a process for detecting the food and its items included in the image, The information processing system according to claim 6, characterized in that the second detection process is a process for detecting the presence or absence of tableware.
8. The information processing system according to claim 1, characterized in that the specific condition includes not detecting an article containing the object to be detected for a predetermined period of time.
9. The information processing system according to claim 1, characterized in that the display control means controls to display first information based on the type of object to be detected in the detection process.
10. The information processing system according to claim 1, characterized in that the first information is information obtained by comparing information relating to the object to be detected with information relating to the object to be compared.
11. The information processing system according to claim 10, characterized in that the comparison object information is information based on a code image obtained from the result information of the detection process.
12. The information processing system according to claim 1, characterized in that the display control means displays a live view image captured by the imaging means and controls the display to superimpose the first information onto the live view image.
13. The information processing system according to claim 1, characterized in that the display control means controls the display means of a wearable device to display the first information.
14. The information processing system according to claim 13, characterized in that the display means of the wearable device is a display means of a glasses-type device.
15. The information processing system according to claim 13, characterized in that the wearable device is a device that can be attached to the user's head.
16. A method for controlling an information processing system, The display control means of the information processing system displays first information relating to the detected object based on the result information of the detection process. A display control step that controls the display to suppress the display of the first information when certain conditions are met, including the fact that no article containing the object to be detected is detected, A control method for an information processing system, characterized by comprising the following:
17. A program for causing at least one computer to function as one of the means of the information processing system described in any one of claims 1 to 15.