Photography support system

Abstract

This system supports the capture of images used to understand the situation at food handling sites for purposes such as hygiene management. [Solution] A method for assisting in the capture of inspection images includes the steps of: transmitting guide images representing a plurality of predetermined targets for photography for hygiene management purposes from an information processing device to a mobile terminal at a food handling site; displaying the received guide images on the mobile terminal while displaying the captured images; and transmitting the captured images from the mobile terminal to the information processing device.

Description

Technical Field

[0001] The present invention relates to a technique for assisting photography.

Background Art

[0002] In a site where food is handled, it may be required to periodically check whether standards, laws, and regulations regarding hygiene management are being met. At this time, if employees or the like at the site are asked to take pictures of predetermined locations (ventilation fans, washrooms, kitchens, and other checkpoints), and based on the taken images, an inspector with specialized knowledge can make a determination such as whether it is kept clean, then there is no need for the inspector to visit the site, which is efficient. However, in order to make an accurate determination, it is necessary for the locations to be inspected to be properly photographed. For example, if the image quality is poor, the brightness is insufficient, or the focus is soft, it becomes difficult to make an accurate determination. Also, generally, there are a large number of checkpoints at a single site, and it is currently difficult for employees who are not used to taking pictures or have little knowledge of hygiene management standards to properly take pictures of all the checkpoints within a limited time.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to assist in taking images used for grasping the situation at a site for purposes such as hygiene management at a site where food or the like is handled.

Means for Solving the Problems

[0005] In one aspect, the present invention provides a method for assisting the capture of inspection images, comprising the steps of: transmitting guide images representing a plurality of predetermined targets for photographic purposes from an information processing device to a mobile terminal at a food handling site; displaying the received guide images on the mobile terminal while displaying the captured images; and transmitting the images captured by the mobile terminal to the information processing device. [Effects of the Invention]

[0006] According to the present invention, the capture of images used to understand the situation at a site where food or other items are handled is supported for purposes such as hygiene management. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram showing an overview of Information Processing System 1. [Figure 2] A diagram showing the functional configuration of server 10. [Figure 3] A diagram showing the functional configuration of the field terminal 20. [Figure 4] A diagram showing an example of a guide image. [Figure 5] A diagram showing example 1 of the operation of information processing system 1. [Figure 6A] An example of the screen displayed on the field terminal 20 (before shooting). [Figure 6B] An example of the screen displayed on the field terminal 20 (during filming). [Figure 7] A diagram showing an example of information stored in the memory of server 10. [Figure 8] A diagram showing example 2 of the operation of information processing system 1. [Figure 9] A diagram showing example 3 of the operation of information processing system 1. [Figure 10] A diagram showing example 4 of the operation of information processing system 1. [Figure 11A] An example of the screen displayed on the field terminal 20 (during filming). [Figure 11B] An example of the screen displayed on the field terminal 20 (during filming). [Figure 12]A diagram showing an operation example 5 of the information processing system 1. [Figure 13] An example of a screen displayed on the on-site terminal 20 (during shooting). [Figure 14] An example of a screen displayed on the on-site terminal 20 (during shooting). [Figure 15] An example of a screen displayed on the on-site terminal 20 (during shooting).

Mode for Carrying Out the Invention

[0008] <Example> 1. Example of System Configuration FIG. 1 shows an overview of the information processing system 1 for assisting shooting. The information processing system 1 includes a server 10, an on-site terminal 20, and a remote terminal 30. The server 10, the on-site terminal 20, and the remote terminal 30 are connected to each other via a communication network 2 such as the Internet.

[0009] The on-site terminal 20 is a portable terminal used for shooting by, for example, employees of a store to be inspected, and is a general smartphone, tablet PC, etc. The on-site terminal 20 displays a shooting image while displaying a guide image indicating a plurality of shooting targets predetermined for hygiene management, and transmits the captured image to the server 10 or the remote terminal 30.

[0010] The server 10 or the remote terminal 30 transmits a guide image representing a plurality of shooting targets predetermined for hygiene management to the on-site terminal 20 at the site where food is handled. The server 10 is an information processing device that performs all or part of the processing based on the shooting data acquired from the on-site terminal 20 and stores the result of the processing. In addition, the on-site terminal 20 transmits a guide image and various information for assisting shooting.

[0011] The remote terminal 30 is a computer such as a PC operated by a person who plans and manages inspections, such as an inspector, who is physically separated from the store being inspected. It generates information for assisting in photography, such as transmitting a guide image, to a photographer such as an employee of the store, and executes part or all of the processing based on the captured data, such as analyzing the acquired captured data.

[0012] The function of storing the captured data and the processing results based on the captured data is only possessed by the on-site terminal 20. For other functions, all or part of the functions may be common or complementary between the server 10 and the remote terminal 30. At least, as a whole in the server 10 and the remote terminal 30, it may function as an information device for providing a guide image to the on-site terminal 20, transmitting various notifications to the on-site terminal 20, and performing functions of acquiring the captured data and performing various processes based on the acquired captured data. Various variations of operation examples can be considered according to the functional configurations of the server 10 and the remote terminal 30, and the operation examples will be described later.

[0013] FIG. 2 shows the functional configuration of the server 10 or the remote terminal 30. The server 10 (remote terminal 30) includes a control unit 11, a display unit 12, a storage unit 13, and a communication unit 14. The control unit 11 is implemented as a dedicated (e.g., dedicated to image processing) or general-purpose processor, reads a program from the storage unit 13 and executes it to control each unit, and realizes an image cutout unit 111, an image suitability determination unit 112, and a hygiene state determination unit 113 as functional modules described later. In this figure, an example in which all of the image cutout unit 111, the image suitability determination unit 112, and the hygiene state determination unit 113 are provided is described. However, in each of the server 10 and the remote terminal 30, it may not have at least any one of the image cutout unit 111, the image suitability determination unit 112, and the hygiene state determination unit 113. Furthermore, the server 10 and the remote terminal 30 may both have at least one of the image extraction unit 111, the image suitability determination unit 112, and the sanitary condition determination unit 113, or only one of them may have at least one of them. If the server 10 and the remote terminal 30 do not have the functional modules for the image extraction unit 111, the image suitability determination unit 112, and the sanitary condition determination unit 113, these functions will be supplemented by visual confirmation by inspectors or the like.

[0014] The image extraction unit 111 generates multiple still image data representing the multiple shooting targets from the captured video data based on the guide image. Specifically, it generates one or more still image data by extracting one or more scenes from a single video data according to a predetermined algorithm. For example, it checks the still images of each frame that make up the video data and compares them with the guide image data, and extracts images that are similar to the guide image data to a predetermined degree or higher. In addition, if the guide image is generated by extracting from the video data, it may obtain timing information within the video in which the extraction process was performed, and use this timing information in addition to the similarity with the guide image to identify the scene to be extracted. This reduces the time required for extraction compared to comparing only images.

[0015] Furthermore, generating still image data based on a guide image may include not only extracting still images that show the same subject as the subject shown in the guide image, but also extracting still images that show other subjects related to the subject in the guide image (e.g., a duct) (e.g., the wall on which the duct is installed). In other words, a guide image only needs to show a subject that the user can recognize as a target for photography, and that subject does not need to be the same as the object that the inspector has set out to actually check.

[0016] The image suitability determination unit 112 uses a predetermined image processing algorithm to determine whether the captured image conforms to the criteria necessary to determine, with a predetermined accuracy, whether a predetermined target object conforms to hygiene standards. These criteria include, for example, whether the entire target object is sufficiently captured, whether any part is missing, whether the shooting distance is not too far, whether the shooting direction and angle of view are appropriate, or whether the image quality (brightness, focus, resolution, etc.) is appropriate. In other words, determining the suitability of the captured image can be understood as determining whether the act of shooting was appropriate or not. The target object is predetermined by the inspector in accordance with standards, specifications, and concepts such as HACCP (Hazard Analysis and Critical Control Point), and includes cooking utensils such as knives, equipment installed in kitchens such as ventilation tags, equipment installed in restaurants such as toilets, and so-called checkpoints for hygiene management.

[0017] Furthermore, if the captured image does not meet the criteria, the image suitability determination unit 112 generates a notification that includes at least information about the subject to be re-shot. This notification is transmitted to the field terminal 20 via the communication unit 14. This notification may include an explanation of why the criteria are not met, points to note when re-shooting (for example, advice to get closer to the subject because it was too far away), and other information to support re-shooting. Such notifications to support shooting may be sent to the server 10 during shooting or after shooting is completed. If the notification is sent during shooting, the notification may be executed at a timing appropriate to the status of the shooting work (for example, issuing a notification if a suitable image is not captured after a certain period of time).

[0018] The hygiene condition determination unit 113 applies a predetermined image determination algorithm to the captured image to determine the hygiene condition of the object being photographed. Specifically, it compares the captured image with a reference image to determine the degree of dirtiness, cleanliness, and tidiness of the object being photographed. The determination result may be a simple pass or fail, or it may be a multi-level classification such as ○ (no problem), △ (passing grade), × (fail).

[0019] Machine learning techniques may be used in image suitability determination and sanitary status determination. For example, for image suitability determination, a learning model is constructed that outputs an image suitability result when image data is input, using multiple images with different shooting conditions and the image suitability determination results created by inspectors for each image as input. Image suitability determination is then performed by inputting the captured image data into this learning model. Similarly, for sanitary status determination, a learning model is constructed using multiple image data captured of the same or substantially the same subject and the sanitary status determination results created by inspectors for each image data. The sanitary status is then determined by inputting the captured image data into this learning model.

[0020] The storage unit 13 is a memory. In addition to the OS program, the storage unit 13 stores guide images, data of images taken by the field terminal 20, the result of determining the suitability of the image data (suitability of taking the image), and the result of determining the sanitary condition based on the image data. The stored information is provided to the remote terminal 30 in response to a request from the remote terminal 30. Here, guide images can be multiple still images or one (or more) videos in which multiple subjects are being photographed. In short, they can be so-called example images, such as those actually photographed on-site by an inspector beforehand. In addition to images of the subjects being photographed, guide images may also include information to support the photography (such as text information or audio data that guides or explains the subject or how to photograph it).

[0021] Figure 4 shows examples of guide images. (a) is a guide image (still image) of a gas range, which is one of the checkpoints. (b) is a guide image (still image) of a ventilation fan, which is another checkpoint. (c) is a guide image (still image) of a handwashing area.

[0022] The display unit 12 is a display device such as a liquid crystal display, which displays various information that can be viewed by the user of the server 10 (remote terminal 30). The communication unit 14 is a communication interface that exchanges various information with the server 10 and the remote terminal 30 via wired or wireless means. The input unit 15 is an input device such as a keyboard or mouse, which is used when the user of the server 10 or the remote terminal 30 inputs, for example, a guide image.

[0023] Figure 3 shows the functional configuration of the field terminal 20. The field terminal 20 has the function of displaying captured images while displaying guide images. The field terminal 20 includes a control unit 21, a storage unit 22, an imaging unit 23, a communication unit 24, an operation unit 25, and a display unit 26.

[0024] The memory unit 22 is a memory that stores the basic program (OS) as well as application programs that run on the OS and implement the functions described later on the field terminal 20. In addition, it stores guide images received from the server 10 or remote terminal 30 and images that have been captured. The control unit 21 is implemented as a general-purpose or dedicated processor and executes application programs stored in the memory unit 22 to control the control unit 21 to the display unit 26, and performs the display control of the shooting function and various information related to shooting, which will be described later.

[0025] The imaging unit 23 is a camera module consisting of an optical system such as a lens and a light-receiving element, and an image processing circuit, which captures images (video or still images) and generates digital data in a predetermined format. The imaging unit 23 may be a physically independent device from other functional units. That is, the imaging unit 23 does not have to be built into the field terminal 20; it may be implemented as an external module connected to the field terminal 20 via wired or wireless connection. In short, the user of the field terminal 20 should be able to capture images at multiple shooting points set up within the store.

[0026] The communication unit 24 communicates with at least one of the server 10 and the remote terminal 30 via wired or wireless connection, transmitting captured images while receiving guide images and information and notifications to assist in shooting. The operation unit 25 is an input device such as a touch panel, used by the user of the field terminal 20 to input instructions related to shooting and shooting. The display unit 26 includes a liquid crystal display and its control circuit, and displays a guide image along with the image being captured.

[0027] 2. Example of operation <Example of operation 1> Figure 5 shows an example of operation 1 of the information processing system 1. In this example, the user of the field terminal 20 (hereinafter simply referred to as "user") takes a video while appropriately referring to guide images which are still images, and sends the captured video data to the server 10. The image suitability judgment and sanitary condition judgment are then performed automatically or manually at the remote terminal 30.

[0028] First, the user launches a predetermined application on the field terminal 20 (S100). The field terminal 20 then accesses the server 10 to acquire a guide image (S101). Here, it is assumed that there are three inspection checkpoints (objects to be photographed), and the guide image contains a total of three still image data G1, G2, and G3, which were generated by photographing each object in advance.

[0029] Next, the field terminal 20 displays a menu screen as shown in Figure 6A. This menu screen includes objects OB1, OB2, OB3, and OB4. These objects are used to start shooting, check the captured images, send the captured images, and end the process, respectively.

[0030] When the user selects OB1, a screen like the one shown in Figure 6B is displayed. This screen includes the captured image IMI, objects OB10 and OB11. The captured image IMI is the image currently being captured (or an image that will be stored as image data if the start operation is performed before capture). Object OB10 is a window that displays the guide image. Object OB11 is a guidance message to assist with capture. This guidance message may be received from the server 10 along with the guide image, or it may be generated at the field terminal 20.

[0031] In this way, the guide image is displayed overlaid on the captured image IMI, allowing the user to take appropriate photos while checking the guide image as needed.

[0032] In this example, one of several guide images (the image of the next object to be photographed) is displayed, but multiple guide images corresponding to all inspection items may be displayed at once. When displaying guide images one by one, as in this example, information including the display order associated with each guide image may be sent from the server 10 to the field terminal 20 along with the guide images, and the field terminal 20 may determine which guide image to display based on the elapsed time since the start of shooting and user operations (such as operations to switch guide images).

[0033] Returning to Figure 5, in S102, the user operates a designated shooting button or similar to move to all checkpoints using the guide images as a reference and record video. Once recording is complete, the user performs a designated operation to display the menu screen shown in Figure 6A, and when OB3 is selected on this screen, all the recorded image data (3 in this case) is sent to the server 10 (S103).

[0034] When server 10 receives video data, it refers to the guide image to extract still images and generates still image data corresponding to each target to be photographed (S104). Next, server 10 sends the generated still image data to remote terminal 30 (S105). Remote terminal 30 performs an image suitability determination based on the received still image data (S106). Then, remote terminal 30 generates information indicating the result of the image suitability determination and sends it to field terminal 20 (S107). Here, for example, suppose the image of the inspection target corresponding to guide image G2 was not appropriate. Then, a notification is generated and sent to field terminal 20 prompting the user to photograph the target again, and containing information indicating the inspection target corresponding to guide image G2 (guide image data and captured image data). In the same figure, information indicating that appropriate photography was not performed for the shooting item corresponding to guide image G2 is included.

[0035] When the field terminal 20 receives this notification, it displays the notification on the display unit 26. The user who sees this notification recognizes that the subject needs to be photographed again and takes another photograph (S108). At this time, the guide image G2 identified in the notification is displayed as the guide image. The user performs a predetermined operation and sends the video data obtained by reshooting while referring to the guide image G2 to the server 10 (S109). Upon receiving the image data, the field terminal 20 performs a still image extraction process (S110). It then generates still image data containing only the reshoot target and sends it to the remote terminal 30 (S111). The remote terminal 30 performs another image suitability check (S112), and if it is determined to be suitable, it means that suitable images have been obtained for all shooting targets.

[0036] The remote terminal 30 then determines the sanitary status of all inspection targets based on the obtained images (S113), generates inspection results, and sends them to the server 10 (S114). The server 10 stores the received inspection results (S115). In this example, of the three checkpoints, those corresponding to guide images 1 and 2 are marked with ◎ (pass), while guide image 3 is marked with ▲ (fail).

[0037] Figure 7 shows an example of inspection results stored in the memory unit 13 of the field terminal 20. In this example, the ID of the store or other facility being inspected, the number of times the inspection was performed, search items (checkpoints), inspection results, captured images (image data used to determine the sanitary condition), the date and time recorded, and comments supplementing the inspection results are stored in association with each other.

[0038] According to the above embodiment, the photographer can check the guide image as needed without interrupting the shooting process, accurately grasp the model subject and composition, and then take the photograph. Therefore, the possibility of providing images unsuitable for inspection to the server 10, and the frequency of situations where the user has to be asked to retake the photograph in such cases, can be reduced. As a result, appropriate and efficient shooting can be achieved. Furthermore, requiring employees to capture videos rather than still images helps ensure the quality of inspections. If employees are only required to submit still images, they might try to pass the inspection by temporarily clearing only the checkpoints. However, if videos are required, the store's condition beyond the checkpoints will also be captured, making it more likely that superficial measures will be exposed. This is expected to provide an incentive for employees to stop taking superficial measures. As a result, the provided images are more likely to more accurately reflect the actual state of hygiene. Moreover, using still images generated from videos in image suitability judgments and hygiene management judgments improves the accuracy of the judgments. In this case, since the server 10 automatically extracts still images, the burden on inspectors is reduced compared to when judgments are made using videos, even if the image suitability judgments and hygiene management judgments are made visually.

[0039] Furthermore, by having the field terminal 20 submit video data, the inspector has greater flexibility in setting checkpoints. For example, if an inspector wants to check areas other than the initially set checkpoints after shooting video data, they can access the video data stored on the server 10 themselves without having to request reshooting from the field terminal 20, and visually check the condition of areas that they later became concerned about.

[0040] <Example of operation 2> Figure 8 shows example 2 of the operation of the information processing system 1. In this example, the server 10 automatically performs still image extraction and image suitability determination, while the remote terminal 30 performs the determination of sanitary conditions based on the images.

[0041] The processing in S201-S204 is the same as in Operation Example 1. In S205, the field terminal 20 makes an image suitability determination based on the generated still image data. The process of notifying the field terminal 20 to retake the image and making another image suitability determination (S206-S210) is the same as in Operation Example 1. When the still image data necessary to determine the sanitary condition has been collected, the server 10 sends the still image data to the remote terminal 30 (S211). The sanitary condition determination is made at the remote terminal 30 (S212), and the field terminal 20 receives the inspection result (S213) and stores it (S214).

[0042] <Example of operation 3> Figure 9 shows example 3 of the operation of the information processing system 1. In this example, the server 10 automatically performs all the tasks of extracting still images, determining the suitability of the captured images, and determining the sanitary condition. The remote terminal 30 is used to generate guide images (S301 and S302). The processing from sending the guide images to the field terminal 20 to recording the inspection results (S303 to S314) is the same as that described in operation examples 1 and 2, so the explanation is omitted.

[0043] <Example of operation 4> Figure 10 shows example 4 of the operation of the information processing system 1. In this example, the images captured by the field terminal 20 and provided to the server 10 are still images, and therefore no still image cropping process is performed. During capture, the support information displayed on the field terminal 20 is updated according to the capture status.

[0044] When the 20 receives a guide image (S401), the user generates still image data by referring to the guide image (and other shooting support information) and performing operations such as pressing the shutter button at each shooting point (S402). A single still image data generated by taking a picture at a shooting point is sent to the server 10 each time (S403).

[0045] When server 10 receives image data, it compares it with a guide image to determine whether the received image data is of one of the target objects. If it finds an image of one of the target objects, it performs an image suitability check on that image. Then, it generates a notification containing the result of this check (whether or not appropriate shooting was performed for that target object, in other words, information on whether the shooting for that target object is complete or incomplete) and sends it to the field terminal 20 (S405). Upon receiving the check result, the field terminal 20 displays a guidance message corresponding to this notification (S406).

[0046] For example, if it is determined that the shooting of one target has been completed, a message OB41 prompting the field terminal 20 to complete the shooting at shooting point 1 and proceed to the next shooting point 2 will be displayed, as shown in Figure 11A. If the captured image is not satisfactory, a message OB42 prompting the user to retake the image because the shooting at shooting point 1 has not been completed will be displayed, as shown in Figure 11B.

[0047] In this way, re-shooting data is acquired as needed (S407, S408), and when it is determined that images have been acquired for all targets (S409: YES), a notification is sent to the field terminal 20. The determination of the sanitary condition and the recording of the inspection results are the same as in operation example 3, so the explanation is omitted (S411, S412).

[0048] Figure 12 shows Operation Example 5 of the Information Processing System 1. This example differs from Operation Example 4 in that the images captured by the field terminal 20 are videos, and the guide images are also videos. Preferably, these guide images are taken by an inspector who actually walks around the store to be inspected in advance. By making the guide images videos, the user of the field terminal 20 only needs to trace the movements of the inspector and imitate the operation such as how to point the camera, which has the advantage of allowing them to concretely visualize the appropriate actions to take when taking pictures, compared to still images. Furthermore, this differs from Operation Example 4 in that the field terminal 20 is equipped with the functions of an image suitability determination unit 112 and a hygiene condition determination unit 113 in order to determine the suitability of the image in real time during shooting.

[0049] In S502, the user starts recording video. At this time, no operation such as pressing the shutter button (operation for generating still images) is required. The user moves the field terminal 20 with them so that they can obtain an image with a similar composition to the guide image, and adjusts the direction of the field terminal 20 (shooting direction), etc. The field terminal 20 monitors the recorded video data in real time and checks in real time whether shooting of the target has been completed by comparing it with the guide image (S503).

[0050] When proper shooting of a target is completed (S503:YES), a message OB52 indicating successful shooting, as shown in Figure 14, is displayed (S504A), and video data containing at least one frame in which the target is captured is extracted from the video data being recorded and sent to the server 10 (S505). If it is determined that proper shooting of a target has not been performed (S503:NO), a message to that effect is displayed (S504B). Figure 13 shows an example of message OB51 displayed on the field terminal 20.

[0051] In this case, for example, if the condition in which proper shooting is determined to have not occurred continues for a predetermined period of time or longer, message OB53, as shown in Figure 15, will be displayed. At this time, the playback of the guide image may be stopped and restarted from the beginning or from the point in time when the condition began. This is because the guide image being played back in such a state is highly likely to no longer be able to properly support the user's condition.

[0052] Returning to Figure 12, when the server 10 determines that it has obtained imaging data for all targets (S506: YES), it notifies the server 10 of this (S507). Then, it determines the sanitary status and records the inspection results based on the acquired video data (S508, S509). When determining the sanitary status, when comparing the guide image, which is video data, with the captured video data, it is possible to extract still images and compare them, or to compare video data with each other.

[0053] In the above-described embodiment, the server 10, the field terminal, and the remote terminal 30 were provided as independent hardware components. However, all or part of the functions realized by these three hardware components may be implemented by one or more devices selected from these three. That is, the present invention is an information processing system for supporting the capture of inspection images, comprising at least a portable terminal, wherein the portable terminal's display unit has means for displaying a captured image while displaying guide images representing a plurality of targets predetermined for hygiene management purposes in a food handling site, and means for acquiring the image captured by the portable terminal. [Explanation of symbols]

[0054] 1: Information Processing System 10: Server 11: Control Unit 12: Display section 13: Storage part 14: Communications Department 15: Input section 111: Image cropping section 112: Image suitability determination unit 113: Hygiene Condition Determination Unit 20: Field terminal 30: Remote terminal 21: Control Unit 22: Storage section 23: Imaging Unit 24: Communications Department 25:Operation unit 26: Display section

Claims

1. In a food handling environment, the process involves transmitting guide images representing multiple predetermined targets for photography from an information processing device to a mobile device, and The steps include displaying the captured image while displaying the received guide image on the aforementioned mobile terminal, The steps include: transmitting the image captured by the mobile terminal to the information processing device; A method for assisting in the acquisition of inspection images, having the following characteristics.

2. The images captured by the aforementioned mobile device are moving images. Based on the aforementioned guide image, multiple still images are generated from the video, each representing one of the multiple targets being photographed. The method according to claim 1.

3. Each of the aforementioned guide images is a still image obtained by extracting a portion of a video image previously taken at a location that includes the multiple targets to be photographed. When generating a still image based on the captured guide image, the time range for which the cropping process is performed is determined based on information about the timing of the cropping process. The method according to claim 2.

4. The method further includes a step of determining, based on the guide image, whether the captured image meets the criteria necessary to determine the appropriateness of hygiene management related to the subject being photographed. The method according to claim 1.

5. If the captured image does not meet the criteria, the information processing device notifies the mobile terminal of the object to be re-captured. The method according to claim 4.

6. While the mobile device is taking a picture, the system notifies the user of information regarding the suitability of the captured image, based on the captured image and the guide image. The method according to claim 1.

7. The images captured by the aforementioned mobile device are moving images. During shooting, the system determines whether each target represented by each guide image has been properly captured, and outputs an image representing each target that has been properly captured. The method according to claim 6.

8. The aforementioned guide image is a moving image. If, during the aforementioned shooting process, it is determined that proper shooting did not occur while a guide image corresponding to a shooting target was displayed for a predetermined period, the playback of the guide image will be restarted. The method according to claim 7.

9. The steps include: using a learning model that has been trained on images taken of a subject and the results of a determination of the hygiene management status of that subject, outputting the results of a determination of the hygiene management status of multiple subjects included in the captured images; The method according to claim 1, further comprising:

10. On the computer of a mobile device with a camera function, In a food handling environment, the process involves receiving guide images representing multiple predetermined targets for photography from an information processing device, and The steps include displaying the captured image while displaying the received guide image, The steps include: transmitting the captured image to the information processing device; A program to execute.

11. Equipped with a mobile terminal and an information processing device, The display unit of the aforementioned mobile terminal includes means for displaying a captured image while simultaneously displaying guide images representing multiple predetermined targets for photographic purposes in a food handling environment, A means for acquiring images taken with the aforementioned mobile device, An information processing system for assisting in the acquisition of inspection images, which has the following features.

Images