Information processor

Abstract

To allow remote involvement in food sanitation inspections.SOLUTION: An inspection image evaluation unit 112 evaluates the quality of an acquired inspection image as an image to be used for an inspection on the basis of the feature quantity of the acquired inspection image. A re-photographing instruction unit 113 transmits instruction data indicating instructions to re-photograph an inspection target included in the inspection image used for the evaluation to a user terminal 20, when the acquired inspection image is evaluated to be of a predetermined quality. A machine determination unit 121 makes a determination on an inspection on the inspection target included in an acquired inspection image on the basis of the feature quantity of the acquired inspection image. The machine determination unit 121 transmits an inspection image used for determination to an instructor terminal 30 when the determination is a prescribed result, and presents the inspection image used for the determination to an instructor. An inspection instruction transfer unit 122 transfers (transmits) instruction data indicating instructions given by the instructor on the basis of the inspection image presented by the machine determination unit 121 to the user terminal 20.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to food hygiene management. [Background technology]

[0002] As a technology related to food hygiene management, Patent Document 1 discloses a technology that sends an alert to a specified notification destination when the inspection results of the hygiene management status of a food-handling establishment and its employees are poor compared to predetermined standard values ​​determined according to the importance of the results. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-216115 A Summary of the Invention [Problem to be solved by the invention]

[0004] In food hygiene inspections at facilities that handle food, the inspections may be carried out by people (such as store clerks) who do not have the inspection skills backed by specialized knowledge or experience in food hygiene. In such cases, it would be ideal if specialized inspectors could be involved in the inspections remotely by conducting the inspections remotely or supporting on-site inspectors, but if inspectors were to always go to the site instead, the travel time would be significant, reducing the number of facilities that can be inspected. Therefore, an object of the present invention is to enable remote participation in food hygiene inspections. [Means for solving the problem]

[0005] In order to achieve the above-mentioned objective, the present invention provides an information processing device comprising an image acquisition unit that acquires images of employees, ingredients, utensils, tableware, storage conditions, or the environment that are the subject of food hygiene inspection in a facility that handles food; a judgment unit that makes a judgment on the inspection subject depicted in the acquired image based on features of the image; a presentation unit that presents the image used for the judgment to an instructor of the inspection if the judgment is a predetermined result; and a first transmission unit that transmits first instruction data indicating instructions to be given by the instructor based on the presented image to a destination associated with the facility.

[0006] In addition, the judgment unit may use an AI that has learned the relationship between the features of an inspection image and a correct inspection result using inspection images used in past inspections and inspection results performed by a person based on those inspection images as training data, to make an inspection judgment for the inspection target shown in the image acquired by the image acquisition unit.

[0007] The image processing apparatus may also include an evaluation unit that evaluates the quality of the acquired image as an image for inspection based on the features of the acquired image, and a second transmission unit that, when the acquired image is evaluated to be of a predetermined quality, transmits to the destination second instruction data indicating an instruction to re-photograph the inspection subject appearing in the image used for the evaluation.

[0008] The second transmission unit may transmit data for displaying a sample image for photographing the object of the inspection as the second instruction data.

[0009] The second transmission unit may transmit data for displaying the sample image according to the type of facility as the second instruction data.

[0010] The presentation unit may also be configured such that the fewer the test results for the test subject for which the determination has been made, the more likely it is that the test result for the test subject will be determined to be the predetermined result.

[0011] In addition, the image acquisition unit may acquire images from a wide-angle camera installed in the facility, and the presentation unit may present to the instructor an image in which distortion of the subject of the test that appears in the image used for the judgment has been corrected.

[0012] The first transmission unit may transmit data for displaying a sample image for photographing the object of the inspection as the first instruction data. Effect of the Invention

[0013] The invention makes it possible to allow remote participation in food hygiene inspections. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of an overall configuration of an examination support system according to an embodiment. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of a server device. [Diagram 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of a user terminal. [Figure 4] A diagram showing the functional configuration realized by each device [Diagram 5] A diagram showing an example of facility information [Figure 6] FIG. 13 is a diagram illustrating an example of printed correspondence information. [Figure 7] FIG. 1 is a diagram illustrating an example of stored examination information. [Figure 8] FIG. 13 is a diagram illustrating an example of a displayed inspection screen. [Figure 9] FIG. 13 is a diagram showing an example of an enlarged display of a selected test item. [Figure 10] FIG. 1 is a diagram illustrating an example of a test result table. [Figure 11] A diagram showing an example of the displayed tally results [Figure 12] FIG. 13 is a diagram illustrating an example of a displayed inspection screen. [Figure 13] FIG. 13 is a diagram illustrating an example of a displayed determination result. [Figure 14] FIG. 13 is a diagram illustrating an example of a displayed instruction screen. [Figure 15]FIG. 13 is a diagram illustrating an example of a displayed inspection screen. [Figure 16] FIG. 1 is a diagram illustrating an example of an operation procedure of each device in a storage process. [Figure 17] FIG. 13 is a diagram illustrating an example of an operation procedure of each device in the training support process. [Figure 18] FIG. 1 is a diagram illustrating an example of a sample image table. [Figure 19] FIG. 1 is a diagram showing an example of a result table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] [1] First Example 1 shows an example of the overall configuration of an inspection support system 1 according to an embodiment. The inspection support system 1 is a system that supports inspection work, which is one of the tasks related to food hygiene management in facilities that handle food. The inspection work is a task of inspecting whether the state of food hygiene in a facility meets the standards required for safety (for example, standards established by HACCP (Hazard Analysis and Critical Control Point)).

[0016] Facilities that handle food include, for example, restaurants, food and beverage retailers, food and beverage warehouses, and food and beverage manufacturing plants, and are also facilities where food hygiene inspections are conducted. In these facilities, a person in charge of food hygiene inspections is designated, but the person in charge of inspection work does not necessarily have specialized knowledge or experience in food hygiene inspections.

[0017] For example, in stores that set up stalls at food sales events, restaurants in shopping malls, or self-employed restaurants in shopping districts, there are no specialized inspectors, so inspections are carried out by employees who are not familiar with food hygiene inspections. Inspection support system 1 particularly supports inspection work in facilities where only staff members have no specialized knowledge or experience in food hygiene inspections.

[0018] The test support system 1 comprises a network 2, a server device 10, a user terminal 20, an instructor terminal 30, and a printing device 3. The network 2 is a communication system including a mobile communication network and the Internet, and relays data exchange between devices accessing the system. The server device 10 accesses the network 2 via wired communication (wireless communication is also acceptable), and the user terminal 20 and the instructor terminal 30 access the network 2 via wireless communication.

[0019] The user terminal 20 is a terminal used by an inspector who performs inspection work in a facility that handles food, and is a so-called mobile terminal that can be carried around, such as a smartphone or tablet terminal. An inspector is, for example, an employee in charge of inspection work among employees of a facility that handles food. The user terminal 20 stores items to be inspected and displays the stored items. The inspector inspects the displayed items and inputs the inspection results into the user terminal 20.

[0020] 1 shows only one user terminal 20, but if there are many facilities to be inspected, multiple inspection personnel will use multiple user terminals 20 to perform multiple inspections in parallel. The instructor terminal 30 is a terminal used by an instructor who supports the inspection personnel from a remote location, and is, for example, a mobile terminal such as a smartphone. In this embodiment, it is assumed that there is only one instructor to simplify the explanation.

[0021] The server device 10 performs processes such as managing information related to tests performed by a plurality of user terminals 20 and instructor terminal 30. The server device 10 is an example of an "information processing device" of the present invention. The information related to tests includes the test items and the test results for each test item described above. The processes performed by the server device 10 will be described in detail later.

[0022] 2 shows an example of a hardware configuration of the server device 10. The server device 10 is a computer including a processor 11, a memory 12, a storage 13, a communication device 14, and a bus 15. The processor 11 controls the entire computer by running, for example, an operating system. The processor 11 includes a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, and the like.

[0023] The processor 11 reads programs, data, etc. from at least one of the storage 13 and the communication device 14 into the memory 12, and executes various processes according to the read programs, etc. The memory 12 is a computer-readable recording medium such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage 13 is a computer-readable recording medium such as a hard disk drive.

[0024] The communication device 14 is hardware (transmitting / receiving device) for performing communication between computers via at least one of a wired network and a wireless network. Each device, such as the processor 11 and the memory 12, is connected by a bus 15 for communicating information. The bus 15 may be configured using a single bus, or may be configured using different buses between each device.

[0025] Fig. 3 shows an example of the hardware configuration of the user terminal 20 and the instructor terminal 30. The user terminal 20 and the instructor terminal 30 are computers equipped with a processor 21, a memory 22, a storage 23, a communication device 24, an input device 25, an output device 26, an imaging device 27, and a bus 28. The processor 21 and the like, which show the same hardware in Fig. 2, are the same type of hardware as in Fig. 2, although there are differences in performance and specifications.

[0026] The input device 25 is an input device (e.g., a switch, a button, a sensor, etc.) that accepts input from the outside. The input device 25 includes an imaging device (e.g., a digital camera, etc.) that takes pictures. The output device 26 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. In the user terminal 20, the input device 25 and the output device 26 include an integrated configuration (a touch screen in which a display and a touch panel are integrated). The imaging device 27 includes a lens, an optical system, an image sensor, etc., and captures an image represented by light that reaches the image sensor.

[0027] The functions of each device in the test support system 1 are realized by loading specific software (programs) onto each hardware such as processor and memory, causing the processor to perform calculations, control communications by each communication device, and control at least one of reading and writing of data in memory and storage.

[0028] 4 shows the functional configuration realized by each device. The server device 10 comprises a facility information storage unit 101, a print control unit 102, a read information acquisition unit 103, an examination information output unit 104, an examination image acquisition unit 111, an examination image evaluation unit 112, a re-photograph instruction unit 113, a machine judgment unit 121, and an examination instruction transfer unit 122. The user terminal 20 comprises a code reading unit 201, an examination information storage unit 202, and an examination processing unit 203. The instructor terminal 30 comprises a machine judgment display unit 301, an instruction operation reception unit 302, and an examination instruction transmission unit 303.

[0029] The facility information storage unit 101 stores facility information, which is information about facilities where food hygiene inspections are performed. Specifically, the facility information storage unit 101 stores information that associates facilities where food hygiene inspections are performed with inspection information indicating inspection methods determined for the facilities as facility information. Appropriate methods for food hygiene inspections are determined depending on factors such as the type of industry, business form, facility equipment environment, facility size, and types of ingredients handled.

[0030] Fig. 5 shows an example of facility information. In the example of Fig. 5, the facility information storage unit 101 stores information as facility information in which "major items," "medium items," "inspection items," "inspection time," and "pass criteria" are associated with facilities A, B, and C. Major items include "employees" and "ingredients." Medium items include "hand washing," "clothing," and the like. Inspection items are represented by symbols in Fig. 5, but in reality they are represented by sentences explaining the contents of the inspection.

[0031] For example, the content of the test item K-1 is assumed to be, for example, "washing hands for a specified period of time or more." For example, this "specified period of time" is set as the pass criterion for the test item K-1. In the example of FIG. 5, different pass criteria (criterion a for facility A, criterion b for facility B, and criterion c for facility C) are associated with the test item K-1.

[0032] Also, for example, test items K-1, K-2, and K-3 are associated with the hand-washing area of ​​facility A, but only test items K-1 and K-3 are associated with the hand-washing area of ​​facility B, and only test items K-1 and K-2 are associated with the hand-washing area of ​​facility C. In this way, facility information storage unit 101 stores facility information in which different test items are associated with each facility.

[0033] The print control unit 102 outputs print data for printing information associated with a facility (hereinafter referred to as "corresponding information"). The corresponding information is, for example, identification information individually assigned to each facility, specifically a facility ID (Identification). In this embodiment, the print control unit 102 outputs print data for printing the coded corresponding information.

[0034] The print control unit 102 outputs print data for printing, for example, a two-dimensional code indicating a facility ID for each facility. In the example of Fig. 5, the facility information storage unit 101 stores facility IDs (Identification) such as "ID001" for facility A, "ID002" for facility B, and "ID003" for facility C as identification information. Therefore, these pieces of identification information are information associated with the facilities.

[0035] For example, for facility A, the print control unit 102 outputs print data for printing a two-dimensional code indicating "ID001," and for facility B, outputs print data for printing a two-dimensional code indicating "ID002." The print control unit 102 outputs the print data to the printing device 3 described above. Note that the coded corresponding information is not limited to a two-dimensional code, and may be a one-dimensional code or a list of symbols obtained by converting identification information using a predetermined algorithm. The printing device 3 prints the corresponding information indicated by the output print data on a medium such as paper.

[0036] Fig. 6 shows an example of the printed correspondence information. In the example of Fig. 6, a two-dimensional code 5, which is correspondence information associated with "facility A," is printed on paper 4. Also printed on paper 4 are the following text: "Food hygiene inspection support information for facility A" and "Please launch the inspection app and read the two-dimensional code below." In this way, the print control unit 102 outputs print data that also shows the procedure for supporting food hygiene inspections using the correspondence information.

[0037] The code reading unit 201 reads the two-dimensional code which is the printed corresponding information. The code reading unit 201 analyzes the image captured by the imaging device 27 of the own device, and if a two-dimensional code is included, reads the information indicated by the two-dimensional code. When the code reading unit 201 reads the two-dimensional code 5 in Fig. 6, it reads the facility ID "ID001" of facility A.

[0038] The code reading unit 201 transmits the read information (hereinafter referred to as "read information") to the server device 10. The read information acquiring unit 103 of the server device 10 acquires the transmitted read information. The read information acquiring unit 103 supplies the acquired read information to the test information output unit 104. The read information thus supplied is the facility ID read by the user terminal 20 as described above.

[0039] When read information (facility ID) is supplied as described above, i.e., when corresponding information read by a user terminal 20 capable of reading corresponding information is transmitted from the user terminal 20, the test information output unit 104 outputs to the user terminal 20 the test information associated with the same facility as the facility to which the corresponding information is associated.

[0040] The test information output unit 104 first reads out the test information stored in association with the supplied read information (facility ID) from the facility information storage unit 101. Then, the test information output unit 104 outputs the read test information to the user terminal 20 which is the sender of the read information. The output test information is supplied to the test information storage unit 202 of the user terminal 20. The facility ID read by the code reading unit 201 is also supplied to the test information storage unit 202.

[0041] The test information storage unit 202 stores the supplied test information in association with the supplied facility ID. Fig. 7 shows an example of stored test information. In the example of Fig. 7(a), test information for facility A is stored. In this case, the user terminal 20 equipped with the test information storage unit 202 is the user terminal 20 that reads the correspondence information printed for facility A.

[0042] In the example of FIG. 7(b), the user terminal 20 equipped with the inspection information storage unit 202 is the user terminal 20 that reads the corresponding information printed for facility B, which is the inspection information at facility B. By storing the inspection information as described above, preparations for the food hygiene inspection are complete. The inspection processing unit 203 performs processing related to the food hygiene inspection based on the stored inspection information. The inspection processing unit 203 first displays an inspection screen for the relevant facility.

[0043] Fig. 8 shows an example of a displayed inspection screen. In the example of Fig. 8, the inspection processing unit 203 displays an inspection list including the major items of "Employee" tab T1, "Foodstuff" tab T2, "Utensils and tableware" tab T3, "Storage status" tab T4, and "Environment" tab T5, as well as a "Judgment" tab T6, which will be described later, with tab T1 open. The display area C1 for "Employee" includes the medium items of "Handwashing" display area D1, "Clothing" display area D2, and "Visual inspection of hands" display area D3.

[0044] The subsequent displays are also performed by the inspection processing unit 203. In the display area for each medium item, inspection items "K-1" and "K-2" are displayed. By scrolling the display area C1, the display area for medium items and inspection items that are not displayed are displayed. To the left of the inspection items, an NG button E1, a photo button E2, a comment button E3, and an OK button E4 are displayed. When the inspector selects an inspection item for which he or she wishes to enter the inspection results by tapping or other operation, the inspection processing unit 203 enlarges and displays only the selected inspection item.

[0045] Fig. 9 shows an example of an enlarged display of a selected test item. In the example of Fig. 9, test item K-4 is displayed in an enlarged state. When the NG button E1 is pressed in the enlarged state, an "x" mark (a mark indicating that the test result is NG, i.e., a failure) is displayed, and when the OK button E4 is pressed, an "o" mark (a mark indicating that the test result is OK) is displayed. When the photo input field E5 is tapped, for example, the imaging device of the user terminal 20 is started and a photo is taken by the user's operation, and the taken photo is displayed in the photo input field E5.

[0046] It should be noted that when the photo input field E5 is tapped, a photograph that has been taken and is stored in the user terminal 20 may be imported. A comment may be manually input into the comment input field E6, or a comment may be input by selecting an example sentence included in a list of comment example sentences E7. When the inspection processing unit 203 receives an input operation of the inspection result, it displays the result of the received input operation.

[0047] When the temporary interruption button B1 is tapped, the test processing unit 203 determines that input to the enlarged test item K-4 is to be temporarily interrupted, and returns to the test screen of Fig. 8 to display the list of test items. When the input completion button B2 is tapped, the test processing unit 203 determines that input to the enlarged test item K-4 is completed, and returns to the test screen of Fig. 8 to display the list of test items.

[0048] Furthermore, the inspection processing unit 203 stores an inspection result table, and when the input completion button B2 is tapped, the result of the input operation up to that point is reflected in the inspection result table as the inspection result. An example of the test result table is shown in Fig. 10. The test result table stores "facility ID", "number of tests", "test item", "NG", "OK", "photo", "comment", and "input date and time" in association with each other.

[0049] The "facility ID" and "number of tests" are stored when the test processing unit 203 generates a test result table. In the example of FIG. 10, since this is the first time that a test result table for the relevant facility ID has been generated, "1" is stored as the "number of tests." "NG" means that the test failed, and "OK" means that the test passed. An "NG" input is represented by an "X," and an "OK" input is represented by an "O."

[0050] Photos are represented by the file name of the image data, and comments are represented by the file name of the text data. The input date and time are represented by year, month, and day, but the exact time may also be represented. The inspection processing unit 203 refers to the inspection results stored in the inspection result table and performs a counting process of the inspection results. The counting process is a process of calculating the total number of common inspection results (e.g., OK inspection results) as the count value.

[0051] The tallying process also includes a process of calculating another value based on the calculated tally. Specifically, the inspection processing unit 203 tally up the OK inspection results and calculates the ratio of the OK tally to the total number of inspection items (hereinafter referred to as the "evaluation rate") for each major item and for the whole. The higher the evaluation rate, the more work that should be performed for sanitation management is being performed, and it can be used as an index that indicates that safer food can be expected to be supplied.

[0052] When calculating the evaluation rate, the inspection processing unit 203 handles the inspection results of each inspection item by weighting rather than treating them uniformly. For example, if there are 50 inspection items, the evaluation rate will decrease by 2% for each NG if they are handled uniformly. In contrast, by weighting each inspection item, the evaluation rate is calculated so that, for example, if an inspection item with high importance is NG, the evaluation rate will decrease by 3%, and if an inspection item with low importance is NG, the evaluation rate will decrease by 1%.

[0053] The importance level means, for example, that the greater the impact of an inspection item on food hygiene, the higher the importance level. By weighting the items with the importance level taken into consideration, the reliability of the evaluation rate as the above-mentioned index can be increased compared to when the importance level is not taken into consideration. When the "Judgment" tab T6 shown in FIG. 8 is pressed, the inspection processing unit 203 displays the results of the previous tabulation.

[0054] 11 shows an example of the displayed tally results. The inspection processing unit 203 displays a result table F1 showing the evaluation rate for each major category, and a radar chart F2 showing the evaluation rate for each major category. The inspection processing unit 203 also displays the overall evaluation rate as an "overall evaluation," and displays the character string "Excellent Store," which is a name determined according to the overall evaluation, and "S," which is an evaluation rank determined according to the overall evaluation.

[0055] These names and evaluation ranks are determined according to the overall evaluation value, for example, "Excellent store" and "S", "Excellent store" and "A", "Good store" and "B", "Needs improvement" and "C". When the input completion button B2 displayed on the inspection screen is pressed, the inspection processing unit 203 updates and stores the inspection results table. The stored inspection results include the facility ID and the number of inspections, so it is possible to know which facility and which inspection result it is.

[0056] The inspection processing unit 203 transmits an image (hereinafter referred to as an "inspection image") input by a user and used for inspection to the server device 10. An inspection image is, for example, a photograph of an inspection target such as a sink, a washbasin, a floor, or a refrigerator. The inspection image acquisition unit 111 receives the transmitted inspection image and acquires the inspection image, i.e., an image of an inspection target for food hygiene in a facility that handles food. The inspection image acquisition unit 111 is an example of the "image acquisition unit" of the present invention.

[0057] The inspection image acquisition unit 111 supplies the acquired inspection image to the inspection image evaluation unit 112. The inspection image evaluation unit 112 evaluates the quality of the inspection image as an image for inspection based on the feature amount of the inspection image acquired by the inspection image acquisition unit 111. The inspection image evaluation unit 112 is an example of the "evaluation unit" of the present invention. The inspection image evaluation unit 112 stores an inspection image table that associates, for example, the feature amounts of inspection images of samples from various facilities that have a common inspection target with the level of quality of the inspection image as an image for inspection.

[0058] After calculating the feature amount of the acquired inspection image, the inspection image evaluation unit 112 compares the calculated feature amount with the feature amount stored in the inspection image table to calculate the similarity between the acquired inspection image and each sample inspection image. The inspection image evaluation unit 112 evaluates the quality associated with the inspection image of the sample having the highest similarity as the quality of the acquired inspection image.

[0059] The inspection image evaluation unit 112 supplies the quality evaluation result to the re-photographing instruction unit 113. When the acquired inspection image is evaluated to have a predetermined quality, the re-photographing instruction unit 113 transmits instruction data indicating an instruction to re-photograph the inspection target shown in the inspection image used for the evaluation to the destination mentioned above (user terminal 20 in this embodiment). The instruction data in this case is an example of the "second instruction data" of the present invention, and the re-photographing instruction unit 113 is an example of the "second transmission unit" of the present invention.

[0060] The re-photographing instruction unit 113 also transmits data as instruction data for displaying a sample image for photographing the test subject on the user terminal 20. For example, for a test subject appearing in a test image evaluated to have a predetermined quality, the re-photographing instruction unit 113 transmits, as a sample image, a test image of a sample that is evaluated to have the highest quality in the test image table. In this embodiment, if the test subject is the same, the same sample image is transmitted to user terminals 20 used in any facility.

[0061] The inspection processing unit 203 of the user terminal 20 displays an inspection screen including a sample image indicated by the transmitted instruction data. Note that the re-photographing instruction unit 113 is not limited to transmitting data indicating the sample image itself, and may transmit, for example, data indicating the save destination and file name of the sample image as the data for display described above.

[0062] Fig. 12 shows an example of the displayed inspection screen. In the example of Fig. 12, the inspection processing unit 203 displays a sample image F1 next to the instruction content in addition to the judgment result etc. shown in Fig. 15. The inspector refers to the displayed sample image F1, photographs the inspection object again, and inputs it as the second inspection image. After that, the inspection input is performed as described in the explanation of Figs. 8 and 9.

[0063] If the acquired inspection image is evaluated as not being of a predetermined quality, re-photographing instruction unit 113 notifies inspection image acquisition unit 111 of that effect. Upon receiving this notification, inspection image acquisition unit 111 supplies the acquired inspection image to machine determination unit 121. When machine determination unit 121 receives an inspection image, i.e., when the inspection image is evaluated as not being of a predetermined quality, it performs the following determination.

[0064] The machine judgment unit 121 judges the inspection of the inspection object shown in the inspection image based on the feature amount of the inspection image acquired by the inspection image acquisition unit 111. The machine judgment unit 121 is an example of the "judgment unit" of the present invention. The machine judgment unit 121 learns the relationship between the feature amount of the inspection image and the correct inspection result using, for example, an inspection image used in a past inspection and an inspection result performed by a person based on the inspection image as training data.

[0065] The feature amount of the inspection image may be, for example, one or more of SIFT (Scaled Invariance Feature Transform) feature amount, SURF (Speed-Up Robust Features) feature amount, HOG (Histograms of Oriented Gradients) feature amount, etc. The machine determination unit 121 determines the inspection result in which a learned relationship holds for the feature amount of the acquired inspection image.

[0066] The machine determination unit 121 may perform the inspection judgment using so-called AI (Artificial Intelligence) technology, or may perform the inspection judgment using an algorithm determined through research by engineers, etc. In short, the machine determination unit 121 may perform the inspection judgment without bothering human hands, and the higher the accuracy of the judgment (the more it matches the judgment of an expert inspector), the more desirable it is.

[0067] When the judgment is a predetermined result, the mechanical judgment unit 121 transmits the test image used for the judgment to the instructor terminal 30. In this embodiment, the predetermined result is a result of the test failing, which is the same as the result when the NG button E1 shown in FIG. 8 is pressed. Note that the predetermined result may be, for example, the judgment result of an inspection item that has a significant impact on reducing the risk of food poisoning even if the inspection is passed. The mechanical judgment unit 121 transmits the inspection image input for the inspection item that was judged to be a failure to the instructor terminal 30.

[0068] The machine judgment display unit 301 of the instructor terminal 30 displays the judgment result transmitted from the server device 10. Fig. 13 shows an example of the displayed judgment results. In the example of Fig. 13, the machine judgment display unit 301 displays the NG judgment result, the inspection image, and a comment for the inspection item "K1-3", and the NG judgment result, the inspection image, and a comment for the inspection item "K1-8".

[0069] The displayed inspection image is viewed by an instructor using the instructor terminal 30. In this way, the machine judgment unit 121 transmits the inspection image used in the judgment, thereby presenting the inspection image used in the judgment to the instructor. The machine judgment unit 121 is an example of the "presentation unit" of the present invention. The machine judgment display unit 301 displays an instruction creation button B3 for displaying an instruction screen for creating instructions for an inspection item.

[0070] The trainer looks at the presented test image and comments and judges whether the judgment result is appropriate or not. If the trainer judges that the judgment result of a certain test item is not appropriate, he or she presses the instruction creation button B3 for that test item to display the instruction screen. Fig. 14 shows an example of the displayed instruction screen. In the example of Fig. 14, the machine judgment display unit 301 displays an instruction input field E8 for the trainer to input instructions below the NG judgment result, the test image, and the comment for the test item "K1-3" shown in Fig. 13.

[0071] When the instructor inputs instructions in the instruction input field E8 and presses the instruction confirmation button B4, the instruction operation receiving unit 302 receives these operations, i.e., the instruction operation by the instructor. The instruction operation receiving unit 302 supplies instruction data indicating the instructions input by the received instruction operation to the examination instruction transmission unit 303. The examination instruction transmission unit 303 transmits the instructions indicated by the supplied instruction data to the inspector using the user terminal 20. In this embodiment, the examination instruction transmission unit 303 transmits the supplied instruction data to the server device 10.

[0072] The inspection instruction transfer unit 122 of the server device 10 transfers (sends) the transmitted instruction data, i.e., instruction data indicating instructions to be given by the instructor based on the inspection image presented by the machine determination unit 121, to the user terminal 20. The user terminal 20, which is the transfer destination, is a destination associated with the facility. The instruction data in this case is an example of the "first instruction data" of the present invention, and the inspection instruction transfer unit 122 is an example of the "first transmission unit" of the present invention. The inspection processing unit 203 of the user terminal 20 displays an inspection screen including instructions indicated by the transmitted instruction data.

[0073] Fig. 15 shows an example of a displayed inspection screen. In the example of Fig. 15, the inspection processing unit 203 displays the NG judgment result of the inspection item "K1-3" shown in Fig. 13, the first inspection image, and the first comment below, instructions from the instructor, an input field E9 for the second inspection image that has been retaken, and an input field E10 for the second comment. Although not shown in Fig. 15, when the screen is scrolled down, a temporary pause button B1 and an input completion button B2 shown in Fig. 9 are displayed, and input is performed in the same manner as in the example of Fig. 9.

[0074] Based on the above configuration, the server device 10 and the user terminal 20 perform a storage process for storing the examination information based on the printed correspondence information, and a training support process for supporting remote training by a trainer. Fig. 16 shows an example of the operation procedure of each device in the storage process. The operation procedure of Fig. 16 is started, for example, when the operator of the examination support system 1 performs an operation to instruct the server device 10 to print the corresponding information for each facility from an external device or the like.

[0075] First, the server device 10 (print control unit 102) outputs print data for printing coded corresponding information (in this embodiment, a two-dimensional code indicating the identification information of the facility) and prints the corresponding information on a medium such as paper (step S11). Next, the user terminal 20 (code reading unit 201) reads the two-dimensional code which is the printed corresponding information (step S21). Next, the user terminal 20 (code reading unit 201) transmits the read information (facility ID indicated by the two-dimensional code) which is the read information to the server device 10 (step S22).

[0076] The server device 10 (read information acquisition unit 103) acquires the transmitted read information (step S23). Next, the server device 10 (test information output unit 104) reads out the test information stored in association with the acquired read information (facility ID) from the facility information storage unit 101 (step S24). Next, the server device 10 (test information output unit 104) outputs the read out test information to the user terminal 20 that is the sender of the read information (step S25). The user terminal 20 (test information storage unit 202) stores the output test information in association with the facility ID (step S26).

[0077] Fig. 17 shows an example of the operation procedure of each device in the training support process. The operation procedure in Fig. 17 is started, for example, when an inspector operates the user terminal 20 to display an inspection screen. First, the user terminal 20 (inspection processing unit 203) displays an inspection screen including the inspection items stored in the operation procedure in Fig. 16, and performs an inspection process to accept an input of an inspection by the inspector (step S31).

[0078] Next, the user terminal 20 (inspection processing unit 203) transmits the inspection image input by the inspection process to the server device 10 (step S32). The server device 10 (inspection image acquisition unit 111) acquires the transmitted inspection image (step S33). Next, the server device 10 (inspection image evaluation unit 112) evaluates the quality of the acquired inspection image as an image for inspection based on the feature amount of the inspection image (step S34).

[0079] In the example of Fig. 17, it is assumed that a predetermined quality is evaluated in step S34. In this case, the server device 10 (re-photographing instruction unit 113) transmits instruction data indicating an instruction to re-photograph the inspection target shown in the inspection image used for the evaluation to the user terminal 20 (step S41). The user terminal 20 (inspection processing unit 203) re-photographs the inspection target by the operation of the inspector (step S42), and transmits the photographed image, i.e., the inspection image, to the server device 10 (step S43).

[0080] In steps S44 and S45, the server device 10 performs the same operations as in steps S33 and S34. In the example of Fig. 17, it is assumed that the image is not evaluated as being of a predetermined quality in step S34. In this case, the server device 10 (machine determination unit 121) performs a determination of the inspection of the inspection target shown in the inspection image based on the feature amount of the inspection image acquired in step S44 (step S46).

[0081] In the example of Fig. 17, it is assumed that the judgment in step S46 is the above-mentioned predetermined result (e.g., a fail result). In this case, the server device 10 (machine judgment unit 121) transmits the input inspection image for the inspection item judged to be the predetermined result to the instructor terminal 30 (step S51). The instructor terminal 30 (machine judgment display unit 301) displays the transmitted judgment result (step S52).

[0082] Next, the instructor terminal 30 (instruction operation receiving unit 302) receives the instruction operation by the instructor (step S53). Then, the instructor terminal 30 (test instruction transmitting unit 303) transmits instruction data indicating the instruction input by the instructor to the user terminal 20 via the server device 10, thereby transmitting the instruction from the instructor to the inspector using the user terminal 20 (step S54).

[0083] The server device 10 (examination instruction transfer unit 122) transfers (sends) the transmitted instruction data, i.e., instruction data indicating the instruction to be given by the instructor based on the presented examination image, to the user terminal 20 (step S55). The user terminal 20 (examination processing unit 203) displays an examination screen including the instructions indicated by the transmitted instruction data (step S56). Then, the user terminal 20 (examination processing unit 203) displays the examination screen and performs examination processing to accept examination input by the examiner (step S57).

[0084] As described above, this embodiment allows the inspector to remotely participate in the food hygiene inspection without the need for the instructor to go to the site. Also, in this embodiment, if the quality of the image for inspection is determined to be low, the image is retaken and a higher quality inspection image is presented to the instructor. This reduces the need for the instructor to instruct retaking of the inspection image, thereby reducing the instructor's workload.

[0085] In this embodiment, a sample image for photographing the subject is displayed as shown in FIG. 12. This makes it possible to improve the quality of the re-photographed examination image as an image for examination, compared to when the sample image is not displayed. In this embodiment, the server device 10 performs the examination mechanically, and only the examination image that is a predetermined result is presented to the instructor. This makes it possible to reduce the number of examination items instructed by the instructor, and to increase the number of facilities where the instructor can provide support remotely, compared to when all examination images are presented to the instructor.

[0086] [2] Variations The above-described embodiment is merely one example of the implementation of the present invention, and may be modified as follows. Furthermore, the embodiment and each modification may be combined as necessary.

[0087] [2-1] Sample image In the embodiment, if the inspection target is the same, the same sample image is transmitted to the user terminal 20 used in any facility, but the sample image may be different depending on the facility. The re-photographing instruction unit 113 in this modified example transmits data for displaying a sample image (sample image for photographing the inspection target) corresponding to the type of facility on the user terminal 20 as instruction data.

[0088] The re-photographing instruction unit 113 uses, for example, a sample image table in which the type of facility is associated with a sample image. An example of the sample image table is shown in Fig. 18. In the example of Fig. 18, the types of facilities (types classified according to the size of the facilities) of "large scale", "medium scale", and "small scale" are associated with the test items and sample images.

[0089] The re-photographing instruction unit 113 stores, for example, a list in which the identification information of the user terminal 20 is associated with the size of the facility associated with the user terminal 20. Then, the re-photographing instruction unit 113 reads out a sample image that is associated in the sample image table with the size of the facility associated with the identification information of the user terminal 20 that is the destination of the instruction data, and transmits the read sample image to the user terminal 20 as a sample image according to the type of the facility.

[0090] The type of facility is not limited to the type classified by the size of the facility. For example, the type of business conducted at the facility may be used as the type of facility. For example, in a case where a facility is provided with equipment specific to a particular business type, the re-photographing instruction unit 113 reads and transmits a sample image using a sample image table in which sample images of the equipment are associated with the business type.

[0091] The area where the facility is located may also be used as the type of facility. For example, when the equipment installed in cold regions and warm regions is different, the re-photographing instruction unit 113 reads and transmits the sample image using a sample image table in which the area and the sample image of the equipment installed in that area are associated. In addition to the equipment, tools or ingredients, etc. may differ depending on the type of facility. If so, the re-photographing instruction unit 113 reads out and transmits sample images associated with those types.

[0092] Whichever type is used, an image showing equipment specific to that type of facility is used as the sample image, so a sample image that is more useful for re-photographing can be displayed compared to a case where a uniform sample image is used regardless of the type of facility.

[0093] [2-2] Reliability of machine judgment The machine judgment unit 121 may perform so-called feedback using an actually judged inspection image and a judgment result judged by a person based on the inspection image as teacher data. In this case, the reliability of the judgment improves as the amount of feedback increases. In other words, the reliability of the judgment is not high when the amount of feedback is small.

[0094] Therefore, the fewer the inspection results for an inspection object to be judged, the easier it is for the machine judgment unit 121 to judge the inspection result of the inspection object as the above-mentioned predetermined result. The machine judgment unit 121 uses, for example, a result table that associates the inspection results with the predetermined results. An example of a result table is shown in Fig. 19. In the example of Fig. 19, the inspection results of "less than Th1", "between Th1 and Th2", and "between Th2 and above" are associated with the predetermined results of "fail two consecutive times", "fail", and "fail and specific inspection target".

[0095] When the mechanical judgment unit 121 first performs an inspection judgment on an inspection object, it adds "1" to the number of inspections associated with the inspection object to update the inspection record. Then, when the mechanical judgment unit 121 performs a new inspection judgment on the inspection object, it reads out the inspection record of the inspection object, and judges whether the inspection judgment result is a predetermined result by using a predetermined result associated with the read out inspection record in the result table.

[0096] For example, when the inspection record is less than Th1, the mechanical judgment unit 121 judges that the predetermined result is "two consecutive failures." Also, when the inspection record is greater than or equal to Th2, the mechanical judgment unit 121 judges that the predetermined result is "failure and specific inspection target."

[0097] According to this modification, the more reliable the test subject, the less likely it is to be presented to the trainer, which reduces the trainer's burden compared to when the predetermined results are uniform. Also, the more reliable the test subject, the more likely it is to be presented to the trainer, which reduces the possibility of an inappropriate test result being overlooked compared to when the predetermined results are uniform.

[0098] [2-3] Inspection image In the embodiment, the inspection image is a photograph, i.e., a still image, but it may be a video. In the case of a video, the inspection image evaluation unit 112 may evaluate the quality of the inspection image based on a still image cut out from the video, or may evaluate the quality of the inspection image based on a video of a certain period. The machine judgment unit 121 may also make an inspection judgment based on a still image cut out from the video, or may make an inspection judgment based on a video of a certain period.

[0099] [2-4] Wide-angle camera A wide-angle camera may be used to capture the inspection target. A wide-angle camera is a camera that uses a so-called wide-angle lens that has a wider angle of view than a standard lens. For example, by installing a wide-angle camera on the ceiling of a store, it is possible to simultaneously capture images of multiple inspection targets such as a kitchen, a dishwashing area, and a hand washing area. The wide-angle camera transmits the captured image to the server device 10.

[0100] The inspection image acquisition unit 111 acquires the transmitted image, i.e., the image of a wide-angle camera installed in the facility. Since an object is distorted in an image taken by a wide-angle camera compared to an image taken by a standard lens, the inspection image evaluation unit 112 corrects the distortion of the inspection target in the image taken by the wide-angle camera. The machine determination unit 121, for example, cuts out a portion of the inspection target from the image taken by the wide-angle camera.

[0101] The machine determination unit 121 cuts out a portion including an object recognized as an inspection target by, for example, image recognition processing. The machine determination unit 121 may cut out a range set manually as a portion including the inspection target. The machine determination unit 121 performs coordinate conversion using conversion parameters between the coordinates of each pixel included in the cut-out image in the image of the wide-angle camera and the coordinates when the distortion and tilt are corrected, and generates a corrected image.

[0102] The mechanical judgment unit 121 uses the generated corrected image as an inspection image to make an inspection judgment on the inspection object shown in the inspection image. The mechanical judgment unit 121 presents the inspection image used for the judgment to the instructor by transmitting the inspection image used for the judgment, i.e., an image in which the distortion of the inspection object shown in the image used for the judgment has been corrected.

[0103] A 360-degree camera may be used as the wide-angle camera. A 360-degree camera is a camera that can capture images in any direction, including front, back, left, and right, by capturing images through multiple lenses and stitching the captured images together. A 360-degree camera has a horizontal viewing angle of 360 degrees and a vertical viewing angle of 180 degrees, for example. For example, by installing a 360-degree camera on the ceiling of a store, it is possible to simultaneously capture images of multiple inspection targets, such as the kitchen, dishwashing area, and hand washing area.

[0104] As described above, when using wide-angle cameras (including 360-degree cameras), the number of cameras that need to be installed can be reduced when installing cameras in a facility to photograph an object to be inspected, compared to cameras with standard lenses. Also, wide-angle cameras tend to capture distorted images of objects compared to cameras with standard lenses, but in this modified example, image distortion is corrected as described above, so the state of the object to be inspected can be grasped more accurately than when this correction is not performed.

[0105] [2-5] Processing before presentation of test images The process performed before presenting the test image to the trainer is not limited to that described in the embodiment. For example, in the embodiment, when the judgment by the mechanical judgment unit 121 is a predetermined result, the test image used for the judgment is presented to the trainer and instructions are given remotely, but regardless of whether the judgment is a predetermined result or not, the mechanical judgment unit 121 may present the test image used for the judgment to the trainer and give instructions remotely.

[0106] In the embodiment, when the acquired inspection image is evaluated as having a predetermined quality by the inspection image evaluation unit 112, the re-photographing instruction unit 113 transmits instruction data indicating an instruction to re-photograph, but the evaluation of the inspection image and the instruction to re-photograph may not be performed. In any case, as in the embodiment, the instructor can remotely support the food sanitation inspection by the inspector without visiting the site.

[0107] [2-6] Instruction data The instruction data transmitted by the re-photographing instruction unit 113 and the examination instruction transmission unit 303 is not limited to data showing a character string and an image, but may be data showing, for example, a voice. In this case, the instruction by the instructor can be conveyed to the person in charge at the site by outputting the voice shown by the instruction data from a speaker or the like in the user terminal 20.

[0108] [2-7] Instruction timing There is no particular limitation on the timing at which the re-photographing instruction unit 113 and the examination instruction transmission unit 303 transmit the instruction data. For example, the instructor may input instructions while the examination is being performed on-site, and the instruction data may be transmitted in real time. In addition, in the case of the examination instruction transmission unit 303, after the examination on-site is completed, the instructor may view the examination results and input instructions at a time convenient for him / her, and transmit the instruction data at a timing different from that of the examination on-site.

[0109] By shifting the timing of instructions as described above, both the person in charge of the inspection and the instructor can carry out the work at a time that is convenient for them. In the inspection support system 1, the instructor can provide instruction not only without geographical restrictions, but also without time restrictions. In addition, a business providing a remote inspection support service may charge different fees for real-time remote instructions and remote instructions with different timing (it is preferable to charge a higher fee for real-time instructions).

[0110] [2-8] Destination of instruction data In the embodiment, the inspection instruction transfer unit 122 and the re-photographing instruction unit 113 transmit the instruction data to the user terminal 20 used by the inspector, but the destination of the instruction data is not limited to this. For example, the destination may be the email address or SNS (Social Networking Service) account of the inspector, or the destination may be a terminal used by a manager who manages a group of restaurants.

[0111] [2-9] Destination for sending inspection images In the embodiment, there is one instructor, but there may be multiple instructors. In that case, the machine judgment unit 121 may select the instructor terminal 30 to which the input inspection image for the inspection item judged as failed is transmitted. For example, it is assumed that multiple instructors each have a different specialty.

[0112] The specialty area is, for example, the type of business of the facility where the inspection is performed (fast food, restaurant, izakaya, food factory, food distribution, etc.). For example, some instructors are knowledgeable about inspections in fast food establishments, while others are knowledgeable about inspections in izakaya establishments. In addition, instructors may have a specialty area for each piece of equipment they inspect. For example, some instructors are knowledgeable about inspections related to water, while others are knowledgeable about inspections related to food ingredients.

[0113] In either case, the machine judgment unit 121 transmits the inspection image to the instructor terminal 30 used by an instructor with expertise in the field of the inspection item that was determined to be failed. This allows the inspector to receive remote support from an instructor who is more knowledgeable about the relevant inspection work than if the inspection image was always transmitted to a specific instructor terminal 30.

[0114] In addition to the above, the machine judgment unit 121 may transmit the inspection image to an instructor terminal 30 used by an instructor with a level according to the difficulty of judging the inspection item judged as failing. The level here refers to the level of skill (knowledge, technique, etc.) required to perform the inspection work correctly. This makes it easier for an instructor with sufficient skills to provide remote support for difficult inspection items, compared to a case where the inspection image is always transmitted to a specific instructor terminal 30.

[0115] [2-10] Subject of inspection The subject of the food hygiene inspection may be a local person in charge or a remote instructor. Even when the instructor is the main inspector and inspects remotely, the instructor operates the instructor terminal 30 to instruct, for example, how to photograph the inspection object. When an image of the inspection object is photographed according to the instruction, the quality of the inspection image is evaluated, an instruction to photograph again is given, the inspection image is presented to the instructor, and the instructor gives instructions, as in the embodiment.

[0116] [2-11] Devices that realize each function The devices that realize the functions shown in Fig. 4 are not limited to the devices described above. For example, the functions realized by the server device 10 may be shared and realized by two or more devices, or the functions realized by the user terminal 20 or the instructor terminal 30 may be shared and realized by two or more devices. In either case, it is sufficient that the functions shown in Fig. 4 are realized by the entire test support system 1.

[0117] [2-12] Invention Categories The present invention can be understood as an information processing system (of which the examination support system 1 is an example) including each information processing device, such as the above-mentioned server device 10, user terminal 20, and instructor terminal 30, as well as each information processing device. The present invention can also be understood as an information processing method for realizing the processing performed by each information processing device, or as a program for causing a computer that controls each information processing device to function. The program understood as the present invention may be provided in the form of a recording medium such as an optical disk on which the program is stored, or may be provided in the form of a program that is downloaded to a computer via a network such as the Internet and the downloaded program is installed and made available. [Explanation of symbols]

[0118] 1...inspection support system, 3...printing device, 10...server device, 20...user terminal, 30...instructor terminal, 101...facility information storage unit, 102...printing control unit, 103...read information acquisition unit, 104...inspection information output unit, 111...inspection image acquisition unit, 112...inspection image evaluation unit, 113...re-photographing instruction unit, 121...mechanical judgment unit, 122...inspection instruction transfer unit, 201...code reading unit, 202...inspection information storage unit, 203...inspection processing unit, 301...mechanical judgment display unit, 302...instruction operation reception unit, 303...inspection instruction transmission unit.

Claims

1. an image acquisition unit that acquires images of employees, ingredients, utensils, tableware, storage conditions, and / or environment that are subject to food hygiene inspection in a food handling facility; a determination unit that determines whether or not an inspection is performed on the subject of the inspection shown in the acquired image based on a feature amount of the image; a presentation unit that presents an image used for the judgment to an instructor of the test when the judgment is a predetermined result; a first transmission unit that transmits first instruction data indicating an instruction to be given by the instructor based on the presented image to a destination associated with the facility; An information processing device comprising:

2. The judgment unit uses an AI that has learned the relationship between the feature amount of an inspection image and a correct inspection result using training data of an inspection image used in a past inspection and an inspection result performed by a person based on the inspection image, and judges the inspection of the inspection target shown in the image acquired by the image acquisition unit. The information processing device according to claim 1 .

3. an evaluation unit that evaluates the quality of the image as an inspection image based on the feature amount of the acquired image; and a second transmission unit that, when the acquired image is evaluated to have a predetermined quality, transmits to the destination second instruction data indicating an instruction to re-photograph the object of examination shown in the image used for the evaluation. The information processing device according to claim 1 .

4. The second transmission unit transmits data for displaying a sample image for photographing the object of the inspection as the second instruction data. The information processing device according to claim 3 .

5. The second transmission unit transmits data for displaying the sample image according to the type of the facility as the second instruction data. The information processing device according to claim 4.

6. The presenting unit makes it easier to determine the test result of the test subject as the predetermined result as the test result of the test subject is smaller in number. The information processing device according to claim 1 .

7. The image acquisition unit acquires an image from a wide-angle camera installed in the facility, The presentation unit presents to the instructor an image in which distortion of the subject of the inspection shown in the image used for the judgment has been corrected. The information processing device according to claim 1 .

8. The first transmission unit transmits data for displaying a sample image for photographing the object of the inspection as the first instruction data. The information processing device according to claim 1 .

Images