Teacher data creation system and teacher data creation method

Abstract

To provide a teacher data generation system which can generate teacher data for use for constructing a machine learning model.SOLUTION: A storing / retrieving management system 10 functions as the teacher data generation system which generates teacher data for use for constructing a machine learning model for identifying objects. The storing / retrieving management system 10 comprises an input unit 51 to which label information on an object is inputted by a user, an imaging unit 31 which captures a moving image including the object, a display unit 52 on which the captured moving image is displayed, and a control unit 39 which displays a guide frame within the displayed moving image to give the user an instruction to locate the object within the guide frame. The control unit 39 associates the inputted label information with a still image including the object, which is captured by the imaging unit 31 after the instruction is given, to generate teacher data and changes a display mode of the guide frame each time the still image is captured.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a teacher data creation system and a teacher data creation method.

Background Art

[0002] Conventionally, techniques related to object recognition and identification have been proposed. Patent Document 1 discloses an information processing apparatus capable of efficiently recognizing different types of sold goods.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a teacher data creation system and the like that can create teacher data used to construct a machine learning model.

Means for Solving the Problems

[0005] A teacher data creation system according to an aspect of the present disclosure is a teacher data creation system that creates teacher data used to construct a machine learning model for identifying an object, including an input unit into which label information of the object is input by a user, a photographing unit that photographs a video in which the object appears, a display unit on which the photographed video is displayed, and a control unit that gives an instruction to the user to position the object within a guide frame by displaying the guide frame within the displayed video. After the instruction is given, the control unit creates the teacher data by associating the input label information with a still image in which the object photographed by the photographing unit appears, and changes a display mode of the guide frame each time the still image is photographed.

Effects of the Invention

[0006] A teacher data creation system and the like according to one aspect of the present disclosure can create teacher data used for constructing a machine learning model.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 10F

DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be specifically described with reference to the drawings. Note that each of the embodiments described below shows general or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, the components not described in the independent claims are described as optional components.

[0009] Note that each drawing is a schematic diagram and is not necessarily drawn precisely. Also, in each drawing, the same reference numerals are given to substantially the same configurations, and redundant descriptions may be omitted or simplified.

[0010] (Embodiment) [Configuration] First, the configuration of the inventory management system according to the embodiment will be described. FIG. 1 is a block diagram showing the functional configuration of the inventory management system according to the embodiment.

[0011] The inventory management system 10 shown in FIG. 1 is a system for managing foods stored in the refrigerator 20. The inventory management system 10 includes a refrigerator 20, an inventory management device 30, a server device 40, and an information terminal 50.

[0012] The refrigerator 20 is an example of a storage compartment, installed in a user's home or the like, and refrigerates food. FIG. 2 is an external view of the refrigerator 20 (and the incoming / outgoing management device 30). As shown in FIG. 2, the refrigerator 20 includes a main body 21, a door 22 for accessing the first storage compartment 23 inside the main body 21, a plurality of shelves 24 provided in the first storage compartment 23, and a drawer 25. When the door 22 is opened, the refrigerator 20 can be in a state where food can be placed on each of the plurality of shelves 24 (that is, food can be stored in the first storage compartment 23). Also, when the door 22 is opened, the refrigerator 20 can be in a state where food can be stored in the door pocket 22a provided inside the door 22. Further, when the drawer 25 is opened, the refrigerator 20 can be in a state where food can be stored in the second storage compartment 26 inside the drawer 25. In the example of FIG. 2, the refrigerator 20 has two doors 22 that cover the first storage compartment 23, but there may be one door 22 that covers the first storage compartment 23.

[0013] The incoming / outgoing management device 30 is, for example, attached to the upper surface of the refrigerator 20 and performs information processing for managing the food stored in the refrigerator 20. The incoming / outgoing management device 30 is, for example, retrofitted to an existing refrigerator 20, but may also be built into the refrigerator 20. That is, the incoming / outgoing management device 30 may be a device separate from the refrigerator 20 or a device manufactured integrally with the refrigerator 20. As shown in FIG. 1, the incoming / outgoing management device 30 includes a photographing unit 31, a distance measuring unit 32, an information processing unit 33, a storage unit 34, a communication unit 35, and a notification unit 36.

[0014] The photographing unit 31 photographs a video showing the state when food is put into or taken out of the refrigerator 20 from above the refrigerator 20. That is, the photographing unit 31, for example, photographs a video looking down on the refrigerator 20 with the door 22 open. The video includes the door 22, the door pocket 22a, the first storage compartment 23, the shelves 24, the drawer 25, and the floor of the indoor space in front of the refrigerator 20 as photographing targets. The photographing unit 31 is realized, for example, by a camera having a fish-eye lens, but may also be realized by a camera having a normal lens.

[0015] Note that the photographing unit 31 may generate a plurality of images (still images) by photographing the state of an object being put into or taken out of the refrigerator 20 at a preset time interval, and it is not necessarily required to photograph a moving image. Also, although the photographing unit 31 is attached to the housing of the inventory management device 30, it may be attached to the refrigerator 20 separately from the housing of the inventory management device 30. That is, the photographing unit 31 may be retrofitted to the refrigerator 20. In this case, the photographing unit 31 has a magnet, a suction cup, a clip, or the like as a structure that can be retrofitted to the refrigerator 20.

[0016] The distance measuring unit 32 measures the distance from the distance measuring unit 32 to the door 22 at the upper part of the refrigerator 20. That is, the distance measuring unit 32 measures the degree of opening of the door 22. The distance measuring unit 32 is realized by, for example, an infrared distance measuring sensor. The distance measuring unit 32 can separately measure the distance from the distance measuring unit 32 to the right door 22 and the distance from the distance measuring unit 32 to the left door 22.

[0017] Note that, as will be described later, the distance measuring unit 32 is used to determine the opening and closing of the door 22. The inventory management device 30 may be provided with another sensor for determining the opening and closing of the door 22, such as an angle sensor, instead of the distance measuring unit 32. Also, the sensor for determining the opening and closing of the door 22 may be provided by the refrigerator 20. In this case, the inventory management device 30 may not be provided with a sensor for determining the opening and closing of the door 22.

[0018] The information processing unit 33 performs information processing for managing the food stored in the refrigerator 20. The information processing unit 33 is realized by, for example, a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 33 has, as functional components, a determination unit 37, a trailing unit 38, and a control unit 39. The functions of the determination unit 37, the trailing unit 38, and the control unit 39 are realized, for example, by a microcomputer or the like constituting the information processing unit 33 executing a computer program stored in the storage unit 34. Details of the functions of the determination unit 37, the trailing unit 38, and the control unit 39 will be described later.

[0019] The storage unit 34 is a storage device that stores the computer programs executed by the information processing unit 33 and various types of information necessary for the above information processing (such as the setting information and food management information described later). A machine learning model for determining (identifying) the type of food is also stored in the storage unit 34. The storage unit 34 is realized by, for example, a semiconductor memory.

[0020] The communication unit 35 is a communication circuit for the incoming / outgoing management device 30 to communicate with the server device 40 via the wide area communication network 60. The communication unit 35 is, for example, a wireless communication circuit that performs wireless communication, but may also be a wired communication circuit that performs wired communication. There is no particular limitation on the communication standard of the communication performed by the communication unit 35.

[0021] The notification unit 36 performs notification when it is impossible to determine the type of food put into or taken out of the refrigerator 20. The notification unit 36 is realized by, for example, a speaker or a buzzer, and performs notification by emitting sound, but may also be realized by a light emitting element such as an LED (Light Emitting Diode) and perform notification by emitting light.

[0022] The server device 40 is a computer located outside the facility where the refrigerator 20 is installed, and specifically, is a cloud server. The server device 40 performs information processing for presenting information about the food stored in the refrigerator 20. In addition, the server device 40 is an example of a notification device, and performs information processing for notifying information about the food stored in the refrigerator 20. The server device 40 includes a communication unit 41, an information processing unit 42, and a storage unit 43.

[0023] The communication unit 41 is a communication circuit for the server device 40 to communicate with the incoming / outgoing management device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There is no particular limitation on the communication standard of the communication performed by the communication unit 41.

[0024] The information processing unit 42 performs information processing for presenting or notifying information regarding the food stored in the refrigerator 20. The information processing unit 42 is realized by, for example, a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 42 has, as functional components, a presentation unit 44 and a notification unit 45. The functions of the presentation unit 44 and the notification unit 45 are realized, for example, by a microcomputer or the like constituting the information processing unit 42 executing a computer program stored in the storage unit 43. Details of the functions of the presentation unit 44 and the notification unit 45 will be described later.

[0025] The storage unit 43 is a storage device that stores the computer program executed by the information processing unit 42 and various types of information necessary for the above information processing. The storage unit 43 is realized by, for example, a semiconductor memory.

[0026] The information terminal 50 is an information terminal owned by the user. The information terminal 50 is used as a user interface for the user to receive notifications from the server device 40. Further, the information terminal 50 is used as a user interface when creating teacher data (described later) for constructing a machine learning model for identifying the type of food. The information terminal 50 is, for example, a portable information terminal such as a smartphone or a tablet terminal, but may also be a stationary information terminal such as a personal computer. The information terminal 50 includes an input unit 51, a display unit 52, a communication unit 53, an information processing unit 54, a storage unit 55, and a photographing unit 56.

[0027] The input unit 51 accepts manual input or voice input from the user. In other words, information is input to the input unit 51 manually or by voice. The input unit 51 is realized by, for example, a touch panel or a microphone.

[0028] The display unit 52 displays a video (or a still image) for the user to view. The display unit 52 is realized by, for example, a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel.

[0029] The communication unit 53 is a communication circuit for the information terminal 50 to communicate with the incoming and outgoing inventory management device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 53 may communicate with the incoming and outgoing inventory management device 30 through a local communication network. The communication unit 53 is, for example, a wireless communication circuit that performs wireless communication, but may also be a wired communication circuit that performs wired communication. There is no particular limitation on the communication standard of the communication performed by the communication unit 53.

[0030] The information processing unit 54 performs information processing for creating teacher data and the like. The information processing unit 54 is realized by, for example, a microcomputer, but may also be realized by a processor or a dedicated circuit. The functions of the information processing unit 54 are realized, for example, by a microcomputer or the like constituting the information processing unit 54 executing a computer program stored in the storage unit 55.

[0031] The storage unit 55 is a storage device that stores the computer program executed by the information processing unit 54 and various types of information necessary for the above information processing. The storage unit 55 is realized by, for example, a semiconductor memory.

[0032] The photographing unit 56 is a camera realized by an image sensor and an optical system such as a lens.

[0033] [Operation Example 1] Next, as operation example 1 of the incoming and outgoing inventory management system 10, the management operation of food incoming or outgoing will be described. FIG. 3 is a flowchart of operation example 1 of the incoming and outgoing inventory management system 10.

[0034] First, when the determination unit 37 determines that at least one of the two doors 22 has opened based on the measurement result of the distance measurement unit 32 (S11), it activates the imaging unit 31 and starts imaging a video by the imaging unit 31 (S12). FIG. 4 is a diagram showing an example of the video imaged by the imaging unit 31. As shown in FIG. 4, the imaging range of the imaging unit 31 includes the main body 21 (the first storage chamber 23), the inside of the two doors 22 (the door pockets 22a), and the upper shelf 24a and the lower shelf 24b provided in the first storage chamber 23. Note that FIG. 4 also shows a boundary line (dashed line) used for food storage determination or food removal determination. This boundary line is not imaged by the imaging unit 31 but is virtual.

[0035] Next, the tracking unit 38 detects the food shown in the video (S13) and tracks the detected food (S14). For food detection, various existing algorithms such as R-CNN (Regions with Convolutional Neural Networks) are used. The tracking unit 38, for example, also detects food in the frame following the frame (hereinafter also referred to as the previous frame) in which the food detected in step S13 (hereinafter also referred to as the tracking target) is shown (hereinafter also referred to as the current frame). The tracking unit 38 determines, as the movement destination of the tracking target, the food that is closest to the position of the tracking target in the previous frame among the food shown in the current frame. By repeating such processing, the tracking unit 38 can track the food detected in step S13.

[0036] Note that the tracking unit 38 may further extract the image feature amount of the food detected in step S13 and use the image feature amount as auxiliary information to track the food. The tracking unit 38 may determine the movement destination of the tracking target, for example, based on the distance of the food detected in the current frame from the tracking target and the similarity of the image feature amount detected in the current frame to the tracking target. Thereby, the tracking accuracy of the food can be improved.

[0037] In addition, such a method of tracking food is just an example, and other existing algorithms may be used for food tracking.

[0038] Next, the determination unit 37 performs an inbound determination or an outbound determination of the food (S15). In the video, when the food being tracked moves from the inside of the refrigerator 20 to the outside of the refrigerator 20 (crosses the boundary line and moves into the indoor side of the room where the refrigerator 20 is installed), the determination unit 37 determines that the food has been shipped out. Also, in the video, when the tracked food moves from the outside of the refrigerator 20 to the inside of the refrigerator 20 and crosses the boundary line, the determination unit 37 determines that the food has been stocked. Note that the boundary line is determined in advance empirically or experimentally.

[0039] Next, the determination unit 37 determines the type of the food that has been stocked or shipped out (S16). The determination unit 37 determines the type of food using, for example, a machine learning model. The type of food is, for example, beverages, vegetables, meat, etc., but it may also be a finer type. Note that the type of food may be determined in accordance with the detection of the food in step S13. In this case, step S16 can be omitted.

[0040] Next, based on the determination results in steps S15 to S16, the determination unit 37 updates the food management information stored in the storage unit 34 (S17). FIG. 5 is a diagram showing an example of the food management information. The food management information is information indicating the breakdown of the food stored in the refrigerator 20. As shown in FIG. 5, the food management information is, for example, information indicating the type and quantity of the food stored in the refrigerator 20. When the determination unit 37 determines that beverages have been shipped out in steps S15 to S16, for example, the determination unit 37 decreases the quantity of beverages in the food management information by one. Also, when the determination unit 37 determines that vegetables have been stocked in steps S15 to S16, the determination unit 37 increases the quantity of vegetables in the food management information by one.

[0041] After that, when it is determined by the determination unit 37 that both of the two doors 22 are closed based on the measurement result of the distance measurement unit 32 (S18), the operation ends.

[0042] As described above, the warehousing and shipping management system 10 (warehousing and shipping management device 30) determines the warehousing and shipping (warehousing of food into the refrigerator 20 or shipping of food out of the refrigerator 20) of food based on whether the food shown in the video exceeds the set boundary line.

[0043] In addition, when the determination of the food type fails in step S16, that is, when the food type cannot be determined, the determination unit 37 may use the notification unit 36 to notify that the food type cannot be determined. For example, when the notification unit 36 is realized by a speaker, the notification unit 36 outputs an audio message or the like indicating that the food type cannot be determined. When the notification unit 36 is realized by a light-emitting element, the notification unit 36 notifies by light emission that the food type cannot be determined.

[0044] In this way, if it is notified when the determination of the food type fails, the user can take measures such as manually correcting the food management information.

[0045] [Operation Example 2] Next, as operation example 2 of the warehousing and shipping management system 10, the presentation operation of the food stored in the refrigerator 20 will be described. FIG. 6 is a sequence diagram of operation example 2 of the warehousing and shipping management system 10.

[0046] The communication unit 35 of the warehousing and shipping management device 30 transmits the food management information to the server device 40 (S21). The communication unit 41 of the server device 40 receives the food management information, and the presentation unit 44 stores the received food management information in the storage unit 43 (S22). The communication unit 35 of the warehousing and shipping management device 30 transmits the food management information to the server device 40, for example, every time a warehousing determination or a shipping determination of food is performed. Alternatively, the communication unit 35 of the warehousing and shipping management device 30 transmits the food management information to the server device 40 every time the opened door 22 is closed. Thereby, the latest food management information is stored in the storage unit 43 of the server device 40. Note that the food management information may be transmitted from the warehousing and shipping management device 30 to the server device 40 periodically, such as once every predetermined time.

[0047] On the one hand, when the user makes a predetermined input to the input unit 51 of the information terminal 50 from a location outside the home or the like to check the food stored in the refrigerator 20 (the content of the food management information), the input unit 51 accepts such a predetermined input (S23), and the information processing unit 54 transmits a presentation request to the server device 40 based on the accepted predetermined input (S24). More specifically, the transmission of the presentation request is performed by the communication unit 53.

[0048] The communication unit 41 of the server device 40 receives the presentation request. The presentation unit 44 transmits presentation information for presenting the content of the food management information stored in the storage unit 43 to the information terminal 50 based on the received presentation request (S25). More specifically, the transmission of the presentation information is performed by the communication unit 41.

[0049] The communication unit 53 of the information terminal 50 receives the presentation information, and the information processing unit 54 displays a display screen showing the content of the food management information on the display unit 52 based on the received presentation information (S26). FIG. 7 is a diagram showing an example of a display screen showing the content of the food management information.

[0050] In this way, the user of the inventory management system 10 can check the food stored in the refrigerator 20 at home from a location outside the home or the like.

[0051] [Operation Example 3] Next, as Operation Example 3 of the inventory management system 10, the operation of creating teacher data for constructing a machine learning model that determines (in other words, identifies) the type of food in step S16 of Operation Example 1 will be described. FIG. 8 is a sequence diagram of Operation Example 3 of the inventory management system 10. In the following Operation Example 3, it is assumed that video is captured by the imaging unit 31 with the two doors 22 of the refrigerator 20 open.

[0052] The user inputs the label information of the food that will be photographed by the photographing unit 31 to the input unit 51 of the information terminal 50, and the input unit 51 accepts such input (S31). In other words, the label information of the food is input by the user to the input unit 51. FIG. 9 is a diagram showing an example of an input screen for label information. The input of the label information is performed, for example, by selecting from among pre-prepared options, but it may also be free input. FIG. 9 shows a display screen when the label information is manually input, but the label information may also be input by voice uttered by the user.

[0053] The label information is, for example, information indicating the type of food (specifically, beverages, vegetables, or meat, etc.), but may also be information indicating a more specific name of the food (such as milk). Further, the label information may be information indicating both the type and name of the food.

[0054] Next, the user inputs shape information indicating the approximate shape of the food that will be photographed by the photographing unit 31 to the input unit 51, and the input unit 51 accepts such input (S32). In other words, the shape information of the food is input by the user to the input unit 51. FIG. 9 is also a diagram showing an example of an input screen for shape information. The input of the shape information is performed, for example, by selecting from among pre-prepared options, and the user selects the shape closest to the food. The options include various three-dimensional shapes such as a quadrangular prism, a cylinder, and a sphere. FIG. 9 shows a display screen when the shape information is manually input, but the shape information may also be input by voice uttered by the user.

[0055] The shape information is information input to display a three-dimensional shape, which will be described later, but there may be cases where the three-dimensional shape of the food can be estimated based on the label information input in step S31. For example, if the label information is "milk carton," it is possible to estimate that the three-dimensional shape is a rectangular prism. In such a case, the control unit 39 may automatically display the three-dimensional shape based on the label information. In other words, manual input of the shape information is not essential and can be omitted.

[0056] Next, the information processing unit 54 transmits a display request for the video captured by the imaging unit 31 to the inventory management device 30 (S33). More specifically, the transmission of the display request is performed by the communication unit 53. The display request is transmitted, for example, in response to an input for selecting the start imaging button on the display screen of FIG. 9. The display request includes the label information input in step S31 and the shape information input in step S32.

[0057] The communication unit 35 of the incoming / outgoing management device 30 receives the display request. The control unit 39 starts shooting by the shooting unit 31 based on the received display request (S34), and transmits video information for displaying the video shot by the shooting unit 31 to the information terminal 50 (S35). The control unit 39 also transmits guide frame information in addition to the video information (S36). The guide frame information is information for displaying a guide frame and a three-dimensional shape inscribed in the guide frame, which will be described later. More specifically, the transmission of the video information and the guide frame information is performed by the communication unit 35. Note that the transmission of the video information continues during the following steps S35 to S40, during which the video shot by the shooting unit 31 is displayed on the display unit 52 substantially in real time.

[0058] The communication unit 53 of the information terminal 50 receives the video information and the guide frame information. The information processing unit 54 displays the video and the guide frame on the display unit 52 based on the received video information and guide frame information (S37). Fig. 10A is a diagram showing a display example 1 of the video and the guide frame.

[0059] The guide frame is a rectangular frame for instructing the user on where to photograph the food, and in the example of Fig. 10A, is superimposed on the upper left corner of the video. In other words, the guide frame is a bounding box. In this way, the control unit 39 instructs the user to position the food within the guide frame by displaying the guide frame (transmitting guide frame information) within the video displayed on the display unit 52.

[0060] Also, within the guide frame, a three-dimensional shape inscribed in the guide frame is displayed. This three-dimensional shape is the three-dimensional shape indicated by the shape information included in the display request (the three-dimensional shape input by the user in step S32), and is intended to instruct the user on the posture to take when photographing the food. In other words, the three-dimensional shape indicates the silhouette of the food when photographing it.

[0061] The three-dimensional shape is simply a frame line, but may also be a 3D model, illustration, or photo of the food. When a 3D model, illustration, or photo is displayed as the three-dimensional shape, the three-dimensional shape may be displayed semi-transparently or flashing so that the user can easily place the food on the three-dimensional shape.

[0062] In this way, the control unit 39 instructs the user on the photographing posture of the food by displaying a three-dimensional shape within the guide frame based on the input shape information (transmitting the guide frame information). The control unit 39 may further instruct the user to face a logo mark or the like of the food toward the photographing unit 31.

[0063] The user moves the food in the actual space while watching the display unit 52 on which the display screen as shown in Fig. 10 is displayed, and when it is determined that the food is positioned so as to be inscribed in the guide frame and is in a position close to the position indicated by the three-dimensional shape, inputs an instruction to take a photograph. The input unit 51 accepts such an input (S38). When the input instructing to take a photograph is accepted, the information processing unit 54 transmits photographing trigger information to the inventory management device 30 (S39). More specifically, the transmission of the photographing trigger information is performed by the communication unit 53.

[0064] The communication unit 35 of the incoming and outgoing inventory management device 30 receives the shooting trigger information. Based on the received shooting trigger information, the control unit 39 causes the shooting unit 31 to shoot a still image at the timing when the shooting trigger information is received, and associates the shot still image with the label information input in step S31, and stores it in the storage unit 34 as teaching data (S40). That is, after the display of the guide frame (after the instruction to position the food within the guide frame is given), the control unit 39 associates the label information input in step S31 with the still image showing the food photographed by the shooting unit 31 to create teaching data, and stores the created teaching data in the storage unit 34. Note that the teaching data may include information indicating the position of the guide frame (coordinates of the center position of the guide frame), that is, position information of the place where the food is considered to be reflected in the still image.

[0065] The processes from step S36 to step S40 are repeated until the number of teaching data (still images) stored in the storage unit 34 reaches a predetermined number, or until the user makes an input intending to end the creation of teaching data. At this time, the display mode of the guide frame displayed on the display unit 52 is changed (updated) each time the guide frame information is transmitted and received in step S36. FIGS. 10B to 10F are diagrams showing display examples 2 to 6 of a moving image and a guide frame.

[0066] The display mode of the guide frame is changed, for example, in the order of FIGS. 10A to 10F. As shown in FIGS. 10A to 10F, each time the guide frame information is transmitted, the display position, shape, size, and aspect ratio of the shape of the guide frame are changed. In other words, the display position, shape, size, and aspect ratio of the shape of the guide frame are changed each time a still image is shot (each time teaching data is created). Note that as long as at least one of the display position, shape, size, and aspect ratio of the shape of the guide frame is changed each time the guide frame information is transmitted, it is not essential that all of the display position, shape, size, and aspect ratio of the shape are changed.

[0067] Here, regarding the size of the guide frame, it should be noted that the size of the guide frame corresponds to the distance from the imaging unit 31 to the food (vertical distance). The smaller the guide frame, the more it instructs the user to position the food farther from the imaging unit 31. Changing the display position of the guide frame means changing the background image of the food.

[0068] When the creation and storage of the teaching data are completed, the control unit 39 updates the machine learning model (food identification algorithm) by having the machine learning model learn the learning data stored in the storage unit 34 (S41).

[0069] As described above, the inventory management system 10 has a learning data creation function and also functions as a learning data creation system. The inventory management system 10 gives an instruction to the user to position the food within the guide frame by displaying the guide frame in the video displayed on the display unit 52. After the instruction is given, the teaching data is created by associating the input label information with the still image showing the object photographed by the imaging unit 31, and the display mode of the guide frame is changed every time a still image is photographed.

[0070] Such an inventory management system 10 can create high-quality teaching data with appropriate variations in the position and posture of the food in the still image by instructing the user on the position to hold the food and the posture of the food. Also, according to the inventory management system 10, even a user with little knowledge about the machine learning model can photograph the teaching data (still images) without using special tools such as a turntable and a tripod.

[0071] Note that there are no particular limitations on where and in what order to change the guide frame within the video. Regarding where and in what order to display the guide frame within the video, it may be determined in advance or may be randomly determined by a random number or the like. The same applies to the orientation (posture) in which the three-dimensional shape is displayed within the guide frame.

[0072] Also, in step S39, the trigger information was transmitted in response to an input instructing the user to take a picture, but it is not essential that the user inputs an instruction to take a picture. For example, the information processing unit 54 may detect that the outer shape of the food is inscribed within the guide frame by image processing, and transmit the trigger information on the basis of such detection. That is, the still image may be automatically captured without requiring a user input.

[0073] [Modification Example] In operation example 1 of the above-described embodiment, the processes from food detection (step S13) to update of food management information (step S17) were performed by the warehousing and shipping management device 30, but some or all of these processes may be performed by the server device 40. For example, the determination of the type of food in step S16 may be performed by the server device 40. In this case, the machine learning model for determining the type of food is stored in the storage unit 43 of the server device 40, and the teacher data created by the warehousing and shipping management device 30 in operation example 3 is transmitted from the warehousing and shipping management device 30 to the server device 40.

[0074] In this case, the server device 40 can collect teacher data from a plurality of warehousing and shipping management devices including the warehousing and shipping management device 30, and cause the machine learning model to learn a large number of teacher data.

[0075] Also, in operation example 3 of the above-described embodiment, the information terminal 50 was used as the user interface, but the refrigerator 20 may be used as the user interface instead of the information terminal 50. That is, the refrigerator 20 may include an input unit into which label information and shape information are input by the user, and a display unit on which the video captured by the imaging unit 31 is displayed. Further, one of the input unit 51 and the display unit 52 may be provided by the information terminal 50, and a component corresponding to the other of the input unit 51 and the display unit 52 may be provided by the refrigerator 20. That is, it is not essential that both the input unit 51 and the display unit 52 are provided by the information terminal 50, and at least one of the input unit 51 and the display unit 52 may be provided by the information terminal 50.

[0076] Further, in the above embodiment, the refrigerator 20 may be a household refrigerator, a refrigerator for displaying products used in a retail store such as a convenience store, or other commercial refrigerators.

[0077] In addition, when a single type of food is stored in the refrigerator 20, the determination of the food type may not be performed.

[0078] The refrigerator 20 is an example of a storage compartment, and the present disclosure can also be realized as other storage compartments without a refrigeration function. Further, the articles (in other words, objects) stored in the storage compartment are not limited to foods and may be other articles. A case where articles other than foods are stored in the refrigerator 20 is also conceivable.

[0079] In addition, the teacher data creation system may create teacher data not only for a machine learning model that identifies foods that are put into and taken out of the refrigerator 20, but also for a machine learning model that performs other identifications. For example, the teacher data creation system may be realized as a single device corresponding to the information terminal 50 that uses the imaging unit 56 provided in the information terminal 50 to capture a video or a still image instead of the imaging unit 31 of the incoming / outgoing management device 30. In this case, the information processing unit 54 of the information terminal 50 includes a control unit 57 (illustrated by a dashed line in FIG. 1) as a functional component corresponding to the control unit 39.

[0080] Such an information terminal 50 functions as a teacher data creation system for creating teacher data used to build a machine learning model for identifying an object. The information terminal 50 includes an input unit 51 into which label information of an object is input by a user, a photographing unit 56 that photographs a video in which the object appears, a display unit 52 on which the photographed video is displayed, and a control unit 57 that gives an instruction to the user to position the object within a guide frame by displaying the guide frame in the displayed video. After the instruction is given, the control unit 57 creates teacher data by associating the input label information with a still image in which the object photographed by the photographing unit 56 appears, and changes the display mode of the guide frame each time a still image is photographed. Such information terminals 50 include digital cameras in addition to smartphones, tablet terminals, and personal computers.

[0081] Such an information terminal 50 can create teacher data not only for a machine learning model for identifying foods that are put into and taken out of the refrigerator 20, but also for other machine learning models for identification.

[0082] [Effects, etc.] As described above, the warehousing and inventory management system 10 functions as a teacher data creation system for creating teacher data used to build a machine learning model for identifying an object. The warehousing and inventory management system 10 includes an input unit 51 into which label information of an object is input by a user, a photographing unit 31 that photographs a video in which the object appears, a display unit 52 on which the photographed video is displayed, and a control unit 39 that gives an instruction to the user to position the object within a guide frame by displaying the guide frame in the displayed video. After the instruction is given, the control unit 39 creates teacher data by associating the input label information with a still image in which the object photographed by the photographing unit 31 appears, and changes the display mode of the guide frame each time a still image is photographed.

[0083] Such a warehousing and inventory management system 10 can create teacher data used to build a machine learning model by giving an instruction on the photographing position of an object.

[0084] Further, for example, shape information of an object is further input to the input unit 51, and the control unit 39 gives an instruction to the user on the shooting posture of the object by displaying a three-dimensional shape within the guide frame based on the input shape information.

[0085] Such an inventory management system 10 can create teacher data used for constructing a machine learning model by giving an instruction on the shooting posture of the object.

[0086] Further, for example, the control unit 39 displays a frame line, a 3D model of an article, an illustration, or a photograph as the three-dimensional shape.

[0087] Such an inventory management system 10 can give an instruction on the shooting posture of the object by a frame line, a 3D model of an article, an illustration, or a photograph.

[0088] Further, for example, each time a still image is captured, the control unit 39 changes the display position of the guide frame.

[0089] Such an inventory management system 10 can create teacher data used for constructing a machine learning model by changing the display position of the guide frame.

[0090] Further, for example, each time a still image is captured, the control unit 39 changes at least one of the display position of the guide frame, the shape, and the size of the guide frame.

[0091] Such an inventory management system 10 can create teacher data used for constructing a machine learning model by changing at least one of the display position of the guide frame, the shape, and the size of the guide frame.

[0092] Further, for example, label information is input to the input unit 51 by voice uttered by the user.

[0093] Such an inventory management system 10 can receive input of label information based on the voice uttered by the user.

[0094] Also, for example, at least one of the input unit 51 and the display unit 52 is provided by the information terminal 50. The control unit 39 is provided by a device that communicates with the information terminal 50.

[0095] Such an inventory management system 10 can create teacher data using the information terminal 50 as a user interface.

[0096] Also, a teacher data creation method for creating teacher data used to build a machine learning model for identifying an object is executed by a computer such as the inventory management system 10. The teacher data creation method includes a reception step S31 of receiving input of label information of the object from the user, a photographing step S34 of photographing a video in which the object appears, a display step S37 of displaying the photographed video, a control step S37 of instructing the user to position the object within a guide frame by displaying the guide frame within the displayed video, a creation step S40 of creating teacher data by associating the input label information with a still image in which the object appears after the instruction is given, and a change step S36 of changing the display mode of the guide frame each time a still image is photographed.

[0097] Such a teacher data creation method can create teacher data used to build a machine learning model by instructing the photographing position of the object.

[0098] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above embodiments.

[0099] For example, in the above embodiment, the guide frame was rectangular, but the guide frame may have a shape other than rectangular such as circular.

[0100] In the above-described embodiment, the inventory management system (in other words, the teacher data creation system; the same applies hereinafter) was realized by a plurality of devices. In this case, the components included in the inventory management system may be distributed among the plurality of devices in any manner.

[0101] Further, the inventory management system may be realized by a single device. For example, the inventory management system may be realized as a single device corresponding to the inventory management device in the above-described embodiment, as a single device corresponding to a server device, or as a single device corresponding to an information terminal.

[0102] Also, the communication method between the devices in the above-described embodiment is not particularly limited. In the communication between devices, a relay device (not shown) may be interposed. Further, the transmission path of the information described in the above-described embodiment is not limited to the transmission path shown in the sequence diagram.

[0103] Also, in the above-described embodiment, the processing executed by a specific processing unit may be executed by another processing unit. Further, the order of a plurality of processes may be changed, or a plurality of processes may be executed in parallel.

[0104] Also, in the above-described embodiment, each component may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0105] Also, each component may be realized by hardware. For example, each component may be a circuit (or an integrated circuit). These circuits may constitute one circuit as a whole, or may be separate circuits respectively. Further, these circuits may be general-purpose circuits or dedicated circuits respectively.

[0106] Furthermore, the general or specific aspects of the present disclosure may be implemented in a system, apparatus, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. Also, the general or specific aspects of the present disclosure may be implemented in any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium.

[0107] For example, the present disclosure may be implemented as a method for creating teacher data to be executed by a computer, or may be implemented as a program for causing a computer to execute the method for creating teacher data. The present disclosure may be implemented as a computer-readable non-transitory recording medium on which such a program is recorded.

[0108] In addition, forms obtained by applying various modifications conceivable by those skilled in the art to each embodiment, or forms realized by arbitrarily combining the components and functions in each embodiment without departing from the spirit of the present disclosure are also included in the present disclosure.

Description of Reference Numerals

[0109] 10 In-out management system 20 Refrigerator (storage) 21 Main body 22 Door 22a Door pocket 23 First storage room 24, 24a, 24b Shelf 26 Second storage room 30 In-out management device 31, 56 Photographing unit 32 Distance measuring unit 33, 42, 54 Information processing unit 34, 43, 55 Storage unit 35, 41, 53 Communication unit 36 Notification unit 37 Determination unit 38 Tracking unit 39 Control unit 40 Server device 44 Presentation unit 45 Notification section 50 Information terminal 51 Input section 52 Display section 57 Control section 60 Wide area communication network

Claims

1. A teacher data creation system for creating teacher data used to build a machine learning model for identifying an object, comprising: an input unit for receiving label information of the object from a user; a photographing unit for photographing a video in which the object appears; a display unit for displaying the photographed video; and a control unit for instructing the user to position the object within a guide frame by displaying the guide frame within the displayed video, wherein the control unit: creates the teacher data by associating the input label information with a still image of the object photographed by the photographing unit after the instruction is given; changes the display mode of the guide frame each time the still image is photographed. Teacher data creation system.

2. Shape information of the object is further input to the input unit, and the control unit instructs the user about the photographing posture of the object by displaying a three-dimensional shape within the guide frame based on the input shape information. The teacher data creation system according to claim 1.

3. The control unit displays a frame line, a 3D model of an article, an illustration, or a photograph as the three-dimensional shape. The teacher data creation system according to claim 2.

4. The control unit changes the display position of the guide frame each time the still image is photographed. The teacher data creation system according to any one of claims 1 to 3.

5. The control unit changes at least one of the display position of the guide frame, the shape, size, and aspect ratio of the shape of the guide frame each time the still image is photographed. The teacher data creation system according to claim 4.

6. The label information is input to the input unit by voice uttered by the user. The teacher data creation system according to any one of claims 1 to 5.

7. At least one of the input unit and the display unit is provided by an information terminal, and the control unit is provided by a device that communicates with the information terminal. The teacher data creation system according to any one of claims 1 to 6.

8. A teacher data creation method executed by a teacher data creation system for creating teacher data used to build a machine learning model for identifying an object, the method comprising: a reception step of receiving input of label information of the object from a user; a photographing step of photographing a video in which the object appears; a display step of displaying the photographed video; A control step of instructing the user to position the object within the guide frame by displaying a guide frame within the displayed video; A creation step of creating the training data by associating the input label information with a still image showing the object captured after the instruction is given; A changing step of changing the display mode of the guide frame each time the still image is captured, and A training data creation method.

Images