Image correction device, storehouse, and image correction method

Abstract

To correct an image of the interior of a storehouse to an image for a common identification model regardless of the type of the storehouse.SOLUTION: An image correction device 100 includes: an image quality acquisition unit 112 configured to acquire the image quality of an image taken inside a storehouse; an interior background acquisition unit 114 configured to acquire the interior background of the storehouse shown in the image; an image quality correction unit 113 configured to correct the image quality of the image to a predetermined image quality (reference image quality information 132); and an interior background correction unit 115 configured to correct the interior background of the image to a predetermined interior background (reference interior background information 133). The image correction device 100 may further include a learning unit 117 configured to train and generate an identification model 131, which is a machine learning model that identifies items shown in the corrected image, using teacher data 150 that includes the corrected image having the predetermined image quality and the predetermined interior background as an explanatory variable and an area of an item shown in the corrected image and the name of the item as objective variables.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image correction device, a storage device, and an image correction method for correcting an image showing the inside of the storage device.

Background Art

[0002] There is a commercially available refrigerator equipped with a function to confirm stocked food by viewing a photographed image showing the inside of the refrigerator at a destination. Also, a system has been proposed to identify, count, and manage food / ingredients based on the photographed image. For this reason, in the future, the spread of refrigerators equipped with such a food management system is expected.

[0003] The refrigerator system described in Patent Document 1 extracts each feature amount of images photographed from different viewpoints for the same area, and updates the learned feature amount of the food material by performing relearning between each feature amount and the food material. From this, the refrigerator system aims to improve the identification accuracy and expand the range of identifiable food materials for the food materials stored in the storage room.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to recognize and identify food / ingredients based on an image showing the inside (inside the storage) of a storage device including a refrigerator, it is common to use an identification model which is a machine learning model. Since the specifications and positions of cameras and the structures inside the storage differ for each model of the storage device, it is desirable to prepare an identification model for each model in order to improve the accuracy of recognition and identification.

[0006] However, constructing an identification model requires creating training data, building the model, and evaluation, and it takes a great deal of cost to construct an identification model for each model type. Therefore, it is desirable to construct an identification model common to multiple model types, but Patent Document 1 does not mention an identification model common to model types.

Means for Solving the Problem

[0007] To solve the above problems, an image correction apparatus according to the present invention includes an image quality acquisition unit that acquires the image quality of an image taken inside a storage, an in-warehouse background acquisition unit that acquires the in-warehouse background inside the storage shown in the image, an image quality correction unit that corrects the image quality of the image to a predetermined image quality, and an in-warehouse background correction unit that corrects the in-warehouse background of the image to a predetermined in-warehouse background.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0009] ≪Outline of Image Correction Device≫ The image correction device in the form (embodiment) for carrying out the present invention will be described below. The image correction device corrects / converts the image quality of the image showing the inside of the storage and the background inside the storage shown in the image into a predetermined image quality and a predetermined background inside the storage. Here, the image quality is, for example, the aspect ratio, contrast, brightness, resolution, noise, size, color, edge, and distortion of the image. Also, the background inside the storage is part or all of the inside of the storage shown in the part excluding the part where food / ingredients are shown within the shooting range of the image showing the inside of the storage. Examples of part of the inside of the storage include shelves, drawers, walls, etc.

[0010] In order to construct an identification model common to multiple models, it is preferable to correct / convert the image taken inside the storage into an image suitable for the identification model. In the image correction device (refrigerator and image correction method) in this embodiment, the image showing the inside of the refrigerator is corrected / converted into an image for a common identification model independent of the model of the refrigerator (image with a predetermined image quality and a predetermined background inside the storage). Here, the storage is, for example, a refrigerator.

[0011] According to such an image correction device, even if the specifications of the camera for shooting inside the refrigerator or the mounting position of the camera for shooting inside the refrigerator on the refrigerator differ depending on the model of the refrigerator, the image taken inside the refrigerator can be corrected into an image with a predetermined image quality. Also, even if the background inside the storage is different, the image taken inside the storage can be corrected into an image with a predetermined background inside the storage. Consequently, even for images taken inside the storage with different models (different image qualities, backgrounds inside the storage, etc.), the food / ingredients shown can be recognized and identified using a common identification model, and the construction cost of the identification model can be reduced.

[0012] ≪Configuration of Image Correction Device≫ FIG. 1 is a functional block diagram of an image correction apparatus 100 according to the first embodiment. The image correction apparatus 100 is a computer, such as a personal computer or a server in the cloud. The image correction apparatus 100 includes a control unit 110, a storage unit 130, and an input / output unit 180. User interface devices such as a display, a keyboard, and a mouse may be connected to the input / output unit 180. The input / output unit 180 may include a communication device and be capable of receiving an image captured by a camera provided in the refrigerator. Also, a media drive may be connected to the input / output unit 180, enabling data exchange using a recording medium.

[0013] ≪Image Correction Apparatus: Storage Unit≫ The storage unit 130 is configured to include storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 130 stores an image database 140, an identification model 131, reference image quality information 132, reference in-refrigerator background information 133, training data 150, and a program 138.

[0014] ≪Storage Unit: Image Database≫ The image database 140 stores images inside the refrigerator captured by a camera 591 provided in the refrigerator 500 (see FIG. 2 described later). FIG. 2 is a diagram showing an example of the camera 591 that captures images according to the first embodiment. The camera 591 captures the inside of the refrigerator when the doors 592, 593 of the refrigerator compartment or the drawers 594 - 597 are opened. Alternatively, the camera 591 may capture the inside of the refrigerator when it can detect by image recognition that food has been placed inside. Instead of being provided outside the refrigerator compartment like the camera 591, it may be installed on the upper part inside the refrigerator compartment, on the doors 592, 593, or on the drawers 594 - 597. The camera 591 is not limited to the inside of the refrigerator compartment and may capture the inside of the freezer compartment or the vegetable compartment, and it is desirable to be installed at an appropriate location according to the imaging target. In the following description, it is assumed that the image is a captured image inside the refrigerator compartment (inside the refrigerator).

[0015] FIG. 3 is a data configuration diagram of the image database 140 according to the first embodiment. The image database 140 is, for example, tabular data, and one row (record) represents one image. The records of the image database 140 include columns (attributes) of identification information (described as "ID" in FIG. 3), model, individual identification information (described as "individual ID" in FIG. 3), acquisition date and time, reference, image, image quality, in-store background, label, corrected image, and type.

[0016] The identification information is the identification information of the image. The model is the model of the refrigerator. The individual identification information is the identification information of each individual refrigerator. The acquisition date and time is the date and time when the image was acquired, that is, the date and time when the camera 591 photographed the inside of the store. The reference indicates the validity ("Y" / "N") of whether the image is an image (also referred to as a reference candidate image) to be referred to when determining the requirement for the image to be a reference image. A reference image is an image whose image quality is a predetermined image quality and the in-store background shown in the image matches a predetermined in-store background. For the reference candidate image, the administrator of the image correction device 100 (the developer of the identification model) may select a reference candidate image from the images in the image database 140, or an image of a specific model may be used as the reference candidate image.

[0017] Examples of image quality include aspect ratio, contrast, brightness, resolution, noise, size, color, edge, and distortion. The predetermined image quality (also referred to as the reference image quality) means that the value indicating the image quality is within a predetermined range. The value indicating the predetermined image quality is stored in the reference image quality information 132 described later.

[0018] The in-store background is part or all of the area (background) in the in-store photographed image where the food / ingredient is shown and the food to be identified by the identification model is not shown, and is information such as the shape, hue, and position of the shelves, drawers, and walls in the refrigerator compartment, for example. The predetermined in-store background (also referred to as the reference in-store background) is, for example, the in-store background (information) where the position of the shelf is within a predetermined range. The information indicating the predetermined in-store background is stored in the reference in-store background information 133 described later.

[0019] Continue the description of the attributes of the image database 140 (see FIG. 3). The image is the data of the image. The image quality indicates the image quality of the image, such as the aspect ratio. The image quality is acquired by the image quality acquisition unit 112 described later. The in-store background indicates the in-store background (information), such as the shape, hue, and position of the shelves, drawers, and walls. The in-store background is acquired by the in-store background acquisition unit 114 described later.

[0020] The label is the position of the area of the food shown in the image and the name of the food. Examples of the name include eggs, natto, milk, and tofu. The label is generated by the image processing unit 111 described later. Note that, as described later, the label becomes the target variable of the teacher data 150 of the identification model 131.

[0021] The corrected image is the data of the image corrected to have a predetermined image quality and a predetermined in-store background. The corrected image is generated by the image quality correction unit 113 and the in-store background correction unit 115 described later. The type indicates whether the image is a teacher (training) image ("T"), an evaluation image ("E"), or other ("N / A"). Note that the teacher image is the image used when training the identification model 131 described later. The evaluation image is the image for evaluating (calculating) the identification accuracy of the identification model 131.

[0022] The type of the corrected image may be randomly assigned to either "T" or "E" for the corrected images in the image database 140, for example. Alternatively, after dividing a number of refrigerators with different individual identification information into a plurality of groups, the type of either "T" or "E" may be assigned to each group.

[0023] ≪Memory Unit: Identification Model≫ Returning to FIG. 1, continue the description of the memory unit 130. The identification model 131 is a machine learning model that recognizes and identifies foods shown in the in-store images captured by the camera 591, and is, for example, a deep learning model. The predetermined image quality of the in-store images that are the explanatory variables (inputs) of the identification model 131 is the reference image quality, and the predetermined in-store background of the images is the reference in-store background. In other words, the in-store images captured by the camera 591 are corrected by the image processing unit 111, the image quality correction unit 113, and the in-store background correction unit 115 described later into images with the reference image quality or / and the reference in-store background, and become the input images of the identification model 131.

[0024] ≪Memory unit: Reference image quality · Reference in-store background≫ The reference image quality information 132 is a predetermined image quality (reference image quality), and indicates a reference value or a reference range (range of values indicating the image quality) of the image quality that becomes the input image of the identification model 131. The reference in-store background information 133 is a predetermined in-store background (reference in-store background), and indicates a reference or a reference range (range of the shape, hue, and position such as a shelf) of the in-store background that becomes the input image of the identification model 131. The range of the hue is, for example, a numerical range when the color of the shelf is represented by RGB values. The range of the shape and position is, for example, the range of the coordinates when the position or area where the shelf etc. appears is represented by the coordinates on the image.

[0025] ≪Memory unit: Teacher data≫ The teacher data 150 is the teacher data (training data, learning data) of the identification model 131. The explanatory variables (inputs) of the teacher data 150 are images with the reference image quality or / and the reference in-store background. The objective variable (correct label) of the teacher data 150 is a label (referencing the label of the image database 140), and is the cut-out area (hereinafter simply referred to as the food area) and name when the food shown in the image is cut out by trimming. The program 138 includes a description of the processing of the control unit 110, and includes a description of the reference image quality · reference in-store background calculation processing (see FIG. 7) and the learning processing (see FIG. 8) described later.

[0026] ≪Image correction device: Control unit≫ The control unit 110 is configured to include a CPU (Central Processing Unit), and is provided with an image processing unit 111, an image quality acquisition unit 112, an image quality correction unit 113, an in-store background acquisition unit 114, an in-store background correction unit 115, a reference information calculation unit 116, a learning unit 117, and an identification accuracy evaluation unit 118. The control unit 110 may be configured to include a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

[0027] <<Control Unit: Image Processing Unit>> The image processing unit 111 processes the acquired in-store image and stores it in the image (image attributes) of the image database 140 (see Fig. 3). For example, when the acquired image is taken with a fish-eye or wide-angle lens, the image processing unit 111 may correct / convert it into a planar image (an image without standard distortion) and then store it in the image database 140. Also, the image processing unit 111 stores the area in the image where the food appears and the name of the food specified by the administrator in the label of the image database 140. Note that the image processing unit 111 may store the area and name of the food generated by existing machine learning processing in the label of the image database 140.

[0028] <<Control Unit: Image Quality Acquisition Unit>> The image quality acquisition unit 112 acquires the image quality of the image by a known method and stores it in the image quality of the image database 140. The image quality acquired by the image quality acquisition unit 112 is, for example, aspect ratio, contrast, brightness, resolution, noise, size, color, edge, and distortion.

[0029] The aspect ratio is the ratio of the horizontal size to the vertical size of the image. Contrast is the difference in brightness between the bright and dark parts in the image. Brightness is the average of the luminance values of the image. Resolution is the density of pixels. Noise is the noise level, for example, the number of noise or the number of pixels that are noise. Color indicates the hue of the image, for example, the distribution of hues or the average of RGB values. An edge is the boundary between the bright and dark parts of the image. Distortion is a distortion level that varies depending on whether the camera is installed inside or outside the warehouse, the nature of the camera lens, etc.

[0030] The image quality acquired by the image quality acquisition unit 112 does not necessarily have to be all of the image qualities such as the aspect ratio described above, and may be only a part. The image quality acquisition unit 112 stores the acquired image quality in the image quality in the image database 140.

[0031] <<Control Unit: Image Quality Correction Unit>> The image quality correction unit 113 corrects the image so that the image quality becomes the reference image quality. Examples of correction methods include histogram conversion, filter processing, and affine transformation. For example, when the reference image quality includes a range of aspect ratios, the image is corrected so as to be within this range of aspect ratios. Regarding noise correction, it is not limited to noise removal, and there may be cases where noise is added so as to be within the range of noise included in the reference image quality. Regarding resolution correction, it is not limited to increasing the resolution, and there may be cases where the resolution is decreased. Also, when the image is reduced and transformed to correct the aspect ratio, the resolution may simultaneously decrease to become the reference image quality. The contrast, brightness, color, edge, distortion, etc. of the image are also appropriately corrected by known image conversion processing methods so as to become the reference image quality.

[0032] <<Control Unit: Inside Warehouse Background Acquisition Unit · Inside Warehouse Background Correction Unit>> The in - store background acquisition unit 114 acquires the shape, hue, and position of the in - store (refrigerator) such as shelves, drawers, and walls shown in the image. Regarding the shape, the in - store background acquisition unit 114 may acquire the shape of the in - store based on the design information (CAD information) corresponding to the model of the image database 140 stored in the storage unit 130 in advance. Regarding the hue and position, the in - store background acquisition unit 114 acquires the hue and position of the part in the image where the in - store background is shown. Note that the in - store background acquisition unit 114 may acquire information such as hue and position stored in the storage unit 130 in advance.

[0033] The in - store background correction unit 115 corrects the image so that the in - store background becomes the reference in - store background. Examples of correction methods include color correction, affine transformation, homography transformation, etc. Regarding the correction of the image, after the image quality correction unit 113 corrects the image quality, it is basically the in - store background correction unit 115 that corrects the in - store background, but the reverse order may also be possible. Also, it may be in a mode where either the image quality correction by the image quality correction unit 113 or the in - store background correction by the in - store background correction unit 115 is executed.

[0034] An example of in - store background correction is shown below. FIG. 4 is a diagram showing an image 410 which is a reference image according to the first embodiment. The aspect ratio of the image 410 is, for example, 1.09, and a range of 1.1 or within 0.05 before and after it is the reference image quality. Also, the shelf 411 has a shape covered by a member 412 with an opaque edge, and this shape becomes the reference background. Note that as described above, the reference image is an image in which the image quality of the image is a predetermined image quality and the in - store background shown in the image matches a predetermined in - store background.

[0035] FIG. 5 is an image 420 before in - store background correction according to the first embodiment. The aspect ratio of the image 420 is, for example, 1.83, which is outside the range of the reference image quality, the entire shelf 421 is transparent, and the edge is not covered. FIG. 6 shows an image 430 after in-warehouse background correction according to the first embodiment. The aspect ratio of the image 430 is, for example, 1.09, and the image quality is corrected so as to be within the range of the reference image quality. Also, the image is corrected to the shape of a reference in-warehouse background in which the edge of the shelf 431 is covered with an opaque member 432.

[0036] As described above, the image correction device 100 includes an image quality acquisition unit 112 that acquires the image quality of an image taken inside the storage warehouse (inside the refrigerator 500). The image correction device 100 includes an in-warehouse background acquisition unit 114 that acquires the in-warehouse background inside the storage warehouse shown in the image. The image correction device 100 also includes an image quality correction unit 113 that corrects the image quality of the image to a predetermined image quality (reference image quality). The image correction device 100 includes an in-warehouse background correction unit 115 that corrects the in-warehouse background of the image to a predetermined in-warehouse background (reference in-warehouse background).

[0037] The image quality includes at least any one of the aspect ratio, contrast, brightness, resolution, noise, size, color, edge, and distortion of the image. The in-warehouse background includes at least any one of the shape, hue, and position inside the storage warehouse shown in the image.

[0038] ≪Control Unit: Reference Information Calculation Unit≫ Returning to FIG. 1, the description of the control unit 110 will be continued. The reference information calculation unit 116 calculates the reference image quality and the reference in-warehouse background based on a reference candidate image that is an image in the image database 140 and has a reference of "Y".

[0039] ≪Control Unit: Learning Unit≫ The learning unit 117 generates teacher data 150 and trains and generates the identification model 131 using the teacher data 150. The image serving as the explanatory variable (input) of the teacher data 150 is an image with the reference image quality and the background within the reference library. In other words, the said image is the image after image quality correction and correction of the background within the library by the image quality correction unit 113 and the background within the library correction unit 115. Note that the image serving as the explanatory variable may be an image with the reference image quality or the background within the reference library.

[0040] The objective variable (correct label) of the teacher data 150 is the label of the image database 140 (see FIG. 3). Note that in image quality correction and correction of the background within the library, when the image is subjected to affine transformation or homography transformation, the area of the food included in the label is similarly transformed. When the identification accuracy of the identification model 131 does not meet a predetermined value, the learning unit 117 adds the teacher data 150 and performs additional learning by retraining the identification model 131 using the added teacher data 150 (added teacher data).

[0041] ≪Control Unit: Identification Accuracy Evaluation Unit≫ The identification accuracy evaluation unit 118 evaluates the identification accuracy of the identification model 131. The identification accuracy evaluation unit 118 evaluates using an image whose type is the evaluation image ("E") among the images in the image database 140.

[0042] As described above, the image correction device 100 includes a learning unit 117 that trains and generates an identification model 131, which is a machine learning model for identifying an article shown in a corrected image, using teacher data 150 having an image corrected to a predetermined image quality (reference image quality) and a predetermined background within the library (reference background within the library) as the explanatory variable, and the area of the article shown in the corrected image and the name (label) of the article as the objective variable.

[0043] The image correction device 100 includes an identification accuracy evaluation unit 118 that calculates the identification accuracy of the identification model 131. If the recognition accuracy is less than a predetermined value, the learning unit 117 uses, as explanatory variables, images corrected to a predetermined image quality (reference image quality) and a predetermined in-store background (reference in-store background) not included in the teacher data 150, and uses additional teacher data with the area of the food shown in the corrected image and the name (label) of the food as objective variables to retrain and generate the recognition model 131 again.

[0044] ≪Reference Image Quality and Reference In-Store Background Calculation Process≫ FIG. 7 is a flowchart of the reference image quality and reference in-store background calculation process according to the first embodiment. With reference to FIG. 7, after an image is stored in the image database 140 and a reference candidate image (reference attribute) is set, the process of calculating the reference image quality and the reference in-store background will be described.

[0045] In step S11, the reference information calculation unit 116 acquires a reference candidate image. The reference candidate image is an image in the images in the image database 140 where the reference is "Y". In step S12, the reference information calculation unit 116 starts the process of repeating steps S13 to S14 for each of the reference candidate images acquired in step S11. Hereinafter, the reference candidate image to be processed in this repeated process will be referred to as the processing target reference candidate image. After the processes of steps S13 to S14 are performed on all the processing target reference candidate images acquired in step S11, the repeated process ends and the process proceeds to step S15.

[0046] In step S13, the image quality acquisition unit 112 acquires the image quality of the processing target reference candidate image. In step S14, the in-store background acquisition unit 114 acquires the in-store background of the processing target reference candidate image.

[0047] In step S15, the reference information calculation unit 116 calculates the reference image quality and the background in the reference library, and stores them in the reference image quality information 132 and the reference library background information 133 respectively. For example, regarding the aspect ratio, which is one of the image qualities, the reference information calculation unit 116 calculates the average and standard deviation of the aspect ratio obtained in step S13, and sets the range of average ± 2 × standard deviation as the reference image quality of the aspect ratio. Also, for example, regarding the position of the shelf, which is one of the backgrounds in the library, the reference information calculation unit 116 calculates the average and standard deviation of the position coordinates of the shelf obtained in step S14, and sets the range of average ± 2 × standard deviation as the reference library background of the position of the shelf. The calculation methods of the reference image quality and the reference library background by the reference information calculation unit 116 are not limited to this, and for example, a certain range before and after the average value may be used.

[0048] ≪Learning Process≫ FIG. 8 is a flowchart of the learning process according to the first embodiment. While referring to FIG. 8, a process of generating the identification model 131 after an image is stored in the image database 140, the reference image quality / reference library background calculation process has been executed, and a teacher image (training image) is set will be described.

[0049] In step S21, the learning unit 117 acquires a teacher image. The teacher image is an image whose type is "T" among the images in the image database 140. In step S22, the learning unit 117 starts a process of repeating steps S23 to S27 for each of the teacher images acquired in step S21. Hereinafter, the teacher image to be processed in this repeated process is referred to as a processing target teacher image. When the processes of steps S23 to S27 are performed for all the processing target teacher images acquired in step S21, the repeated process ends and the process proceeds to step S28.

[0050] In step S23, the image quality acquisition unit 112 acquires the image quality of the processing target teacher image. In step S24, if the image quality obtained in step S23 is not the reference image quality, the image quality correction unit 113 corrects the image quality of the teacher image to be processed to obtain the teacher image to be processed after image quality correction. If the image quality obtained in step S23 is the reference image quality, the image quality correction unit 113 uses the teacher image to be processed itself as the teacher image to be processed after image quality correction.

[0051] In step S25, the in-store background acquisition unit 114 acquires the in-store background of the teacher image to be processed after image quality correction. In step S26, if the in-store background obtained in step S25 is not the reference in-store background, the in-store background correction unit 115 corrects the in-store background of the teacher image to be processed after image quality correction to obtain the teacher image to be processed after correction. If the in-store background obtained in step S25 is the reference in-store background, the in-store background correction unit 115 uses the teacher image to be processed after image quality correction itself as the teacher image to be processed after correction.

[0052] In step S27, the learning unit 117 adds the teacher image to be processed after correction and the label to the teacher data 150. In step S28, the learning unit 117 trains and generates the identification model 131 using the teacher data 150.

[0053] ≪Additional learning process≫ FIG. 9 is a flowchart of the additional learning process according to the first embodiment. With reference to FIG. 9, the additional learning process of the identification model 131 executed at a predetermined timing will be described. The predetermined timing is, for example, the timing of a predetermined cycle or the timing of developing a new model of the refrigerator 500. Also, the additional learning process may be performed at any timing such as when a problem occurs in machine learning (for example, the food identification process using the identification model 131) or when an improvement in identification accuracy is required.

[0054] In step S31, the identification accuracy evaluation unit 118 acquires an evaluation image. The evaluation image is an image whose type is "E" among the images in the image database 140. In step S32, the discrimination accuracy evaluation unit 118 starts a process of repeating step S33 for each of the evaluation images acquired in step S31. Hereinafter, the evaluation image to be processed in this repeated process is referred to as a processing target evaluation image.

[0055] Step S33 is the same process as steps S23 to S26 (see FIG. 8) for the processing target evaluation image, and is a process of correcting the processing target evaluation image. In step S34, the discrimination accuracy evaluation unit 118 evaluates the accuracy of the discrimination model 131 using the evaluation image corrected in step S33. As indicators of accuracy, there are precision, recall, F-value, accuracy rate, specificity, uncertainty, AUC, etc., and these may be combined.

[0056] In step S35, if the accuracy evaluated in step S34 satisfies a predetermined accuracy (step S35 → OK), the discrimination accuracy evaluation unit 118 ends the additional learning process, and if it does not satisfy (step S35 → NG), the process proceeds to step S36. In step S36, the learning unit 117 acquires additional teacher images. For example, the learning unit 117 uses, as additional teacher images, a predetermined number of images among the images added to the image database 140 after the generation of the previous discrimination model 131 and whose type is teacher use ("T").

[0057] In step S37, the learning unit 117 starts a process of repeating step S38 for each of the teacher images acquired in step S36. Hereinafter, the teacher image to be processed in this repeated process is referred to as a processing target teacher image. Step S38 is the same process as steps S23 to S27 (see FIG. 8) for the processing target teacher image, and is a process of correcting the processing target teacher image and adding the corrected processing target teacher image and label to the additional teacher data. In step S39, the learning unit 117 retrains and generates the discrimination model 131 using the additional teacher data.

[0058] <<Features of the Image Correction Device>> The image correction device 100 corrects the image quality of an image capturing the interior of the refrigerator 500 and the interior background to a predetermined image quality (reference image quality) and a predetermined interior background (reference interior background). According to such an image correction device 100, even if the specifications of the camera 591 for interior shooting of the refrigerator 500 differ depending on the model of the refrigerator 500 or the installation position of the camera for interior shooting on the refrigerator is different, the interior captured image can be corrected to an image of a predetermined image quality. Also, even if the interior structure is different, it can be corrected to an image of a predetermined interior background. Consequently, even for interior captured images of different models, foods shown using the common identification model 131 can be recognized and identified, and the construction cost of the identification model can be reduced.

[0059] ≪Second Embodiment: Refrigerator≫ The above-described image correction device 100 generates an identification model 131 for identifying foods. The image correction device 100 may be in the form of a refrigerator 500 that identifies foods using the generated identification model 131.

[0060] ≪Configuration of Refrigerator≫ FIG. 10 is a functional block diagram of a refrigerator 500 having a food identification function according to the second embodiment. The refrigerator 500 includes a food identification unit 510, a storage unit 530, a camera 591, a communication unit 580, and a refrigeration control unit 570. The refrigerator 500 has a freezer compartment, a refrigerating compartment, doors 592, 593, drawers 594 to 597, a heat pump, etc. as the original configuration of a refrigerating and freezing refrigerator, but these are not shown in FIG. 10.

[0061] The refrigerator 500 can perform data communication with a distribution system 600 and a terminal 690 via a network. The distribution system 600 is configured to include the image correction device 100 and transmits the identification model 131, the reference image quality information 132, and the reference interior background information 133 to the refrigerator 500. The terminal 690 is a communication terminal such as a smartphone used by a user who uses the refrigerator. By using the terminal 690, the user of the refrigerator can refer to an image of the interior captured by the camera 591 and a list of foods in the interior.

[0062] ≪Refrigerator: Memory Unit≫ The memory unit 530 is configured to include a memory device such as a flash memory or a RAM. The memory unit 530 stores an identification model 531, reference image quality information 532, reference in-refrigerator background information 533, an inventory database 540, and a program 538.

[0063] The identification model 531 is the identification model 531 generated by the image correction device 100. More specifically, the identification model 531 is a machine learning model that recognizes foods shown in the in-refrigerator images captured by the camera 591 and identifies the types and names of the foods. The image quality of the in-refrigerator images that are the explanatory variables (inputs) of the identification model 531 is the reference image quality, and the in-refrigerator background of the images is the reference in-refrigerator background. The in-refrigerator images that are the explanatory variables may be images with the reference image quality or the reference in-refrigerator background. The target variables of the identification model 531 are the areas and names of the foods shown in the images. The target variables may include the probability (reliability) that the food names in the areas match the actual food names.

[0064] The reference image quality information 532 and the reference in-refrigerator background information 533 are the same as the reference image quality information 132 and the reference in-refrigerator background information 133, respectively. The identification model 531, the reference image quality information 532, and the reference in-refrigerator background information 533 are distributed from the distribution system 600. The program 538 includes a description of the processing of the food identification unit 510 described later.

[0065] The inventory database 540 stores the planar images corrected by the image processing unit 512 described later from the images captured by the camera 591, the shooting date and time, and the list of foods shown in the images. The inventory database 540 is tabular data, and each row (each record) may include the planar image, the shooting date and time, and the list of foods shown in the image.

[0066] ≪Refrigerator: Food Identification Unit≫ The food identification unit 510 includes a microprocessor and is provided with a photographing unit 511, an image processing unit 512, an image quality acquisition unit 513, an image quality correction unit 514, an in-warehouse background acquisition unit 515, an in-warehouse background correction unit 516, an identification unit 517, and a communication unit 518. The food identification unit 510 may include a GPU, an FPGA, an ASIC, etc.

[0067] The photographing unit 511 uses the camera 591 to photograph the inside of the refrigerator when the doors 592, 593 of the refrigerator 500 or the drawers 594 - 597 are opened. Alternatively, the camera 591 may photograph the inside of the refrigerator at the timing when it can detect by image recognition that the food has entered the refrigerator. The image processing unit 512 processes the image of the inside of the refrigerator photographed by the photographing unit 511 and stores it in the inventory database 540 together with the photographing date and time. The processing content is the same as that of the image processing unit 111, and it corrects to a planar image.

[0068] The image quality acquisition unit 513 acquires the image quality of the image in the same manner as the image quality acquisition unit 112. The image quality correction unit 514 corrects / converts the image so that the image quality becomes the reference image quality in the same manner as the image quality correction unit 113.

[0069] The in-warehouse background acquisition unit 515 acquires the shape, hue, position, etc. of the inside of the refrigerator (refrigerating chamber) such as the shelves, drawers, and walls shown in the image in the same manner as the in-warehouse background acquisition unit 114. The in-warehouse background correction unit 516 corrects the image so that the in-warehouse background becomes the reference in-warehouse background in the same manner as the in-warehouse background correction unit 115.

[0070] The identification unit 517 recognizes and identifies the food shown in the image corrected by the image quality correction unit 514 and the in-warehouse background correction unit 516 using the identification model 531. The identification unit 517 stores the area and name where the food, which is the identification result, is shown in the inventory database 540.

[0071] The communication unit 518 receives the identification model 131, the reference image quality information 132, and the reference in - storage background information 133 transmitted by the distribution system 600, and stores them in the identification model 531, the reference image quality information 532, and the reference in - storage background information 533 respectively. Also, the communication unit 518 transmits the image inside the storage taken by the camera 591 to the distribution system 600. The transmitted image is stored in the image database 140 and used as a teacher image or an evaluation image.

[0072] In response to the request of the terminal 690, the communication unit 518 transmits the data in the inventory database 540. For example, the communication unit 518 transmits a list of shooting dates in the inventory database 540, or transmits an image and an identification result (a list of foods shown in the image) of a specified shooting date. Also, for example, the communication unit 518 may search for the name of the food that is the keyword included in the request, and transmit a list of shooting dates as a search result. The list may include images and identification results.

[0073] As described above, the refrigerator 500 includes an image quality acquisition unit 513 that acquires the image quality of the image taken inside the storage. The refrigerator 500 includes an in - storage background acquisition unit 515 that acquires the in - storage background inside the storage shown in the image. The refrigerator 500 includes an image quality correction unit 514 that corrects the image quality of the image to a predetermined image quality (reference image quality). The refrigerator 500 includes an in - storage background correction unit 516 that corrects the in - storage background of the image to a predetermined in - storage background (reference in - storage background).

[0074] The refrigerator 500 is trained using the teacher data 150 with the image corrected to a predetermined image quality and a predetermined in - storage background as the explanatory variable, and the area of the article shown in the corrected image and the name of the article as the target variable, and is generated using an identification model 531, which is a machine - learning model for identifying the article shown in the corrected image. The refrigerator 500 includes an identification unit 517 that identifies the article shown in the image using the identification model 531.

[0075] ≪Food Identification Process≫ FIG. 11 is a flowchart of food identification processing according to the second embodiment. The food identification processing is processing executed after the camera 591 photographs the inside of the storage. In step S51, the image processing unit 512 processes the image of the inside of the storage photographed by the photographing unit 511 and stores it in the inventory database 540 together with the photographing date and time.

[0076] In step S52, the image quality acquisition unit 513 acquires the image quality of the image stored in the inventory database 540 in step S51. In step S53, the image quality correction unit 514 corrects / converts the image so that the image quality becomes the reference image quality.

[0077] In step S54, the inside storage background acquisition unit 515 acquires the shape, hue, and position of the shelves, drawers, walls, etc. inside the storage shown in the image corrected in step S53. In step S55, the inside storage background correction unit 516 corrects the image so that the inside storage background becomes the reference inside storage background. In step S56, the identification unit 517 recognizes and identifies the food shown in the images corrected in steps S53 and S55 using the identification model 531, and stores the identification result in the inventory database 540.

[0078] ≪Features of the refrigerator≫ The user of the refrigerator can check the information stored in the inventory database 540 using the terminal 690. Thereby, the user of the refrigerator can check the inventory of the food in the refrigerator when away from home, and can prevent purchasing food with sufficient inventory wastefully. Also, the user can prevent forgetting to buy food that is out of stock by checking those with unknown inventory availability.

[0079] ≪Modification example: refrigerator≫ The refrigerator 500 according to the second embodiment corrects the image inside the refrigerator taken by the camera 591, identifies food, and transmits the identification result to the terminal 690. The processing after correction may be performed not by the refrigerator 500 but by a server on the cloud. The refrigerator 500 sends the image taken by the camera 591 to the server. A server including an image processing unit 512, an image quality acquisition unit 513, an image quality correction unit 514, an in-refrigerator background acquisition unit 515, an in-refrigerator background correction unit 516, an identification unit 517, an identification model 531, and an inventory database 540 may identify the food shown in the image, store it in the inventory database 540, and transmit the image and the identification result in response to a request from the terminal 690. In this case, since the computing power of the food identification unit 510 required for the processing after correction becomes unnecessary, the cost of the refrigerator 500 can be reduced. Also, the refrigerator 500 may be a storage for storing articles including, but not limited to, food, and having a freezing and refrigerating function. Examples of such a storage include a warm storage, a dehumidifying storage, a medicine storage, etc., and the present invention is applicable to various storages.

[0080] ≪Other Modification Examples≫ As described above, some embodiments of the present invention have been described, but these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and furthermore, various changes such as omission and substitution can be made without departing from the gist of the present invention. These embodiments and their modifications are included in the scope and gist of the invention described in this specification, etc., and are also included in the scope of the invention described in the claims and its equivalents.

[0081] For example, although the in-warehouse background was exemplified as something that affects the identification accuracy by the identification model, it is not limited to this. An image correction device may be provided that includes a background acquisition unit that acquires, for example, the shape, hue, pattern, position, and other properties of part or all of the background, which is other than the object to be identified such as an article, and a background correction unit that corrects the background of the image to a predetermined background. For example, as an example other than the in-warehouse background, there are articles and structures other than the object to be identified indoors, such as walls, pillars, furniture with low movement frequency (e.g., wardrobes, etc.), and home appliances with low movement frequency (e.g., air conditioners, large TVs).

Explanation of Signs

[0082] 100 Image correction device 111 Image processing unit 112 Image quality acquisition unit 113 Image quality correction unit 114 In-warehouse background acquisition unit 115 In-warehouse background correction unit 116 Reference information calculation unit 117 Learning unit 118 Identification accuracy evaluation unit 131 Identification model 132 Reference image quality information 133 Reference in-warehouse background information 138 Program 140 Image database 150 Teacher data 500 Refrigerator 511 Photographing unit 512 Image processing unit 513 Image quality acquisition unit 514 Image quality correction unit 515 In-warehouse background acquisition unit 516 In-warehouse background correction unit 517 Identification unit 518 Communication unit 531 Identification model 532 Reference image quality information 533 Reference in-warehouse background information 540 Inventory database 591 Camera

Claims

1. An image quality acquisition unit that acquires the image quality of an image taken inside a storage; An in-warehouse background acquisition unit that acquires the in-warehouse background inside the storage shown in the image; An image quality correction unit that corrects the image quality of the image to a predetermined image quality; An in-warehouse background correction unit that corrects the in-warehouse background of the image to a predetermined in-warehouse background, comprising An image correction device.

2. Using the image corrected to the predetermined image quality and the predetermined in-warehouse background as explanatory variables, and using teacher data with the area of the article shown in the corrected image and the name of the article as target variables, a learning unit that trains and generates an identification model, which is a machine learning model for identifying the article shown in the corrected image, is further provided The image correction device according to Claim 1.

3. Further comprising an identification accuracy evaluation unit that calculates the identification accuracy of the identification model, The learning unit If the identification accuracy is less than a predetermined value, using the image corrected to the predetermined image quality and the predetermined in-warehouse background not included in the teacher data as explanatory variables, and using additional teacher data with the area of the food shown in the corrected image and the name of the food as target variables, the identification model is retrained and generated The image correction device according to Claim 2.

4. The image quality Includes at least any one of the aspect ratio, contrast, brightness, resolution, noise, size, color, edge, and distortion of the image The image correction device according to Claim 1.

5. The in-warehouse background Includes at least any one of the shape, hue, and position inside the storage shown in the image The image correction device according to Claim 1.

6. An image quality acquisition unit that acquires the image quality of an image taken inside a storage; An in-warehouse background acquisition unit that acquires the in-warehouse background inside the storage shown in the image; An image quality correction unit that corrects the image quality of the image to a predetermined image quality; An in-warehouse background correction unit that corrects the in-warehouse background of the image to a predetermined in-warehouse background; An identification unit that uses an identification model, which is a machine learning model for identifying an article shown in the corrected image, trained and generated using teacher data with the image corrected to the predetermined image quality and the predetermined in-warehouse background as explanatory variables, and the area of the article shown in the corrected image and the name of the article as target variables, to identify the article shown in the image, is provided A storage.

7. When the image correction device Performs a step of acquiring the image quality of an image taken inside a storage; Performs a step of acquiring the in-warehouse background inside the storage shown in the image; Performs a step of correcting the image quality of the image to a predetermined image quality; Executing a step of correcting the in-warehouse background of the image to a predetermined in-warehouse background Image correction method.

8. A teacher data used for training an identification model, which is a machine learning model for identifying an object to be identified shown in an image, and an image correction apparatus for correcting the image, A background acquisition unit that acquires properties other than the object to be identified shown in the image, A background correction unit that corrects the background of the image to a background predetermined as the background of the teacher data, and is provided with Image correction apparatus.

Images