Display method, program, and display system

Abstract

This display method comprises: an acquisition step (S13) of acquiring detection frame information indicating the positions and sizes of detection frames indicating food captured in an image of the inside of a refrigerator (20) detected by a machine learning model; a selection step (S14) of selecting a plurality of first detection frames indicating the same type of food from among the detection frames indicated by the acquired detection frame information; a generation step (S14) of generating a second detection frame obtained by merging a pair of first detection frames satisfying a predetermined requirement among the plurality of selected first detection frames; and an output step (S15) of outputting information for displaying the generated second detection frame on the image.

Description

Display method, program, and display system

[0001] The present disclosure relates to a display method, a program, and a display system.

[0002] Conventionally, technologies relating to storage cabinets for storing objects have been proposed. Patent Document 1 discloses a recipe display system that can suggest recipes for foods stored in a refrigerator. In the display system of Patent Document 1, a food selected by a user is surrounded by a square frame in an image of the entire vegetable compartment.

[0003] JP 2023-79899 A

[0004] When a detection frame is superimposed on each of the multiple foods in an image showing multiple foods, there is room for further consideration regarding the method of displaying the detection frames.

[0005] The present disclosure provides a display method that can reduce the number of detection frames superimposed on multiple foods of the same type and improve the visibility of the detection frames.

[0006] A display method according to one aspect of the present disclosure is a computer-executed method for displaying a plurality of foods stored in a storage cabinet, the storage cabinet having a function of refrigerating the plurality of foods, the display method including: an acquisition step of acquiring detection frame information indicating positions and sizes of detection frames indicating the foods reflected in an image inside the storage cabinet, detected by a machine learning model; a selection step of selecting a plurality of first detection frames indicating the same type of food from the detection frames indicated by the acquired detection frame information; a generation step of generating a second detection frame obtained by merging pairs of first detection frames that satisfy predetermined requirements from the selected plurality of first detection frames; and an output step of outputting information for displaying the generated second detection frames on the image.

[0007] A display method according to one aspect of the present disclosure can reduce the number of detection frames superimposed on multiple foods of the same type, thereby improving the visibility of the detection frames.

[0008] FIG. 1 is a block diagram illustrating a functional configuration of a display system according to an embodiment. FIG. 2 is a diagram illustrating an example of a display screen for an image inside a refrigerator. FIG. 3 is a diagram illustrating another example of a display screen for an image inside a refrigerator. FIG. 4 is an external view of a refrigerator and an imaging device provided in a display system according to an embodiment. FIG. 5 is a sequence diagram illustrating a display operation of a food display screen. FIG. 6 is a first diagram illustrating a distance between X coordinates. FIG. 7 is a second diagram illustrating a distance between X coordinates. FIG. 8 is a first diagram illustrating a distance between Y coordinates. FIG. 9 is a second diagram illustrating a distance between Y coordinates. FIG. 10 is a diagram illustrating a problem with example 1 of detection frame merging processing. FIG. 11 is a diagram illustrating occupancy rates. FIG. 12 is a flowchart of example 2 of detection frame merging processing. FIG. 13 is a diagram illustrating changes in detection frames in example 2 of detection frame merging processing. FIG. 14 is a flowchart of example 3 of detection frame merging processing. FIG. 15 is a diagram illustrating changes in detection frames in example 3 of detection frame merging processing. FIG. 16 is a flowchart of example 4 of detection frame merging processing. FIG. 17 is a flowchart of Example 5 of the detection frame merging process.

[0009] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0010] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0011] (Embodiment) [Configuration] First, the configuration of a display system according to an embodiment will be described. Fig. 1 is a block diagram showing the functional configuration of a display system according to an embodiment.

[0012] The display system 10 shown in Fig. 1 is a system for displaying an image of the inside of a refrigerator 20, which shows food stored in the refrigerator 20, on a display unit 51 of an information terminal 50. Fig. 2 is a diagram showing an example of a display screen of the image of the inside of the refrigerator 20 displayed on the display unit 51.

[0013] 2, a rectangular detection frame is superimposed on the food. The detection frame is added by the server device 40 using a machine learning model, for example. The detection frame is also sometimes called a bounding box.

[0014] Generally, when multiple foods of the same type are shown in an image, a detection frame is superimposed on each of the multiple foods individually. However, when multiple foods are placed close to each other, if a detection frame is superimposed on each of the multiple foods individually, the detection frames may overlap and be difficult to see.

[0015] Therefore, the display system 10 merges the detection frames and displays them. Fig. 3 is a diagram showing an example of a display screen of an image inside a refrigerator with merged detection frames. As can be seen by comparing Fig. 2 with Fig. 3, in Fig. 3, the two detection frames superimposed on two eggplants are merged into one, the two detection frames superimposed on two potatoes are merged into one, and the two detection frames superimposed on two cucumbers are merged into one.

[0016] In this way, display system 10 can reduce the number of detection frames superimposed on an image and improve the visibility of the detection frames. Display system 10 specifically includes refrigerator 20, image capturing device 30, server device 40, and information terminal 50.

[0017] The refrigerator 20 is an example of a storage unit capable of refrigerating stored items, and is installed in a user's home or the like to refrigerate food. Fig. 4 is an external view of the refrigerator 20 (and the image capturing device 30).

[0018] The image capturing device 30 captures an image of a drawer (e.g., a vegetable drawer) of the refrigerator 20 in an open state from above the refrigerator 20. That is, the image capturing device 30 captures, for example, an image of the drawer of the refrigerator 20 viewed from above the refrigerator 20. The image capturing device 30 is realized, for example, by a camera having a wide-angle lens and a telephoto lens.

[0019] Server device 40 is a computer located outside the facility where refrigerator 20 is installed, and specifically, is a cloud server. Server device 40 performs information processing for merging detection frames and displaying the merged detection frames on information terminal 50. Server device 40 includes a communication unit 41, an information processing unit 42, and a storage unit 43.

[0020] The communication unit 41 is a communication circuit that enables the server device 40 to communicate with the image capturing device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard used for communication by the communication unit 41.

[0021] The information processing unit 42 performs information processing for merging the detection frames and displaying them on the information terminal 50. The information processing unit 42 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 42 has, as functional components, a detection unit 44, an acquisition unit 45, a selection unit 46, a generation unit 47, and an output unit 48. The functions of the detection unit 44, the acquisition unit 45, the selection unit 46, the generation unit 47, and the output unit 48 are realized, for example, by a microcomputer or the like constituting the information processing unit 42 executing a computer program stored in the storage unit 43. The functions of the detection unit 44, the acquisition unit 45, the selection unit 46, the generation unit 47, and the output unit 48 will be described in detail below.

[0022] The storage unit 43 is a storage device that stores the computer programs executed by the information processing unit 42 and various information required for the information processing. The storage unit 43 is realized by, for example, a semiconductor memory.

[0023] The information terminal 50 is an information terminal owned by the user. The user uses the information terminal 50 to check the food stored in the refrigerator 20 while away from home. The information terminal 50 is, for example, a portable information terminal such as a smartphone or a tablet terminal, but may also be a stationary information terminal such as a personal computer. The information terminal 50 includes a display unit 51.

[0024] A food display screen such as that shown in Fig. 3 is displayed on the display unit 51. The display unit 51 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel.

[0025] [Display Operation of Food Display Screen] Next, a description will be given of a display operation of the food display screen on the information terminal 50, which is performed by the display system 10. Fig. 5 is a sequence diagram of the display operation of the food display screen.

[0026] The communication unit 41 of the server device 40 communicates with the photographing device 30 to receive an image (image information) of the inside of the refrigerator 20 from the photographing device (S11).

[0027] The detection unit 44 performs object detection processing on the received image to detect food appearing in the image, and generates detection frame information indicating the position and size of a detection frame showing the detected food (S12). Specifically, the detection unit 44 detects objects (food) in the image using a machine learning model (trained model), and generates detection frame information indicating the position and size of a detection frame surrounding the detected food.

[0028] The acquisition unit 45 acquires the generated detection frame information (S13). That is, the acquisition unit 45 acquires detection frame information indicating the position and size of the detection frame that indicates the food in the image inside the refrigerator 20, which has been detected by the machine learning model.

[0029] The selection unit 46 and the generation unit 47 perform a process of merging detection frames that satisfy predetermined requirements among the detection frames indicated by the acquired detection frame information (S14). Details of the process of step S14 (also referred to as a detection frame merging process) will be described later.

[0030] The generation unit 47 generates merged detection frame information that indicates the position and size of the detection frame after the processing of step S14 has been performed (S15). The merged detection frame information is an example of information for displaying the merged detection frame in the image acquired in step S11.

[0031] The output unit 48 communicates with the information terminal 50 using the communication unit, and outputs (transmits) the image and post-merged detection frame information received in step S11 to the information terminal 50 (S16).

[0032] The information terminal 50 receives the image and the merged detection frame information, and displays a food display screen such as that shown in FIG. 3 on the display unit 51 based on the received image and merged detection frame information (S17).

[0033] In this way, the display system 10 acquires detection frame information indicating the position and size of the detection frames that indicate the food in the image inside the refrigerator 20, which was detected by the machine learning model, merges pairs of detection frames that satisfy specified requirements from among the detection frames indicated by the acquired detection frame information, and outputs merged detection frame information for displaying the merged detection frames on the image.

[0034] This allows the display system 10 to reduce the number of detection frames superimposed on the image, and improve the visibility of the detection frames.

[0035] [Example 1 of Detection Frame Merging Process] In step S14, the selection unit 46 selects multiple detection frames that represent the same type of food, and the generation unit 47 merges pairs of detection frames that satisfy a distance requirement from among the selected multiple detection frames. Specifically, the generation unit 47 merges pairs of detection frames when the requirement that the distance between the pairs of detection frames is equal to or less than a threshold is satisfied.

[0036] Here, the distance between the pair of detection frames is expressed by the number of pixels in the image of the inside of refrigerator 20. More specifically, the above-mentioned predetermined requirement is that the sum of the distance between the X coordinates of the pair of detection frames and the distance between the Y coordinates of the pair of detection frames be equal to or smaller than a threshold value.

[0037] The distance between the X coordinates of a pair of detection frames is expressed by the following formula.

[0038] Distance between X coordinates = max (Xmin of the right detection frame - Xmax of the left detection frame, 0)

[0039] 6 and 7 are diagrams for explaining the distance between X coordinates. In the example of Fig. 6, the distance between X coordinates is X2min-X1max (>0). In the example of Fig. 7, X2min-X1max<0, so the distance between X coordinates is 0.

[0040] The distance between the Y coordinates of a pair of detection frames is expressed by the following formula.

[0041] Distance between Y coordinates = max (Ymin of the lower detection frame - Ymax of the upper detection frame, 0)

[0042] 8 and 9 are diagrams for explaining the distance between Y coordinates. In the example of Fig. 8, the distance between Y coordinates is Y2min-Y1max (>0). In the example of Fig. 9, Y2min-Y1max<0, so the distance between Y coordinates is 0.

[0043] The merged detection frame can be defined by the Xmin of the left detection frame, the Xmax of the right detection frame, the Ymin of the upper detection frame, and the Ymax of the lower detection frame. In the example of Fig. 9, the merged detection frame is a rectangular detection frame with four vertices: (X1min, Y1min), (X2max, Y1min), (X1min, Y2max), and (X2max, Y2max).

[0044] [Issues and Solutions for Example 1 of Detection Frame Merging Processing] Figure 10 is a diagram for explaining the issues with Example 1 of detection frame merging processing. When two elongated vegetables such as cucumbers are arranged as shown in Figure 10, the distance between the pair of detection frames is short, so the detection frames are merged. However, the area in which the cucumbers are present within the merged detection frames is small, and there is a high possibility that vegetables other than cucumbers will be included within the merged detection frames.

[0045] Therefore, a requirement regarding occupancy may be defined as a requirement for whether or not to merge a pair of detection frames. The occupancy indicates the ratio of the area corresponding to the union of the detection frames from which the detection frames are merged to the area of ​​the detection frames after the merge. FIG. 11 is a diagram for explaining the occupancy.

[0046] 11 , when detection frame C is obtained by merging detection frame A and detection frame B, the occupancy rate is the ratio of the area equivalent to the union of detection frame A and detection frame B (A∪B) (corresponding to the diagonal hatching in FIG. 11 ) to the area of ​​detection frame C. By merging pairs of detection frames with the requirement that such an occupancy rate be equal to or greater than a predetermined rate, it is possible to prevent pairs of detection frames from being merging in cases where the occupancy rate is low, as in FIG. 10 .

[0047] [Detection Frame Merging Process Example 2] An example of merging detection frames using a requirement related to occupancy in addition to a requirement related to the distance between the detection frames will be described below. Fig. 12 is a flowchart of detection frame merging process example 2. The detection frame merging process in Fig. 12 corresponds to the process performed in step S14 in Fig. 5.

[0048] For simplicity of explanation, the following describes an example of merging the detection frames of one type of food. However, if the detection frame information obtained in step S13 of FIG. 5 includes m types of food (m is a natural number greater than or equal to 2), the processing of FIG. 12 is performed for each of the m types of food.

[0049] The selection unit 46 of the server device 40 selects first detection frames that indicate the same type of food from the detection frames indicated by the detection frame information acquired in step S13 (S14a), and generates a candidate list that lists the selected first detection frames (S14b). The same type means, for example, that the specific variety of vegetable (e.g., cucumber, tomato) is the same, but may also mean that a more general type (e.g., vegetable, meat, beverage) is the same.

[0050] The generation unit 47 generates a pair list that lists all pairs of first detection frames that can be obtained from the first detection frames listed in the candidate list (S14c). The generation unit 47 determines whether the pair list includes a pair of first detection frames whose distance is equal to or less than a threshold (S14d). If it is determined that the pair list does not include a pair of first detection frames whose distance is equal to or less than the threshold (No in S14d), the processing ends.

[0051] When generating unit 47 determines that the pair list includes a pair of first detection frames whose distance is equal to or less than the threshold (Yes in S14d), it extracts a pair from the pair list whose distance between the first detection frames is the shortest (S14e). That is, in step S14e, pairs whose distance between the first detection frames is shorter are preferentially extracted. When there are two or more pairs whose distance between the first detection frames is the shortest, pairs whose degree of overlap between the detection frames (IoU: Intersection over Union) is the largest are preferentially extracted.

[0052] The generation unit 47 determines whether the occupancy rate when the extracted pair of first detection frames is merged is equal to or greater than a predetermined rate (S14f). If the generation unit 47 determines that the occupancy rate is equal to or greater than the predetermined rate (Yes in S14f), the generation unit 47 merges the extracted pair of first detection frames (S14g). Furthermore, the generation unit 47 updates the candidate list by deleting the two first detection frames to be merged from the candidate list generated in step S14b and adding the merged first detection frames (new first detection frames) to the candidate list (S14h). Thereafter, the generation unit 47 generates a pair list using the updated candidate list (S14c).

[0053] On the other hand, if the generation unit 47 determines that the occupancy rate is less than the predetermined rate (No in S14f), the generation unit 47 updates the pair list by not merging the extracted pair of first detection frames and deleting the pair of first detection frames from the pair list (S14i). Thereafter, the generation unit 47 determines whether the updated pair list includes any first detection frames whose distance is equal to or less than the threshold (S14d).

[0054] The above-described second example of the detection frame merging process will be specifically described with reference to the detection frames. Fig. 13 is a diagram showing changes in the detection frames in the second example of the detection frame merging process.

[0055] Initially, four first detection frames A to D are listed in the candidate list, and six pairs of first detection frames are listed in the pair list ((a) of FIG. 13). If the distance is equal to or less than the threshold and the pair with the shortest distance is first detection frame A and B, generation unit 47 determines whether the occupancy rate when first detection frame A and B are merged is equal to or greater than a predetermined rate ((b) of FIG. 13).

[0056] The generation unit 47 determines that the occupancy rate is less than a predetermined rate and does not merge the first detection frames A and B. If the distance is equal to or less than the threshold and the pair with the second shortest distance is the first detection frames B and C, the generation unit 47 determines whether the occupancy rate when the first detection frames B and C are merged is equal to or greater than a predetermined rate ((c) of FIG. 13 ).

[0057] Generation unit 47 determines that the occupancy rate is equal to or greater than a predetermined rate, and generates a new first detection frame E by merging first detection frames B and C ((d) of FIG. 13). As a result, three first detection frames, A, D, and E, are listed in the candidate list, and three pairs of first detection frames are listed in the pair list. Specifically, the three pairs are the pair of first detection frames A and E, the pair of first detection frames D and E, and the pair of first detection frames A and D.

[0058] If the distance is below the threshold and the pair with the shortest distance is the first detection frame A and E, the generation unit 47 determines whether the occupancy rate when the first detection frames A and E are merged is above a predetermined percentage.

[0059] The generation unit 47 determines that the occupancy rate is less than a predetermined rate and does not merge the first detection frames A and E. If the distance is equal to or less than the threshold and the pair with the second shortest distance is the first detection frames D and E, the generation unit 47 determines whether the occupancy rate when the first detection frames D and E are merged is equal to or greater than a predetermined rate ((e) of FIG. 13 ).

[0060] The generation unit 47 determines that the occupancy rate is less than a predetermined ratio, and does not merge first detection frames D and E. The remaining pair of first detection frames A and D does not satisfy the requirement that the distance is equal to or less than the threshold, and therefore the operation ends.

[0061] 13 is performed in step S14 of Fig. 5 , output unit 48 generates post-merged detection frame information in step S15 of Fig. 5 that indicates the position and size of each of first detection frames A, D, and E. Note that new first detection frame E obtained by merging at least one pair of first detection frames is also referred to as a second detection frame.

[0062] In this way, server device 40 (generation unit 47) extracts pairs of first detection frames that satisfy the distance requirement, and if the extracted pairs of first detection frames satisfy the occupancy requirement, generates new first detection frames by merging the pairs of first detection frames, and adds the generated new first detection frames to the extraction candidates. This process is repeated until no new first detection frames can be generated, and the last new first detection frame generated is generated as a second detection frame. When extracting pairs of first detection frames that satisfy the distance requirement, pairs with shorter distances between their first detection frames are given higher extraction priority. The occupancy requirement is that the proportion of the area of ​​the new first detection frame that corresponds to the union of the pairs of first detection frames to be merged is equal to or greater than a predetermined proportion.

[0063] Such a server device 40 can prevent unintended foods from being included in the second detection frame by repeatedly merging the first detection frame, with the requirement that the occupancy rate be greater than a predetermined percentage.

[0064] [Detection Frame Merging Process Example 3] In detection frame merging process example 2, two first detection frames that satisfy both the distance requirement between the pair of detection frames and the occupancy requirement are repeatedly merged. In detection frame merging process example 3, an example will be described in which, after repeating the merging of two first detection frames that satisfy the distance requirement between the pair of detection frames, the merging is finalized when the last obtained first detection frame satisfies the occupancy requirement. Figure 14 is a flowchart of detection frame merging process example 3. The detection frame merging process in Figure 14 corresponds to the processing performed in step S14 of Figure 5.

[0065] For simplicity of explanation, the following describes an example of merging the detection frames of one type of food. However, if the detection frame information obtained in step S13 of FIG. 5 includes m types of food (m is a natural number greater than or equal to 2), the processing of FIG. 14 is performed for each of the m types of food.

[0066] The selection unit 46 of the server device 40 selects a first detection frame that indicates the same type of food from the detection frames indicated by the detection frame information acquired in step S13 (S14a), and generates a candidate list that lists the selected first detection frames (S14b).

[0067] The generation unit 47 generates a pair list that lists all pairs of first detection frames that can be obtained from the first detection frames listed in the candidate list (S14c). The generation unit 47 determines whether the pair list includes a pair of first detection frames whose distance is equal to or less than a threshold (S14d). If it is determined that the pair list does not include a pair of first detection frames whose distance is equal to or less than the threshold (No in S14d), the generation unit 47 determines whether merging of first detection frames has been performed at least once (S14j). In this case, it is determined that merging of first detection frames has not been performed even once (No in S14j), and the processing ends.

[0068] When generating unit 47 determines that the pair list includes a pair of first detection frames whose distance is equal to or less than the threshold (Yes in S14d), it extracts a pair from the pair list whose distance between the first detection frames is the shortest (S14e). That is, in step S14e, pairs whose distance between the first detection frames is shorter are preferentially extracted. When there are two or more pairs whose distance between the first detection frames is the shortest, pairs whose degree of overlap between the detection frames (IoU: Intersection over Union) is the largest are preferentially extracted.

[0069] The generation unit 47 merges the extracted pairs of first detection frames (S14g). The generation unit 47 also updates the candidate list by deleting the two first detection frames to be merged from the candidate list generated in step S14b and adding the merged first detection frames (new first detection frames) to the candidate list (S14h). Thereafter, the generation unit 47 generates a pair list using the updated candidate list (S14c).

[0070] If the generation unit 47 determines that the pair list includes a pair of first detection frames whose distance is equal to or less than the threshold (Yes in S14d), the processes of steps S14e, S14g, S14h, and S14c are performed again. On the other hand, if the generation unit 47 determines that the pair list does not include a pair of first detection frames whose distance is equal to or less than the threshold (No in S14d), the generation unit 47 determines whether merging of first detection frames has been performed at least once (S14j). In this case, it is determined that merging of first detection frames has been performed (Yes in S14j), and the generation unit 47 determines whether the occupancy rate of each of the one or more last-obtained first detection frames is equal to or greater than a predetermined rate (S14k).

[0071] If the one or more first detection frames obtained last are detection frames obtained by merging n (N, n is a natural number greater than or equal to 2, N≧n) first detection frames out of the N first detection frames included in the original candidate list generated in step S14b, the occupancy rate is the proportion of the area of ​​the last new first detection frame generated that corresponds to the union of the n first detection frames. In other words, the occupancy rate is the proportion of the area of ​​the last new first detection frame generated that corresponds to the union of the n original first detection frames.

[0072] For a first detection frame whose occupancy rate is determined to be equal to or greater than a predetermined rate among the one or more last-obtained first detection frames (Yes in S14k), the generation unit 47 confirms the merging up to that point as valid (S14l). On the other hand, for a first detection frame whose occupancy rate is determined to be less than a predetermined rate among the one or more last-obtained first detection frames (No in S14k), the generation unit 47 cancels all merging of the detection frames up to that point (S14m). In other words, the detection frames are not merged, and are returned to the original n number of detection frames to be merged.

[0073] The above-described third example of the detection frame merging process will be specifically described with reference to the detection frames. Fig. 15 is a diagram showing changes in the detection frames in the third example of the detection frame merging process.

[0074] Initially, four first detection frames A to D are listed in the candidate list, and six pairs of first detection frames are listed in the pair list ((a) of FIG. 15). If the distance is equal to or less than the threshold and the pair with the shortest distance is first detection frame A and B, generation unit 47 generates a new first detection frame E by merging first detection frames A and B ((b) of FIG. 15).

[0075] As a result, three first detection frames, C, D, and E, are listed in the candidate list, and three pairs of first detection frames are listed in the pair list. Specifically, the three pairs are the pair of first detection frames C and E, the pair of first detection frames C and D, and the pair of first detection frames D and E.

[0076] If the distance is equal to or less than the threshold and the pair with the shortest distance is first detection frame C and E, generation unit 47 generates a new first detection frame F by merging first detection frames C and E (FIG. 15(c)). As a result, the two first detection frames, first detection frames D and F, are listed in the candidate list, and only the pair of first detection frames D and F is listed in the pair list.

[0077] If the distance between the pair of first detection frames D and F is equal to or less than the threshold, the generating unit 47 generates a new first detection frame G by merging first detection frames D and F ((d) of FIG. 15).

[0078] First detection frame G is a detection frame obtained by merging the initial first detection frames A, B, C, and D. Therefore, generation unit 47 determines whether the area corresponding to the union of first detection frames A, B, C, and D is equal to or greater than a predetermined ratio relative to the area of ​​first detection frame G. If generation unit 47 determines that the area corresponding to the union of first detection frames A, B, C, and D is equal to or greater than the predetermined ratio relative to the area of ​​first detection frame G, it finalizes the merging process that has been carried out so far.

[0079] 15 is performed in step S14 of Fig. 5 , output unit 48 generates merged detection frame information in step S15 of Fig. 5 that indicates the position and size of each detection frame of first detection frame G. Note that a new first detection frame G obtained by merging at least one pair of first detection frames is also referred to as a second detection frame.

[0080] Note that if generation unit 47 determines that the area corresponding to the union of first detection frames A, B, C, and D relative to the area of ​​first detection frame G is less than a predetermined ratio, generation unit 47 cancels the merging that has been performed so far. In this case, detection frames A, B, C, and D are displayed individually in the image.

[0081] In this way, server device 40 (generation unit 47) extracts pairs of first detection frames that satisfy the distance requirement, generates new first detection frames by merging the pairs of first detection frames, and adds the generated new first detection frames to extraction candidates, repeating this process until no new first detection frames can be generated. When extracting pairs of first detection frames that satisfy the distance requirement, pairs with shorter distances between the first detection frames are preferentially extracted. If the last generated new first detection frame satisfies the occupancy requirement, the new first detection frame is generated as a second detection frame. If the last generated new first detection frame is obtained by merging n (N, n is a natural number greater than or equal to 2, N≧n) first detection frames out of the N first detection frames selected by selection unit 46, the occupancy requirement is that the proportion of the area of ​​the last generated new first detection frame that corresponds to the union of the n first detection frames is equal to or greater than a predetermined proportion.

[0082] Such a server device 40 can prevent unintended foods from being included in the second detection frame by determining whether to merge the first detection frame, provided that the occupancy rate is equal to or greater than a predetermined percentage.

[0083] Note that example 2 of the detection frame merging process and example 3 of the detection frame merging process may be combined. For example, in example 2 of the detection frame merging process, first detection frames A, B, and C are listed in a candidate list, and first detection frame D, which is obtained by merging first detection frame A and B, is obtained by merging first detection frame C with first detection frame C. In this case, the requirement regarding occupancy rate imposed for merging first detection frame C and D is that the area equivalent to the union of first detection frame C and D with respect to the area of ​​first detection frame E is equal to or greater than a predetermined ratio. However, in consideration of example 3 of the detection frame merging process, the requirement regarding occupancy rate imposed for merging first detection frame C and D may also be that the area equivalent to the union of first detection frame A, B, and C with respect to the area of ​​first detection frame E is equal to or greater than a predetermined ratio.

[0084] [Detection Frame Merging Process Example 4] In detection frame merging process examples 2 and 3, pairs with shorter distances between their first detection frames were preferentially extracted. Here, when extracting pairs of first detection frames, pairs with higher occupancy rates may be preferentially extracted. An example in which pairs with higher occupancy rates are preferentially extracted will be described. Figure 16 is a flowchart of detection frame merging process example 4. The detection frame merging process in Figure 16 corresponds to the process performed in step S14 of Figure 5.

[0085] For simplicity of explanation, the following describes an example of merging the detection frames of one type of food. However, if the detection frame information obtained in step S13 of FIG. 5 includes m types of food (m is a natural number greater than or equal to 2), the processing of FIG. 16 is performed for each of the m types of food.

[0086] The selection unit 46 of the server device 40 selects a first detection frame that indicates the same type of food from the detection frames indicated by the detection frame information acquired in step S13 (S14a), and generates a candidate list that lists the selected first detection frames (S14b).

[0087] The generation unit 47 generates a pair list that lists all pairs of first detection frames that can be obtained from the first detection frames listed in the candidate list (S14c). The generation unit 47 determines whether the pair list includes a pair of first detection frames whose distance is equal to or less than a threshold and whose occupancy rate is equal to or greater than a predetermined rate (S14n). If it is determined that the pair list does not include a pair of first detection frames whose distance is equal to or less than the threshold and whose occupancy rate is equal to or greater than a predetermined rate (No in S14n), the processing ends.

[0088] When the generation unit 47 determines that the pair list includes a pair of first detection frames whose distance is equal to or less than the threshold and whose occupancy rate is equal to or greater than a predetermined rate (Yes in S14n), the generation unit 47 extracts the pair with the highest occupancy rate from the pair list (S14o). That is, in step S14o, pairs with higher occupancy rates are preferentially extracted. When there are two or more pairs with the same occupancy rate, the pair with the shortest distance is preferentially extracted.

[0089] The generation unit 47 merges the extracted pairs of first detection frames (S14g). The generation unit 47 also updates the candidate list by deleting the two first detection frames to be merged from the candidate list generated in step S14b and adding the merged first detection frames (new first detection frames) to the candidate list (S14h). Thereafter, the generation unit 47 generates a pair list using the updated candidate list (S14c).

[0090] In this way, server device 40 (generation unit 47) extracts pairs of first detection frames that satisfy the distance-related requirements and the occupancy rate requirements, and repeats the process of generating new first detection frames by merging the pairs of first detection frames until no new first detection frames can be generated, and generates the last new first detection frame as a second detection frame. The distance-related requirement is a requirement that the distance between the pairs of first detection frames is equal to or less than a threshold, with pairs with shorter distances being given priority. The occupancy rate requirement is a requirement that the proportion of the area corresponding to the union of the pairs of first detection frames to the area of ​​the detection frame obtained by merging the pairs of first detection frames is equal to or greater than a predetermined proportion, with pairs with higher area proportions being given priority. In extracting pairs of first detection frames that satisfy the distance-related requirements and the occupancy rate requirements, pairs of first detection frames with a higher occupancy rate are preferentially extracted over pairs of first detection frames with a short distance between them.

[0091] By preferentially merging pairs of first detection frames with high occupancy rates, such server device 40 can prevent unintended foods from being included in the second detection frames. According to the inventors' studies, example 4 of the detection frame merging process is more effective at preventing unintended foods from being included in the second detection frames after merging than examples 2 and 3 of the detection frame merging process.

[0092] [Detection Frame Merging Process Example 5] It is not necessary to impose a distance requirement on a pair of first detection frames in order for the pair to be merged. When merging a pair of first detection frames, it is sufficient that at least one of a distance requirement and an occupancy requirement is imposed as a predetermined requirement.

[0093] For example, in order to merge a pair of first detection frames, a distance requirement may not be imposed on the pair, but an occupancy requirement may be imposed on the pair. A fifth example of such a detection frame merging process will now be described. Fig. 17 is a flowchart of the fifth example of the detection frame merging process. The detection frame merging process of Fig. 17 corresponds to the process performed in step S14 of Fig. 5.

[0094] For simplicity of explanation, the following describes an example of merging the detection frames of one type of food. However, if the detection frame information obtained in step S13 of FIG. 5 includes m types of food (m is a natural number greater than or equal to 2), the processing of FIG. 17 is performed for each of the m types of food.

[0095] The selection unit 46 of the server device 40 selects a first detection frame that indicates the same type of food from the detection frames indicated by the detection frame information acquired in step S13 (S14a), and generates a candidate list that lists the selected first detection frames (S14b).

[0096] The generation unit 47 generates a pair list that lists all pairs of first detection frames that can be obtained from the first detection frames listed in the candidate list (S14c). The generation unit 47 determines whether the pair list includes a pair of first detection frames whose occupancy rate is equal to or greater than a predetermined rate (S14p). If it is determined that the pair list does not include a pair of first detection frames whose occupancy rate is equal to or greater than a predetermined rate (No in S14p), the processing ends.

[0097] When the generation unit 47 determines that the pair list includes a pair of first detection frames whose occupancy rate is equal to or greater than a predetermined rate (Yes in S14p), the generation unit 47 extracts the pair with the highest occupancy rate from the pair list (S14o). That is, in step S14o, pairs with higher occupancy rates are preferentially extracted. When there are two or more pairs with the same occupancy rate, the pair with the shortest distance is preferentially extracted.

[0098] The generation unit 47 merges the extracted pairs of first detection frames (S14g). The generation unit 47 also updates the candidate list by deleting the two first detection frames to be merged from the candidate list generated in step S14b and adding the merged first detection frames (new first detection frames) to the candidate list (S14h). Thereafter, the generation unit 47 generates a pair list using the updated candidate list (S14c).

[0099] In this way, the server device 40 (generation unit 47) extracts pairs of first detection frames that satisfy the requirements regarding occupancy, generates new first detection frames by merging the pairs of first detection frames, and adds the newly generated first detection frames to the candidates for extraction, repeating this process until no new first detection frames can be generated, and finally generates the new first detection frame generated as a second detection frame.In extracting pairs of first detection frames that satisfy the requirements regarding occupancy, pairs with a higher proportion of the area equivalent to the union of the pairs of first detection frames to the area of ​​the detection frame obtained by merging the pairs of first detection frames are extracted with higher priority.

[0100] Such a server device 40 can prevent unintended foods from being included in the second detection frame by preferentially merging pairs of first detection frames with high occupancy rates.

[0101] [Modifications] In the above-described embodiment, refrigerator 20 may be a refrigerator for general household use, a refrigerator used for product display in a retail store such as a convenience store, or a refrigerator for other commercial use.

[0102] Furthermore, the refrigerator 20 is an example of a storage cabinet, and the present disclosure can also be realized as other storage cabinets that have a function of cooling stored items, such as a freezer. Furthermore, the items (i.e., objects) stored in the storage cabinet are not limited to food, and may be other items. Cases in which items other than food are stored in the refrigerator 20 are also conceivable.

[0103] [Effects, etc.] Hereinafter, examples of techniques that can be obtained from the disclosure of this specification will be given, and effects, etc. that can be obtained from these techniques will be described.

[0104] Technique 1 is a computer-executed method for displaying multiple food items stored in a refrigerator (20), the refrigerator (20) having a function of refrigerating multiple food items. The display method includes an acquisition step (S13) of acquiring detection frame information indicating positions and sizes of detection frames that indicate the food items displayed in an image inside the refrigerator (20) detected by a machine learning model, a selection step (S14) of selecting multiple first detection frames that indicate the same type of food from the detection frames indicated by the acquired detection frame information, a generation step (S14) of generating second detection frames obtained by merging pairs of first detection frames that satisfy predetermined requirements from the selected multiple first detection frames, and an output step (S15) of outputting information for displaying the generated second detection frames on the image. The computer is, for example, a server device (40) (information processing unit (42)).

[0105] This display method can reduce the number of detection frames superimposed on multiple foods of the same type by merging pairs of first detection frames that indicate the same type of food, thereby improving the visibility of the detection frames.

[0106] Technique 2 is a display method of Technique 1 in which the first detection frame and the second detection frame are each rectangular.

[0107] Such a display method can generate a rectangular second detection frame by merging a pair of rectangular first detection frames.

[0108] Technique 3 is a display method according to Technique 1 or 2, in which the predetermined requirement is a requirement related to distance, that is, a requirement that the distance between a pair of first detection frames is equal to or less than a threshold.

[0109] This display method makes it possible to prevent other types of food from being included in the second detection frame by requiring that pairs of first detection frames be close to each other in order to merge the pairs.

[0110] Technique 4 is a display method of Technique 3, in which, in generation step S14, pairs of first detection frames that satisfy a distance requirement are extracted, and if the extracted pair of first detection frames satisfies a requirement regarding occupancy, the pair of first detection frames is merged to generate a new first detection frame, and the process of adding the generated new first detection frame to the extraction candidates is repeated until no new first detection frames can be generated, and the last new first detection frame generated is generated as a second detection frame, and in extracting pairs of first detection frames that satisfy the distance requirement, pairs with shorter distances between them are extracted with higher priority, and the requirement regarding occupancy is that the proportion of the area of ​​the new first detection frame that corresponds to the union of the pairs of first detection frames to be merged is equal to or greater than a predetermined proportion. Technique 4 corresponds to Example 2 of the detection frame merging process disclosed in FIGS. 12 and 13 .

[0111] This type of display method can prevent other types of food from being included in the second detection frame by repeatedly merging the first detection frame, with the requirement that the occupancy rate be greater than or equal to a predetermined percentage.

[0112] Technique 5 is a display method of Technique 3, in which in the generation step S14, pairs of first detection frames that satisfy a distance requirement are extracted, and new first detection frames are generated by merging the pairs of first detection frames, and the generated new first detection frames are added to extraction candidates, and this process is repeated until no new first detection frames can be generated, and in the extraction of pairs of first detection frames that satisfy the distance requirement, pairs with shorter distances between the first detection frames are preferentially extracted, and if the last generated new first detection frame satisfies a requirement regarding occupancy, the new first detection frame is generated as a second detection frame, and if the last generated new first detection frame is obtained by merging n first detection frames (n is a natural number greater than or equal to 2) out of the first detection frames selected in the selection step, the requirement regarding occupancy is that the proportion of the area of ​​the last generated new first detection frame that corresponds to the union of the n first detection frames is equal to or greater than a predetermined proportion. Technique 5 corresponds to Example 3 of the detection frame merging process disclosed in FIGS. 14 and 15 .

[0113] This type of display method can prevent unintended foods from being included in the second detection frame by generating a second detection frame that is determined by merging the first detection frame, with the requirement that the occupancy rate be greater than a specified percentage.

[0114] Technique 6 is a display method of Technique 1 or 2, in which the specified requirement is a requirement regarding occupancy, that is, the proportion of the area corresponding to the union of pairs of first detection frames to the area of ​​the detection frame obtained by merging pairs of first detection frames is equal to or greater than a specified proportion.

[0115] This display method prevents other types of food from being included in the second detection frame by requiring that the occupancy rate of a pair in the first detection frame be greater than a predetermined percentage in order to merge the pair.

[0116] Technique 7 is a display method of Technique 6, in which in generation step S14, pairs of first detection frames that satisfy the requirements regarding occupancy are extracted, the extracted pairs of first detection frames are merged to generate new first detection frames, and the process of adding the generated new first detection frames to the extraction candidates is repeated until no new first detection frames can be generated, and the last new first detection frame generated is generated as a second detection frame, and in extracting pairs of first detection frames that satisfy the requirements regarding occupancy, pairs with a higher proportion of the area corresponding to the union of the pairs of first detection frames to the area of ​​the detection frame obtained by merging the pairs of first detection frames are extracted with higher priority. Technique 7 corresponds to Example 5 of the detection frame merging process disclosed in FIG. 17 .

[0117] This display method can prevent other types of food from being included in the second detection frame by preferentially merging pairs of first detection frames with high occupancy rates.

[0118] Technique 8 is a display method of Technique 1 or 2, in which the predetermined requirements are requirements related to distance and requirements related to occupancy, and in generation step S14, pairs of first detection frames that satisfy the requirements related to distance and the requirements related to occupancy are extracted, and a process of generating new first detection frames by merging the pairs of first detection frames is repeated until no new first detection frames can be generated, and the last new first detection frame generated is generated as a second detection frame, and the requirement related to distance is that the distance between the pairs of first detection frames must be equal to or less than a threshold, with pairs with shorter distances being given priority, and the requirement related to occupancy is that the proportion of the area corresponding to the union of the pairs of first detection frames to the area of ​​the detection frame obtained by merging the pairs of first detection frames must be equal to or greater than a predetermined proportion, with pairs with higher area proportions being given priority, and in extracting pairs of first detection frames that satisfy the requirements related to distance and the requirements related to occupancy, pairs of first detection frames with a higher occupancy rate are preferentially extracted over pairs of first detection frames with a short distance between their first detection frames. Technique 8 corresponds to Example 4 of the detection window merging process disclosed in FIG.

[0119] This display method can prevent other types of food from being included in the second detection frame by preferentially merging pairs of first detection frames with high occupancy rates.

[0120] Technique 9 is a program for causing a computer to execute any one of the display methods of Techniques 1 to 8.

[0121] According to such a program, the computer can merge pairs of first detection frames that indicate the same type of food, thereby reducing the number of detection frames superimposed on multiple foods of the same type and improving the visibility of the detection frames.

[0122] Technique 10 is a display system 10 for displaying a plurality of food items stored in a refrigerator 20, the refrigerator 20 having a function of refrigerating a plurality of food items, and the display system 10 includes an acquisition unit 45 that acquires detection frame information indicating the positions and sizes of detection frames that indicate the food items shown in an image inside the refrigerator 20 detected by a machine learning model, a selection unit 46 that selects a plurality of first detection frames that indicate the same type of food from the detection frames indicated by the acquired detection frame information, a generation unit 47 that generates a second detection frame obtained by merging pairs of first detection frames that satisfy predetermined requirements from the selected plurality of first detection frames, and an output unit 48 that outputs information for displaying the generated second detection frames on an image.

[0123] Such a display system 10 can reduce the number of detection frames superimposed on multiple foods of the same type by merging pairs of first detection frames that indicate the same type of food, thereby improving the visibility of the detection frames.

[0124] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0125] For example, in the above embodiment, the display system is realized by multiple devices, but it may also be realized by a single device. For example, the display system may be realized as a single device corresponding to a server device. When the display system is realized by multiple devices, the components (especially functional components) of the display system may be distributed in any way among the multiple devices. For example, some or all of the processing described in the above embodiment as being executed by the server device may also be executed by an information terminal.

[0126] Furthermore, for example, the communication method between the devices in the above-described embodiment is not particularly limited. Furthermore, a relay device (not shown) may be involved in the communication between the devices. Furthermore, the information transmission path described in the above-described embodiment is not limited to the transmission path shown in the sequence diagram.

[0127] For example, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit. Also, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0128] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0129] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0130] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0131] For example, the present disclosure may be realized as a display method executed by a computer, or as a program for causing a computer to execute the display method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0132] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0133] The display system of the present disclosure is useful as a system that can present the results of detection of items in a storage facility to a user.

[0134] REFERENCE SIGNS LIST 10 Display system 20 Refrigerator (storage) 30 Imaging device 40 Server device 41 Communication unit 42 Information processing unit 43 Storage unit 44 Detection unit 45 Acquisition unit 46 Selection unit 47 Generation unit 48 Output unit 50 Information terminal 51 Display unit 60 Wide area communication network

Claims

1. A method for displaying a plurality of food items stored in a storage unit, the storage unit having a function of cooling the plurality of food items, the display method including: an acquisition step for acquiring detection frame information indicating positions and sizes of detection frames indicating the food items shown in an image inside the storage unit, detected by a machine learning model; a selection step for selecting a plurality of first detection frames indicating the same type of food from the detection frames indicated by the acquired detection frame information; a generation step for generating a second detection frame obtained by merging pairs of first detection frames that satisfy predetermined requirements from the selected plurality of first detection frames; and an output step for outputting information for displaying the generated second detection frames on the image.

2. The display method according to claim 1, wherein each of the first detection frame and the second detection frame is rectangular.

3. The display method according to claim 1 or 2, wherein the predetermined requirement is a requirement regarding distance, that is, a requirement that the distance between the pair of first detection frames is equal to or less than a threshold value.

4. The display method according to claim 3, wherein in the generating step, a pair of first detection frames that satisfy the distance requirement is extracted, and if the extracted pair of first detection frames satisfies a requirement regarding occupancy, a new first detection frame is generated by merging the pair of first detection frames, and the process of adding the generated new first detection frame to the candidates for extraction is repeated until a new first detection frame cannot be generated, and the last new first detection frame generated is generated as the second detection frame, and in the extraction of a pair of first detection frames that satisfy the distance requirement, pairs having a shorter distance between them are given priority in extraction, and the requirement regarding occupancy is that the proportion of the area of ​​the new first detection frame that corresponds to the union of the pairs of first detection frames to be merged is equal to or greater than a predetermined proportion.

5. The display method according to claim 3, wherein in the generating step, a pair of first detection frames that satisfy the distance requirement is extracted, a new first detection frame is generated by merging the pair of first detection frames, and the generated new first detection frame is added to the candidates for extraction, and this process is repeated until a new first detection frame cannot be generated; in extracting a pair of first detection frames that satisfy the distance requirement, a pair having a shorter distance between them is preferentially extracted; if the last generated new first detection frame satisfies a requirement regarding occupancy, the new first detection frame is generated as the second detection frame; and if the last generated new first detection frame is obtained by merging n first detection frames (n is a natural number equal to or greater than 2) out of the first detection frames selected in the selecting step, the requirement regarding occupancy is that a proportion of an area of ​​the last generated new first detection frame that corresponds to a union of the n first detection frames is equal to or greater than a predetermined proportion.

6. The display method according to claim 1 or 2, wherein the specified requirement is a requirement regarding occupancy rate, and is a requirement that the proportion of the area of ​​the detection frame obtained by merging the pairs of first detection frames that corresponds to the union of the pairs of first detection frames is equal to or greater than a specified proportion.

7. The display method according to claim 6, wherein in the generating step, a pair of first detection frames that meets the requirement regarding occupancy is extracted, the extracted pair of first detection frames is merged to generate a new first detection frame, and the generated new first detection frame is added to the candidates for extraction. This process is repeated until a new first detection frame cannot be generated, and finally the new first detection frame generated is generated as the second detection frame, and in the extraction of a pair of first detection frames that meets the requirement regarding occupancy, a pair having a higher proportion of an area equivalent to the union of the pairs of first detection frames in an area of ​​the detection frame obtained by merging the pairs of first detection frames is extracted with higher priority.

8. The display method according to claim 1 or 2, wherein the predetermined requirements are requirements regarding distance and requirements regarding occupancy, and in the generating step, a process of extracting pairs of first detection frames that satisfy the requirements regarding distance and the requirements regarding occupancy and generating new first detection frames by merging the pairs of first detection frames is repeated until a new first detection frame cannot be generated, and the last new first detection frame generated is generated as the second detection frame, and the requirement regarding distance is that the distance between the pairs of first detection frames is equal to or less than a threshold, with pairs with shorter distances being given priority, and the requirement regarding occupancy is that a ratio of an area corresponding to a union of the pairs of first detection frames to an area of ​​the detection frame obtained by merging the pairs of first detection frames is equal to or greater than a predetermined ratio, with pairs with higher area ratios being given priority, and in the extraction of pairs of first detection frames that satisfy the requirements regarding distance and the requirements regarding occupancy, pairs of first detection frames with a higher occupancy are preferentially extracted over pairs of first detection frames with a shorter distance between the first detection frames.

9. A program for causing a computer to execute the display method according to claim 1 or 2.

10. A display system for displaying a plurality of food items stored in a storage unit, the storage unit having a function of cooling the plurality of food items, the display system comprising: an acquisition unit that acquires detection frame information indicating positions and sizes of detection frames indicating the foods reflected in an image inside the storage unit detected by a machine learning model; a selection unit that selects a plurality of first detection frames indicating the same type of food from the detection frames indicated by the acquired detection frame information; a generation unit that generates a second detection frame obtained by merging pairs of first detection frames that satisfy specified requirements from the selected plurality of first detection frames; and an output unit that outputs information for displaying the generated second detection frames on the image.

Images