Target detection method, target detection device, kitchen equipment and storage medium

Abstract

The invention discloses a target detection method, a target detection device, kitchen equipment and a storage medium. The target detection method comprises the following steps: respectively acquiring tableware images of tableware in kitchen equipment under different illumination conditions; performing tableware identification on each tableware image by using the target detection model to obtain a tableware identification result of each tableware image; and fusing the tableware identification results of all the tableware images to obtain a final tableware identification result of the tableware in the kitchen equipment. According to the invention, the tableware identification results of the tableware images of the same scene under different illumination conditions are fused, so that the accuracy of tableware identification is improved.

Description

Technical Field

[0001] The present application relates to the field of kitchen equipment, and in particular to a target detection method, a target detection device, kitchen equipment, and a computer-readable storage medium for kitchen equipment. Background Art

[0002] Automatically identifying the tableware that users put into kitchen equipment such as dishwashers and disinfection cabinets has become one of the most basic requirements of smart kitchen equipment. Accurate judgment of tableware allows kitchen equipment to automatically adjust water output intensity, control oil pollution, protect tableware and other functions. This requires adding image acquisition devices such as cameras and fill light components to kitchen equipment. Especially in a relatively closed cavity, the machine can only provide lighting inside it through the fill light component. Therefore, under the condition of using the fill light component, how to improve the efficiency and accuracy of tableware recognition has become a focus in this field. Summary of the invention

[0003] The present application provides a target detection method, a target detection device, a kitchen device, and a computer-readable storage medium for kitchen equipment, so as to improve the detection accuracy of tableware when a fill light component is used to illuminate the kitchen equipment.

[0004] To solve the above problems, a technical solution adopted by the present application is to provide a target detection method for kitchen equipment. The target detection method includes: obtaining tableware images of tableware in the kitchen equipment under different lighting conditions respectively; using a target detection model to perform tableware recognition on each tableware image respectively to obtain a tableware recognition result for each tableware image; and fusing the tableware recognition results of all tableware images to obtain a final tableware recognition result of the tableware in the kitchen equipment.

[0005] To solve the above problems, another technical solution adopted by the present application is to provide a target detection device. The target detection device includes a tableware image acquisition module, a target recognition module and a target fusion module. The tableware image acquisition module is configured to respectively acquire tableware images of tableware in the kitchen equipment under different lighting conditions. The target recognition module is configured to use the target detection model to perform tableware recognition on each tableware image respectively to obtain a tableware recognition result for each tableware image. The target fusion module is configured to fuse the tableware recognition results of all tableware images to obtain a final tableware recognition result of the tableware in the kitchen equipment.

[0006] To solve the above problem, another technical solution adopted by the present application is: to provide a kitchen device. The kitchen device includes a memory and a processor. The memory stores a computer program. The processor is used to execute the computer program to implement the above target detection method.

[0007] To solve the above problem, another technical solution adopted by the present application is to provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The computer program can be executed by a processor to implement the above target detection method for kitchen equipment.

[0008] Different from the prior art, the present application obtains tableware images of tableware in kitchen equipment under different lighting conditions, and uses the target detection model to perform tableware recognition on each tableware image to obtain the tableware recognition result of each tableware image, and fuses the tableware recognition results of all tableware images to obtain the final tableware recognition result of the tableware in the kitchen equipment. The present application improves the accuracy of tableware recognition by fusing the tableware recognition results of tableware images under different lighting conditions of the same scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly describe the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0010] Figure 1 A structural schematic diagram of a kitchen device according to an embodiment of the present application is shown.

[0011] Figure 2 A schematic diagram of a kitchen equipment system according to an embodiment of the present application is shown.

[0012] Figure 3 A schematic flow chart of a method for detecting a target for kitchen equipment according to an embodiment of the present application is shown.

[0013] Figure 4 Shown in Figure 3 Schematic diagram of the target detection system used in the target detection method.

[0014] Figure 5 Schematic diagram showing the intersection-over-union ratio of two object detection boxes.

[0015] Figure 6 A schematic diagram showing the intersection-over-union ratio of multiple object detection boxes.

[0016] Figure 7 A schematic flow chart of a method for detecting a target for kitchen equipment according to yet another embodiment of the present application is shown.

[0017] Figure 8 A schematic flow chart of a method for determining whether each tableware image is a difficult sample according to an embodiment of the present application is shown.

[0018] Fig. 9 A schematic structural diagram of a target detection device according to an embodiment of the present application is shown.

[0019] Fig.10 A structural schematic diagram of a kitchen device according to an embodiment of the present application is shown.

[0020] Fig.11 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] Before describing the embodiments of the present application in detail, the kitchen equipment of the present application is briefly described. The kitchen equipment may be, for example, a dishwasher, a disinfection cabinet, a dishware drying machine, or other types of kitchen equipment. The kitchen equipment may include a closable cavity inside to place and hold tableware, so as to perform various operations such as washing, disinfecting, and drying the tableware. The following description takes a dishwasher as an example.

[0024] See also Figure 1 , Figure 1: is a schematic diagram of the structure of the kitchen device 100 provided according to an embodiment of the present invention. The kitchen device 100 may be, for example, a dishwasher. The kitchen device 100 defines a cleaning area inside thereof for placing tableware in the area for cleaning. An image acquisition device 10 may be provided on the inner wall of the cleaning area for acquiring images of the cleaning area, especially the tableware. Further, a fill light component 20 is also provided on the inner wall of the cleaning area to fill light for the cleaning area of ​​the kitchen device 100. As shown in the figure, the fill light component 20 may be a ring-shaped fill light, which is arranged on the peripheral side of the image acquisition device 10 to save space in the cleaning area and facilitate wiring on one side of the inner wall. The fill light component 20 and the image acquisition device 10 may also be arranged on the top, side or other positions of the cleaning area. The present application does not limit the arrangement position of either the fill light component 20 or the image acquisition device 10.

[0025] Specifically, the fill light assembly 20 may include a single fill light or a plurality of fill light. The fill light assembly 20 is configured to provide a plurality of different lighting conditions to the cleaning area. Different lighting conditions may include different lighting intensities. For example, at least one fill light of the fill light assembly 20 may emit light of several different brightness levels. The brightness of the fill light may be adjusted by adjusting the resistance or voltage of the fill light or by adjusting the pulse width or duty cycle of the current input to the fill light, and the present application does not limit this.

[0026] Different lighting conditions may also include different light frequencies. For example, the fill light assembly 20 may include fill lights configured to emit light having different frequencies, respectively. The light having different frequencies may include any one of visible light, infrared light, ultraviolet light, etc. Visible light may include visible light of different colors such as red, blue, yellow, etc.

[0027] In the present application, the fill light assembly 20 can provide N different lighting conditions, where N is an integer greater than 1. Each lighting condition can correspond to a different combination of lighting frequency and intensity. The fill light assembly 20 can be configured to emit different lights in different orders to produce different lighting conditions. The present application does not limit the order in which the fill light assembly 20 emits different lights, and the order can be defined by the user, operator, or maintainer of the kitchen device 100.

[0028] See also Figure 2 , Figure 2 Schematic diagram of a kitchen equipment system 200 according to an embodiment of the present application is shown. The kitchen equipment system 200 may include at least one kitchen equipment 100 and a server 30. The server 30 may communicate and exchange data with a plurality of kitchen equipment 100 via a network.

[0029] The network includes but is not limited to the Internet, Ethernet, mobile communication network, wide area network, metropolitan area network, etc. The above mobile communication network includes but is not limited to the third generation (3G) mobile communication network, the third and a half generation (3.5G) mobile communication network, the fourth generation (4G) mobile communication network and the fifth generation (5G) mobile communication network, etc.

[0030] According to different division methods, the server 30 can be a centralized server, a distributed server, a cloud server, etc., and this application does not limit this. Compared with a single kitchen device 100, the server 30 may have more powerful computing power and computing speed. In the specific application of modern intelligent kitchen equipment 100, services with high computing requirements such as target recognition are often performed by the server 30 rather than a single kitchen device 100 to provide better recognition accuracy and speed. In addition, deploying a target detection model, such as the tableware detection model in this application, on the server 30 can also better maintain the target detection model, thereby providing the latest and most advanced target recognition services for all kitchen devices 100 that are connected to the server 30 in communication. Of course, the target detection model can also be deployed on a single kitchen device 100, and this application does not limit this.

[0031] In the present application, the fill light component 20 of the kitchen device 100 can provide N different lighting conditions, where N is a positive integer. After long-term research, the inventor of the present application found that in order to achieve a good tableware recognition effect, tableware of different colors and materials often requires different optimal lighting conditions. However, in a specific use environment, the various tableware of the user is often mixed together, and the user's tableware is often of various types, and the tableware combination and placement habits are also very different. It is difficult to provide the best recognition effect for various tableware at the same time under the same lighting conditions. For example, for some white porcelain plates, the better lighting conditions are darker, while for transparent wine glasses, the better lighting conditions are brighter. When white porcelain plates and transparent wine glasses are placed together, the scene becomes complicated and tableware recognition becomes difficult. Due to the diversity of users' tableware, the manufacturer of the kitchen device 100 cannot exhaust the various tableware combinations and placement habits in the kitchen device 100, so how to deal with these numerous and complex scenes becomes a problem. In this regard, the present application proposes a solution to improve the accuracy of tableware recognition by fusing target detection results of tableware images of the same scene in the kitchen device 100 under different lighting conditions.

[0032] refer to Figure 3 and Figure 4 . Figure 3 A schematic flow chart of a target detection method for a kitchen device 100 according to an embodiment is shown. Figure 4 Shown in Figure 3 Schematic diagram of an object detection system 400 used in an object detection method.

[0033] like Figure 4 As shown, the target detection system 400 of the present application may include a target detection model 410 and a target fusion model 420 that are communicatively connected to each other. In some embodiments, the target detection model 410 and the target fusion model 420 may be integrated in a software package or executable file, or may be two separate different modules. For example, the target detection model 410 may be a deep learning model, especially a neural network model, and the present application does not limit its specific type. The target fusion model 420 may be software, an executable file, an APP, etc. that are independent of the target detection model 410.

[0034] The object detection model 410 is configured to receive a plurality of tableware images taken under different lighting conditions, and identify or predict the tableware in each tableware image to generate an object detection frame surrounding the tableware. In a specific operation process, the object detection model 410 performs object detection on the tableware images under different lighting conditions of the same scene inside the kitchen device 100, and inputs the object detection results into the object fusion model 420.

[0035] The target fusion model 420 is configured to receive the tableware prediction or recognition results from the target detection model 410 and fuse the prediction or recognition results of multiple tableware images. For example, the target fusion model 420 can filter and / or merge the target detection frames of the tableware images under different lighting conditions to obtain a final target detection frame as the final tableware recognition result.

[0036] like Figure 3 As shown, according to an embodiment of the present application, the target detection method of the present application includes the following steps S11 to S13.

[0037] Step S11: acquiring images of tableware in the kitchen device 100 under different lighting conditions respectively.

[0038] Specifically, the fill light assembly 20 is controlled to work in different working conditions to generate different lighting conditions in the kitchen device 100, and the image acquisition device 10 is controlled to acquire tableware images under different lighting conditions.

[0039] In the present application, the fill light component 20 can provide N different lighting conditions. The fill light component 20 can work in a certain order and in different working conditions to traverse N different lighting conditions for fill light, so that the cleaning area in the kitchen device 100 is in different lighting environments, and cooperate with the image acquisition device 10 to collect, so as to obtain all tableware images under different lighting conditions.

[0040] Taking N=10 as an example, the fill light component 20 is controlled to adjust the working condition to perform fill light in the first to tenth lighting conditions in sequence. After each adjustment or change of the lighting condition, the image acquisition device 10 is triggered to perform an image acquisition, thereby acquiring 10 tableware images corresponding to the first to tenth lighting conditions. In one embodiment, the brightness of the fill light component 20 changes from high to low, and after each brightness change, the image acquisition device 10 captures at least one tableware image. The captured tableware image is input into the target detection model 410 for target detection.

[0041] In some embodiments, a trigger may be provided at the door opening mechanism of the kitchen device 100. When the door of the kitchen device 100 is closed, the trigger is triggered and sends a signal. The controller of the kitchen device 100 receives the signal, thereby starting to initialize the image acquisition device 10, and controlling the fill light component 20 and the image acquisition device 10 to collaboratively capture images of tableware under different lighting conditions.

[0042] Step S12: using the object detection model 410 to perform tableware recognition on each tableware image respectively, so as to obtain a tableware recognition result for each tableware image.

[0043] by Figure 4 Taking the target detection system 400 shown in FIG. 4 as an example, the target detection system 400 can be deployed on the kitchen device 100 or the server 30. In the case where the target detection model is deployed on the server 30, tableware images under different lighting conditions can be transmitted from the kitchen device 100 to the server 30 for recognition.

[0044] Specifically, the target detection model 410 in the target detection system 400 is used to perform tableware recognition on each tableware image respectively to obtain the tableware recognition result of each tableware image. Taking the target detection model 410 as a trained neural network model as an example, the target detection model 410 can be a two-stage target detection model and a single-stage target detection model. After the tableware picture is input into the two-stage target detection model, the two-stage target detection model first generates a candidate area to roughly locate the tableware, and then accurately locates and classifies the candidate area. After the tableware picture is input into the single-stage target detection model, the single-stage target detection model directly regresses the category probability and position information of the tableware based on the features extracted by the model backbone network. The present application does not limit the specific type of target detection model used.

[0045] After the tableware image is input into the target detection model 410, the target detection model 410 can output the tableware recognition result of the tableware image. Specifically, the tableware recognition result includes at least one target detection frame and a corresponding confidence level. Among them, the target detection frame is also called a bounding box, which can surround the identified tableware with the smallest possible area. The target detection frame can be represented by the coordinates of its diagonal points or the center point, length and width. The tableware recognition result also includes the type of tableware in the target detection frame, such as a porcelain plate, a glass, etc. The confidence level (score), also known as the confidence score, is a number between 0 and 1, indicating the possibility that the target detection model 410 believes that the corresponding target detection frame contains a tableware target.

[0046] Step S13: fusing the tableware recognition results of all tableware images to obtain a final tableware recognition result of the tableware in the kitchen device 100 .

[0047] The method can fuse the tableware recognition results of tableware images under all different lighting conditions in the same scene in the kitchen device 100 to prevent missed detection of tableware targets.

[0048] In some embodiments, step S13 may include screening and / or merging the object detection frames of all the tableware images based on the confidence level to obtain a final object detection frame.

[0049] In some embodiments, a non-maximum suppression (NMS) algorithm may be used to screen the object detection frames of all tableware images.

[0050] Specifically, the screening using the NMS algorithm may include the following first to third steps. The first step is to perform an initialization operation. Specifically, two empty first sets and second sets are prepared, and all target detection frames of the tableware images under different lighting conditions in the same scene inside the kitchen device 100 are placed in the first set.

[0051] In the second step, all the target detection frames are sorted based on their confidence levels. For example, the target detection frames are sorted from high to low or from low to high according to their confidence levels.

[0052] The third step is to perform a screening operation on the target detection frame. In a screening operation: first, select a target detection frame with the highest current confidence or a target detection frame among multiple target detection frames with the highest confidence in the first set as the current reference target detection frame, and remove the current reference target detection frame from the first set and put it into the second set; next, compare the intersection over union (IOU) of each remaining target detection frame in the first set with the current reference detection frame; in response to the intersection over union (IOU) of the target detection frame and the current reference target detection frame being greater than a first threshold, remove the target detection frame from the first set, and the first threshold can be any value set by the user or manufacturer based on theory or experience, such as 0.3, 0.5, 0.7, etc., generally between 0.3-0.5. Repeat the screening operation until the first set becomes an empty set. After all screening operations are completed, the target detection frame remaining in the second set is used as the final target detection frame output.

[0053] Through step S13, the best target detection frame can be found as the final target detection frame, and redundant target detection frames can be eliminated to obtain a set of final target detection frames.

[0054] refer to Figure 5 and Figure 6 , Figure 5 and Figure 6 Schematic diagrams showing the concept of the intersection over union (IoU) of two target detection boxes and multiple target detection boxes, respectively.

[0055] like Figure 5 As shown, for two target detection boxes D 1 and D 2 , the intersection over union (IoU) can be defined as the overlapping area (or intersection area) D of two target detection boxes. 1 ∩D 2 The union area D of the two target detection boxes 1 ∪D 2 The ratio between Figure 5 The intersection area D 1 ∩D 2 Highlighted by shading, the union area D 1 ∪D 2 is composed of two target detection boxes D 1 and D 2 The area included by at least one of .

[0056] Similarly, if Figure 6 As shown in , for multiple target detection frames, the intersection-over-union ratio can be defined as the ratio of the union of the overlapping areas (or intersection areas) in the multiple target detection frames to the union area of ​​the multiple target detection frames. The intersection area can be the overlapping area of ​​two, three or more target detection frames. Figure 6 In the figure, the intersection area is highlighted with shade. The union area is the area contained by at least one of the multiple target detection boxes.

[0057] In some embodiments, some variants of the NMS algorithm may be used to screen the target detection frame. These variants of the NMS algorithm may include Soft-NMS, Softer-NMS, Relation Network, ConvNMS, etc., which will not be described in detail in this application.

[0058] In some embodiments, the final target detection frame can also be obtained based on a target detection frame screening method called confidence weighted by the applicant. The target detection frame screening method includes deleting and weighted merging operations of the target detection frame. The method may include the following first to third operations.

[0059] The first step is to perform an initialization operation. Prepare two empty third and fourth sets, and put all target detection frames of tableware images under different lighting conditions of the same scene in the kitchen device 100 into the third set.

[0060] In the second step, all the target detection frames are sorted based on their confidence levels. For example, the target detection frames are sorted from high to low or from low to high according to their confidence levels.

[0061] The third step is to perform a screening operation on the target detection frame. In one screening operation: first, a target detection frame with the highest current confidence or a target detection frame among multiple target detection frames with the highest confidence in the third set is selected as the current reference target detection frame, and the current reference target detection frame is removed from the third set and placed in the fourth set; the intersection over union (IoU) of each remaining target detection frame in the third set is compared with the current reference target detection frame; when the intersection over union (IoU) of the target detection frame and the current reference target detection frame is greater than the second threshold and less than the third threshold, the target detection frame is removed from the third set, and when the intersection over union (IoU) of the target detection frame and the current target detection frame is greater than or equal to the third threshold, the target detection frame is marked as a candidate merged target detection frame of the current reference target detection frame and removed from the third set; a confidence-weighted merge operation is performed on the current reference target detection frame and all corresponding candidate merged target detection frames to replace the current reference target detection frame in the fourth set. At this point, one screening operation is completed. Repeat the screening operation until the third set becomes an empty set.

[0062] Finally, the target detection box in the fourth set is used as the final target detection box output.

[0063] The second threshold and the third threshold can be set by the user or manufacturer based on experience or theory, etc., and the present application does not limit this. For example, the second threshold can be 0.3, and the third threshold can be 0.7.

[0064] The above confidence-weighted merging of the current reference target detection frame and the corresponding candidate merged target detection frame can be performed according to the following formulas E1 and E2:

[0065]

[0066]

[0067] Wherein, k and n are integers, X represents the coordinate value or coordinate vector value of the point, L represents the length of the target detection box, and n is the number of candidate merged target detection boxes corresponding to the current reference target detection box.

[0068] Among them, X 最终 Represents the coordinate value or coordinate vector value of the point in the final target detection frame obtained after weighted merging. 0 Indicates the coordinate value or coordinate vector value of the corresponding point of the current reference target detection frame, X k Represents the coordinate value or coordinate vector value of the corresponding point of the k-th corresponding candidate merged target detection box.

[0069] Among them, L 最终 Represents the length of the final target detection box obtained after weighted merging. L 0 Indicates the corresponding length of the current reference target detection box, L k Represents the corresponding length of the kth corresponding candidate merged target detection box.

[0070] Object detection boxes can be represented in many ways.

[0071] In one case, the target detection frame can be represented by the coordinates of a pair of diagonal points of a rectangular detection frame. The pair of diagonal points may include a first diagonal point P1 and a second diagonal point P2. Applying the above formula E1, the first corner point P1 of the final target detection frame is 最终 The calculation formula is:

[0072]

[0073] Among them, P1 0 is the coordinate value or coordinate vector value of the first corner point P1 of the current reference target detection frame, P1 k is the coordinate value or coordinate vector value of the first corner point P1 of the k-th corresponding candidate merged target detection box.

[0074] The second corner point P2 of the final target detection box 最终 The calculation formula is:

[0075]

[0076] Among them, P2 0 is the coordinate value or coordinate vector value of the second corner point P2 of the current reference target detection frame, P2 k It is the coordinate value or coordinate vector value of the second corner point P2 of the k-th corresponding candidate merged target detection box.

[0077] In another case, the target detection box can be represented by the geometric center point T, width W and height H. In this case, applying formula E1, the geometric center point T of the final target detection box is 最终 The calculation formula is:

[0078]

[0079] Among them, T 0 is the coordinate value or coordinate vector value of the geometric center point T of the current reference target detection frame, T k is the coordinate value or coordinate vector value of the geometric center point T of the kth corresponding candidate merged target detection box.

[0080] Applying formula E2, the length of the final target detection box is W 最终 and height H 最终 The calculation formulas are:

[0081]

[0082]

[0083] Among them, W 0 and H 0 are the width and height of the current reference target detection box, W k and H k are the width and height of the kth corresponding candidate merged target detection box, respectively.

[0084] Through the confidence-weighted target detection frame screening method, the present application can further improve the recognition rate of the final target detection frame obtained. The final target detection frame obtained can be used to count the number and type of tableware in the kitchen device 100, so as to determine the water output, water output strength and detergent dosage of the kitchen device 100, so as to achieve the purpose of accurate cleaning.

[0085] refer to Figure 7 , Figure 7 A schematic flow chart of a target detection method for a kitchen device 100 according to another embodiment of the present application is shown. The target detection method includes the following steps S21-S25.

[0086] Step S21: acquiring images of tableware in the kitchen device 100 under different lighting conditions respectively.

[0087] Step S21 is the same as the above step S11 and will not be described again.

[0088] Step S22: using the object detection model 410 to perform tableware recognition on each tableware image respectively, so as to obtain a tableware recognition result for each tableware image.

[0089] Step S22 is the same as the above step S12 and will not be described again.

[0090] Step S23 : fusing the tableware recognition results of each tableware image to obtain a final tableware recognition result of the tableware in the kitchen device 100 .

[0091] Step S23 is the same as the above step S13 and will not be described again.

[0092] Step S24: comparing the difference between the final tableware recognition result and the tableware recognition result of each tableware image, and determining the tableware image under the corresponding lighting condition as a difficult sample based on the difference.

[0093] Specifically, refer to Figure 8 , Figure 8 A flow chart showing a method for determining whether each tableware image is a difficult sample is shown. The method comprises the following steps S51 to S53.

[0094] Step S51: comparing the number n1 of the final target detection frames of the final tableware recognition result with the number n2 of the target detection frames of the tableware recognition result of the tableware image, and determining whether n1 and n2 are consistent.

[0095] If the final number n1 of target detection frames is inconsistent with the number n2 of target detection frames of the tableware image, go to step S53.

[0096] If the number n1 of the final target detection frames is consistent with the number n2 of the target detection frames of the tableware image, go to step S52.

[0097] Specifically, if the number of target detection frames n1 is inconsistent with the number of target detection frames n2 of the tableware image, it means that the target recognition of the tableware image under the corresponding lighting conditions is wrong, such as missed detection of tableware, and the tableware image under the corresponding lighting conditions can be judged as a difficult sample. If the number of target detection frames n1 is consistent with the number of target detection frames n2 of the tableware image, it means that the number of target recognition of the tableware image under the corresponding lighting conditions is correct, and it is necessary to further combine other conditions to determine whether the tableware image is a difficult sample.

[0098] Difficult samples are image samples with low accuracy when the target detection model 410 used in this application performs target recognition or prediction on them.

[0099] Step S52: Calculate the intersection-and-union ratio of the final target detection frame of the final tableware recognition result and the target detection frame of the tableware image, and determine whether the intersection-and-union ratio is lower than the fourth threshold. If the intersection-and-union ratio is lower than the fourth threshold, go to step S53. If the intersection-and-union ratio is greater than the fourth threshold, end the process. The fourth threshold can be set by the user, maintainer or manufacturer of the kitchen equipment based on theory or experience, for example, 0.9 or 0.8, etc., and this application does not limit this.

[0100] For details, see the reference above. Figure 6 As described above, for multiple target detection frames, the intersection-over-union ratio can be defined as the ratio of the union of the overlapping areas (or intersection areas) in the multiple target detection frames to the union area of ​​the multiple target detection frames. In step S52, by calculating the intersection-over-union ratio between the final target detection frame and the target detection frame of the tableware image, the degree of deviation between the target detection frame predicted by the target detection model 410 for the tableware image and the final target detection frame can be measured. If the intersection-over-union ratio is too low, it means that the target detection frame predicted by the target detection model 410 for the tableware image has a high error rate, and the corresponding tableware image can be judged as a difficult sample.

[0101] Step S53: Determine that the tableware image is a difficult sample.

[0102] As described above, difficult samples are image samples for which the target detection model 410 used in the present application has a low accuracy rate when performing target recognition or prediction, which means that the target detection model 410 has difficulty in recognizing tableware in the kitchen device 100 under the current lighting conditions.

[0103] This application can put all the determined difficult samples into one set.

[0104] The present application may perform the above process of determining whether a tableware image is a difficult sample on all tableware images of the same scene, and put all determined difficult samples into a set for subsequent operations such as maintaining the target detection model 410.

[0105] Step S25: perform incremental learning training on the target detection model 410 using difficult samples to update the target detection model 410.

[0106] Incremental learning training of the target detection model 410 refers to enabling the target detection model 410 to perform incremental learning. Incremental learning, also known as continuous learning or lifelong learning, is a machine learning method that allows a machine learning model to continuously learn new data instead of retraining the entire model. The incremental learning method allows the target detection model 410 to continuously learn new knowledge to adapt to the complex and changing recognition environment in practical applications.

[0107] The incremental learning methods that can be used in the present application include, for example, fine-tuning, joint training, LwF (Learning without Forgetting), etc. Specifically, the fine-tuning method generally only needs to use the newly added samples to train the model (such as the target recognition model 410). The advantage is that the amount of calculation is small, and the disadvantage is that catastrophic forgetting is prone to occur. The joint training method retrains the model on all known data including previous samples and newly added samples. The training effect is the best, but the amount of calculation and time required for training are the largest, and the training cost is relatively high. The LwF method is a training method between the joint training method and the fine-tuning method. The characteristic of the LwF method is that it uses the idea of ​​knowledge distillation, and it can be updated without using old samples. The present application may preferably use the LwF method for incremental learning. The present application may also use other incremental learning methods, which are not limited by the present application.

[0108] In the present application, when the number of collected difficult samples exceeds a certain number or a predetermined number, these difficult samples can be used to perform incremental learning training on the object detection model 410.

[0109] In the present application, difficult samples may be manually labeled and then input into the target detection model 410 for incremental learning.

[0110] In some embodiments, the final tableware recognition result corresponding to the difficult sample may also be input as a label to the object detection model 410 for incremental learning, thereby enhancing the degree of automation of the method of the present application.

[0111] After the incremental learning training is completed, all tableware images in the difficult sample set can be cleared. The object detection model 410 after the incremental learning training can be used to replace the currently used object detection model 410, thereby updating the object detection model 410 and the object detection system 400.

[0112] In the method of the present application, by determining difficult samples and using the difficult samples as input for incremental training of the target detection model 410, the target detection model 410 can be trained to recognize different tableware under various lighting conditions, thereby improving the generalization ability of the target detection model 410.

[0113] refer to Fig. 9 , Fig. 9 A schematic diagram of a target detection device 900 according to an embodiment of the present application is shown. The target detection device 900 comprises a tableware image acquisition module 91, a target recognition module 92, a target fusion module 93, a difficult sample determination module 94 and an incremental learning training module 95 which are interconnected and communicated with each other.

[0114] The tableware image acquisition module 91 is configured to respectively acquire tableware images of the tableware in the kitchen device 100 under different lighting conditions. The tableware image acquisition module 91 is also configured to control the fill light component 20 to work in different working conditions to generate different lighting conditions in the kitchen device 100, and control the image acquisition device to respectively acquire tableware images under different lighting conditions. The different lighting conditions include different light intensities and / or different light frequencies.

[0115] The object recognition module 92 is configured to perform tableware recognition on each tableware image respectively using the object detection model 410 to obtain a tableware recognition result for each tableware image.

[0116] The target fusion module 93 is configured to fuse the tableware recognition results of each tableware image to obtain the final tableware recognition results of the tableware in the kitchen device 100. The target fusion module 93 is also configured to filter and / or merge the target detection frames of all tableware images based on the confidence level to obtain the final target detection frame. The target fusion module 93 is also configured to filter and / or merge the target detection frames of all tableware images using any of the following methods: NMS, Soft-NMS, Softer-NMS, Relation Network, ConvNMS or weighted frame fusion method.

[0117] The difficult sample determination module 94 is configured to compare the difference between the final tableware recognition result and the tableware recognition result of each tableware image, and determine that the tableware image under the corresponding lighting conditions is a difficult sample based on the difference. The difficult sample determination module 94 is also configured to compare the number of final target detection frames of the final tableware recognition result with the number of target detection frames of the tableware recognition result of the tableware image, and in response to the number of the final target detection frames being inconsistent with the number of target detection frames, determine that the tableware image is a difficult sample. The difficult sample determination module 94 is also configured to calculate the intersection and union ratio of the final target detection frame and the target detection frame in response to the number of the final target detection frames being consistent with the number of target detection frames, and in response to the intersection and union ratio being less than a threshold, determine that the tableware image is a difficult sample.

[0118] The incremental learning training module 95 performs incremental learning training on the target detection model 410 using the difficult samples to update the target detection model 410. The updated target detection model 410 can replace the target detection model 410 used by the target recognition module 92 to update the target detection model 410. When the target detection model 410 is incrementally trained using the difficult samples, the corresponding final target detection box and the corresponding category can be used as the label of the tableware image.

[0119] The above-mentioned target detection device 900 can work as a closed-loop system. The target detection device 900 can continuously and automatically collect all washing scenes of different users of multiple kitchen devices 100, and continuously update the target detection model 410, so that the target detection model 410 is more accurate and robust. The target detection model 410 can gradually adapt to various different lighting environments. In the target detection device 900, data collection and training of the target detection model 410 can be fully automatic, thereby greatly reducing the manpower cost required for the maintenance of the target detection device 900.

[0120] refer to Fig.10 , Fig.10 FIG. 8 is a schematic diagram showing the structure of a kitchen device 800 according to an embodiment of the present application. The kitchen device 800 may be, for example, a dishwasher or a disinfection cabinet. Fig.10 As shown, the kitchen device 800 further includes a processor 810 and a memory 820. The above-mentioned target detection system 400 may be stored in the memory 820. A computer program is also stored in the memory 820. The processor 810 is used to execute the computer program to implement the above-mentioned target detection method.

[0121] refer to Fig.11 The present application also provides a computer-readable storage medium 700. The computer-readable storage medium 700 stores a computer program 710. The computer program 710 can be executed by a processor of a computer to implement the above-mentioned target detection method.

[0122] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0123] The above are only implementation methods of the present application, and are not intended to limit the patent scope of the present application. Equivalent structures or equivalent process changes made by utilizing the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target detection method for kitchen equipment, characterized in that: include: respectively acquiring tableware images of the tableware in the kitchen equipment under different lighting conditions; Using the target detection model to perform tableware recognition on each tableware image respectively, so as to obtain a tableware recognition result of each tableware image; as well as The tableware recognition results of all the tableware images are fused to obtain a final tableware recognition result of the tableware in the kitchen device.

2. The method according to claim 1, characterized in that Also includes: comparing the difference between the final tableware recognition result and the tableware recognition result of each tableware image, and determining that the tableware image under the corresponding lighting condition is a difficult sample based on the difference; as well as The target detection model is incrementally trained using the difficult samples to update the target detection model.

3. The method according to claim 2, characterized in that The step of comparing the difference between the final tableware recognition result and the tableware recognition result of each tableware image, and determining the tableware image under the corresponding lighting condition as a difficult sample based on the difference comprises: comparing the number of final object detection frames of the final tableware recognition result with the number of object detection frames of the tableware recognition result of the tableware image; and In response to the number of the final object detection frames being inconsistent with the number of the object detection frames, the tableware image is determined to be a difficult sample.

4. The method according to claim 3, characterized in that The step of comparing the difference between the final tableware recognition result and the tableware recognition result of each tableware image, and determining the tableware image under the corresponding lighting condition as a difficult sample based on the difference further includes: In response to the number of the final object detection frames being consistent with the number of the object detection frames, calculating an intersection-over-union ratio between the final object detection frame and the object detection frame; and In response to the intersection-over-union ratio being less than a threshold, the tableware image is determined to be a difficult sample.

5. The method according to claim 1, characterized in that The tableware recognition result includes a target detection frame and a corresponding confidence level. The tableware recognition results of all tableware images are integrated to obtain the final tableware recognition results of the tableware in the kitchen equipment, including: Based on the confidence level, the object detection frames of all the tableware images are screened and / or merged to obtain a final object detection frame.

6. The method according to claim 5, characterized in that The step of screening and / or merging the target detection frames of all tableware images based on the confidence level to obtain the final target detection frame includes: Filter and / or merge the object detection boxes of all tableware images using any of the following methods: NMS, Soft-NMS, Softer-NMS, Relation Network, ConvNMS, or weighted box fusion method.

7. The method according to claim 1, characterized in that The kitchen equipment includes an image acquisition device and a fill light component arranged in the kitchen equipment, The step of respectively acquiring tableware images of the tableware in the kitchen equipment under different lighting conditions comprises: Controlling the fill light assembly to operate in different working conditions to generate different lighting conditions in the kitchen device; and The image acquisition device is controlled to acquire the tableware images under the different lighting conditions respectively.

8. The method according to claim 7, characterized in that The different lighting conditions include different lighting intensities and / or different light frequencies.

9. A target detection device, characterized in that: include: A tableware image acquisition module, configured to respectively acquire tableware images of tableware in the kitchen equipment under different lighting conditions; an object recognition module, configured to perform tableware recognition on each tableware image respectively using the object detection model to obtain a tableware recognition result for each tableware image; as well as The target fusion module is configured to fuse the tableware recognition results of all the tableware images to obtain a final tableware recognition result of the tableware in the kitchen device.

10. The target detection device according to claim 9, characterized in that: Also includes: a difficult sample determination module, configured to compare the difference between the final tableware recognition result and the tableware recognition result of each tableware image, and determine that the tableware image under the corresponding lighting condition is a difficult sample based on the difference; as well as The incremental learning training module is configured to perform incremental learning training on the target detection model using the difficult samples to update the target detection model.

11. A kitchen device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images