Maintenance method, kitchen equipment system, and computer-readable storage medium

Abstract

The application discloses a maintenance method, a kitchen equipment system and a computer readable storage medium. The maintenance method comprises: continuously performing a maintenance operation of a tableware sample set of the kitchen equipment. The maintenance operation further comprises: in response to the tableware sample set not being updated in the next N continuous maintenance operations after the tableware sample set is updated in the maintenance operation, incrementally training a target detection model by using the tableware sample set and updating the target detection model. Through the scheme, the application can continuously incrementally train the target detection model by using images of new tableware categories from the user of the kitchen equipment, so as to ensure that the target detection model continuously enhances the categories and range of tableware recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of kitchen equipment, in particular to a target detection model maintenance method for kitchen equipment, a kitchen equipment system and a computer readable storage medium. BACKGROUND

[0002] In today's booming artificial intelligence, users have increasingly strong demand for smart home. A dishwasher with artificial intelligence function can bring users convenient operation and greatly improve product quality and user experience. Automatically identifying the tableware put into the dishwasher by the user is a basic requirement for the dishwasher. Precise judgment of the items inside can enable the dishwasher to automatically adjust water intensity, control oil stains, and protect tableware.

[0003] However, the types and materials of tableware on the market are diverse, and new tableware styles are constantly emerging. The laboratory data used by the manufacturer to train the model of the kitchen equipment often cannot cover such a wide variety of tableware, so that the actual application effect of the intelligent kitchen equipment with target detection function in the user's home after being put on the market is greatly discounted. Therefore, how to maintain the target detection model of the kitchen equipment becomes a problem. SUMMARY

[0004] The present application provides a target detection model maintenance method for kitchen equipment, a kitchen equipment system and a computer readable storage medium to maintain the target detection model of the kitchen equipment and enhance its tableware detection capability.

[0005] To achieve the above technical effects, one technical solution adopted by the present application is to provide a target detection model maintenance method for kitchen equipment. The maintenance method comprises: continuously performing a maintenance operation of a tableware sample set of the kitchen equipment. The maintenance operation comprises: in response to the start of the kitchen equipment, acquiring a tableware image in the kitchen equipment; predicting a target detection frame and a corresponding target category in the tableware image by using a target detection model; calculating a first difference between each target detection frame and a reference sample of the corresponding target category; in response to the first difference being greater than a first threshold, performing: in response to the tableware sample set being an empty set, merging the target detection frame into the tableware sample set of the kitchen equipment to update the tableware sample set of the kitchen equipment; or in response to the tableware sample set being a non-empty set, calculating a second difference between the target detection frame and each target detection frame in the tableware sample set, and in response to each second difference corresponding to the target detection frame being greater than a second threshold, merging the target detection frame into the tableware sample set of the kitchen equipment to update the tableware sample set of the kitchen equipment. The maintenance operation further comprises: in response to the tableware sample set being updated in the maintenance operation, and not being updated in the next N consecutive maintenance operations, incrementally training the target detection model by using the tableware sample set and updating the target detection model. Wherein, N is a positive integer.

[0006] To achieve the above technical effects, another technical scheme adopted by the present application is to provide a kitchen equipment system. The kitchen equipment system comprises a memory and a processor. The memory stores a computer program. The processor is configured to execute the computer program to implement the above maintenance method.

[0007] To achieve the above technical effects, another technical scheme adopted by the present application is to provide a computer readable storage medium. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the above maintenance method.

[0008] Different from the prior art, the present application continuously performs the maintenance operation of the kitchen equipment utensil sample set, and in response to the utensil sample set not being updated in the next N continuous maintenance operations after being updated in the maintenance operation, the target detection model is incrementally trained using the utensil sample set and the target detection model is updated. Through this scheme, the present application can continuously incrementally train the target detection model using images of new utensil categories from the user of the kitchen equipment, to ensure that the target detection model continuously enhances the category and range of utensil recognition. BRIEF DESCRIPTION OF DRAWINGS

[0009] To more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0010] Figure 1 A structural schematic diagram of a kitchen equipment according to an embodiment of the present application is shown.

[0011] Figure 2 A schematic diagram of a kitchen equipment system according to an embodiment of the present application is shown.

[0012] Figure 3 A flowchart of a maintenance method for a target detection model of kitchen equipment according to an embodiment of the present application is shown.

[0013] Figure 4 A flowchart of a maintenance operation in the method according to an embodiment of the present application is shown. Figure 3

[0014] Figure 5 A flowchart of a method for determining new outlier samples in the method according to an embodiment of the present application is shown. Figure 4

[0015] Figure 6 A flowchart of a maintenance method for a target detection model of kitchen equipment according to another embodiment of the present application is shown.​​

[0016] Figure 7 A flowchart illustrating a deduplication operation according to an embodiment is shown.

[0017] Figure 8 A structural diagram of a target detection device according to an embodiment of the present application is shown.

[0018] Figure 9 A structural diagram of a kitchen equipment system according to an embodiment of the present application is shown.

[0019] Figure 10 A structural diagram of a computer readable storage medium according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0022] Before the embodiments of the present application are described in detail, a brief description of the kitchen equipment of the present application is given. The kitchen equipment may, for example, be a dishwasher, a sterilizer, a dishware dryer or other types of kitchen equipment. The kitchen equipment can internally include a closable cavity to place and hold dishware, so as to perform various operations such as washing, sterilizing, drying and the like of the dishware. The following is described taking a dishwasher as an example.

[0023] Reference is made to Figure 1 , Figure 1is a structural schematic diagram of a kitchen device 100 according to an embodiment of the present application. The kitchen device 100 may, for example, be a dishwasher. The kitchen device 100 defines a washing area inside the kitchen device 100 for placing dishes to be washed. An image acquisition device 10 may, for example, be arranged on an inner wall of the washing area for acquiring images of the washing area, in particular of the dishes. The image acquisition device 10 may, for example, also be arranged on top of the washing area, on a side of the washing area or at another location. The present application does not limit the arrangement of the image acquisition device 10.

[0024] In some embodiments, the kitchen device 100 may, for example, also comprise at least one spray head or spray arm (not shown). The at least one spray head may, for example, spray cleaning liquid (e.g. water, cleaning agent or a mixture of both) onto the dishes. The spray flow, the spray direction and the like of the cleaning liquid of the at least one spray head may, for example, be adjusted depending on the recognition of the dishes in the kitchen device 100.

[0025] Referring to Figure 2 , Figure 2 Fig. 2 shows a schematic diagram of a kitchen device system 200 according to an embodiment of the present application. The kitchen device system 200 may, for example, comprise at least one kitchen device 100 and a server 20. The server 20 may, for example, communicate and exchange data with a plurality of kitchen devices 100 via a network.

[0026] The network may, for example, comprise, but is not limited to, the Internet, an Ethernet network, a mobile communication network, a wide area network, a metropolitan area network and the like. The mobile communication network may, for example, comprise, but is not limited to, a third generation (3G) mobile communication network, a third point five generation (3.5G) mobile communication network, a fourth generation (4G) mobile communication network, a fifth generation (5G) mobile communication network and the like.

[0027] Depending on the division, the server 20 may, for example, be a centralized server, a distributed server, a cloud server and the like, which is not limited in the present application. Compared with a single kitchen device 100, the server 20 may, for example, have stronger computing power and computing speed. In the specific application of modern intelligent kitchen devices 100, services with high computing requirements such as target recognition are often executed by the server 20 rather than a single kitchen device 100, so as to provide better recognition accuracy and speed. In addition, deploying a target detection model, such as a dish detection model in the present application, on the server 20 can also better maintain the target detection model, thereby providing the latest and most advanced target recognition service for all kitchen devices 100 connected with the server 20. Of course, the target detection model may, for example, also be deployed on a single kitchen device 100, which is not limited in the present application.

[0028] In the present application, the kitchen equipment system 200 can use a target detection model to identify dishware information such as the number of dishware, the type of dishware, and / or the degree of dirtiness of each dishware, and determine specific dishware cleaning parameters, including the amount of water and detergent, cleaning intensity, cleaning time, etc., according to these information. The type of dishware can include information such as the type of dishware and / or the material of dishware, for example, a glass goblet, a ceramic bowl, a melamine plate, a stainless steel spoon holder, etc.

[0029] The target detection model can use a target detection algorithm to identify dishware information. Currently mainstream target detection algorithms can be classified into two categories: two-stage detection algorithms and one-stage detection algorithms. The two-stage detection algorithm first uses an RPN network to generate region proposals for rough positioning, and then performs precise positioning (regression) and classification prediction on the candidate regions in the model head. Typical two-stage detection algorithms include the FasterRCNN, RFCN, and other series of methods. The one-stage detection algorithm directly regresses the class probability and position information of the target object based on the features extracted by the backbone network, without generating region proposals for rough positioning. Typical algorithms include YOLO v1 / v2 / v3, SSD, etc. Compared with the two-stage detection algorithm, the one-stage detection algorithm has faster detection speed, which is helpful for realizing real-time target detection.

[0030] The target detection model of the present application can use any of the above two-stage detection algorithms and one-stage detection algorithms. Specifically, the target detection model can be deployed in any of the kitchen equipment 100 and the server 20. In some embodiments, the kitchen equipment system 200 can include at least two different target detection models, for example, a first target detection model and a lightweight second target detection model. The first target detection model can be deployed on the server 20, and the relatively lightweight second target detection algorithm can be deployed on the kitchen equipment 100 to meet the demand for pre-detection of dishware images. The first target detection model and the second target detection model can correspond to each other. After the first target detection model is updated, the second target detection model can be updated correspondingly to maintain synchronization between the two.

[0031] In some embodiments, by way of example only and not limitation, the target detection model deployed on the kitchen equipment 100 can be a one-stage target detection model using a one-stage detection algorithm to meet the requirements of real-time response and embedded deployment in the field of dishware cleaning, while the target detection model deployed on the server 20 can be a two-stage target detection model using a two-stage detection algorithm to make full use of the computing resources of the server 20 and improve the detection accuracy.

[0032] Reference Figure 3 ,Figure 4 and Figure 5 , Figure 3 FIG. 1 shows a flowchart of a method for maintaining a target detection model of a kitchen device 100 according to an embodiment of the present application, Figure 4 FIG. 2 shows a flowchart of a method for maintaining a target detection model of a kitchen device 100 according to an embodiment of the present application, Figure 3 FIG. 3 shows a flowchart of a method for maintaining a target detection model of a kitchen device 100 according to an embodiment of the present application, Figure 5 FIG. 4 shows a flowchart of a method for maintaining a target detection model of a kitchen device 100 according to an embodiment of the present application. Figure 4 FIG. 5 shows a flowchart of a method for determining a new outlier sample according to an embodiment of the present application.

[0033] As shown in FIG. 1, the method for maintaining a target detection model of a kitchen device 100 includes the following steps S11-S12. Figure 3

[0034] Step S11: Continuously performing a maintenance operation of a utensil sample set of the kitchen device 100.

[0035] Specifically, in some embodiments, the maintenance operation is performed once for each start of the kitchen device 100.

[0036] In some embodiments, the maintenance operation of the utensil sample set of the kitchen device 100 is continuously performed within a period of time or a number of starts after the kitchen device 100 is first sold after leaving the factory.

[0037] In some embodiments, the kitchen device 100 can obtain information of a user identification or a specific use address of the kitchen device 100, and when the user identification of the kitchen device 100 changes or the specific use address changes significantly, the kitchen device 100 continuously performs the maintenance operation of the utensil sample set of the kitchen device 100 within a period of time or a number of starts thereafter.

[0038] In some embodiments, the step S11 can also be initiated by the user. For example, when the user newly purchases a batch of utensils, the user can actively initiate the step S11. The user can send an instruction to the kitchen device 100 through a button provided on the kitchen device 100, a touchable button on the screen, or through other wired or wireless means to initiate the step S11.

[0039] Specifically, the utensil sample set can be a utensil sample set specific to a kitchen device 100. In particular, the utensil sample set can be a utensil sample set specific to a user or a use scenario of the kitchen device 100.

[0040] ​The tableware sample set may, for example, consist of outliers. Specifically, outliers are images containing the object detection bounding boxes of tableware, where the object category of the tableware does not belong to the object categories included in the current object detection model of the kitchen equipment 100. For example, the tableware that the kitchen equipment 100 needs to clean includes a tortoiseshell bowl. However, the object detection bounding box corresponding to the kitchen equipment 100 cannot identify tortoiseshell bowls. Therefore, the image portion of the object detection bounding box corresponding to the tortoiseshell bowl can be considered an outlier. The manifestations and detection methods of outliers will be described in detail below.

[0041] The tableware sample set may be created and deployed for the kitchen device 100, for example, on server 20 (e.g., in the cloud) or locally on the kitchen device 100. In some embodiments, the tableware sample set of the kitchen device 100 is stored in a cloud that is communicatively connected to the kitchen device 100. For example, a corresponding tableware sample set may be created for the kitchen device 100 when it is manufactured, when it is first sold, or when it is first used. When created, the tableware sample set may be initialized to an empty set, for example.

[0042] In some embodiments, when a change in the user of the kitchen appliance 100 or a significant change in the usage scenario (e.g., the location of use) is detected, the tableware sample set corresponding to the kitchen appliance 100 is cleared again.

[0043] like Figure 4 As shown, each maintenance operation includes the following operations S21 to S24.

[0044] Step S21: In response to the startup of the kitchen equipment 100, acquire an image of the tableware inside the kitchen equipment 100.

[0045] In some embodiments, a trigger may be provided at the door opening mechanism of the kitchen appliance 100. When the door of the kitchen appliance 100 is closed, the trigger is activated and sends a trigger signal. The controller of the kitchen appliance 100 receives the trigger signal and, in response to the trigger signal, begins to initialize the image acquisition device 10 and controls the image acquisition device 10 to capture images of the tableware inside the kitchen appliance 100.

[0046] Step S22: Use the object detection model to predict the object detection box and the corresponding object category in the tableware image.

[0047] Specifically, the object detection model may be, for example, an object detection model deployed at kitchen equipment 100 or an object detection model deployed at server 20. In the following description, the object detection model deployed at server 20 is used as an example rather than a limitation.

[0048] In particular, the tableware images obtained in step S21 can be uploaded to the server 20 to predict target detection boxes and corresponding target categories in the tableware images using a target detection model. The target detection box can be, for example, a rectangular box surrounding at least a portion of a tableware. In some embodiments, the target detection box can also be a tilted box or a box of other shapes, etc. The target detection model is further configured to predict a target category of the tableware within the target detection box, such as a glass goblet, a ceramic soup bowl, a melamine plate, a stainless steel spoon rack, etc. These target categories can be, for example, target categories learned by the target detection model from samples in a supervised or unsupervised manner. Generally speaking, when a new tableware category that the target detection model has never learned appears, the target detection model is often unable to identify the new tableware category. Here, the target detection model needs to be further trained to learn the ability to identify the target category. The target detection model can also output a confidence score of each target detection box, which generally represents the possibility that the target detection box exists a real tableware.

[0049] In some embodiments, the target detection model only outputs target detection boxes with a confidence score greater than a certain threshold, and target detection boxes with a confidence score lower than the threshold are considered invalid target detection boxes and are ignored, not entering the subsequent steps.

[0050] Step S23: Determine whether each target detection box is a new outlier sample.

[0051] In particular, it is determined whether each target detection box is a new outlier sample that has not been detected in previous maintenance operations.

[0052] In this application, as described above, an outlier sample refers to a target detection box or an image portion corresponding to the target detection box, and the target category of the tableware contained in the target detection box does not exist in the current target detection model or cannot be accurately identified by the current target detection model. As can be seen, whether a target detection box is an outlier sample depends on the detection ability of the current target detection model. A target detection box can be an outlier sample for a period of time, but after that, when the target detection model is updated, retrained or incrementally trained and can identify the tableware category surrounded by the target detection box, the target detection box is no longer an outlier sample.

[0053] A new outlier sample is an outlier sample that is sufficiently different from the outlier samples existing in the current tableware sample set. The application determines whether a sample is a new outlier sample to prevent the target detection boxes of the tableware in the kitchen equipment 100 from being repeatedly added to the tableware sample set as much as possible. For example, when no new outlier sample is detected for several times (e.g., 3 times, 7 times, 12 times or any other multiple times) of maintenance operations, it can be considered that the tableware that needs to be cleaned in the kitchen equipment 100 has been photographed and identified once.

[0054] In particular, with reference toFigure 5 The method for determining whether each target detection frame is a new outlier sample in step S23 includes procedures S31-S38.

[0055] Specifically, the procedures S31-S38 are performed on each target detection frame in the predicted tableware image in step S22 to determine whether the target detection frame is a new outlier sample.

[0056] In some embodiments, all target detection frames in the predicted tableware image are put into a set and sorted, for example, from the 1st target detection frame to the pth target detection frame. In fact, p is the total number of target detection frames. Then, the procedures S31-S38 are performed on each target detection frame in sequence according to the sorting result.

[0057] In some embodiments, the procedures S31-S38 can be performed on each target detection frame according to the position of the predicted target detection frame in the tableware image.

[0058] Step S31: Obtain a target detection frame.

[0059] Specifically, taking the example that there are p target detection frames in the tableware image and whether they are new outlier images need to be determined, a flag f can be set to mark the index value of the target detection frame currently to be obtained. That is, in the current step S31, the fth target detection frame is obtained. The initial value of f is 1.

[0060] Step S32: Calculate the first difference between the target detection frame and the reference sample of the corresponding target category.

[0061] Specifically, the corresponding target category is the target category of the target detection frame output by the target detection model. The target category in this application is the tableware category, for example, a glass goblet, a ceramic soup bowl, a melamine plate, a stainless steel spoon holder, etc.

[0062] Specifically, when training the target detection model, the input training data is labeled with a label (ground truth) including the target category of the above-mentioned tableware. The trained target detection model can learn the features of the tableware of the target category and identify the target category labeled in the training set. In some embodiments, the trained target detection model can identify K target categories. Generally, the target detection model cannot identify the target category not labeled in the training set. For example, if the target detection model does not include a tortoise shell bowl in the training set, the target detection model will not be able to identify the tortoise shell bowl. However, the inventors of the present application have found in long-term practice and research that although the target detection model of the tableware cannot correctly identify the new target category, such as the tortoise shell bowl, it will still mistakenly identify the tableware of the new target category as one of the K target categories included in the current target detection model. For example, by way of example but not limitation, the target detection model can mistakenly identify the tortoise shell bowl as a ceramic soup bowl, and the identified target category, i.e., the ceramic soup bowl, is the target category corresponding to the target detection box, regardless of whether it is actually correct or not.

[0063] A first difference between the target detection box and the reference sample of the corresponding target category is calculated. Specifically, the target detection box can be normalized, and the first difference between the normalized target detection box and the reference sample is calculated. The first difference can be, for example, any distance for representing the degree of difference between the target detection box and the reference sample, including Euclidean distance, Mahalanobis distance, Manhattan distance, or Hamming distance, etc.

[0064] Step S33: determining whether the first difference is greater than a first threshold.

[0065] The step S33 can be used to determine whether the target detection box is an outlier sample.

[0066] In some embodiments, a database corresponding to the target detection model can be stored at the server 20. The database can, for example, store target detection boxes classified by the target category of the tableware.

[0067] The first threshold can be a threshold based on the corresponding target category. For example, the threshold can be based on the target detection boxes of the target category in the above-mentioned database. Specifically, the first threshold can be a certain multiple, such as 1.5 times, 2 times, etc., of the average distance between the target detection boxes of the target category and the reference sample of the target category. In some embodiments, the first threshold can be a certain multiple, such as 1 times, 1.5 times, etc., of the maximum distance between the target detection boxes of the target category and the reference sample. The first threshold can also be set in other ways, for example, the first threshold can also be a fixed value, which is not limited in the present application.

[0068] Specifically, if the first difference is greater than the first threshold, the target bounding box is an outlier sample, and the process proceeds to step S34. If the first difference is not greater than the first threshold, the target bounding box is not an outlier sample, and the process proceeds to step S38.

[0069] Step S34: Determine whether the tableware sample set is empty. In response to the tableware sample set being empty, determine that the target bounding box is a new outlier sample, and the process proceeds to step S37. Otherwise, in response to the tableware sample set not being empty, the process proceeds to step S35.

[0070] Step S35: Calculate a second difference between the target bounding box and each target bounding box in the tableware sample set.

[0071] This step can be used to determine whether the target bounding box is a new outlier sample. Specifically, the second difference can be used to represent the degree of difference between the target bounding box and each target bounding box in the tableware sample set.

[0072] Similar to the first difference, the second difference can be, for example, any distance used to represent the degree of difference between the target bounding box and each reference sample in the current tableware sample set, including Euclidean distance, Mahalanobis distance, Manhattan distance, or Hamming distance, etc. The first difference and the second difference can belong to the same type of distance, for example, both belong to Euclidean distance.

[0073] Step S36: Determine whether the second difference between the target bounding box and each target bounding box in the tableware sample set is greater than a second threshold. In response to the second difference being greater than the second threshold, it can be determined that the target bounding box is a new outlier sample, and the process proceeds to step S37. Otherwise, the process proceeds to step S38.

[0074] Specifically, if the second difference between the target bounding box and each target bounding box in the tableware sample set is greater than the second threshold, it indicates that there is a significant difference between the target bounding box and each target bounding box in the current tableware sample set, and the target bounding box is a new outlier sample. The second threshold can be a threshold set according to experience or theory, and the specific implementation is not limited.

[0075] Step S37: Determine that the target bounding box is a new outlier sample, and incorporate the target bounding box into the tableware sample set of the kitchen device 100. The process proceeds to step S38.

[0076] In this step S37, the tableware sample set of the kitchen device 100 is updated.

[0077] Step S38: Determine whether all target bounding boxes have been traversed.

[0078] If yes, the process ends. For example, it can be judged whether the flag f is equal to the number of target detection boxes p. When f = p, it can be considered that the above process of judging whether it is a new outlier sample has been performed for all target detection boxes, and the process can end.

[0079] If no, go to step S31 to perform the above process for the next target detection box. In some embodiments, if the flag f is less than the number of target detection boxes p, the value of f can be increased by 1, i.e. f = f + 1, go to step S31, and continue to perform the process of judging whether the target detection box is a new outlier sample for the next target detection box.

[0080] Step S24: Incorporate the new outlier sample into the tableware sample set of the kitchen device 100 to update the tableware sample set of the kitchen device 100.

[0081] The above steps S21-S24 are the specific process of the maintenance operation of the tableware sample set.

[0082] Step S12: In response to the tableware sample set not being updated in the next N maintenance operations after being updated in the maintenance operation, incrementally train the target detection model using the tableware sample set and update the target detection model. N is a positive integer.

[0083] Specifically, after the tableware sample set is updated last time, the tableware sample set is not updated in the next N maintenance operations (i.e. the start of the kitchen device 100), it is considered that the current tableware has been photographed once.

[0084] In some embodiments, before incrementally training the target detection model using the tableware sample set, a deduplication operation can be performed on the target detection model in the current tableware sample set. In some embodiments, due to reasons such as placement position, posture and illumination, multiple target detection models of the same tableware can exist repeatedly in the tableware sample set. In order to remove as many repeated target detection boxes as possible, to reduce the workload of marking the target detection boxes and reduce the computational burden of incremental training, a deduplication operation can be performed on the target detection model in the tableware sample set.

[0085] In some embodiments, the deduplication operation can be performed manually or using a clustering algorithm. The clustering algorithm that can be used includes K-Means clustering, K-center clustering, density-based clustering, etc.

[0086] In some embodiments, the deduplication operation is performed using K-Means clustering. Specifically, refer to Figure 7 , Figure 7 A flowchart of a deduplication operation according to an embodiment of the present application is shown. As shown in Figure 7 , the deduplication operation includes steps S701-S703.

[0087] Step S701: Clustering operation is performed on the target bounding boxes in the cutlery sample set to obtain at least one cluster and a corresponding cluster center.

[0088] In some embodiments, normalization is needed for the target bounding boxes before the clustering operation is performed. K-Means clustering for the target bounding boxes requires setting the number of clusters for clustering. The number can be set by the user or be other preset values, which are not limited in the present application. K-Means clustering can divide the target bounding boxes into several clusters. The cluster center of each cluster can be the arithmetic mean of all target bounding boxes in the cluster, which is calculated similarly to the reference sample described above. In some embodiments, the cluster center can be exactly one of the target bounding boxes.

[0089] Step S702: For each cluster, the distance between the target bounding boxes in the cluster and the corresponding cluster center is obtained, and the minimum value among the distances is obtained. For example, for a cluster including W target bounding boxes, the distance D i between each of the W target bounding boxes and the cluster center can be calculated, 1≤i≤W. The distance can be, for example, any distance for characterizing the difference between different images, including Euclidean distance, Mahalanobis distance, Manhattan distance, or Hamming distance, etc. Further, the minimum value D i among the W distances D 最小 is obtained.

[0090] Step S703: For each cluster, the target bounding boxes with a distance greater than the minimum value D 最小 and less than a third threshold value are removed from the cutlery sample set. This step can be used to remove the target bounding boxes in each cluster that are close to the cluster center, but keep the target bounding boxes that can be used as the cluster center, or in the case where the cluster center is not one of the target bounding boxes, the target bounding box closest to the cluster center can be kept. The third threshold value can be set according to experience or theory. The present application does not limit this.

[0091] Specifically, the incremental learning training of the target detection model means that the target detection model is subjected to incremental learning. Incremental learning, also known as continuous learning or lifelong learning, is a machine learning method that allows machine learning models to continuously learn new data rather than retrain the entire model. The incremental learning method allows the target detection model to continuously learn new knowledge to adapt to the complex and changing recognition environment in practical applications.

[0092] The incremental learning method that can be used in the present application includes, for example, a fine-tuning method, a joint training method, a learning without forgetting (LwF) method, and the like. Specifically, the fine-tuning method generally only needs to use new samples to train a model (for example, a target recognition model), and has the advantages of small amount of calculation and the disadvantage of easy catastrophic forgetting. The joint training method re-trains the model on all known data including previous samples and new samples, and has the best training effect, but requires the largest amount of calculation and time for training, and has a relatively high training cost. The LwF method is a training method between the joint training method and the fine-tuning method. The LwF method uses the idea of knowledge distillation and can update without using old samples. The LwF method can be preferably used for incremental learning in the present application. Other incremental learning methods can also be used in the present application, and the present application does not limit this.

[0093] In the present application, as described above, if the tableware sample set is updated in one maintenance operation and is not updated in the next N consecutive maintenance operations, it can be considered that the tableware in the current user or application scenario of the kitchen device 100 has been photographed at least once. At this time, the tableware sample set can be used to train the target detection model.

[0094] In the present application, the step of using the tableware sample set to incrementally train and update the target detection model can include: labeling the target class of the target detection frame in the tableware sample set; and using the labeled target detection frame to incrementally train the target detection model. In some embodiments, the target class of the target detection frame can be manually labeled. In some embodiments, an artificial intelligence large model or other types of artificial intelligence products can also be used to automatically label the target detection frame with labels including the target class. The labeled target detection frame can be used as a training set to incrementally train the target detection model. The specific process of this incremental training is not repeated here.

[0095] In some embodiments, in response to the target category being a new category different from the current target categories recognized by the current target detection model, the baseline sample corresponding to the target category is calculated based on all target detection boxes in the cutlery sample set that are labeled as the target category. For example, as described above, the current target detection model can recognize k target categories, but the k model does not include tortoiseshell bowls and bamboo cups. The user of the current kitchen device 100 has a batch of tortoiseshell bowls and bamboo cups that need to be cleaned in the kitchen device 100. Through the above method for maintaining the target detection model of the kitchen device 100, the target detection boxes of these tortoiseshell bowls and bamboo cups are identified as new outlier samples. Further, the application can respectively collect the target detection boxes of the tortoiseshell bowls and bamboo cups to respectively calculate the baseline sample corresponding to the target category of the tortoiseshell bowls and the baseline sample corresponding to the target category of the bamboo cups.

[0096] Specifically, the method for calculating the baseline sample of the target category is: performing normalization processing on the target detection boxes under the target category to obtain standard target detection boxes; and calculating the arithmetic mean of all standard target detection boxes of the target category as the baseline sample. Specifically, each pixel point value of the baseline sample is the arithmetic mean of the pixel point values at the corresponding positions of the standard target detection boxes.

[0097] Specifically, the normalized image can have the same size, for example, 120*160. In some embodiments, the normalized image can also resist the influence of geometric transformation.

[0098] In some embodiments, the normalization method can include linear function conversion, logarithmic function conversion, inverse tangent function conversion, and invariant moment-based conversion, etc. Among them, the image normalization process based on invariant moments can include steps such as coordinate centering, x-shearing normalization, scaling normalization, and rotation normalization.

[0099] In some embodiments, in response to the target category belonging to the current target categories recognized by the target detection model, all target detection boxes in the cutlery sample set that are labeled as the target category are transmitted to the database of the target detection model, and the baseline sample corresponding to the target category is updated using the updated database.

[0100] For example, as described above, the current target detection model can recognize k target categories, but at least one of the target categories, such as the stainless steel spoon holder target category, has a deviation between the baseline sample corresponding to the target category and the ideal baseline sample due to insufficient training samples or different distribution of training samples from the distribution of products in reality during training. In this case, some target detection boxes of cutlery that originally belong to this target category can be incorrectly identified as new outlier samples.

[0101] To improve the sample library of the target category and correct the benchmark sample to be as close to the ideal benchmark sample as possible, the application can transmit all the target detection boxes labeled as the target category in the tableware sample set to the database or sample library corresponding to the target category of the target detection model, and update the benchmark sample corresponding to the target category by using the updated database. Specifically, the target detection box can be converted into a standard target detection box after normalization processing, and then merged into the database of the target category. The application can calculate the arithmetic mean of all standard target detection boxes of the target category in the database as the updated benchmark sample. Specifically, each pixel point value of the benchmark sample is the arithmetic mean of the pixel point values at the corresponding position of the standard target detection box.

[0102] Optionally, as shown in Figure 3 , the maintenance method of the target detection model of the kitchen device 100 can further include step S13: resetting the tableware sample set to an empty set.

[0103] Specifically, the application can empty all the target detection boxes in the tableware sample set after step S12, and / or mark the tableware sample set as an empty set. Resetting the tableware sample set to an empty set can facilitate the use of the tableware sample set in subsequent maintenance operations.

[0104] In some embodiments, step S13 further includes incorporating the target detection boxes in the tableware sample set and their labels and shooting information into the database corresponding to the target detection model. Specifically, the database corresponding to the target detection model can be stored at the server 20. The database can store, for example, target detection boxes classified by the target category of the tableware. These target detection boxes can be, for example, normalized standard target detection boxes. Optionally, the database can also store the shooting information of each target detection box, including the shooting time and shooting location, etc. The target detection boxes in the database can be used, for example, to maintain the target detection model, calculate the benchmark sample, etc.

[0105] As described above, Figure 3 , the above steps S11-S12 or S11-S13 can be performed after the kitchen device 100 is first sold, or initiated by the user of the kitchen device 100, or when the user identification or specific use address of the kitchen device 100 changes significantly.

[0106] In some embodiments, after step S12, the target detection boxes in the tableware sample set can be stored as the basic data set of the kitchen device 100 at the kitchen device 100 or the server 20.

[0107] Referring to Figure 6 , Figure 6 shows a flowchart of a maintenance method of a target detection model of a kitchen device 100 according to another embodiment of the application.

[0108] As Figure 6 shown, the maintenance method comprises steps S41-S47.

[0109] Step S41: Continuously performing maintenance operation of the dishware sample set of the kitchen device 100.

[0110] This step is similar to the step S11 described above, and will not be repeated here.

[0111] Step S42: In response to the dishware sample set not being updated in the next N consecutive maintenance operations after being updated in the maintenance operation, incrementally training the target detection model using the dishware sample set and updating the target detection model. N is a positive integer.

[0112] This step is similar to the step S12 described above, and will not be repeated here.

[0113] Step S43: Resetting the dishware sample set to an empty set.

[0114] This step is similar to the step S13 described above, and will not be repeated here.

[0115] In some embodiments, steps S41-S43 can be performed when the kitchen device 100 is first used by the user.

[0116] In some embodiments, the user of the kitchen device 100 can also actively start step S41. For example, after the user purchases a batch of new dishware, the user can actively start step S41.

[0117] Step S44: Continuously performing maintenance operation of the dishware sample set of the kitchen device 100.

[0118] Wherein, the maintenance operation of the dishware sample set can refer to the description of the maintenance method of the dishware sample set in the above. Figure 4

[0119] Step S45: In response to the kitchen device 100 updating the dishware sample set in M consecutive maintenance operations, checking the recognition result of the target detection frame in the dishware sample set, wherein M is a positive integer.

[0120] Specifically, the value of M can be 3, 5, 6, 7 or any other positive integer set according to experience or theory.

[0121] ​Specifically, the operation of updating the tableware sample set indicates that new outlier samples are added in the tableware sample set. This situation can be caused by, for example, the kitchen device 100 captures tableware belonging to a new target category, or the kitchen device 100 captures new difficult-to-identify tableware. At this time, it can be checked by manual or other means whether the identification result of the target detection frame of the new outlier sample is correct, and the error rate of the identification result is calculated. For example, the current tableware sample set includes a total of s outlier samples, of which t outlier samples have incorrect identification results, and the error rate of the identification result is (t / s)*100%.

[0122] Step S46: In response to the error rate of the identification result being higher than the fourth threshold value, incrementally training the target detection model using the tableware sample set and updating the target detection model.

[0123] Specifically, the fourth threshold value can be a numerical value set according to experience or theory. As an example but not limitation, the fourth threshold value can be 20%, 30%, 40%, or 70%, etc.

[0124] The error rate of the identification result being higher than the fourth threshold value indicates that the kitchen device 100 is very likely to capture a new tableware target category or a difficult-to-identify tableware type. At this time, the target detection model can be incrementally trained using the new outlier samples in the tableware sample set as described above. The method of incrementally training is as described above, and will not be repeated here.

[0125] Step S47: Reset the tableware sample set to an empty set.

[0126] Similar to step S13, the tableware sample set is reset to an empty set by the present application. In some embodiments, as described in step S13 above, the present application can incorporate the target detection frame and its label and shooting information in the tableware sample set into the database corresponding to the target detection model. In some embodiments, the present application can store the target detection frame in the tableware sample set as update data in the kitchen device 100 or the server 20.

[0127] After step S47, the method can return to step S44, and the above process is repeatedly executed.

[0128] Reference Figure 8 , Figure 8 A schematic diagram of a maintenance device 800 for the target detection model of the kitchen device 100 according to an embodiment of the present application is shown. As shown in FIG. 8, the maintenance device 800 includes a target detection model database 810, a tableware sample set 820, and a target detection model updating unit 830. Figure 8As shown, the maintenance device 800 comprises a first tableware sample set maintenance operation module 810, a second tableware sample set maintenance operation module 820, a first tableware sample set resetting module 830, a second tableware sample set resetting module 840, a first target detection model incremental training module 850, a second target detection model incremental training module 860, and a recognition result checking module 870, which are connected in communication with each other.

[0129] Specifically, the first tableware sample set maintenance operation module 810 can be configured to continuously perform the maintenance operation of the tableware sample set of the kitchen device 100. For example, the first tableware sample set maintenance operation module 810 can be configured to continuously perform the maintenance operation of the tableware sample set of the kitchen device 100 when the kitchen device 100 is first used by a user.

[0130] Specifically, the second tableware sample set maintenance operation module 820 can be configured to continuously perform the maintenance operation of the tableware sample set of the kitchen device 100. For example, the second tableware sample set maintenance operation module 820 can be configured to continuously perform the maintenance operation of the tableware sample set of the kitchen device 100 during the subsequent use of the kitchen device 100.

[0131] In some embodiments, the first tableware sample set maintenance operation module 810 and the second tableware sample set maintenance operation module 820 can be the same module. The present application does not limit this.

[0132] Specifically, the first tableware sample set resetting module 830 can be configured to reset the tableware sample set to an empty set.

[0133] Specifically, the second tableware sample set resetting module 840 can be configured to reset the tableware sample set to an empty set.

[0134] In some embodiments, the first tableware sample set resetting module 830 and the second tableware sample set resetting module 840 can be the same module, and the present application does not limit this.

[0135] Specifically, the first target detection model incremental training module 850 can be configured to, in response to the tableware sample set being updated in the maintenance operation and not being updated in the next N consecutive maintenance operations, perform incremental training on the target detection model using the tableware sample set and update the target detection model. Wherein, N is a positive integer.

[0136] Specifically, the recognition result checking module 870 can be configured to, in response to the kitchen device 100 updating the tableware sample set in M consecutive maintenance operations, check the recognition result of the target detection frame in the tableware sample set. M is a positive integer.

[0137] Specifically, the second target detection model incremental training module 860 can be configured to, in response to the error rate of the identification result being higher than the fourth threshold value, perform incremental training on the target detection model using the tableware sample set and update the target detection model.

[0138] Reference Figure 9 , Figure 9 A structural schematic diagram of a kitchen equipment system 900 according to an embodiment of the present application is shown. The kitchen equipment system 900 may, for example, include the kitchen equipment 100 and the server 20. The kitchen equipment 100 may, for example, be a dishwasher or a sterilizer. As shown, the kitchen equipment system 900 further includes a processor 910 and a memory 920. The memory 920 may, for example, store the target detection system described above. The memory 920 further stores a computer program. The processor 910 is configured to execute the computer program to implement the maintenance method described above. Figure 9

[0139] Reference Figure 10 The present application also provides a computer-readable storage medium 1000. The computer-readable storage medium 1000 stores a computer program 1010. The computer program 1010 can be executed by a processor of a computer to implement the maintenance method described above.

[0140] The embodiments of the present application are realized in the form of software functional units and sold or used as independent products. Based on such understanding, the technical solutions of the present application or the whole or part of the technical solutions that make essential contributions to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0141] The above is only an embodiment of the present application, and does not limit the patent scope of the present application. The equivalent structure or equivalent flow transformation made by the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.​

Claims

1. A method for maintaining a target detection model for kitchen equipment, characterized in that, The method comprises: continuously performing a maintenance operation on a utensil sample set of the kitchen device, the maintenance operation comprising: in response to the kitchen device being started, obtaining an image of a utensil within the kitchen device; using the object detection model to predict an object detection bounding box and a corresponding object class in the image of the utensil; calculating a first difference between each of the object detection bounding box and a reference sample of the corresponding object class; in response to the first difference being greater than a first threshold, performing: in response to the utensil sample set being an empty set, incorporating the object detection bounding box into the utensil sample set of the kitchen device to update the utensil sample set of the kitchen device; or in response to the utensil sample set being a non-empty set, calculating a second difference between the object detection bounding box and each of the object detection bounding boxes in the utensil sample set, in response to each of the second differences corresponding to the object detection bounding box being greater than a second threshold, incorporating the object detection bounding box into the utensil sample set of the kitchen device to update the utensil sample set of the kitchen device; and in response to the utensil sample set not being updated in the next N consecutive maintenance operations after being updated in the maintenance operation, using the utensil sample set to incrementally train and update the object detection model, wherein N is a positive integer, wherein the using the utensil sample set to incrementally train and update the object detection model comprises: annotating the object classes of the object detection bounding boxes in the utensil sample set; and using the annotated object detection bounding boxes to incrementally train the object detection model; wherein the annotating the object classes of the object detection bounding boxes in the utensil sample set further comprises: in response to the object class being a new class different from a current object class recognizable by the object detection model, based on all the object detection bounding boxes in the utensil sample set annotated as the object class, calculating a reference sample corresponding to the object class; and / or in response to the object class belonging to the current object class recognizable by the object detection model, transmitting all the object detection bounding boxes in the utensil sample set annotated as the object class to a database of the object detection model, and using the updated database to update the reference sample corresponding to the object class.

2. The method of claim 1, wherein the using the utensil sample set to incrementally train and update the object detection model comprises: performing a clustering operation on the object detection bounding boxes in the utensil sample set to obtain at least one cluster and a corresponding cluster center; for each cluster, obtaining a distance between the object detection bounding boxes in the cluster and the corresponding cluster center, and a minimum value in the distances; and for each cluster, removing the object detection bounding boxes in the utensil sample set having a distance greater than the minimum value and less than a third threshold.

3. The method of claim 1, wherein after the using the utensil sample set to incrementally train and update the object detection model, the method further comprises: resetting the utensil sample set to an empty set. ​ ​ 4. The maintenance method of claim 3, wherein the step of resetting the tableware sample set to an empty set comprises: incorporating the target detection frame and its label and shooting information in the tableware sample set into a database corresponding to the target detection model. After the step of resetting the tableware sample set to an empty set, the method further comprises:

5. The method of maintenance of claim 3, wherein, continuously performing the maintenance operation of the tableware sample set of the kitchen equipment, in response to the kitchen equipment updating the tableware sample set in consecutive M maintenance operations, checking an identification result of the target detection frame in the tableware sample set, wherein M is a positive integer; and in response to the error rate of the identification result being higher than a fourth threshold, incrementally training the target detection model using the tableware sample set and updating the target detection model.

6. The maintenance method of claim 1, wherein the tableware sample set of the kitchen equipment is stored in a cloud end in communication connection with the kitchen equipment. a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to implement the maintenance method of any one of claims 1-6. the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the maintenance method of any one of claims 1-6.

7. A kitchen device system characterized in that, ​ 8. A computer-readable storage medium, characterized in that, ​

Images