Dish-washing machine cleaning strategy determination method and device based on image recognition

Abstract

The invention discloses a dish-washing machine cleaning strategy determination method and device based on image recognition, and relates to the technical field of smart home, and the dish-washing machine cleaning strategy determination method based on image recognition comprises the steps: determining an image recognition result of the interior of a dish-washing machine; wherein the image recognition result comprises attribute information of a cleaning object existing in the dish washing machine; the attribute information is at least used for indicating the function, the material, the number and the orientation of the cleaning object; determining a washing mode of the cleaning object according to the function and the material of the cleaning object, and determining a washing position and washing time of a washing device used for executing the washing mode in the dish washing machine according to the number and the orientation of the cleaning object; and according to the washing mode, the washing position and the washing time, a washing strategy for the washing object is determined.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart home, in particular to a dish washer cleaning strategy determination method and device based on image recognition. BACKGROUND

[0002] Most dish washers on the market are of the manual setting type, that is, users need to select the washing mode and set the washing time on the control panel of the dish washer. Such a cleaning strategy is usually fixed when the dish washer is shipped from the factory, including multiple modes such as daily washing, light oil washing, fruit and vegetable washing, crystal washing, steam washing, and intelligent washing, aiming to cover different cleaning needs. However, such a dish washer with a fixed cleaning strategy has many problems in actual use, limiting its performance and user experience. Specifically, in a fast-paced life, although the dish washer provides multiple washing modes, users often make mistakes due to neglect or forget the details of the instruction manual, resulting in low cleaning efficiency, poor effect, and even damage to kitchen utensils, especially for materials sensitive such as wooden or plastic tableware. In addition, users have difficulty determining the optimal washing time, and improper time setting also affects the cleaning effect. The dish washer often cannot autonomously determine the characteristics of the cleaning object and whether it needs to be cleaned, increasing the operation complexity and possibly wasting resources. That is, in the related art, the cleaning strategy provided by the dish washer to the user is usually fixed when the dish washer is shipped from the factory, which cannot fully adapt to the cleaning object (such as kitchen utensils, fruits and vegetables, etc.) that the user actually needs to clean, resulting in poor cleaning effect on the cleaning object.

[0003] In the related art, the cleaning strategy provided by the dish washer to the user is usually fixed when the dish washer is shipped from the factory, which cannot fully adapt to the cleaning object that the user actually needs to clean, resulting in poor cleaning effect on the cleaning object. An effective solution has not yet been proposed.

[0004] Therefore, it is necessary to improve the related art to overcome the defects in the related art. SUMMARY

[0005] The embodiments of the present application provide a dish washer cleaning strategy determination method and device based on image recognition, at least to solve the problem that the cleaning strategy provided by the dish washer to the user in the related art is usually fixed when the dish washer is shipped from the factory, which cannot fully adapt to the cleaning object that the user actually needs to clean, resulting in poor cleaning effect on the cleaning object.

[0006] According to one aspect of the embodiments of this application, a dishwasher cleaning strategy determination method based on image recognition is provided, comprising: determining image recognition results of the interior of the dishwasher; wherein, the image recognition results include: attribute information of cleaning objects present in the dishwasher; the attribute information is at least used to indicate the function, material, quantity, and orientation of the cleaning objects; determining a washing mode for the cleaning objects based on the function and material of the cleaning objects, and determining the washing position and washing time of a cleaning device in the dishwasher for executing the washing mode based on the quantity and orientation of the cleaning objects; and determining a cleaning strategy for the cleaning objects based on the washing mode, the washing position, and the washing time.

[0007] In an exemplary embodiment, before determining the washing mode for the object to be cleaned based on its function and material, the method further includes: if the image recognition result also includes the size of the object to be cleaned, determining whether there is an unwashable object in the object to be cleaned based on the size, the function, and the material; if the unwashable object is present in the object to be cleaned, determining that the object to be cleaned is unwashable and instructing a target object to remove the unwashable object; if the unwashable object is not present in the object to be cleaned, determining that the object to be cleaned is washable, and allowing the determination of the washing mode for the object to be cleaned based on its function and material.

[0008] In an exemplary embodiment, determining the washing mode for the object to be cleaned based on its function and material includes: mapping multiple materials corresponding to the object to be cleaned to a target material through a first mapping relationship, wherein the target material is allowed to be used as the bottom-line material for cleaning the object; inputting the function and the target material into a second mapping relationship to obtain an output result; if the output result indicates that a special washing mode is required, determining the washing mode for the object to be cleaned as the special washing mode; if the output result is null, determining the washing mode for the object to be cleaned as the mode allowed for the target material in the daily washing mode.

[0009] In an exemplary embodiment, determining the washing position of the washing device for performing the washing mode within the dishwasher based on the number and orientation of the objects to be washed includes: determining a first number of objects to be washed facing a first direction and a second number of objects to be washed facing a second direction; determining a third number of objects to be washed in each of the multiple washing areas divided by the washing basket in the dishwasher; determining the operating direction of the washing device based on the first and second numbers, and determining the operating position of the washing device based on the third number; and determining the combination of the operating direction and the operating position as the washing position.

[0010] In an exemplary embodiment, determining the operating direction of the cleaning device using the first quantity and the second quantity includes: determining a first sum of the first quantity and the second quantity, and determining a first difference between the first quantity and the second quantity; if a second difference between the first sum and the quantity is less than a first threshold, determining a magnitude relationship between the first difference and the second threshold; and determining the operating direction using the magnitude relationship; wherein the operating direction is the operating direction of the cleaning device relative to the image acquisition device and the cleaning basket inside the dishwasher.

[0011] In an exemplary embodiment, determining the operating position of the cleaning device using the third quantity includes: when the difference between the second sum and the quantity is less than a first threshold, determining the operating position of the cleaning device using the third quantity includes one of the following: determining the target area corresponding to the largest third quantity, and obtaining the center position of the target area; determining the operating position as the center position of the area, or the position of the center of the cleaning basket offset from the center position of the area by a preset distance; wherein, the second sum is the sum of multiple third quantities; determining the regional order of the multiple cleaning areas according to the size order of the multiple third quantities; and determining the regional center positions corresponding to the multiple cleaning areas respectively as the operating positions according to the regional order.

[0012] In an exemplary embodiment, determining a cleaning strategy for the object to be cleaned based on the washing mode, the washing position, and the washing time includes: displaying the washing mode, the washing position, and the washing time to the target object; receiving the processing result of the target object on the washing mode, the washing position, and the washing time; and, if the processing result is within the allowable execution range of the dishwasher, determining a cleaning strategy for the object to be cleaned based on the processing result.

[0013] According to another aspect of the embodiments of this application, a dishwasher cleaning strategy determination device based on image recognition is also provided, comprising: a first determining module, configured to determine image recognition results of the interior of the dishwasher; wherein the image recognition results include: attribute information of cleaning objects present in the dishwasher; the attribute information is at least used to indicate the function, material, quantity, and orientation of the cleaning objects; a second determining module, configured to determine a washing mode for the cleaning objects based on the function and material of the cleaning objects, and to determine the washing position and washing time of a cleaning device in the dishwasher used to execute the washing mode based on the quantity and orientation of the cleaning objects; and a third determining module, configured to determine a cleaning strategy for the cleaning objects based on the washing mode, the washing position, and the washing time.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described image recognition-based dishwasher cleaning strategy determination method at runtime.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described image recognition-based dishwasher cleaning strategy determination method through the computer program.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described image recognition-based dishwasher cleaning strategy determination method.

[0017] This application determines the image recognition results inside a dishwasher. The image recognition results include: attribute information of the objects to be cleaned inside the dishwasher; the attribute information at least indicates the function, material, quantity, and orientation of the objects to be cleaned; a washing mode for the objects to be cleaned is determined based on their function and material; and the washing position and washing time of the cleaning device inside the dishwasher for executing the washing mode are determined based on the quantity and orientation of the objects to be cleaned. A cleaning strategy for the objects to be cleaned is determined based on the washing mode, the washing position, and the washing time. In other words, after performing image recognition on the inside of the dishwasher to obtain the attribute information of the objects to be cleaned, the washing mode, the washing position of the dishwasher's cleaning device, and the washing time are further determined based on the function, material, quantity, and orientation indicated by the attribute information, ultimately obtaining the dishwasher's cleaning strategy for the objects to be cleaned. Therefore, this technical solution solves the problem in related technologies where the cleaning strategies provided by dishwashers to users are often fixed at the factory and cannot be fully applied to the objects that users actually need to clean, resulting in poor cleaning effects; thus improving the cleaning effect of the objects to be cleaned by the dishwasher. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the hardware environment for an optional image recognition-based dishwasher cleaning strategy determination method according to an embodiment of this application.

[0021] Figure 2 This is a flowchart of an optional image recognition-based dishwasher cleaning strategy determination method according to an embodiment of this application;

[0022] Figure 3 This is another flowchart of an optional image recognition-based dishwasher cleaning strategy determination method according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the spray arm position according to an embodiment of this application;

[0024] Figure 5 This is a structural block diagram of an optional image recognition-based dishwasher cleaning strategy determination device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] According to one aspect of the embodiments of this application, a dishwasher cleaning strategy determination method based on image recognition is provided. This image recognition-based dishwasher cleaning strategy determination method is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligencehouse ecosystems. Optionally, in this embodiment, the above-mentioned image recognition-based dishwasher cleaning strategy determination method can be applied to, for example... Figure 1 The hardware environment shown consists of multiple terminal devices 102 and a server 104. For example... Figure 1 As shown, server 104 is connected to multiple terminal devices 102 via a network and can be used to provide services (such as application services) to terminals or clients installed on terminals. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0028] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0029] This embodiment provides a dishwasher cleaning strategy determination method based on image recognition, including but not limited to applications in dishwashers. Figure 2 This is a flowchart of a dishwasher cleaning strategy determination method based on image recognition according to an embodiment of this application. The process includes the following steps:

[0030] Step S202: Determine the image recognition result of the inside of the dishwasher; wherein the image recognition result includes: attribute information of the objects to be cleaned inside the dishwasher; the attribute information is at least used to indicate the function, material, quantity and orientation of the objects to be cleaned;

[0031] The image recognition results for the inside of the dishwasher can be determined by acquiring images of the cleaning basket and performing image recognition after a pressure sensor or similar device detects that an item has been placed inside the dishwasher. The image acquisition device can be deployed on the upper or lower inner wall of the dishwasher, specifically at the intersection of a plumb line passing through the center of the cleaning basket and the upper or lower inner wall. One or more cleaning baskets may be used.

[0032] The function of the object being cleaned can be considered its purpose. When the object is fruits and vegetables, the function is consumption. When the object is kitchenware, kitchenware can be categorized based on its function of assisting in eating (tableware) or its function of cooking food (cooking utensils). Tableware can be further divided into eating tableware and serving tableware based on its function of eating and serving. Eating tableware can include chopsticks, knives, and forks, while serving tableware can include bowls and plates. Similarly, cooking utensils can be categorized based on their function of cooking and baking (cooking utensils can include pots and pans, while baking utensils can include molds and baking trays).

[0033] The material can be wood, plastic, metal, ceramic, glass, etc. The orientation can be: the side of the object being cleaned that is in contact with food should face upwards, and the side of the object not in contact with food should face downwards.

[0034] Step S204: Determine the washing mode for the object to be washed based on its function and material, and determine the washing position and washing time of the washing device in the dishwasher for executing the washing mode based on the number and orientation of the object to be washed.

[0035] Optional washing modes include a regular wash and a special wash. Regular wash mode: This mode is suitable for most washable kitchen utensils. Special wash mode: For some washable kitchen utensils, special washing methods are required. For example, the crystal wash mode (plus recommended descaling detergent) is best suited for glassware such as wine glasses. This mode can include a wine glass function and can be used for glassware. The dishwasher's cleaning devices can include spray arms, spray nozzles, etc.

[0036] Step S206: Determine the cleaning strategy for the object to be cleaned based on the washing mode, the washing location, and the washing time.

[0037] Through the above steps, the image recognition results inside the dishwasher are determined. These results include: attribute information of the objects to be cleaned inside the dishwasher; the attribute information at least indicates the function, material, quantity, and orientation of the objects; the washing mode for the objects is determined based on their function and material; and the washing position and washing time of the cleaning device inside the dishwasher for executing the washing mode are determined based on the quantity and orientation of the objects; finally, a cleaning strategy for the objects is determined based on the washing mode, the washing position, and the washing time. In other words, after performing image recognition on the inside of the dishwasher to obtain the attribute information of the objects to be cleaned, the washing mode, the washing position of the dishwasher's cleaning device, and the washing time are further determined based on the function, material, quantity, and orientation indicated by the attribute information, ultimately resulting in the dishwasher's cleaning strategy for the objects. Therefore, this technical solution solves the problem in related technologies where the cleaning strategies provided by dishwashers to users are often fixed at the factory and cannot be fully applied to the objects that users actually need to clean, leading to poor cleaning results; thus, it improves the cleaning effect of the objects to be cleaned by the dishwasher.

[0038] In an exemplary embodiment, before determining the washing mode for the object to be cleaned based on its function and material, the method further includes: if the image recognition result also includes the size of the object to be cleaned, determining whether there is an unwashable object in the object to be cleaned based on the size, the function, and the material; if the unwashable object is present in the object to be cleaned, determining that the object to be cleaned is unwashable and instructing a target object to remove the unwashable object; if the unwashable object is not present in the object to be cleaned, determining that the object to be cleaned is washable, and allowing the determination of the washing mode for the object to be cleaned based on its function and material.

[0039] Optionally, the unwashable objects may further include: a first unwashable object and a second unwashable object.

[0040] The first unwashable object can be any object whose size or weight exceeds the capacity of the dishwasher's cleaning basket. For example, cookware may exceed the basket's capacity in both size and weight. The second unwashable object can be materials that are not resistant to high temperatures and pressure, such as plastic. Identifying the size of the target object involves using a target detection algorithm to accurately select each cleaning object and obtain its size data. This size data is then compared with the dishwasher's physical size limitations. If the size data exceeds the cleaning basket's physical size limitations, the cleaning object is designated as the first unwashable object, and the user is displayed "XX cleaning object needs to be removed." Physical size limitations include the cleaning basket's width and length limits, and the dishwasher's height limit. Optionally, if a single cleaning object's size data is not detected to exceed the physical size limit, but the combined size of multiple cleaning objects exceeds the physical size limit, the presence of a first unwashable object is also output, and the user is displayed "Some cleaning objects need to be removed."

[0041] Through this embodiment, the dishwasher can automatically select suitable items for cleaning and exclude those that may cause negative consequences, thereby greatly improving washing efficiency and safety. Furthermore, accurate category recognition provides a basis for intelligent selection of washing modes, enabling the dishwasher to adjust water flow intensity, temperature, and washing time according to actual conditions to achieve optimal cleaning results.

[0042] If it is determined that all items in the washing basket are washable, the washing mode for the washing items is determined based on their function and material. This includes: mapping multiple materials corresponding to the washing items to a target material through a first mapping relationship, wherein the target material is allowed as the baseline material for washing the washing items; inputting the function and the target material into a second mapping relationship to obtain an output result; if the output result indicates that a special washing mode is required, the washing mode for the washing items is determined to be the special washing mode; if the output result is null, the washing mode for the washing items is determined to be the mode allowed for the target material in the daily washing mode.

[0043] Understandably, dishwashers use image analysis to identify the various materials of the objects being cleaned and map these materials to a target material. The selection of the target material is based on the principle that it must be a material that ensures the object will not be damaged during the cleaning process; this is the user's basic requirement. For example, if both object A and object B are present in the cleaning basket, and object A can withstand higher temperatures and pressures than object B, then object B can be used as the baseline material (i.e., the target material). Optionally, the first mapping relationship can be: f({material} i＝1 N = f(material1) & ... & f(material2) N= target material.

[0044] Next, the function definition (such as serving, cooking, baking, etc.) and the selected target material are input into the second mapping relationship. This mapping relationship is used to determine the most suitable washing mode based on the function and material. The second mapping relationship can be: f(function, material) = True and f(function) or f(material) = output result.

[0045] If the second mapping outputs a special wash mode, it means that for a given combination of function and target material, there exists a specially designed wash mode that optimizes cleaning results; for example, the "Crystal Wash" mode is more suitable for glass wine glasses. In this case, the dishwasher will automatically select this special wash mode. If the second mapping outputs a null value, it means there is no special wash mode specific to the combination of function and target material. In this case, the dishwasher will select the wash mode from the daily wash modes that is suitable for the target material. The daily wash mode is a general mode that can provide good cleaning results in most cases, but it is not optimized for specific materials or functions.

[0046] Through this automated process based on image processing and material mapping, the dishwasher can intelligently adapt to different cleaning needs, ensuring cleaning effectiveness while avoiding damage to unsuitable materials, greatly improving the user experience and the applicability of the equipment. Furthermore, this method can flexibly handle the diversity of objects placed for cleaning by the user, reducing operational complexity.

[0047] While determining the selectable mode, the washing position of the washing device for executing the washing mode in the dishwasher is determined according to the number and orientation of the objects to be washed, including: determining a first number of objects to be washed facing a first direction and a second number of objects to be washed facing a second direction; determining a third number of objects to be washed in each of the multiple washing areas divided in the washing basket of the dishwasher; determining the operating direction of the washing device by the first number and the second number, and determining the operating position of the washing device by the third number; and determining the combination of the operating direction and the operating position as the washing position.

[0048] This application deeply integrates visual AI technology with the dishwasher's automated control strategy to ensure optimal cleaning efficiency and results. Determining the washing position includes: First, the dishwasher's camera captures images of the washing basket, and image recognition technology determines the orientation of the objects to be washed (i.e., kitchenware). Orientation is divided into two directions: upward and downward, corresponding to a first quantity and a second quantity, respectively. Simultaneously, the system identifies the number of objects to be washed in multiple washing zones divided within the washing basket, i.e., a third quantity. The operating direction of the washing device is determined by the first and second quantities corresponding to the orientation, and the operating position of the washing device is determined by the third quantity.

[0049] Specifically, such as Figure 3 As shown, determining the operating direction of the cleaning device using the first quantity and the second quantity includes:

[0050] Step S302: Determine the first sum of the first quantity and the second quantity, and determine the first difference between the first quantity and the second quantity;

[0051] Step S304: If the second difference between the first sum and the quantity is less than the first threshold, determine the size relationship between the first difference and the second threshold;

[0052] Step S306: Determine the running direction based on the size relationship; wherein, the running direction is the running direction of the cleaning device relative to the image acquisition device and the cleaning basket inside the dishwasher.

[0053] The sum of the first and second quantities should theoretically be the same as the number of objects detected in the image from the washing basket. Therefore, this embodiment first verifies the second difference between the sum of the first and second quantities and the stated number of objects to be washed. If this second difference is small, i.e., less than the first threshold, it can be confirmed that the items in the washing basket have been correctly categorized and quantified by the dishwasher's AI vision function. Otherwise, it means that an image recognition error has occurred, or that some items have not been correctly identified. The system will automatically re-capture and analyze the image until the accuracy requirements are met, or prompt the user to check and reposition the items, ensuring that each washing process starts accurately, thereby achieving the best cleaning effect and user experience. The first threshold can be a small integer value, such as 3.

[0054] The side of the baking tray or other object being cleaned that is in contact with food is considered upwards when facing the food collection device, and the side that is not in contact with food is considered downwards when facing the device. If the first difference is greater than the second threshold, it indicates that there are more upward-facing objects being cleaned, and the cleaning device's operating direction is determined as follows: a spray arm extending between the food collection device and the cleaning basket sprays water towards the cleaning basket. If the first difference is less than the second threshold, it indicates that there are more downward-facing objects being cleaned, and the cleaning device's operating direction is determined as follows: a spray arm extending between the cleaning basket and the inner wall of the dishwasher without a food collection device is sprayed water towards the cleaning basket. If the first difference equals the second threshold, it indicates that the number of upward-facing and downward-facing objects is balanced, and the cleaning device's operating direction is determined as follows: a spray arm extending simultaneously between the food collection device and the cleaning basket, and a spray arm extending between the cleaning basket and the inner wall of the dishwasher without a food collection device, controlling the spray arm to spray water towards the cleaning basket.

[0055] Specifically, determining the operating position of the cleaning device using the third quantity includes: when the difference between the second sum and the quantity is less than a first threshold, determining the operating position of the cleaning device using the third quantity includes one of the following: determining the target area corresponding to the largest third quantity, and obtaining the center position of the target area; determining the operating position as the center position of the area, or the position of the center of the cleaning basket offset from the center position of the area by a preset distance; wherein, the second sum is the sum of multiple third quantities; determining the regional order of the multiple cleaning areas according to the size order of the multiple third quantities; and determining the regional center positions corresponding to the multiple cleaning areas respectively as the operating positions according to the regional order.

[0056] The difference between the sum of multiple third quantities and the number of objects to be cleaned is compared with the first threshold. The purpose of this comparison is the same as that of comparing the sum of the first and second quantities and the difference between the number of objects to be cleaned with the first threshold. Both are to verify whether the dishwasher's AI vision function is working properly and whether the items need to be repositioned.

[0057] Through the above embodiments, regarding determining the operating position of the cleaning device, it can be understood that the spray arm can be shifted from the center of the cleaning basket towards the center of the target area containing the largest number of cleaning objects before spraying. Alternatively, the spray arm can be moved sequentially between the centers of multiple cleaning areas according to the order of the number of cleaning objects included in the multiple cleaning areas from highest to lowest. Furthermore, the spray arm can be shifted sequentially from the center of the cleaning basket towards multiple cleaning areas according to the order of the number of cleaning objects included in the multiple cleaning areas from highest to lowest.

[0058] Optionally, the washing time is determined by the number and orientation of the items to be washed in the washing basket. The washing time for each orientation is determined based on the first and second quantities corresponding to different orientations. Specifically, the time corresponding to the range of quantities covered by the first and second quantities is defined as the washing time for the first and second quantities, respectively.

[0059] In an exemplary embodiment, determining a cleaning strategy for the object to be cleaned based on the washing mode, the washing position, and the washing time includes: displaying the washing mode, the washing position, and the washing time to the target object; receiving the processing result of the target object on the washing mode, the washing position, and the washing time; and, if the processing result is within the allowable execution range of the dishwasher, determining a cleaning strategy for the object to be cleaned based on the processing result.

[0060] Regarding the processing results received from the target object concerning the washing mode, washing position, and washing time corresponding to the washing position, it is understood that after displaying the washing mode, washing position, and washing time to the target object, the target object can select the washing mode, adjust the washing time, and add special detergent for a specific washing mode according to the displayed information within the allowed adjustment range. After receiving the processing results from the target object regarding mode and time adjustments and detergent additions, if the dishwasher determines that the processing is indeed within the allowed execution range, it executes the cleaning strategy corresponding to the processing result, i.e., starts washing according to the adjusted mode and time or using special detergent. Optionally, if the target object does not perform any processing action, i.e., the dishwasher does not receive the target object's processing result, the dishwasher automatically executes the cleaning strategy, which includes the washing mode, washing position, and washing time.

[0061] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the above-described method for determining dishwasher cleaning strategies based on image recognition, the process is described below with reference to embodiments, but this is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0062] This application embodiment utilizes visual AI capabilities to automatically select the optimal washing mode and washing time based on the current type and quantity of kitchen utensils, and provides users with detergent recommendations, thus maximizing the resolution of user pain points. Specifically, this application embodiment uses image recognition to identify the category of items (this category is a broad concept and must be combined with the core objective of the dishwasher) and its strong correlation with the subsequent washing mode. For a dishwasher, the most important thing is to identify the category, quantity, and positional relationship of the tableware, thereby controlling the washing position, time, and washing mode. These will be described in detail below.

[0063] Regarding the categories in this application's embodiments, they are defined as follows: Kitchen utensil categories are specifically divided into three attributes, and the total number of categories is the product of the quantity of each of these three attributes. Of course, not all combinations exist; the combination categories defined in this scheme are subsets. For example:

[0064] The functional definitions are shown in Table 1, and include at least 8 types; the material definitions include at least 3 types: wood, plastic, and metal; the orientation definitions include at least 2 types, facing up or down relative to the image acquisition device. Therefore, there are a total of 8*3*2=48 categories. However, for the baking pan, there is only one material, metal, so there will be 1*2*2=4 fewer categories. Therefore, after removing these, there are a total of 48-4=44 categories.

[0065] Table 1 Functional Definitions

[0066]

[0067]

[0068] Furthermore, the optional washing modes of the dishwasher in this application embodiment are defined as follows:

[0069] 1) Washable modes, including daily wash mode and special wash mode.

[0070] The "Daily Wash" mode is suitable for most washable kitchen utensils.

[0071] Among them, special washing modes: For washable kitchen utensils, some categories require special washing methods. For example, the best choice is the crystal clean mode (plus the recommended descaling detergent). The function definition includes wine glasses, and the material definition includes glass.

[0072] 2) Non-washable modes, including non-washable modes based on size and non-washable modes based on material.

[0073] Among them, the size-unwashable mode: Due to the physical size limitations of the dishwasher basket itself, even if it can be washed, not all washable kitchen utensils are suitable to be put in the dishwasher. For example, pots are large and heavy, making it difficult to put them in the basket. In addition, the weight can easily damage the basket, or even if it is not damaged immediately, it can easily cause hidden damage to the basket, reducing its lifespan.

[0074] Among them, the non-washable material mode: because the dishwasher sprays hot, high-pressure water from the spray arm during the washing process, it can easily damage or deform materials that are not hard enough, such as plastic materials.

[0075] The mapping relationship between the categories and washing modes in this application embodiment is as follows: Assuming that the washable mode is True and the non-washable mode is False, f: the functional relationship between attribute and whether it is washable mode, then by means of the definition of mathematical logic relationship: f(function, material) = f(function) & f(material).

[0076] Additionally, if f(function, material) = True and f(function) or f(material) = special wash mode, then in addition to indicating that it is washable, it should also include additional information about the special wash mode (including the special wash mode and recommended detergent, etc.).

[0077] The mapping relationship between image categories and categories in this application embodiment is as follows: According to the category definition above, there is an implicit condition, namely, a single kitchen utensil can only contain the same material. This is insufficient in the real environment, meaning that there are often cases where the same utensil contains multiple materials. For example, some chopsticks have a wooden lower half and a metal upper half. The perspective adopted in this application embodiment is related to the user's bottom-line goal. From the user's perspective, the bottom line that a dishwasher cannot break is that it cannot damage the user's kitchen utensils, because this means that the dishwasher not only does not create value, but also causes a loss of value. Therefore, for the definition of multiple materials, the following definition method is given: f({material} i＝1 N = f(material1) & ... & f(material2) N ).

[0078] Furthermore, in this embodiment, the dishwasher cleans using water jets sprayed from the spray arm. Given a fixed position for the kitchen appliances, the relative positional relationship between the spray arm and the washing basket is defined along two axes: horizontal axis (left-right) and vertical axis (up-down). Therefore, there are a total of four possible positional relationships: Washing position definition:

[0079] Horizontal axis (left and right) definition: Before leaving the factory, the position of the camera is adjusted so that its field of view covers the washing basket, and then the image can be divided into two equal left and right parts.

[0080] Vertical axis (upper and lower): Based on the length of the spray arm that should extend, two types of spray arms are defined: upper spray arm and lower spray arm, such as... Figure 4 As shown.

[0081] Washing time is defined as follows: each 5-minute interval is a maximum of 15 minutes, i.e., [5, 10, 15].

[0082] Orientation definition: "Upward" means that the inside of the kitchen utensil faces the camera, and "downward" means that the outside of the utensil faces the camera. For example, for a bowl, "upward" means that the rim of the bowl faces the camera, and "downward" means that the bottom of the bowl faces the camera.

[0083] Relationship between orientation and vertical axis (top and bottom): For users, the cleanliness of the interior of tableware, especially bowls, plates, and dishes, is more important than the exterior. This is because the interior is where food is ultimately placed and enters the mouth, and food residue and grease are mainly found inside. Therefore, the spray arm needs to primarily clean the interior. The orientation of kitchen utensils can be identified through images, and based on the orientation of all categories, N of the identified kitchen utensils can be determined. 朝上 and -N 朝下 These represent the total number of items facing upwards and downwards in the washing basket, respectively. Given a threshold θ, three patterns of the washing basket are derived:

[0084] Cleaning basket facing upwards: N 朝上 -N 朝下 >θ (Extend the upper spray arm).

[0085] Cleaning basket facing down: N 朝下 -N 朝上 >θ (Extend the lower spray arm).

[0086] The cleaning basket has two sections: |N 朝下 -N 朝上 |<=θ (extend the upper and lower spray arms).

[0087] The relationship between the number of items and the horizontal axis (left and right): Based on the category recognition of image object detection, all utensils and their detection bounding boxes on the image can be identified. For a bounding box that spans both the left and right sides, the criterion is that the left and right sides occupy the larger pixel area, thus obtaining the number of categories detected on each side. Similar to the vertical axis pattern, three patterns for the horizontal axis can be derived (η is the threshold):

[0088] Cleaning basket left: N 左 -N 右 >η (Spray arm extends to the left center).

[0089] Cleaning basket right: N 右 -N 左 >η (Spray arm extends to the right center).

[0090] There are cleaning baskets on both sides: |N 朝下 -N 朝上 |<=η(The spray arms will extend to the left and right centers respectively).

[0091] Relationship between quantity and washing time:

[0092] 0→0 (If there are 0, do not clean).

[0093] [1,3]→5 (1-3, wash for 5 minutes).

[0094] [4,6]→10 (4-6 pieces, wash for 10 minutes).

[0095] [7,+∞)→15 (6 or more, clean for 10 minutes).

[0096] In summary, the overall washing process of this application embodiment includes:

[0097] 1) The user puts the kitchenware they want to wash into the dishwasher basket.

[0098] 2) After the dishwasher starts, it first detects all categories of kitchen utensils through image target detection. If a non-washable mode is detected, it will pause and issue an alarm (informing the user which utensil cannot be washed and needs to be removed). It will continue to run only after the user removes the utensil, presses the continue button, and the image detects that there are no non-washable modes.

[0099] 3) Plan which spray arm should extend based on the relationship between the orientation and the longitudinal axis.

[0100] 4) Based on the relationship between the quantity and the horizontal axis, plan which side the spray arm should extend to. If it should extend to both sides, further determine which side has a higher priority (the side with more quantity has a higher priority; if they are of the same level, then the left side first, then the right side).

[0101] 5) Based on the relationship between quantity and washing time, plan the washing time for the corresponding area.

[0102] 6) Based on the planning results from 3) to 5), issue the program to the dishwasher for execution.

[0103] Therefore, the embodiments of this application realize the automated washing process of the dishwasher. In this process, visual AI capabilities are also used to automatically select the optimal washing mode and washing time based on the current type and quantity of kitchen utensils, and to provide the user with suggestions on the detergent to be added, so as to solve the user's pain points to the greatest extent.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0105] This embodiment also provides a dishwasher cleaning strategy determination device based on image recognition, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0106] Figure 5 This is a structural block diagram of an optional image recognition-based dishwasher cleaning strategy determination device according to an embodiment of this application. The device includes:

[0107] The first determining module 52 is used to determine the image recognition result of the inside of the dishwasher; wherein, the image recognition result includes: attribute information of the objects to be cleaned existing inside the dishwasher; the attribute information is used to indicate at least the function, material, quantity and orientation of the objects to be cleaned;

[0108] The second determining module 54 is used to determine the washing mode for the object to be washed based on the function and material of the object to be washed, and to determine the washing position and washing time of the washing device in the dishwasher for performing the washing mode based on the number and orientation of the object to be washed.

[0109] The third determining module 56 is used to determine a cleaning strategy for the object to be cleaned based on the washing mode, the washing position, and the washing time.

[0110] The aforementioned device determines the image recognition results of the dishwasher's interior. These results include: attribute information of the objects to be cleaned within the dishwasher; the attribute information at least indicates the function, material, quantity, and orientation of the objects; a washing mode is determined based on the function and material of the objects; and the washing position and washing time of the cleaning device within the dishwasher for executing the washing mode are determined based on the quantity and orientation of the objects. Finally, a cleaning strategy is determined based on the washing mode, the washing position, and the washing time. In other words, after performing image recognition on the dishwasher's interior to obtain the attribute information of the objects to be cleaned, the washing mode, the washing position of the dishwasher's cleaning device, and the washing time are further determined based on the function, material, quantity, and orientation indicated by the attribute information, ultimately resulting in the dishwasher's cleaning strategy for the objects. Therefore, this technical solution solves the problem in related technologies where the cleaning strategies provided by dishwashers to users are often fixed at the factory and cannot be fully applied to the objects that users actually need to clean, leading to poor cleaning results; thus, it improves the cleaning effect of the dishwasher's cleaning tools.

[0111] In an exemplary embodiment, the device further includes a fourth determining module, configured to determine whether there is an unwashable object in the cleaning object by means of the size, the function, and the material when the image recognition result also includes the size of the cleaning object; if the unwashable object exists in the cleaning object, determine that the cleaning object is unwashable and instruct the target object to remove the unwashable object; if the unwashable object does not exist in the cleaning object, determine that the cleaning object is washable and allow the determination of the washing mode for the cleaning object based on the function and material of the cleaning object.

[0112] In an exemplary embodiment, the second determining module 54 is further configured to map multiple materials corresponding to the cleaning object to a target material through a first mapping relationship, wherein the target material is allowed to be used as the bottom line material for cleaning the cleaning object; input the function and the target material into the second mapping relationship to obtain an output result; if the output result indicates that a special washing mode is required, determine the washing mode for the cleaning object as the special washing mode; if the output result is null, determine the washing mode for the cleaning object as the mode allowed by the target material in the daily washing mode.

[0113] In an exemplary embodiment, the second determining module 54 is further configured to determine a first number of objects to be cleaned facing a first direction and a second number of objects to be cleaned facing a second direction; determine a third number of objects to be cleaned in the multiple cleaning areas divided into the cleaning basket in the dishwasher; determine the operating direction of the cleaning device by the first number and the second number, and determine the operating position of the cleaning device by the third number; and determine the combination of the operating direction and the operating position as the washing position.

[0114] In an exemplary embodiment, the second determining module 54 is further configured to determine a first sum of the first quantity and the second quantity, and determine a first difference between the first quantity and the second quantity; if the second difference between the first sum and the quantity is less than a first threshold, determine the magnitude relationship between the first difference and the second threshold; determine the running direction based on the magnitude relationship; wherein the running direction is the running direction of the cleaning device relative to the image acquisition device and the cleaning basket inside the dishwasher.

[0115] In an exemplary embodiment, the second determining module 54 is further configured to determine the operating position of the cleaning device by means of the third quantity when the difference between the second sum and the quantity is less than a first threshold, including one of the following: determining the target area corresponding to the largest third quantity, and obtaining the area center position of the target area; determining the area center position, or the position of the center position of the cleaning basket offset from the area center position by a preset distance, as the operating position; wherein, the second sum is the sum of multiple third quantities; determining the area order of the multiple cleaning areas by the size order of the multiple third quantities; and determining the area center positions corresponding to the multiple cleaning areas respectively as the operating positions according to the area order.

[0116] In an exemplary embodiment, the third determining module 56 is further configured to display the washing mode, the washing position, and the washing time to the target object; receive the processing result of the target object on the washing mode, the washing position, and the washing time; and, if the processing result is within the allowable execution range of the dishwasher, determine a cleaning strategy for the object to be cleaned based on the processing result.

[0117] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0118] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0119] S1, determine the image recognition result of the inside of the dishwasher; wherein, the image recognition result includes: attribute information of the objects to be cleaned inside the dishwasher; the attribute information is at least used to indicate the function, material, quantity and orientation of the objects to be cleaned;

[0120] S2, determine the washing mode for the object to be cleaned based on its function and material, and determine the washing position and washing time of the cleaning device in the dishwasher for executing the washing mode based on the number and orientation of the object to be cleaned.

[0121] S3, determine the cleaning strategy for the object to be cleaned based on the washing mode, the washing location, and the washing time.

[0122] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0123] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0124] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0125] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0126] S1, determine the image recognition result of the inside of the dishwasher; wherein, the image recognition result includes: attribute information of the objects to be cleaned inside the dishwasher; the attribute information is at least used to indicate the function, material, quantity and orientation of the objects to be cleaned;

[0127] S2, determine the washing mode for the object to be cleaned based on its function and material, and determine the washing position and washing time of the cleaning device in the dishwasher for executing the washing mode based on the number and orientation of the object to be cleaned.

[0128] S3, determine the cleaning strategy for the object to be cleaned based on the washing mode, the washing location, and the washing time.

[0129] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0130] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0131] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0132] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0133] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0134] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0135] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for determining a washing strategy of a dishwasher based on image recognition, characterized in that, The method comprises: determining an image recognition result of the inside of the dishwasher; wherein the image recognition result comprises attribute information of a cleaning object present in the dishwasher; the attribute information at least indicates the function, material, quantity and orientation of the cleaning object; determining a washing mode for the cleaning object according to the function and material of the cleaning object, and determining a washing position and washing time of a cleaning device in the dishwasher for executing the washing mode according to the quantity and orientation of the cleaning object; determining a cleaning strategy for the cleaning object according to the washing mode, the washing position and the washing time. 2.The image recognition-based determination method of a dish washer cleaning strategy according to claim 1, characterized in that, Before determining the washing mode for the cleaning object according to the function and material of the cleaning object, the method further comprises: in the case that the image recognition result further comprises the size of the cleaning object, determining whether there is an unwashable object in the cleaning object through the size, the function and the material; in the case that there is the unwashable object in the cleaning object, determining that the cleaning object is unwashable, and instructing a target object to remove the unwashable object; in the case that there is no unwashable object in the cleaning object, determining that the cleaning object is washable, and allowing the washing mode for the cleaning object to be determined according to the function and material of the cleaning object. 3.The image recognition-based determination method of a dish washer cleaning strategy according to claim 1, characterized in that, Determining the washing mode for the cleaning object according to the function and material of the cleaning object comprises: mapping a plurality of materials corresponding to the cleaning object into a target material through a first mapping relationship, wherein the target material is allowed to be used as the cleaning bottom line material of the cleaning object; inputting the function and the target material into a second mapping relationship to obtain an output result; in the case that the output result indicates that a special washing mode needs to be adopted, determining that the washing mode for the cleaning object is the special washing mode; in the case that the output result is a null value, determining that the washing mode for the cleaning object is a mode allowed to be used by the target material in a daily washing mode. 4.The image recognition-based determination method of a dish washer cleaning strategy according to claim 1, wherein, Determining the washing position of the cleaning device in the dishwasher for executing the washing mode according to the quantity and orientation of the cleaning object comprises: determining a first quantity of cleaning objects with a first orientation and a second quantity of cleaning objects with a second orientation; determining a third quantity of cleaning objects included in each of a plurality of cleaning areas divided in a cleaning basket in the dishwasher; determining a running direction of the cleaning device through the first quantity and the second quantity, and determining a running position of the cleaning device through the third quantity; determining a combination of the running direction and the running position as the washing position. 5.The image recognition-based determination method of a dish washer cleaning strategy according to claim 4, characterized in that, Determining the running direction of the cleaning device through the first quantity and the second quantity comprises: determining a first sum of the first quantity and the second quantity, and determining a first difference between the first quantity and the second quantity; in the case that the first sum and a second difference between the quantity are less than a first threshold value, determining a size relationship between the first difference and a second threshold value; The running direction is determined by the size relationship; wherein the running direction is a running direction of the cleaning device relative to an image collection device and the cleaning basket in the dishwasher. 6.The image recognition-based determination method of a dish washer cleaning strategy according to claim 4, characterized in that, The running position of the cleaning device is determined by the third quantity, including: In a case where a difference between the second sum and the quantity is less than a first threshold, the running position of the cleaning device is determined by the third quantity, including one of: A target region corresponding to a maximum third quantity is determined, and a region center position of the target region is obtained; a position of the region center position or a center position of the cleaning basket offset by a preset distance from the region center position is determined as the running position; wherein the second sum is a sum of the third quantities; A region order of the multiple cleaning regions is determined by a size order of the multiple third quantities; and region center positions respectively corresponding to the multiple cleaning regions are determined as the running position according to the region order. 7.The image recognition-based determination method of a dish washer cleaning strategy according to claim 1, wherein, A cleaning strategy for the cleaning object is determined according to the washing mode, the washing position, and the washing time, including: The washing mode, the washing position, and the washing time are displayed to a target object; A processing result of the target object on the washing mode, the washing position, and the washing time is received; In a case where the processing result is within an allowable execution range of the dishwasher, a cleaning strategy for the cleaning object is determined by the processing result. 8.A dishwasher cleaning strategy determination apparatus based on image recognition, characterized by, including: The first determining module is configured to determine an image recognition result of the inside of the dishwasher; wherein the image recognition result includes attribute information of a cleaning object existing in the dishwasher; and the attribute information is used at least to indicate a function, a material, a quantity, and an orientation of the cleaning object. The second determining module is configured to determine a washing mode for the cleaning object according to the function and the material of the cleaning object, and to determine a washing position and a washing time of a cleaning device for executing the washing mode in the dishwasher according to the quantity and the orientation of the cleaning object. The third determining module is configured to determine a cleaning strategy for the cleaning object according to the washing mode, the washing position, and the washing time.

9. A computer readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program executes the method of any one of claims 1 to 7 when running. 10.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 7 by using the computer program.

Images