Item assisted cleaning method and electronic device

Abstract

The embodiment of the application discloses an article auxiliary cleaning method and an electronic device, relates to the computer technology and the medical field, and includes the following steps: in response to collecting first image information of an auxiliary cleaning work area through a monitoring module, performing feature extraction on the first image information to obtain a target object in the first image information and a corresponding target feature vector; performing three-dimensional modeling processing on the first image information to obtain three-dimensional scene information corresponding to the first image information; and controlling a mechanical arm module to perform an auxiliary cleaning operation according to the target feature vector and the three-dimensional scene information. Compared with the mode in which the mechanical arm operates according to a preset action, the embodiment of the application can make the mechanical arm module adapt to the actual situation of the placement of tableware in the auxiliary cleaning work area, accurately determine the position, type and litter distribution of the target object and the like by collecting image information in real time and combining feature extraction and three-dimensional modeling, and improve the accuracy of the mechanical arm in performing the auxiliary cleaning operation on the tableware.

Description

Technical Field

[0001] This application relates to computer technology and the medical field, and in particular to an auxiliary cleaning method and electronic device for items. Background Technology

[0002] In the healthcare industry, the cleaning of items such as tableware after meals is a crucial part of daily care for inpatients in hospitals, sanatoriums, and other medical institutions, especially those recovering from surgery or with limited mobility. Medical institutions have a high demand for tableware cleaning, and using devices such as robotic arms to assist in tableware cleaning has become an important direction for the intelligent development of medical care.

[0003] In related technologies, robotic arms typically operate according to preset actions when performing auxiliary cleaning operations on tableware, which makes it difficult to adapt to the actual placement of tableware in medical work areas. As a result, robotic arms cannot accurately perform auxiliary cleaning operations on target tableware.

[0004] It is evident that current robotic arms have a problem with low precision when performing auxiliary cleaning operations on tableware. Summary of the Invention

[0005] In view of this, one of the objectives of this application is to provide an auxiliary cleaning method and electronic device for items, which can improve the accuracy of a robotic arm in performing auxiliary cleaning operations on items such as tableware.

[0006] To achieve the above objectives, the technical solution of this application is implemented as follows: In a first aspect, embodiments of this application provide an article-assisted cleaning method, applied to an article-assisted cleaning device, the article-assisted cleaning device including a monitoring module and a robotic arm module, the method comprising: In response to the first image information of the auxiliary cleaning operation area collected by the monitoring module, feature extraction is performed on the first image information to obtain the target object in the first image information and its corresponding target feature vector; The first image information is subjected to 3D modeling processing to obtain the 3D scene information corresponding to the first image information; Based on the target feature vector and 3D scene information, the robotic arm module is controlled to perform auxiliary cleaning operations.

[0007] In one possible implementation, the target object includes the target tableware and the debris attached to the target tableware, the three-dimensional scene information includes the tableware position information of the target tableware, and the robotic arm module includes a first robotic arm and a second robotic arm. Based on the target feature vector and 3D scene information, the robotic arm module is controlled to perform auxiliary dishwashing operations, including: Based on the tableware location information, control the first robotic arm to grab the target tableware and move it to the preset cleaning area; The second robotic arm is controlled to perform auxiliary cleaning operations in a preset cleaning area through a cleaning module. The second robotic arm includes a cleaning module.

[0008] In one possible implementation, the target feature vector includes the type of tableware and the size of the target tableware; Before controlling the first robotic arm to grasp the target tableware and move it to the preset cleaning area, based on the tableware's position information, the method also includes: Determine the target grasping parameters based on the type and size of the tableware; Based on the position information of the tableware, the first robotic arm is controlled to grasp the target tableware and move it to the preset cleaning area, including: Based on the location information of the tableware, the first robotic arm is controlled to grab the target tableware and move it to the preset cleaning area according to the grabbing parameters.

[0009] In one possible implementation, the item cleaning aid further includes a cleaning module, and the target feature vector includes the location information and size of the debris attached to the target tableware; Before controlling the second robotic arm to perform auxiliary cleaning operations in the preset cleaning area via the cleaning module, the method further includes: Determine auxiliary cleaning parameters based on the location and size of the debris; Controlling the second robotic arm to perform auxiliary cleaning operations within a preset cleaning area via a cleaning module includes: The second robotic arm is controlled to perform auxiliary cleaning operations in the preset cleaning area according to the auxiliary cleaning parameters through the cleaning module.

[0010] In one possible implementation, the cleaning module includes a spray head and a scraping brush, and the auxiliary cleaning parameters include cleaning pressure and scraping pressure; Controlling the second robotic arm to perform auxiliary cleaning operations within a preset cleaning area according to auxiliary cleaning parameters via the cleaning module includes: Control the second robotic arm to spray and rinse debris in a preset cleaning area using a spray head at cleaning pressure; and control the second robotic arm to scrape debris in the preset cleaning area using a scraper brush.

[0011] In one possible implementation, in a preset cleaning area, the target surface of the target tableware faces the cleaning module, and the target surface is the side of the target tableware that holds the food.

[0012] In one possible implementation, the item cleaning aid further includes a first communication module, which is used to establish a communication connection with the tableware cleaning equipment; Based on the target feature vector and 3D scene information, after controlling the robotic arm module to perform auxiliary cleaning operations, the method also includes: In response to receiving a first target signal through the first communication module, the control robot arm module transfers the target tableware to the tableware cleaning equipment for cleaning; wherein, the first target signal is used to indicate that the tableware cleaning equipment is in an idle state.

[0013] In one possible implementation, after the robotic arm module transfers the target tableware to the tableware cleaning equipment for cleaning in response to receiving the target signal via the first communication module, the method further includes: In response to receiving a second target signal through the first communication module, the robotic arm module is controlled to move to a preset cleaning area for self-cleaning; wherein, the second target signal is used to indicate whether the tableware cleaning equipment is in either the state of starting cleaning or completing cleaning.

[0014] In one possible implementation, the item-assisted cleaning device further includes a second communication module for communicating with floor cleaning equipment; In response to receiving the second target signal via the first communication module and controlling the robotic arm module to move to the preset cleaning area for self-cleaning, the method further includes: In response to receiving a third target signal via the second communication module, a ground cleaning command is generated; wherein the third target signal is used to indicate that the robotic arm module has completed self-cleaning. Send a floor cleaning instruction to the floor cleaning equipment. The floor cleaning instruction is used to instruct the floor cleaning equipment to clean the floor in the auxiliary cleaning work area.

[0015] Secondly, embodiments of this application provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it implements the method provided in the first aspect.

[0016] The item-assisted cleaning method provided in this application is applied to an item-assisted cleaning device, which includes a monitoring module and a robotic arm module. In response to first image information of the assisted cleaning work area acquired by the monitoring module, feature extraction is performed on the first image information to obtain the target object and its corresponding target feature vector. Subsequently, three-dimensional modeling processing is performed on the first image information to obtain three-dimensional scene information corresponding to the first image information. Finally, the robotic arm module is controlled to perform assisted cleaning operations based on the target feature vector and the three-dimensional scene information. Compared to the robotic arm operating according to preset actions in related technologies, this application embodiment, by acquiring image information in real time and combining feature extraction and three-dimensional modeling, enables the robotic arm module to adapt to the actual situation of tableware placement in the assisted cleaning work area, accurately determine the position, type, and debris distribution of the target object, and thus accurately control the robotic arm module to perform assisted cleaning operations on the target object, thereby improving the accuracy of the robotic arm in performing assisted cleaning operations on tableware. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. It should be understood that the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] The attached diagram is described below: Figure 1 This is a schematic diagram illustrating the application environment of an auxiliary cleaning method for articles provided in this application embodiment; Figure 2 A flowchart illustrating an auxiliary cleaning method for articles provided in this application embodiment; Figure 3 This is a schematic diagram of the functional modules of a tableware auxiliary cleaning system provided in an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0019] Explanation of reference numerals in the attached figures: 300. Dishware auxiliary cleaning system; 310. First processing module; 320. Second processing module; 330. Control module; 401. Processor; 402. Memory; 403. Communication interface; 410. Bus. Detailed Implementation

[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0022] To facilitate a better understanding of the solutions in the embodiments of this application, the relevant technologies will be introduced first below.

[0023] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0024] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0025] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0026] To address the technical problems in the background art, embodiments of this application provide an article-assisted cleaning method, a tableware-assisted cleaning system, an electronic device, a computer-readable storage medium, and a computer program product.

[0027] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the application environment of an article-assisted cleaning method provided in an embodiment of this application. The article-assisted cleaning method provided in this embodiment can be applied to, for example... Figure 1 The application environment shown. In Figure 1 In this device, electronic device A can communicate with item cleaning aid B (including wireless and / or wired communication).

[0028] In some embodiments, the electronic device may include a personal computer, a server, a mobile device, a cloud computing platform, and a supercomputer, etc.

[0029] The article continues to describe the article-assisted cleaning method provided in the embodiments of this application.

[0030] Please see Figure 2 , Figure 2 This is a flowchart illustrating an auxiliary cleaning method for items provided in an embodiment of this application, such as... Figure 2 The method shown can be applied to electronic devices. The following will describe its application to electronic devices. It can be applied to an item-assisted cleaning device, which includes a monitoring module and a robotic arm module. The item-assisted cleaning method specifically includes the following steps 110 to 130.

[0031] Step 110: In response to the first image information of the auxiliary cleaning operation area collected by the monitoring module, feature extraction is performed on the first image information to obtain the target object in the first image information and its corresponding target feature vector; Step 120: Perform 3D modeling processing on the first image information to obtain the 3D scene information corresponding to the first image information; Step 130: Based on the target feature vector and 3D scene information, control the robotic arm module to perform auxiliary cleaning operations.

[0032] The item-assisted cleaning method provided in this application is applied to an item-assisted cleaning device, which includes a monitoring module and a robotic arm module. In response to first image information of the assisted cleaning work area acquired by the monitoring module, feature extraction is performed on the first image information to obtain the target object and its corresponding target feature vector. Subsequently, three-dimensional modeling processing is performed on the first image information to obtain three-dimensional scene information corresponding to the first image information. Finally, the robotic arm module is controlled to perform assisted cleaning operations based on the target feature vector and the three-dimensional scene information. Compared to the robotic arm operating according to preset actions in related technologies, this application embodiment, by acquiring image information in real time and combining feature extraction and three-dimensional modeling, enables the robotic arm module to adapt to the actual situation of tableware placement in the assisted cleaning work area, accurately determine the position, type, and debris distribution of the target object, and thus accurately control the robotic arm module to perform assisted cleaning operations on the target object, thereby improving the accuracy of the robotic arm in performing assisted cleaning operations on tableware.

[0033] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. AI refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0034] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0035] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0036] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0037] In step 110, the electronic device can collect image information of the auxiliary cleaning operation area in real time through the monitoring module, namely the first image information, in order to identify the target object and its corresponding target feature vector.

[0038] The aforementioned auxiliary cleaning areas refer to the areas where auxiliary cleaning operations for tableware are required. Specifically, auxiliary cleaning areas may include the sink area, countertop area, and the area surrounding tableware cleaning equipment.

[0039] In some embodiments, the auxiliary cleaning area in a home setting can be the kitchen area. Specifically, home settings often involve elderly people living at home, especially those with limited mobility, physical strength, and reaction speed. After meals, the elderly are often unable to clean, move, and finish cleaning the tableware, and the sink area is slippery and prone to falls. Manually cleaning large pieces of debris from the tableware can easily result in cuts from sharp objects, and handling hot water also poses a risk of burns. When the tableware auxiliary cleaning method provided in this application is applied to a home setting, the elderly only need to start the device to complete all the work without manually participating in the core operations, thus meeting the usage needs of the elderly population.

[0040] The aforementioned first image information refers to image data of the auxiliary cleaning operation area collected by the monitoring module. The first image information may include visual information of various items in the auxiliary cleaning operation area, such as the outline, color, and material of tableware, as well as the visual characteristics of debris attached to the surface of the tableware.

[0041] In some embodiments, the monitoring module may include a camera unit and an image processing unit.

[0042] In some embodiments, the camera unit in the monitoring module includes multiple sets of high-definition industrial cameras, which can be installed and deployed above and around the auxiliary cleaning area. For example, three sets of high-definition industrial cameras can be installed above the sink and around the dishwashing equipment, each set of cameras can capture image information of the auxiliary cleaning area from different angles. The camera unit can also be equipped with an infrared supplementary lighting module to adapt to dimly lit environments, such as the dishwashing room during night shifts in medical institutions or the dimly lit food preparation area in the corridor of a nursing home.

[0043] Specifically, in home settings, the kitchens of elderly people are often dimly lit in the evening or at night, and their eyesight is generally declining, making it difficult for them to accurately place tableware in dim environments. The infrared supplementary lighting module equipped in the aforementioned camera unit can provide supplementary lighting for image acquisition in low-light environments, enabling the monitoring module to accurately capture clear first image information even in dim conditions.

[0044] In some embodiments, the assistive cleaning device may also be equipped with a one-button start button and a voice guidance module. When the elderly start the assistive cleaning device, they only need to press the button once, and the voice guidance module can announce the current work progress in a soothing and adjustable voice, without requiring the elderly to learn complicated operating procedures.

[0045] In some embodiments, the image information acquired by the camera unit can be transmitted to the image processing unit in the monitoring module, where the image processing unit performs preprocessing and feature extraction on the image information.

[0046] The aforementioned target object refers to the object to be processed in the auxiliary cleaning operation. For example, it may include the target tableware itself and the debris attached to the target tableware.

[0047] The aforementioned target feature vector is vector data used to characterize various features of the target tableware. For example, the target feature vector may include features such as the type of tableware, size, and material information of the target tableware, as well as features such as the location coordinates and size of debris attached to the surface of the target tableware.

[0048] In some embodiments, feature extraction is performed on the first image information to obtain the target object in the first image information and its corresponding target feature vector, including: An adaptive threshold segmentation algorithm is used to denoise and enhance the first image information to remove the effects of water stains and light interference. The contour, material, and size features of the target tableware, as well as the position and size features of the debris attached to the surface of the target tableware, are extracted by a convolutional neural network (CNN) to generate a target feature vector.

[0049] This application embodiment extracts multi-dimensional features of the target object and its surface debris using CNN, which can improve the accuracy and richness of the target feature vector, thereby improving the accuracy of subsequent robotic arm modules performing auxiliary cleaning operations.

[0050] The adaptive threshold segmentation algorithm described above can be understood as an algorithm that dynamically determines the segmentation threshold based on local image features. By using the adaptive threshold segmentation algorithm to denoise and enhance images, the influence of interference factors such as water stain reflections and uneven lighting in the auxiliary cleaning area on image quality can be effectively removed.

[0051] In some embodiments, the aforementioned debris can be defined as food residue with a diameter greater than or equal to 5 millimeters, such as bones, vegetable leaves, fruit peels, etc. In medical settings, debris can also include medical-related residues such as medicine dregs and gauze fragments.

[0052] For example, the target feature vector can be represented as a multi-dimensional vector, where the first dimension corresponds to the type of tableware (such as bowls, plates, spoons, etc.), the second dimension corresponds to the size of the tableware, the third dimension corresponds to the coordinates of the debris's location, and the fourth dimension corresponds to the size of the debris. Through this target feature vector, the auxiliary cleaning device can comprehensively grasp the feature information of the target tableware, which is beneficial for the precise control operation of the robotic arm module.

[0053] In step 120, the electronic device can perform three-dimensional modeling processing on the first image information acquired in the aforementioned embodiment to obtain three-dimensional scene information corresponding to the first image information.

[0054] The aforementioned 3D scene information is information that can be used to characterize the 3D spatial layout of the auxiliary cleaning operation area and the spatial positional relationships of various objects. For example, the 3D scene information may include the position information of the target tableware, the position coordinates of the tableware cleaning equipment, the position coordinates of the trash can, the position coordinates of the sink, etc.

[0055] In some embodiments, the first image information is subjected to three-dimensional modeling processing to obtain three-dimensional scene information corresponding to the first image information, including: The preprocessed first image information is input into the visual encoder. The collaborative interaction of multi-view visual information is realized through the joint attention mechanism. Combined with the SLAM algorithm, a three-dimensional environment model of the auxiliary cleaning operation area is constructed to accurately locate the spatial coordinates of each target tableware. At the same time, the location coordinates of the sink, the tableware cleaning equipment inlet, and the trash can are marked to generate three-dimensional scene information.

[0056] This application embodiment uses 3D modeling to transform the actual layout of the auxiliary cleaning operation area into 3D scene information, enabling the auxiliary cleaning device to accurately grasp the precise position of each target tableware in 3D space. This provides reliable spatial data for the subsequent motion path planning and operation execution of the robotic arm module, thereby improving the accuracy and reliability of the robotic arm module in performing auxiliary cleaning operations.

[0057] The aforementioned joint attention mechanism is a technical mechanism that enables information interaction and collaborative processing among multiple visual information sources. Through the joint attention mechanism, image information acquired by different cameras can complement and verify each other, improving the accuracy of 3D modeling.

[0058] The aforementioned SLAM algorithm is an algorithm that simultaneously performs localization and mapping in an unknown environment. In the embodiments of this application, the SLAM algorithm can construct a three-dimensional environmental model of the area to assist in cleaning operations based on image information acquired by multiple sets of cameras.

[0059] For example, in the dishwashing area of ​​a medical institution's cafeteria, the target tableware may be scattered in multiple locations such as the sink, countertop, and next to the conveyor belt, and the tableware is diverse and haphazardly arranged. Through 3D modeling, the auxiliary cleaning device can accurately determine the coordinate position of each plate and medicine cup in 3D space, while marking the entrance position of the medical dishwasher and the location of the medical waste bin, providing complete spatial information for the subsequent collaborative operation of the robotic arm module.

[0060] In step 130, the electronic device can control the robotic arm module to perform auxiliary cleaning operations based on the target feature vector and three-dimensional scene information determined in the foregoing embodiments.

[0061] The aforementioned auxiliary cleaning operations can be understood as operations performed by the robotic arm module under the control of the auxiliary cleaning device to assist in cleaning tableware. For example, auxiliary cleaning operations may include cleaning debris adhering to the surface of the target tableware, or moving the target tableware.

[0062] This application embodiment controls the robotic arm module to perform auxiliary cleaning operations by using target feature vectors and three-dimensional scene information. The target feature vectors provide feature information such as the type, size, and debris distribution of the target tableware, while the three-dimensional scene information provides the precise location information of the target tableware in space. The combination of the two enables the robotic arm module to accurately know the various attributes and spatial position of the target tableware, thereby accurately performing the corresponding auxiliary cleaning operations and effectively improving the accuracy of the robotic arm module in performing auxiliary cleaning operations.

[0063] In one possible implementation, the target object includes the target tableware and the debris attached to the target tableware, the three-dimensional scene information includes the tableware position information of the target tableware, and the robotic arm module includes a first robotic arm and a second robotic arm. Based on the target feature vector and 3D scene information, the robotic arm module is controlled to perform auxiliary dishwashing operations, including: Based on the location information of the tableware, control the first robotic arm to grab the target tableware and move it to the preset cleaning area; The second robotic arm is controlled to perform auxiliary cleaning operations in a preset cleaning area through a cleaning module. The second robotic arm includes a cleaning module.

[0064] This embodiment of the application achieves efficient collaborative operation between the first robotic arm and the second robotic arm. The first robotic arm is responsible for grasping and fixing the target tableware, while the second robotic arm is responsible for performing auxiliary cleaning operations through the cleaning module. This enables efficient collaborative operation between the two arms, which can significantly improve the efficiency and accuracy of auxiliary cleaning operations compared to single-arm operation. At the same time, it can avoid the time-consuming and error problems caused by the frequent switching between fixing tableware and cleaning debris by a single arm.

[0065] The aforementioned tableware location information refers to the coordinates of the target tableware in three-dimensional space within the three-dimensional scene information. For example, the tableware location information may include the three-dimensional coordinates of the target tableware within the auxiliary cleaning area.

[0066] The first and second robotic arms mentioned above are two independent robotic arms in the robotic arm module, each undertaking different operational tasks. If they are divided into left and right arms, in one embodiment, the first robotic arm, corresponding to the left arm, is responsible for fixing and grasping the target tableware; the second robotic arm, the right arm, is responsible for cleaning up debris through the cleaning module.

[0067] The aforementioned preset cleaning area is a pre-defined area within the auxiliary cleaning work area used for performing auxiliary cleaning operations. The preset cleaning area can be a pre-designated washing location within a sink or a specially designated cleaning station on a countertop. For example, in medical institutions, the preset cleaning area could be a dedicated washing station in the center of the sink in the hospital's dishwashing room, facilitating the centralized disposal of food residue from medical staff's plates.

[0068] The aforementioned cleaning module is a modular component mounted on the second robotic arm for performing cleaning operations. The cleaning module may include cleaning tools such as spray nozzles and scraper brushes.

[0069] In some embodiments, based on the tableware position information, controlling a first robotic arm to grasp the target tableware and move it to a preset cleaning area includes: The end effector of the first robotic arm moves to the location of the target tableware based on the tableware position information, adjusts the gripping force, grabs the target tableware, and transfers it to a preset cleaning area for fixation.

[0070] The aforementioned end effector is a device at the end of a robotic arm used for directly contacting and manipulating objects. In some embodiments, the end effector of the first robotic arm employs a flexible silicone gripper, which can prevent scratching the surface of the target tableware.

[0071] In some embodiments, controlling a second robotic arm to perform auxiliary cleaning operations in a preset cleaning area via a cleaning module includes: The second robotic arm moves to the preset cleaning area and uses its cleaning module to clean the debris attached to the surface of the target tableware fixed by the first robotic arm, and pushes the cleaned debris to the debris collection area.

[0072] For example, in a medical facility cafeteria, a first robotic arm picks up a medical dining plate with food residue from a sink, fixes it in a pre-designated washing position in the sink, and keeps the plate open upwards. A second robotic arm then moves to that position, rinsing the surface of the plate with a high-pressure micro-spray nozzle, and then scraping off any stubborn residue with a flexible scraper. After cleaning, the first robotic arm transfers the plate to a temporary placement platform at the edge of the sink, awaiting loading into a medical dishwasher.

[0073] For example, in a home setting, an elderly person living alone might place a plate with bones and vegetable scraps in the sink after a meal, as they are too weak to stand and wash it for an extended period. The first robotic arm retrieves the plate from the sink based on its position and positions it in a pre-defined washing location. A second robotic arm then cleans the surface of the plate using a cleaning module. Throughout this process, the elderly person does not need to manually handle sharp objects like bones, avoiding the risk of cuts. They also avoid contact with hot water, preventing burns, and do not need to stand or bend over for extended periods, significantly reducing their physical burden.

[0074] In one possible implementation, the target feature vector includes the type of tableware and the size of the target tableware; Before controlling the first robotic arm to grasp the target tableware and move it to the preset cleaning area, based on the tableware's position information, the method also includes: Determine the target grasping parameters based on the type and size of the tableware; Based on the position information of the tableware, the first robotic arm is controlled to grasp the target tableware and move it to the preset cleaning area, including: Based on the location information of the tableware, the first robotic arm is controlled to grab the target tableware and move it to the preset cleaning area according to the grabbing parameters.

[0075] This application embodiment determines the target gripping parameters based on the type and size of the tableware, enabling the first robotic arm to adopt corresponding gripping force and gripping method for different types and sizes of target tableware. This avoids tableware slipping or being damaged due to improper gripping force, thereby improving the reliability and safety of the first robotic arm gripping the target tableware.

[0076] The tableware types mentioned above are classifications of the target tableware. Tableware types can include bowls, plates, spoons, cups, etc.

[0077] The tableware dimensions mentioned above are the physical dimensions of the target tableware, which may include parameters such as the diameter, height, and depth of the tableware.

[0078] The above-mentioned target grasping parameters are the parameters used by the first robotic arm when grasping the target tableware, and may include parameters such as grasping force, grasping position, and grasping angle.

[0079] In some embodiments, the target grasping parameters are determined based on the type and size of the cutlery, including: Based on the preset mapping relationship between tableware type and gripping force, the gripping force corresponding to the target tableware type is determined as the gripping force in the target gripping parameters.

[0080] For example, the gripping force for bowl-type tableware can be 5 to 8 Newtons, and the gripping force for plate-type tableware can be 8 to 12 Newtons.

[0081] In some embodiments, determining the target grasping parameters based on the type and size of the cutlery further includes: Based on the size of the target tableware, determine the opening angle and gripping position of the end effector of the first robotic arm.

[0082] For example, for plate-type tableware with a diameter of 20 cm, the opening angle of the end effector can be set to 45 degrees, and the gripping position is on both sides of the symmetrical edge of the tableware. For bowl-type tableware with a diameter of 10 cm, the opening angle can be set to 30 degrees, and the gripping position is on the side of the bowl.

[0083] In one possible implementation, the item cleaning aid further includes a cleaning module, and the target feature vector includes the location information and size of the debris attached to the target tableware; Before controlling the second robotic arm to perform auxiliary cleaning operations in the preset cleaning area via the cleaning module, the method further includes: Determine auxiliary cleaning parameters based on the location and size of the debris; Controlling the second robotic arm to perform auxiliary cleaning operations within a preset cleaning area via a cleaning module includes: The second robotic arm is controlled to perform auxiliary cleaning operations in the preset cleaning area according to the auxiliary cleaning parameters through the cleaning module.

[0084] This application embodiment determines auxiliary cleaning parameters based on the location and size of the debris, enabling the second robotic arm to adopt appropriate cleaning force and method for debris of different locations and sizes. This avoids debris splashing or damage to the tableware surface due to improper cleaning force, thereby improving the accuracy and reliability of the second robotic arm in performing auxiliary cleaning operations.

[0085] The location information of the aforementioned debris can be obtained from the target feature vector.

[0086] The above-mentioned dimensions of miscellaneous items can be understood as the size parameters of the miscellaneous items, which can be expressed by parameters such as diameter, area, or volume.

[0087] The aforementioned auxiliary cleaning parameters can be understood as the parameters used by the second robotic arm when performing auxiliary cleaning operations. For example, auxiliary cleaning parameters may include cleaning pressure, scraping pressure, cleaning duration, and other parameters.

[0088] In some embodiments, auxiliary cleaning parameters are determined based on the location information and size of the debris, including: Determine the cleaning pressure based on the size of the debris.

[0089] For example, for large debris such as bones with a diameter of 10 mm or more, the cleaning pressure can be set to 0.4 to 0.5 MPa to provide sufficient rinsing force; for smaller debris such as vegetable leaves with a diameter between 5 and 10 mm, the cleaning pressure can be set to 0.3 to 0.4 MPa.

[0090] In some embodiments, determining auxiliary cleaning parameters based on debris location information and debris size further includes: The scraping pressure is determined based on the degree of adhesion and size of the debris.

[0091] For example, for debris with strong adhesion, such as dried rice residue, the scraping pressure can be set to a higher value, and the pressure sensor can provide real-time feedback and adjustment to avoid damaging the tableware; for debris with weak adhesion, such as fruit peels, the scraping pressure can be set to a lower value.

[0092] In one possible implementation, the cleaning module includes a spray head and a scraping brush, and the auxiliary cleaning parameters include cleaning pressure and scraping pressure; Controlling the second robotic arm to perform auxiliary cleaning operations within a preset cleaning area according to auxiliary cleaning parameters via the cleaning module includes: Control the second robotic arm to spray and rinse debris in a preset cleaning area using a spray head at cleaning pressure; and control the second robotic arm to scrape debris in the preset cleaning area using a scraper brush.

[0093] This application embodiment utilizes a combined cleaning method of a spray head and a scraping brush. First, the spray head uses high-pressure rinsing to remove most loose debris, then the scraping brush removes stubborn, attached debris. This achieves thorough cleaning of the target tableware surface, improving the effectiveness of the auxiliary cleaning operation. Simultaneously, by separately controlling the cleaning pressure and scraping pressure, appropriate cleaning force can be applied to different types and degrees of adhesion of debris, avoiding damage to the tableware.

[0094] The aforementioned spray head is a component in the cleaning module used to spray water to rinse away debris. In some embodiments, the spray head may be a high-pressure micro-spray head with adjustable pressure. In medical facilities, the spray head can use sterilized hot water to spray, providing initial disinfection of tableware while cleaning debris.

[0095] The aforementioned scraper brush is a component in the cleaning module used to scrape away stubborn debris adhering to the surface of tableware. In some embodiments, the scraper brush may be made of a flexible material, such as silicone or soft bristles, to avoid scratching the surface of the tableware.

[0096] In some embodiments, controlling a second robotic arm to spray and rinse debris in a preset cleaning area using a spray head at a cleaning pressure includes: The second robotic arm is controlled to aim the spray head at the location of the debris, and the spray head is activated according to the determined cleaning pressure to perform high-pressure rinsing on the debris. During the rinsing process, the angle and distance of the spray head can be adjusted in real time according to the location information of the debris.

[0097] In some embodiments, controlling a second robotic arm to scrape debris in a preset cleaning area using a scraping brush includes: After the spray rinsing is completed, the second robotic arm is controlled to scrape the area of ​​the tableware surface where the debris is located by contacting the scraping brush according to the corresponding scraping pressure.

[0098] During the scraping process, the pressure sensor can monitor the scraping force in real time and provide feedback for adjustment, ensuring that the scraping force is moderate, which can effectively remove debris without damaging the tableware.

[0099] For example, in the catering room of a nursing home, the second robotic arm first washes the food residue on the surface of the medical dining plates with a spray head at a pressure of 0.4 MPa. Most of the loose residue is washed into the debris collection tank at the bottom of the sink. Subsequently, the second robotic arm uses a flexible scraper to scrape off the dried rice paste residue on the surface of the plates with appropriate force. The pressure sensor provides real-time feedback that the scraping force is 6 Newtons to ensure that the surface of the plates is not scratched.

[0100] In some embodiments, while cleaning up debris, the second robotic arm pushes large pieces of debris scraped and rinsed off into a debris collection trough at the bottom of a pre-designated cleaning area. The debris collection trough has a built-in infrared sensor that monitors the debris level in real time. When the level reaches 80%, the automatic slag discharge function is activated, transporting the debris through a pipe to a waste bin, completing the debris collection. In medical institutions, the debris can be transported to dedicated medical waste bins to meet the regulatory requirements for medical waste disposal.

[0101] In one possible implementation, in a preset cleaning area, the target surface of the target tableware faces the cleaning module, and the target surface is the side of the target tableware that holds the food.

[0102] This application embodiment controls the target surface of the target tableware to face the cleaning module, enabling the cleaning module to directly target the food-containing side of the tableware for cleaning. Since the food-containing side of the tableware is usually the area where debris adheres most severely, facing the target surface towards the cleaning module ensures that the cleaning module thoroughly cleans this area, thereby improving the thoroughness and effectiveness of the auxiliary cleaning operation.

[0103] The target surface mentioned above refers to the surface of the target tableware used to hold food. For bowls, the target surface is the inner concave surface of the bowl; for plates, the target surface is the upper surface of the plate; and for spoons, the target surface is the concave part of the spoon.

[0104] In some embodiments, when the first robotic arm grasps the target tableware to the preset cleaning area, it can adjust the orientation of the target tableware according to the tableware type information in the target feature vector, so that the target surface faces the cleaning module of the second robotic arm, which facilitates the cleaning module to clean the target surface of debris.

[0105] For example, for bowl-type tableware, the first robotic arm fixes the bowl in a preset cleaning area and keeps the rim of the bowl facing upwards, so that the concave surface of the bowl faces the cleaning module of the second robotic arm, making it easier for the spray head and scraper brush to clean the inner surface of the bowl. For plate-type tableware, the first robotic arm lays the plate flat or fixes it at an appropriate angle, so that the upper surface of the plate faces the cleaning module.

[0106] In one possible implementation, the item cleaning aid further includes a first communication module, which is used to establish a communication connection with the tableware cleaning equipment; Based on the target feature vector and 3D scene information, after controlling the robotic arm module to perform auxiliary cleaning operations, the method also includes: In response to receiving a first target signal through the first communication module, the control robot arm module transfers the target tableware to the tableware cleaning equipment for cleaning; wherein, the first target signal is used to indicate that the tableware cleaning equipment is in an idle state.

[0107] In this embodiment, a communication connection is established between the first communication module and the tableware cleaning equipment. After receiving a first target signal indicating that the tableware cleaning equipment is in an idle state, the robotic arm module is controlled to transfer the target tableware to the tableware cleaning equipment for cleaning. This enables the linkage between the auxiliary cleaning device and the tableware cleaning equipment. After completing the pre-treatment of debris, the tableware is automatically transferred to the tableware cleaning equipment for deep cleaning. There is no need for manual handling of tableware, which can improve the automation level and efficiency of the entire tableware cleaning process.

[0108] The aforementioned first communication module can be understood as a module in the auxiliary cleaning device used to communicate with the tableware cleaning equipment. Specifically, the first communication module can achieve communication connection with the tableware cleaning equipment through Internet of Things (IoT) communication technology, including wireless and wired communication methods.

[0109] The aforementioned tableware cleaning equipment is used for deep cleaning of tableware. This equipment can be a dishwasher or similar cleaning device. In medical institutions, tableware cleaning equipment can be a medical dishwasher, equipped with high-temperature sterilization and disinfection functions.

[0110] The aforementioned first target signal can be understood as a signal sent by the tableware cleaning equipment to the auxiliary cleaning device through the first communication module, indicating that the tableware cleaning equipment is currently in an idle state and can receive new tableware for cleaning.

[0111] In some embodiments, after the auxiliary cleaning device establishes a communication connection with the tableware cleaning equipment through the first communication module, it can detect the working status and door status of the tableware cleaning equipment in real time. The working status may include an idle status and a busy status, and the door status may include an open status and a closed status.

[0112] In some embodiments, when the tableware cleaning equipment is idle, after receiving the first target signal, the auxiliary cleaning device controls the robotic arm module to grab the target tableware that has completed the debris pretreatment from the temporary placement table and transfer it into the tableware cleaning equipment.

[0113] In some embodiments, controlling the robotic arm module to transfer the target tableware to a tableware cleaning device for cleaning includes: Based on the type of tableware, the robotic arm module is controlled to classify and load the target tableware.

[0114] For example, bowls are placed upside down on the upper shelf of the dishwashing equipment, plates are laid flat on the middle shelf, and spoons are placed in a dedicated storage slot on the lower shelf.

[0115] In some embodiments, when the door of the tableware cleaning device is closed, the auxiliary cleaning device can send a signal to the door tapping sensor of the tableware cleaning device to trigger the door to open automatically at a preset tapping frequency, without the need for manual operation.

[0116] For example, the preset tapping frequency can be 3 times per second for 2 seconds.

[0117] Once loaded, the electronic device can automatically select the appropriate cleaning mode based on the type of tableware.

[0118] For example, bowls should be cleaned using the gentle cycle, while plates should be cleaned using the standard cycle. In medical facilities, medical plates can be cleaned using the high-temperature, high-intensity sterilization cycle to ensure thorough disinfection.

[0119] In a home setting, the design that triggers the door to open automatically by tapping a sensor can prevent elderly people from bending over to manually open the dishwasher door, further reducing the difficulty of operation for them and enabling them to start and use the device independently.

[0120] In a home setting, the elderly have relatively weaker immune systems. If the stains left on the cleaning aids are not cleaned in time, bacteria can easily grow and contaminate the tableware that is cleaned later, affecting the health of the elderly.

[0121] In one possible implementation, after the robotic arm module transfers the target tableware to the tableware cleaning equipment for cleaning in response to receiving the target signal via the first communication module, the method further includes: In response to receiving a second target signal through the first communication module, the robotic arm module is controlled to move to a preset cleaning area for self-cleaning; wherein, the second target signal is used to indicate whether the tableware cleaning equipment is in either the state of starting cleaning or completing cleaning.

[0122] This application embodiment triggers the self-cleaning of the robotic arm module when the tableware cleaning equipment starts or finishes cleaning. This allows the auxiliary cleaning device to clean itself during the idle time of the tableware cleaning equipment, without taking up extra time and improving overall efficiency. At the same time, self-cleaning prevents the growth of bacteria from debris and water stains on the auxiliary cleaning device, ensuring the hygiene and safety of the usage environment, especially meeting the needs of medical institutions and other scenarios with strict hygiene requirements.

[0123] The aforementioned second target signal is a signal sent by the tableware cleaning equipment to the auxiliary cleaning device through the first communication module, used to indicate that the tableware cleaning equipment has started or completed the cleaning process.

[0124] The self-cleaning described above can be understood as the operation of the robotic arm module cleaning its own components. The objects of self-cleaning may include the end effectors of the first and second robotic arms, the water tank in the preset cleaning area, and the temporary placement platform, etc.

[0125] In some embodiments, in response to receiving a second target signal via a first communication module, controlling a robotic arm module to move to a preset cleaning area for self-cleaning includes: The first and second robotic arms are moved above the water tank, activating the self-cleaning function of the end effectors. The flexible silicone grippers are then rinsed with high-pressure spray to remove residual debris and water stains from the surface, while simultaneously opening and closing to ensure no residue remains in the gripper crevices. A flexible scraper brush then rotates to clean, working in conjunction with the water spray to remove any remaining debris. After cleaning, both arms move to a designated drying area and are dried with hot air to prevent water residue.

[0126] For example, during the self-cleaning process, the spray pressure of the flexible silicone gripper can be 0.4 MPa, and the number of opening and closing cycles can be greater than or equal to 10. The rotation speed of the flexible scraping brush can be 300 rpm. The hot air drying temperature can be 40 to 50 degrees Celsius, and the drying time can be 3 minutes.

[0127] In some embodiments, self-cleaning also includes cleaning the pool and temporary placement platform. The built-in high-pressure spray system of the pool is activated to thoroughly rinse the inner walls, bottom, and temporary placement platform for up to 2 minutes. Simultaneously, the drain at the bottom of the pool is opened to discharge the wastewater. After rinsing, the hot air drying function is activated for up to 2 minutes to prevent mold growth on the inner walls of the pool.

[0128] This application embodiment uses a self-cleaning process to promptly remove residual debris and water stains from the end effectors of the dual arms, the water tank, and the temporary placement platform, and uses hot air drying to inhibit mold growth, thus ensuring the food hygiene and safety of the elderly.

[0129] In some embodiments, the auxiliary cleaning device can flexibly switch between fully automatic and semi-automatic modes according to the elderly person's physical strength and operational ability. In fully automatic mode, no manual intervention is required from the elderly person throughout the entire process; in semi-automatic mode, the elderly person can manually participate in simple operations such as setting the table, which can adapt to the usage habits of elderly people with different physical conditions.

[0130] In one possible implementation, the item-assisted cleaning device further includes a second communication module for communicating with floor cleaning equipment; In response to receiving the second target signal via the first communication module and controlling the robotic arm module to move to the preset cleaning area for self-cleaning, the method further includes: In response to receiving a third target signal via the second communication module, a ground cleaning command is generated; wherein the third target signal is used to indicate that the robotic arm module has completed self-cleaning. Send a floor cleaning instruction to the floor cleaning equipment. The floor cleaning instruction is used to instruct the floor cleaning equipment to clean the floor in the auxiliary cleaning work area.

[0131] This embodiment of the application communicates with the ground cleaning equipment through the second communication module, and sends a ground cleaning command to the ground cleaning equipment after the robotic arm module completes self-cleaning. This enables the linkage between the auxiliary cleaning device and the ground cleaning equipment to complete the cleaning of the ground in the auxiliary cleaning operation area without the need for manual mopping, which can further improve the automation level of the entire process.

[0132] Furthermore, in home settings, elderly people find it strenuous to bend over and mop the floor, and residual water on the kitchen floor can easily cause them to slip and fall, posing a significant safety hazard. This application's embodiment utilizes a floor cleaning device to automatically clean the auxiliary cleaning area, leaving no standing water after cleaning and reducing the risk of falls for the elderly.

[0133] The aforementioned second communication module is a module in the auxiliary cleaning device used for communication with the floor cleaning equipment.

[0134] The aforementioned floor cleaning equipment can be understood as equipment used for cleaning floors. Floor cleaning equipment can be a floor cleaning module integrated into the bottom of an auxiliary cleaning device, or it can be a standalone robot vacuum cleaner or robot mop. In medical institutions, floor cleaning equipment may also have floor disinfection functions.

[0135] The aforementioned third target signal is generated by the auxiliary cleaning device after determining that the robotic arm module has completed self-cleaning, and can be used to trigger ground cleaning operations.

[0136] The aforementioned floor cleaning instructions can be understood as instructions sent by the auxiliary cleaning device to the floor cleaning equipment, which are used to instruct the floor cleaning equipment to begin cleaning the floor in the auxiliary cleaning work area.

[0137] In some embodiments, after the floor cleaning equipment is started, the side brushes first gather water stains and small debris from the floor in both directions, and then a high-speed fan sucks the debris into the dust box. At the same time, the wet mopping function is activated, using a microfiber cloth to adhere tightly to the floor and remove water stains and small dirt. During the cleaning process, a lidar sensor is used to avoid obstacles in real time, preventing collisions with furniture, dishwashing equipment, and other objects.

[0138] In some embodiments, after the dishwashing equipment completes cleaning, it sends a cleaning completion signal to the auxiliary cleaning device. Electronic devices can control the door of the dishwashing equipment to open automatically via a tap sensor, and simultaneously control a robotic arm module to move to the side of the dishwashing equipment, preparing for dish unloading. At this time, after the floor cleaning equipment completes floor cleaning, it automatically returns to the designated storage location, and the dustbin automatically seals, awaiting subsequent unified cleaning.

[0139] In some embodiments, after the entire process is completed, the visual perception system takes a picture of the auxiliary cleaning area to confirm that there are no debris or water stains on the ground, no residue from the auxiliary cleaning device, and that the tableware is placed in a standardized manner. Then, it generates a complete cleaning signal and issues a voice prompt to inform the user that the auxiliary cleaning of the tableware is complete.

[0140] It should be noted that the various embodiments described in this application can be combined with each other or implemented individually without conflict, and this application does not limit this.

[0141] Corresponding to the above method embodiments, this application also provides a tableware auxiliary cleaning system. Please refer to [link / reference]. Figure 3 , Figure 3 This is a functional module diagram of the tableware auxiliary cleaning system provided in this application embodiment. The tableware auxiliary cleaning system 300 is applied to an item auxiliary cleaning device, which includes a monitoring module and a robotic arm module. The tableware auxiliary cleaning system 300 includes: The first processing module 310 is used to extract features from the first image information of the auxiliary cleaning operation area collected by the monitoring module in response to the first image information collected by the monitoring module, and to obtain the target object in the first image information and its corresponding target feature vector. The second processing module 320 performs three-dimensional modeling processing on the first image information to obtain three-dimensional scene information corresponding to the first image information. The control module 330 controls the robotic arm module to perform auxiliary cleaning operations based on the target feature vector and the three-dimensional scene information.

[0142] The tableware auxiliary cleaning system provided in this application embodiment can achieve the following: Figure 1The various processes implemented in the Chinese method embodiments can achieve similar or the same technical effects, and will not be described again here to avoid repetition.

[0143] In addition to the methods and systems provided in the above embodiments, this application also provides an electronic device. Please see [link to documentation]. Figure 4 , Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0144] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0145] Specifically, the processor 401 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0146] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 402 is non-volatile solid-state memory.

[0147] In some embodiments, memory 402 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described in the methods provided according to embodiments of this application.

[0148] The processor 401 implements the method provided in the above embodiments by reading and executing computer program instructions stored in the memory 402.

[0149] In one example, the electronic device may also include a communication interface 403 and a bus 410. The processor 401, memory 402, and communication interface 403 are connected via the bus 410 and communicate with each other.

[0150] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0151] Bus 410 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0152] Furthermore, in conjunction with the methods provided in the above embodiments, this application embodiment can be implemented using a computer-readable storage medium. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the methods in the above embodiments.

[0153] Furthermore, in conjunction with the methods provided in the above embodiments, this application embodiment can provide a computer program product to implement the methods. This program product is stored in a storage medium and executed by at least one processor to implement the various processes of the embodiments of the methods provided in the above embodiments, achieving similar or identical technical effects. To avoid repetition, further details are omitted here.

[0154] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0155] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0156] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0157] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0158] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for assisting in cleaning items, characterized in that, An auxiliary cleaning device for items, the auxiliary cleaning device including a monitoring module and a robotic arm module, the method comprising: In response to the first image information of the auxiliary cleaning operation area collected by the monitoring module, feature extraction is performed on the first image information to obtain the target object in the first image information and its corresponding target feature vector; Perform three-dimensional modeling processing on the first image information to obtain the three-dimensional scene information corresponding to the first image information; Based on the target feature vector and the three-dimensional scene information, the robotic arm module is controlled to perform auxiliary cleaning operations.

2. The method as described in claim 1, characterized in that, The target object includes the target tableware and the debris attached to the target tableware; the three-dimensional scene information includes the tableware position information of the target tableware; and the robotic arm module includes a first robotic arm and a second robotic arm. The step of controlling the robotic arm module to perform assisted dishwashing operations based on the target feature vector and the three-dimensional scene information includes: Based on the tableware position information, the first robotic arm is controlled to grab the target tableware and move it to a preset cleaning area; The second robotic arm is controlled to perform the auxiliary cleaning operation in the preset cleaning area through the cleaning module, and the second robotic arm includes the cleaning module.

3. The method as described in claim 2, characterized in that, The target feature vector includes the type of tableware and the size of the target tableware; Before controlling the first robotic arm to grasp the target tableware to the preset cleaning area based on the tableware position information, the method further includes: Determine the target grasping parameters based on the type and size of the tableware; The step of controlling the first robotic arm to grasp the target tableware and move it to a preset cleaning area based on the tableware position information includes: Based on the tableware position information, the first robotic arm is controlled to grab the target tableware and move it to the preset cleaning area according to the grabbing parameters.

4. The method as described in claim 2, characterized in that, The item-assisted cleaning device also includes a cleaning module, and the target feature vector includes the location information and size of the debris attached to the target tableware. Before controlling the second robotic arm to perform the auxiliary cleaning operation in the preset cleaning area via the cleaning module, the method further includes: Based on the location information and size of the debris, determine the auxiliary cleaning parameters; The control of the second robotic arm to perform the auxiliary cleaning operation in the preset cleaning area via the cleaning module includes: The second robotic arm is controlled to perform the auxiliary cleaning operation in the preset cleaning area through the cleaning module according to the auxiliary cleaning parameters.

5. The method as described in claim 4, characterized in that, The cleaning module includes a spray head and a scraping brush, and the auxiliary cleaning parameters include cleaning pressure and scraping pressure. The control of the second robotic arm to perform the auxiliary cleaning operation in the preset cleaning area via the cleaning module according to the auxiliary cleaning parameters includes: The second robotic arm is controlled to spray and rinse the debris in the preset cleaning area through the spray head at the cleaning pressure. In addition, the second robotic arm is controlled to scrape the debris in the preset cleaning area using the scraping brush.

6. The method as described in claim 4, characterized in that, In the preset cleaning area, the target surface of the target tableware faces the cleaning module, and the target surface is the side of the target tableware that holds the food.

7. The method as described in claim 2, characterized in that, The item cleaning auxiliary device also includes a first communication module, which is used to establish a communication connection with the tableware cleaning equipment. After controlling the robotic arm module to perform auxiliary cleaning operations based on the target feature vector and the three-dimensional scene information, the method further includes: In response to receiving a first target signal through the first communication module, the robotic arm module is controlled to transfer the target tableware to the tableware cleaning equipment for cleaning; wherein, the first target signal is used to indicate that the tableware cleaning equipment is in an idle state.

8. The method as described in claim 7, characterized in that, After receiving a target signal through the first communication module and controlling the robotic arm module to transfer the target tableware to the tableware cleaning equipment for cleaning, the method further includes: In response to receiving a second target signal through the first communication module, the robotic arm module is controlled to move to a preset cleaning area for self-cleaning; wherein, the second target signal is used to indicate that the tableware cleaning equipment is in either the state of starting cleaning or completing cleaning.

9. The method as described in claim 8, characterized in that, The item-assisted cleaning device also includes a second communication module, which is used to communicate with the floor cleaning equipment. After receiving the second target signal through the first communication module and controlling the robotic arm module to move to the preset cleaning area for self-cleaning, the method further includes: In response to receiving a third target signal via the second communication module, a ground cleaning command is generated; wherein the third target signal is used to indicate that the robotic arm module has completed self-cleaning. The floor cleaning instruction is sent to the floor cleaning equipment, which instructs the floor cleaning equipment to clean the floor in the auxiliary cleaning operation area.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method of any one of claims 1 to 9.

Images