Interaction control method and electronic device

Abstract

The embodiment of the application relates to the technical field of robots, and discloses an interactive control method and electronic equipment. First, an image set is acquired, the image set including images collected by a cleaning robot in a period of time (for example, consecutive days) from an environment in which the cleaning robot is located. The images in the image set are identified to determine a degree of dirtiness change of the environment compared with a standard state of the environment. Whether an abnormal state (for example, a user is busy or in a low mood, etc.) occurs is determined based on the degree of dirtiness change. If the abnormal state occurs, the cleaning robot is controlled to actively start an interactive action. In this embodiment, the ground of the environment is visually monitored, and in the case where it is determined that the abnormal state occurs, the cleaning robot is controlled to start the interactive action to provide a humanized caring service, for example, through an interactive mode such as a voice greeting or an app pop-up reminder, greeting the user whether the user needs cleaning, providing comfort and encouragement, or reminding the user to clean up articles, etc.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of robots, and in particular to an interactive control method and an electronic device. BACKGROUND

[0002] As an important device in smart home, a cleaning robot is gradually developing towards intelligence, for example, the interaction between the cleaning robot and people or pets is increasing. Meanwhile, with the development of natural language processing technology, cleaning robots with voice control functions are gradually emerging. The cleaning robot with voice control function can automatically recognize the voice instruction spoken by the user, and then perform corresponding operations in response to the recognition result of the voice instruction.

[0003] However, these interactive functions are single, mainly realizing mechanical operations such as cleaning, and cannot monitor the environment where the cleaning robot is located, and cannot provide personalized care services. SUMMARY

[0004] Therefore, some embodiments of the present application provide an interactive control method and an electronic device, which monitor the environment, control the cleaning robot to start interactive actions when an abnormal state is detected, and provide personalized care services.

[0005] In a first aspect, some embodiments of the present application provide an interactive control method, comprising:

[0006] obtaining an image set, the image set comprising images collected by a cleaning robot in a preset time in an environment where the cleaning robot is located;

[0007] identifying the images in the image set to determine a degree of dirtiness and disorder of the environment compared with a standard state of the environment;

[0008] determining whether an abnormal state occurs based on the degree of dirtiness and disorder;

[0009] if the abnormal state occurs, controlling the cleaning robot to start an interactive action.

[0010] In some embodiments, the foregoing identifying the images in the image set to determine the degree of dirtiness and disorder of the environment compared with the standard state of the environment comprises:

[0011] performing semantic recognition on the images in the image set to determine an item condition of the environment in a current state;

[0012] comparing the item condition of the environment in the current state with an item condition of the environment in a standard state to determine the degree of dirtiness and disorder of the environment compared with the standard state of the environment.

[0013] In some embodiments, the item condition comprises a type and a quantity of garbage on the ground, a pose and a quantity of daily necessities, or a pose of furniture.

[0014] In some embodiments, the aforementioned comparing the item status of the environment in the current state with the item status of the environment in the standard state to determine the degree of messiness change of the environment compared with the standard state of the environment comprises:

[0015] The item status in the current state and the item status in the standard state are respectively counted to determine the abnormal quantity as the degree of messiness change, wherein the abnormal quantity comprises the increase quantity of garbage, the increase quantity of daily necessities, the quantity of the occurrence of pose change, or the quantity of the occurrence of pose change of furniture.

[0016] In some embodiments, the aforementioned determining whether an abnormal state occurs based on the degree of messiness change comprises:

[0017] According to the abnormal quantity, the duration that the abnormal quantity exceeds the quantity threshold, and the change trend of the abnormal quantity, it is determined whether an abnormal state occurs.

[0018] In some embodiments, the aforementioned determining whether an abnormal state occurs according to the abnormal quantity, the duration that the abnormal quantity exceeds the quantity threshold, and the change trend of the abnormal quantity comprises:

[0019] The abnormal quantity, the duration, and the change trend are weighted to obtain an abnormal score;

[0020] If the abnormal score is greater than or equal to a score threshold, it is determined that an abnormal state occurs.

[0021] In some embodiments, the aforementioned controlling the cleaning robot to start the interactive action comprises:

[0022] Controlling the cleaning robot to initiate a voice greeting or controlling an application associated with the cleaning robot to initiate a pop-up reminder; or,

[0023] Calling audio data and controlling the cleaning robot to play the audio data.

[0024] In some embodiments, the method further comprises: determining the interaction mode according to historical feedback records.

[0025] In a second aspect, some embodiments of the present application provide an electronic device, comprising:

[0026] At least one processor;

[0027] A memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the interactive control method of the first aspect.

[0028] In a third aspect, some embodiments of the present application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the interaction control method of the first aspect.

[0029] The interaction control method provided by the embodiments of the present application has the following advantages. Different from the prior art, the interaction control method provided by the embodiments of the present application first acquires an image set, the image set including images collected by a cleaning robot in a period of time (for example, consecutive days) in an environment in which the cleaning robot is located. The images in the image set are identified to determine a degree of dirtiness change of the environment compared with a standard state of the environment. Based on the degree of dirtiness change, it is determined whether an abnormal state (for example, a user is busy or in a low mood, etc.) occurs. If the abnormal state occurs, the cleaning robot is controlled to actively start an interactive action.

[0030] In this embodiment, the ground of the environment is visually monitored to collect an image set that can reflect the degree of dirtiness change of the environment. The image set is identified to extract key data (for example, the number of garbage, the degree of disorder of the arrangement of articles, etc.) reflecting the degree of dirtiness change. The key data is comprehensively analyzed, and in the case where it is determined that an abnormal state occurs, the cleaning robot is controlled to start an interactive action to provide a humanized care service, for example, through an interactive mode such as a voice greeting or an app pop-up reminder, to inquire whether the user needs cleaning, to provide comfort and encouragement, or to remind the user to clean articles, etc. BRIEF DESCRIPTION OF DRAWINGS

[0031] One or more embodiments are illustrated by way of example in the figures that are part of this document, and which illustrate key principles of the embodiments. The examples do not constitute an exhaustive list of examples that can be made by a person of ordinary skill in the art in light of the present disclosure. Elements having the same reference number designations in the figures indicate like elements, unless otherwise expressly provided. The figures do not constitute a proportional limitation.

[0032] Figure 1 A scenario diagram for the application of the interaction control method in some embodiments of the present application;

[0033] Figure 2 A flowchart of the interaction control method in some embodiments of the present application;

[0034] Figure 3 A diagram of article identification and positioning in some embodiments of the present application;

[0035] Figure 4 A structural diagram of an electronic device in some embodiments of the present application. DETAILED DESCRIPTION

[0036] The application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the application. These are within the scope of protection of the application.

[0037] In order to make the purpose, technical solutions and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0038] It should be noted that the various features in the embodiments of the application can be combined with each other without conflict, and are within the scope of protection of the application. In addition, although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the order in the flowchart. In addition, the "first", "second", "third" and the like used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same function and effect.

[0039] Unless otherwise defined, all technical and scientific terms used in the specification have the same meaning as understood by those skilled in the art to which the application belongs. The terms used in the specification of the application are only for the purpose of describing the specific embodiments and are not used to limit the application. The term "and / or" used in the specification includes any and all combinations of one or more related listed items.

[0040] In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as there is no conflict.

[0041] As Figure 1 is a schematic diagram of the application scenario of the interactive control method of the embodiments of the application. The application scenario includes a cleaning robot 10, a server 20 and a terminal 30, the cleaning robot 10 and the server 20 are communicatively connected, and the server 20 and the terminal 30 are communicatively connected.

[0042] Among them, the server 20 can be a physical server or a cloud server, etc. The terminal 30 can be a mobile terminal such as a smart phone, a tablet computer, etc.

[0043] In some embodiments, the terminal 30 downloads an application program (app) associated with the cleaning robot 10, and a backend module of the application program (app) is arranged in the server 20. Thus, the user sets the cleaning robot 10, such as setting the use permission, etc., by operating the application program on the terminal 30. The cleaning robot 10 can send sharing information to the server 20, and the server 20 further sends the sharing information to the terminal 30.

[0044] In this application scenario, the user can control the cleaning robot 10 through voice, that is, voice interaction with the cleaning robot. Specifically, the controller in the cleaning robot 10 calls a voice recognition system to perform semantic recognition and speaker recognition on the voice signal, and controls the cleaning robot 10 to perform a corresponding task based on the recognition result. Thus, voice interaction between the user and the cleaning robot 10 is realized.

[0045] In some embodiments, the cleaning robot 10 is installed with a laser radar and a camera. The laser radar is arranged on the body of the cleaning robot 10, for example, the laser radar is arranged on the mobile chassis of the body of the cleaning robot 10. In some embodiments, the laser radar includes a pulsed laser radar, a continuous wave laser radar, etc., and the mobile chassis includes a general-purpose chassis, a waist-hunched mobile chassis, etc. The camera is installed on the body of the cleaning robot 10, for example, on the top of the body of the cleaning robot 10.

[0046] The laser radar scans the environment where the cleaning robot 10 is located to obtain a laser point cloud, and the camera photographs the environment where the cleaning robot 10 is located to obtain an image. The laser radar and the camera are respectively in communication connection with the controller, and the laser point cloud and the image are respectively sent to the controller. The controller calls a program for constructing a map which is previously preset in the memory of the cleaning robot 10, and constructs a map based on the laser point cloud and / or the image. The program for constructing a map can include a program corresponding to a SLAM algorithm (Simultaneous Localization and Mapping, SLAM), which is not described in detail herein. In some embodiments, the map is a grid map. The map is saved in the memory of the cleaning robot 10. When the robot moves for operation, the controller calls the map as a basis for autonomous positioning, path planning and obstacle avoidance.

[0047] It can be understood that the SLAM algorithm has both positioning and navigation functions. In the positioning process, the laser radar is controlled to rotate at high speed to emit laser, measure the distance between the cleaning robot and the obstacle, and determine the relative position between the cleaning robot and the obstacle in combination with the map, so as to realize positioning. In some embodiments, the cleaning robot 10 can be visually positioned based on the camera. In the navigation process, the cleaning path is planned based on the positioning and the cleaning task, the full-coverage cleaning path is planned, and the corresponding cleaning task is completed.

[0048] The cleaning robot 10 can be configured in any suitable shape to achieve cleaning services. The cleaning robot 10 includes, but is not limited to, a sweeping robot, a dusting robot, a mopping robot, a scrubbing robot, or the like.

[0049] In some embodiments, the cleaning robot 10 includes a body and a drive wheel component, a camera, a sensor, and a controller. The body can be generally oval, triangular, D-shaped, or other shapes. The controller is arranged in the body, which is the main structure of the cleaning robot 10, and can be selected according to the actual needs of the cleaning robot 10, such as a relatively flat cylindrical shape commonly used for cleaning robots, and the manufacturing material (such as hard plastic or aluminum, iron, and other metals).

[0050] The drive wheel component is mounted to the body for driving the cleaning robot 10 to move on the surface to be cleaned. In some embodiments, the drive wheel component includes a left drive wheel, a right drive wheel, and an omni-wheel, and the left drive wheel and the right drive wheel are respectively mounted to opposite sides of the main body. The omni-wheel is mounted to the front position of the bottom of the main body, and the omni-wheel is a movable caster wheel that can rotate horizontally by 360 degrees to enable the cleaning robot 10 to turn flexibly. The left drive wheel, the right drive wheel, and the omni-wheel are mounted in a triangular shape to improve the stability of the cleaning robot 10 when walking.

[0051] In some embodiments, the sensor is used to collect some motion parameters and environmental space data of the cleaning robot 10, and the sensor includes various suitable sensors, such as a gyroscope, an infrared sensor, an odometer, a magnetic field meter, an accelerometer, or a speed meter, and the like.

[0052] In some embodiments, the controller is arranged inside the body and is the electronic computing core of the cleaning robot 10, which is used to perform logical operation steps to realize intelligent control of the cleaning robot 10. The controller is electrically connected to the left drive wheel, the right drive wheel, and the omni-wheel, respectively. As the control core of the cleaning robot 10, the controller is used to control the cleaning robot 10 to walk, retreat, and process some business logic. For example, the controller is used to receive voice instructions sent by the microphone, and control the cleaning robot 10 to complete the corresponding task based on the voice instructions.

[0053] It will be appreciated that the controller can be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single chip computer, an ARM (Acorn RISC Machine), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. The controller can also be any conventional processor, controller, microcontroller, or state machine. The controller can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP and / or any other such configuration, or a combination of one or more of a microcontroller unit (MCU), a field-programmable gate array (FPGA), a system on chip (SoC).

[0054] It will be appreciated that the memory of the robot 10 in the embodiments of the present application includes, but is not limited to, one or more of the following devices: FLASH, NAND, vertical NAND (VNAND), NOR, resistive random access memory (RRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), spin-transfer torque random access memory (STT-RAM), etc.

[0055] It should be noted that, according to the task to be completed, in addition to the above functional modules, one or more other different functional modules (such as a water storage tank, a cleaning device, etc.) can be mounted on the main body of the cleaning robot 10 to cooperate with each other to perform corresponding tasks.

[0056] Some interactive control methods for cleaning robots known to the present inventors have a single interactive mode, for example, most of them are controlled interactively through keys, applications, or remote controls. In addition, the interactive functions are single, mainly realizing mechanical operations such as cleaning, and cannot monitor the environment and provide personalized care services.

[0057] To solve the above problems, the embodiments of the present application provide an interactive control method and an electronic device. First, an image set is obtained, the image set including images collected by a cleaning robot in a preset period of time (for example, for several consecutive days) in an environment in which the cleaning robot is located. The images in the image set are identified to determine a degree of dirtiness and disorder of the environment compared with a standard state. Based on the degree of dirtiness and disorder, it is determined whether an abnormal state (for example, a user is busy or in a low mood, etc.) occurs. If the abnormal state occurs, the cleaning robot is controlled to actively start an interactive action.

[0058] In this embodiment, by visually monitoring the environment ground, a set of images reflecting the degree of dirtiness and disorder of the environment ground is collected. The set of images is identified to extract key data (such as the number of garbage, the degree of disorder of the placement of articles, etc.) reflecting the degree of dirtiness and disorder, and a comprehensive analysis is performed on the key data. In the case where an abnormal state is determined, the cleaning robot is controlled to start an interactive action to provide a humanized care service, for example, through an interactive mode such as a voice greeting or an app pop-up reminder to inquire whether the user needs cleaning, provide comfort and encouragement, or remind the user to clean up articles, etc.

[0059] According to the foregoing, it can be understood that the interactive control method provided by the embodiments of the present application can be implemented and executed by an electronic device with computing processing capability, for example, by a server in communication connection with the cleaning robot. In some embodiments, the interactive control method provided by the embodiments of the present application can also be implemented by a cleaning robot including a microphone, for example, by a controller or processor of the cleaning robot.

[0060] The interactive control method provided by the embodiments of the present application will be described below in combination with an exemplary application and implementation of a server provided by the embodiments of the present application. Please refer to Figure 2 , Figure 2 is a flowchart of the interactive control method provided by the embodiments of the present application. It can be understood that the execution subject of the interactive control method can be one or more processors of a server.

[0061] As shown in Figure 2 , the method S100 can specifically include the following steps:

[0062] S10: acquiring a set of images, the set of images including images collected by a cleaning robot in a preset time from an environment where the cleaning robot is located.

[0063] It can be understood that one or more cameras are installed on the cleaning robot, for example, a camera is installed on the side of the body of the cleaning robot, so that the environment can be image collected. The environment is the space where the cleaning robot is located, for example, it can be a living room where a user lives or an office where a user works, etc. The cleaning robot is located on the ground of the environment and can freely walk in the area without obstacles. During the process of staying or stopping, the environment can be photographed at a certain frequency, and the images photographed in a period of time constitute a set of images. For example, the preset time is 48 hours, the cleaning robot photographs the environment once every 4 hours, 12 images are collected in 48 hours, and the 12 collected images constitute a set of images.

[0064] It can be understood that the set of images records the changes of the environment in this period of time, for example, the changes of the ground garbage, the changes of the placement of daily necessities or furniture, etc., so as to reflect the degree of dirtiness and disorder of the ground of the environment.

[0065] In this embodiment, based on the cleaning robot being in communication connection with the server, the cleaning robot sends the image set collected to the server, so that the server can obtain the image set reflecting the degree of change of the ground of the environment.

[0066] S20: identifying the images in the image set and determining the degree of change of the environment compared with its standard state.

[0067] The standard state of the environment refers to the condition of the environment under normal circumstances, for example, the condition of the ground, the condition of the daily necessities, or the condition of the furniture, etc. Taking user A as an example, if user A pays attention to the environment, he will not throw garbage, shoes, socks and other items on the ground, and will place the daily necessities and furniture neatly on the table or sofa, so the clean and tidy ground, the neatly placed daily necessities, and the neatly placed furniture are the standard state of user A. Taking user B as an example, if user B pays attention to convenience, he will throw some daily necessities on the ground, for example, he will directly put shoes and socks in the entrance area, and will place the daily necessities on the table or sofa randomly, and will place the furniture randomly, so the scattered daily necessities on the ground, the randomly placed daily necessities, and the randomly placed furniture are the standard state of user B. It can be seen that the standard state is personalized and varies from person to person, and the degree of change is calculated with respect to the standard state of the environment, which is suitable for the individuality of the user and is conducive to the accuracy of subsequent judgment of the abnormal state.

[0068] Based on the cleaning robot uploading the image set to the server at regular intervals, the server can determine the standard state of the environment by continuously monitoring the image set, identifying the items in the image, and storing the data representing the standard state (such as image data or statistical data) in the memory, which can be called in subsequent monitoring. Thus, by identifying the images in the image set, the degree of change of the environment compared with its standard state can be determined.

[0069] In some embodiments, the foregoing step S20 specifically comprises:

[0070] S21: performing semantic recognition on the images in the image set to determine the condition of the items in the environment under the current state.

[0071] S22: comparing the condition of the items in the environment under the current state with the condition of the items in the environment under the standard state to determine the degree of change of the environment compared with its standard state.

[0072] In this embodiment, a trained semantic segmentation model can be used to identify the categories of objects in the images within the image set. The semantic segmentation model is trained on a neural network using several images as a training set (each object in the image is labeled), and is an existing technique in the field of machine learning, which will not be described in detail here.

[0073] Those skilled in the art will understand that semantic segmentation models can identify the category to which each pixel region in an image belongs, thereby identifying the category of items in the image, such as sofas, shoes, stools, cabinets, tissues, and garbage bags. Specifically, for each image in the image set, the semantic segmentation model is input, and a semantic segmentation image is output. From the semantic segmentation image, the category of each pixel block can be determined, such as sofas, shoes, stools, cabinets, tissues, and garbage bags. In some embodiments, the location and pose of each item can be determined based on visual positioning and navigation functions. Those skilled in the art will understand that visual positioning and navigation functions are prior art and will not be described in detail here.

[0074] like Figure 3 As shown, Figure 3 This is a diagram illustrating object identification and localization. As can be understood, the semantic segmentation model can identify the types of objects in a single frame of an image. Then, based on visual localization capabilities, the location and pose of each object are determined. For example... Figure 3 As shown in the image, items such as shoes, socks, and watches were identified and located on the ground.

[0075] Because the images in the image set are acquired in a temporal sequence, the degree of change in dirtiness or disorder compared to its standard state can be analyzed and determined. Specifically, the state of items in the current state of the environment is compared with the state of items in the standard state to determine the degree of change in dirtiness or disorder.

[0076] Therefore, by comparing the condition of items at different acquisition times in the current image set with the condition of items under standard conditions, the degree of change in dirtiness can be determined. In some embodiments, the degree of change in dirtiness includes the degree of change in the amount of litter on the ground, the degree of change in the arrangement of daily necessities, or the degree of change in the arrangement of furniture, etc.

[0077] In some embodiments, the items include the type and quantity of trash on the ground, the position and quantity of everyday items, or the position of furniture.

[0078] The degree of messiness change of the environment compared with the standard state is determined based on comparing the item condition of the environment in the current state with the item condition of the environment in the standard state. Thus, when the item condition includes the type and quantity of garbage on the ground, the pose and quantity of daily necessities, and the pose of furniture, the current state and the standard state are actually compared in the dimensions of the type and quantity of garbage on the ground, the pose and quantity of daily necessities, and the pose of furniture, so that the degree of messiness change includes the degree of type change, the degree of quantity change, the degree of pose change, and the degree of quantity change of daily necessities, and the degree of pose change of furniture.

[0079] The type change of garbage refers to a change in the type of garbage. For example, if the type of garbage in the standard state is a plastic bag, and a new type of garbage, i.e., a paper towel, is detected, it is determined that the type of garbage has changed, and a new type of garbage, i.e., a paper towel, is added. At the same time, the quantity of the new type of garbage, i.e., the paper towel, is monitored. It can be understood that when a user is in a low mood and crying, more paper towel garbage may be generated. Therefore, by setting the item condition to include the type and quantity of garbage, the degree of messiness change can more accurately reflect the state change, which is beneficial to the accuracy of subsequent interactive control.

[0080] The quantity change of garbage refers to a change in the quantity of garbage of the same type in the standard state. In some embodiments, if the type of garbage does not change, but the quantity of garbage changes, the quantity of garbage of the same type is also monitored. For example, if the type of garbage in the standard state is a plastic bag, and the quantity of plastic bags is abnormally increased, it may also indicate that the user is in a busy state and cannot clean up in time. Therefore, by setting the item condition to include the quantity of garbage, the degree of messiness change can more accurately reflect the state change, which is beneficial to the accuracy of subsequent interactive control.

[0081] The pose change of daily necessities refers to a change in the placement position or placement form of daily necessities. For example, a pair of shoes is placed side by side near a shoe cabinet in the standard state. If the two shoes of the pair of shoes are placed in different areas or are placed far apart in the same area in the current state, it indicates that the pose of the shoes has changed. It can be understood that if the two shoes of the same pair of shoes are placed far apart, it may also indicate that the user is in an angry state. Therefore, by setting the item condition to include the pose of daily necessities, the degree of messiness change can more accurately reflect the state change, which is beneficial to the accuracy of subsequent interactive control.

[0082] The quantity change of daily necessities refers to a change in the quantity of daily necessities. For example, there are 5 pairs of shoes, socks, and other daily necessities on the ground in the standard state, and there are 10 pairs of shoes, socks, and other daily necessities on the ground in the current state, indicating that the quantity of daily necessities has changed. It can be understood that if the quantity of daily necessities scattered on the ground in the environment is monitored to increase, it is possible that the user is in a busy state. Therefore, by setting the article condition to include the quantity of daily necessities, the mess change degree can more accurately reflect the state change, which is beneficial to the accuracy of subsequent interactive control.

[0083] The pose change of furniture refers to a change in the placement position or placement form of furniture. For example, the chairs are neatly placed around the table in the standard state, and the chairs are tilted and placed at the entrance of the kitchen in the current state, indicating that the pose of the furniture has changed. It can be understood that if a large change in the pose of furniture in the environment is monitored, it is possible that the user is in a busy state and does not have time to return the furniture. Therefore, by setting the article condition to include the pose of furniture, the mess change degree can more accurately reflect the state change, which is beneficial to the accuracy of subsequent interactive control.

[0084] In this embodiment, by setting the article condition to include the type and quantity of garbage on the ground, the pose and quantity of daily necessities, or the pose of furniture, the type and quantity of garbage on the ground, the pose and quantity of daily necessities, or the pose of furniture are compared and analyzed from these dimensions, so that the mess change degree relative to the standard state includes the type change and quantity change of garbage, the pose change and quantity change of daily necessities, and the pose change of furniture, which can more accurately reflect the state change and is beneficial to the accuracy of subsequent interactive control.

[0085] In some embodiments, the foregoing step S22 specifically includes:

[0086] The article conditions in the current state and the standard state are respectively counted to determine the abnormal quantity as the mess change degree, wherein the abnormal quantity includes the growth quantity of garbage, the growth quantity of daily necessities, the quantity of daily necessities that have changed in pose, or the quantity of furniture that has changed in pose.

[0087] In this embodiment, the article conditions used for comparing and analyzing the changes between the current state and the standard state are numerically counted to determine the abnormal quantity, which is used to represent the mess change degree. Thus, whether an abnormal state occurs can be determined according to the size of the abnormal quantity and its change trend.

[0088] Based on the article condition including the type and quantity of garbage on the ground, the pose and quantity of daily necessities, or the pose of furniture, these dimensions are counted to determine the abnormal quantity, so that the abnormal quantity includes the growth quantity of garbage, the growth quantity of daily necessities, the quantity of daily necessities that have changed in pose, or the quantity of furniture that has changed in pose.

[0089] In this embodiment, the dirty and messy change degree is represented by the abnormal quantity by numerically counting the state of the articles, and the abnormal quantity is set to include the increase in the amount of garbage, the increase in the amount of daily articles, the number of changes in the pose, or the number of changes in the pose of furniture, so that the abnormal quantity can more accurately reflect the state change, which is beneficial to the accuracy of subsequent interactive control.

[0090] S30: Determine whether an abnormal state occurs based on the dirty and messy change degree.

[0091] It can be understood that the dirty and messy change degree includes changes in the state of ground garbage, changes in the state of daily articles, or changes in the state of furniture, etc. These dirty and messy change degrees are changes based on the standard state, so that whether an abnormal state occurs can be determined based on the dirty and messy change degree. For example, if a large difference from the standard state is detected, it is determined that an abnormal state occurs. It can be understood that the standard state is different for different people and has individualization, and the dirty and messy change degree is calculated relative to the environmental standard state. Due to the relativity of the dirty and messy change degree, the judgment result of the abnormal state is more accurate.

[0092] In some embodiments, the dirty and messy change degree is represented by the abnormal quantity, so that the abnormal quantity can be compared with a threshold value, and if the threshold value is exceeded, it is determined that an abnormal state occurs.

[0093] In some embodiments, the foregoing step S30 specifically includes:

[0094] S31: Determine whether an abnormal state occurs according to the abnormal quantity, the duration for which the abnormal quantity exceeds the quantity threshold value, and the change trend of the abnormal quantity.

[0095] It can be understood that the image set includes a plurality of images taken at different times, which has a time sequence, so that for each shooting time, it has an abnormal quantity. Therefore, the image set corresponds to an abnormal quantity sequence.

[0096] The abnormal quantity can be the mean or mode of the abnormal quantity sequence. It can be understood that the smaller the mean or mode, the closer to the standard state; the larger the mean or mode, the more likely an abnormal state occurs.

[0097] The duration that the abnormal quantity exceeds the quantity threshold can be determined by comparing each value in the abnormal quantity sequence with the quantity threshold. It can be understood that the shorter the duration, the more timely the user cleans up the environment; the longer the duration, the more likely the user is in an abnormal state such as being busy or in a low mood. In some embodiments, the change trend of the abnormal quantity can be represented by a numerical value, for example, 1 represents an increasing trend, 0 represents no change, and -1 represents a decreasing trend. It can be understood that if the change trend of the abnormal quantity is increasing, it means that the degree of mess in the environment is further deteriorating, and the user is likely to be in an abnormal state such as being busy or in a low mood; if the change trend of the abnormal quantity is decreasing, it means that the degree of mess in the environment is improving.

[0098] In this embodiment, for the abnormal quantity sequence, the abnormal state can be accurately determined from the three aspects of the abnormal quantity, the duration that the abnormal quantity exceeds the quantity threshold, and the change trend of the abnormal quantity.

[0099] In some embodiments, the foregoing step S31 specifically comprises:

[0100] S311: The abnormal quantity, the duration, and the change trend are weighted to obtain an abnormal score.

[0101] S312: If the abnormal score is greater than or equal to a score threshold, it is determined that an abnormal state occurs.

[0102] For example, the abnormal quantity A is configured with a weight α, the duration B is configured with a weight β, and the change trend C is configured with a weight γ, and then the abnormal score = A*α+B*β+C*γ. In some embodiments, the weights α, β, and γ are different, for example, α>β>γ, so that the abnormal quantity and the duration have a greater weight, and the abnormal score is more accurate.

[0103] The score threshold is a threshold for determining whether an abnormal state occurs. When the abnormal score is greater than or equal to the score threshold, it is determined that an abnormal state occurs.

[0104] In this embodiment, by weighting the abnormal quantity, the duration, and the change trend, these indicators are converted into an abnormal score, which can accurately monitor the abnormal state and the algorithm is simple and effective.

[0105] S40: If an abnormal state occurs, the cleaning robot is controlled to start an interactive action.

[0106] When it is determined that an abnormal state occurs, it is possible that the user is in a busy or low mood state. In this embodiment, the server controls the cleaning robot to actively initiate an interactive action to provide personalized services such as comfort, encouragement or reminders for the user. The initiation of the interactive action refers to triggering the interactive action. For example, when the interactive action is a voice greeting, the cleaning robot plays a pre-set greeting speech after receiving a signal for initiating the interactive action.

[0107] In some embodiments, the foregoing step S40 specifically comprises: controlling the cleaning robot to initiate a voice greeting or controlling an application associated with the cleaning robot to initiate a pop-up reminder; or, calling audio data and controlling the cleaning robot to play the audio data.

[0108] For example, when the server monitors that an abnormal state occurs, the cleaning robot is controlled to initiate a voice greeting as follows: "Master, I have noticed that you have been busy recently, do you need me to clean for you". It can be understood that the user will make a voice feedback after receiving the voice greeting, such as "I have been busy recently", "I feel a little depressed recently" or "I have solved the problem, thank you for your concern" and the like.

[0109] For another example, the server can also control the application associated with the cleaning robot to initiate a pop-up reminder. It can be understood that the pop-up content can be text or pattern, expressing concern. Generally, the application associated with the cleaning robot is downloaded and installed in the mobile terminal (such as a smart phone, a tablet or a smart watch and the like) of the user. When the pop-up reminder is initiated, the user can click the pop-up reminder to make a corresponding feedback. For example, the pop-up reminder includes two options reflecting "I need concern" and "Thank you, I am fine", and the user can select the options to make a feedback.

[0110] In some embodiments, the server can also call audio data on the cloud platform and control the cleaning robot to play the audio data. The audio data can be a data packet of a related audio device, such as music, cross-talk or sketch audio data. Specifically, the server sends corresponding audio data to the cleaning robot, so that the cleaning robot plays the audio data to provide a musical atmosphere and calm the user's mood.

[0111] In some embodiments, the interactive mode can include actively sweeping garbage and cleaning the ground. In some embodiments, if the cleaning robot is configured with a mechanical arm, the interactive mode can also include picking up daily necessities and putting them back to the original area (an area where the daily necessities are usually placed), such as putting socks back to the original area, pushing shoes to a shoe cabinet area and the like. By setting these interactive modes, the cleaning robot is more intelligent and humanized.

[0112] In some embodiments, the method S100 further comprises: determining the interactive mode according to the historical feedback record.

[0113] Here, the historical feedback record is the cumulative user feedback in a period of time. It can be understood that the historical feedback record can represent the user's preferred caring manner, such as voice greeting, pop-up reminder, or music program, etc.

[0114] Therefore, according to the historical feedback record of the user, the user's preferred caring interaction manner can be determined. For example, if the user often feeds back "play a song for me", it is determined that the interaction manner is "playing music". In addition, according to the historical feedback record of the user, it can also be determined whether to start the interaction to reduce the interference to the user. For example, the user often feeds back "thank you, I don't need it", and the interaction is stopped from being started to avoid interfering with the user.

[0115] In this embodiment, the interaction manner is determined according to the historical feedback record, so that the caring service is more humanized, can be continuously upgraded, and better product experience and service can be provided.

[0116] In summary, the interaction control method provided by the embodiments of the present application first acquires an image set, which includes images collected by the cleaning robot in a preset time (for example, for several consecutive days) in the environment where the cleaning robot is located. The images in the image set are identified to determine the degree of dirtiness change of the environment compared with the standard state. Based on the degree of dirtiness change, it is determined whether an abnormal state (for example, the user is busy or in a low mood, etc.) occurs. If the abnormal state occurs, the cleaning robot is controlled to actively start the interaction action.

[0117] In this embodiment, the ground of the environment is visually monitored to collect an image set that can reflect the degree of dirtiness change of the ground of the environment. The image set is identified and processed to extract key data (for example, the number of garbage, the degree of disorder of the placement of articles, etc.) reflecting the degree of dirtiness change. These key data are comprehensively analyzed, and in the case where it is determined that an abnormal state occurs, the cleaning robot is controlled to start the interaction action to provide humanized caring service, for example, through the interaction manner of voice greeting or app pop-up reminder, to inquire whether the user needs cleaning, to provide comfort and encouragement, or to remind the user to clean the articles, etc.

[0118] The embodiments of the present application also provide an electronic device. Please refer to Figure 4 , Figure 4 is a hardware structure schematic diagram of an electronic device provided by the embodiments of the present application. In some embodiments, the electronic device is a cleaning robot. In some embodiments, the electronic device can also be a smart device, such as a server, etc., which is in communication connection with the cleaning robot.

[0119] As Figure 4 shown, the electronic device 300 includes at least one processor 301 and a memory 302 in communication connection. Figure 4The electronic device 300 can be connected to a bus, and a processor is taken as an example.

[0120] The processor 301 is configured to provide computing and control capabilities to control the electronic device 300 to perform corresponding tasks, for example, to control the electronic device 300 to perform the interaction control method in any of the above method embodiments, which includes: obtaining an image set including images collected by a cleaning robot in a preset time (for example, for several consecutive days) in an environment in which the cleaning robot is located. The images in the image set are identified to determine the degree of dirtiness change of the environment compared with the standard state. Based on the degree of dirtiness change, it is determined whether an abnormal state (for example, the user is busy or in a low mood, etc.) occurs. If the abnormal state occurs, the cleaning robot is controlled to actively start an interactive action.

[0121] In this embodiment, the image set reflecting the degree of dirtiness change of the environment ground is collected by visual monitoring of the ground of the environment. The image set is identified and processed to extract key data (for example, the number of garbage, the degree of messy arrangement of articles, etc.) reflecting the degree of dirtiness change. These key data are comprehensively analyzed, and in the case where it is determined that an abnormal state occurs, the cleaning robot is controlled to start an interactive action to provide a humanized care service, for example, through an interactive mode such as a voice greeting or an app pop-up reminder, to inquire whether the user needs cleaning, to provide comfort and encouragement, or to remind the user to clean up articles, etc.

[0122] The processor 301 can be a general processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0123] The memory 302, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the interaction control method in the embodiments of the present application. The processor 301 can implement the interaction control method in any of the above method embodiments by running the non-transitory software programs, instructions and modules stored in the memory 302, and details are not repeated here to avoid repetition.

[0124] Specifically, the memory 302 can include a volatile memory (VM), such as a random access memory (RAM); the memory 302 can also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD) or other non-transitory solid-state storage device; the memory 302 can also include a combination of the above types of memories.

[0125] In the embodiments of the present application, the memory 302 can also include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0126] In the embodiments of the present application, the electronic device 300 can also have a wired or wireless network interface, a keyboard and an input / output interface, etc. to perform input / output, and the electronic device 300 can also include other components for implementing device functions, which are not described here.

[0127] The embodiments of the present application also provide a computer readable storage medium, such as a memory including program codes, which can be executed by a processor to complete the interaction control method in the above embodiments. For example, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0128] The embodiment of the present application further provides a computer program product, which comprises one or more program codes stored in a computer readable storage medium. The processor of the electronic device reads the program codes from the computer readable storage medium, and the processor executes the program codes to complete the method steps of the interactive control method provided in the above embodiment.

[0129] It should be noted that the above-described apparatus embodiments are merely illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course, can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the above embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0131] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; under the idea of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of the different aspects of the present application as described above. In order to be brief, they are not provided in detail; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An interactive control method, characterized in that, include: Acquire an image set, which includes images collected by the cleaning robot of its environment within a preset time period; Identify the images in the image set and determine the degree of dirtiness or disorder of the environment compared to its standard state; Based on the degree of change in dirt and disorder, determine whether an abnormal state has occurred; If an abnormal state occurs, the cleaning robot is controlled to initiate interactive actions. The abnormal state includes a busy state or a depressed state. The control of the cleaning robot to initiate interactive actions includes: Control the cleaning robot to initiate a voice greeting or control the application associated with the cleaning robot to initiate a pop-up reminder; or, The system calls upon audio data to control the cleaning robot to play the audio data.

2. The method according to claim 1, characterized in that, Identifying images in the image set and determining the degree of dirtiness or disorder of the environment compared to its standard state includes: Semantic recognition is performed on the images in the image set to determine the item status of the environment in the current state; The current state of the environment is compared with its state under standard conditions to determine the degree of change in the environment's dirtiness compared to its standard state.

3. The method according to claim 2, characterized in that, The items described include the type and quantity of litter on the ground, the position and quantity of everyday items, or the position of furniture.

4. The method according to claim 3, characterized in that, The step of comparing the state of items in the current environment with that in a standard state to determine the degree of change in the environment's dirtiness compared to its standard state includes: The status of items under the current state and the standard state are statistically analyzed to determine the number of anomalies as the degree of soiling change. The number of anomalies includes the increase in the amount of garbage, the increase in the amount of daily necessities, the number of items that have changed position, or the number of furniture items that have changed position.

5. The method according to claim 4, characterized in that, Based on the degree of soiling change, determine whether an abnormal state has occurred, including: Whether an abnormal state has occurred is determined based on the number of abnormalities, the duration for which the number of abnormalities exceeds the threshold, and the trend of the number of abnormalities.

6. The method according to claim 5, characterized in that, The step of determining whether an abnormal state has occurred based on the number of abnormalities, the duration for which the number of abnormalities exceeds a threshold, and the trend of the number of abnormalities includes: The anomaly score is obtained by weighting the number of anomalies, the duration, and the trend of change. If the abnormal score is greater than or equal to the score threshold, then an abnormal state is determined to have occurred.

7. The method according to claim 1, characterized in that, The method further includes: Based on historical feedback records, determine the interaction method.

8. An electronic device, characterized in that, include: At least one processor; A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the interactive control method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer device to perform the interactive control method as described in any one of claims 1-7.

Images