Intelligent equipment control method, intelligent equipment control device, medium and electronic equipment

Abstract

The invention provides a control method of intelligent equipment, a control device of the intelligent equipment, an arrangement robot, a computer readable storage medium and electronic equipment. The method comprises the following steps: acquiring environment data of a working scene; identifying a to-be-processed object in the working scene according to the environment data; determining a target processing mode in the plurality of processing modes according to a preset mode judgment strategy; the plurality of processing modes at least comprise a first processing mode and a second processing mode, the first processing mode is a mode of passively processing an object, and the second processing mode is a mode of actively processing the object; and controlling the intelligent equipment to process the to-be-processed object by adopting the target processing mode. According to the method, a proper mode can be adaptively selected to process the to-be-processed object according to the environment state.

Description

Technical Field

[0001] This disclosure relates to the field of smart homes, and more specifically, to a smart device control method, a smart device control device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] In the field of intelligent equipment technology, such as smart cleaning equipment, intelligent devices can not only perform basic tasks like cleaning, but also manage items in a given environment, such as storage. However, existing intelligent devices have relatively simple response and processing methods for different work scenarios. They can usually only execute operations according to fixed procedures and cannot adapt to the actual environment. They also lack the ability to select appropriate processing modes based on reasonable strategies, which makes it difficult to meet the needs of intelligent devices for scene adaptability and automation levels, thus limiting the effectiveness of intelligent applications. Summary of the Invention

[0003] The purpose of this disclosure is to provide a control method, control device, computer-readable storage medium, and electronic device for intelligent devices, in order to improve the adaptability of intelligent devices to different scenarios and the level of automation. The specific solution is as follows: According to specific embodiments of this disclosure, in a first aspect, this disclosure provides a control method for an intelligent device, the method comprising: acquiring environmental data of a work scene; identifying an object to be processed in the work scene based on the environmental data; determining a target processing mode among multiple processing modes according to a preset mode judgment strategy; the multiple processing modes including at least a first processing mode and a second processing mode, wherein the first processing mode is a mode for passively processing objects, and the second processing mode is a mode for actively processing objects; and controlling the intelligent device to process the object to be processed using the target processing mode.

[0004] In one exemplary embodiment of this disclosure, the step of identifying the object to be processed in the work scene based on the environmental data includes: inputting the environmental data into an object processing model for processing to obtain an object recognition result, and determining the object to be processed in the work scene based on the object recognition result.

[0005] In one exemplary embodiment of this disclosure, the object recognition result includes multiple candidate objects and object attribute information of each candidate object; determining the object to be processed in the work scene based on the object recognition result includes: determining the object to be processed in the work scene from the multiple candidate objects according to the object attribute information of each candidate object.

[0006] In one exemplary embodiment of this disclosure, determining the target processing mode among multiple processing modes according to a preset mode judgment strategy includes: detecting the user's location in a work scenario; determining the first processing mode as the target processing mode when the user's location is detected to be in the work scenario; and determining the second processing mode as the target processing mode when the user's location is detected to be not in the work scenario.

[0007] In one exemplary embodiment of this disclosure, determining the target processing mode among multiple processing modes according to a preset mode judgment strategy includes: obtaining the user's historical preference information; the historical preference information includes the object type of the historical processing object corresponding to multiple processing modes; and determining the target processing mode among multiple processing modes based on the historical preference information and the object type of the object to be processed.

[0008] In one exemplary embodiment of this disclosure, determining the target processing mode among multiple processing modes according to a preset mode judgment strategy includes: determining object feature data of the object to be processed; determining the target processing mode among multiple processing modes based on the matching result of the object feature data of the object to be processed and the mode selection rule; wherein, the mode selection rule includes multiple processing modes and object feature data corresponding to each processing mode.

[0009] In one exemplary embodiment of this disclosure, when the target processing mode is determined to be a first processing mode, the step of using the target processing mode to control the smart device to process the object to be processed includes: in response to detecting that the smart device has moved to the user's location, sending an inquiry prompt message about the object to be processed to the user, and receiving a processing instruction from the user regarding the inquiry prompt message; and performing corresponding operations on the object to be processed according to the processing instruction.

[0010] In one exemplary embodiment of this disclosure, sending an inquiry prompt message to the user about the processing object includes: determining whether the user is in a busy state; if the user is not in a busy state, sending an inquiry prompt message to the user about the processing object.

[0011] In one exemplary embodiment of this disclosure, when the target processing mode is determined to be a second processing mode, the step of using the target processing mode to control the smart device to process the object to be processed includes: controlling the smart device to perform automatic processing operations on the object to be processed.

[0012] In one exemplary embodiment of this disclosure, after controlling the smart device to perform automatic processing operations on the object to be processed, the method further includes: sending the object processing result to the user via voice information and / or prompt information.

[0013] In one exemplary embodiment of this disclosure, sending the object processing result to the user via voice information and / or prompt information includes: determining whether the user is in a busy state; if the user is not in a busy state, sending the object processing result to the user via voice information and / or prompt information.

[0014] In one exemplary embodiment of this disclosure, the method further includes: identifying restricted areas in the work scene based on the environmental data; determining a target processing mode among multiple processing modes based on a preset mode judgment strategy includes: determining the target processing mode among multiple processing modes based on the type of the restricted area and / or the positional relationship between the restricted area and the object to be processed.

[0015] In one exemplary embodiment of this disclosure, determining a target processing mode among multiple processing modes based on the type of the restricted area and / or the positional relationship between the restricted area and the object to be processed includes: determining a first processing mode as the target processing mode when it is determined that the object to be processed is located within the restricted area, or the distance between the object to be processed and the restricted area is less than a first preset distance.

[0016] In one exemplary embodiment of this disclosure, the plurality of processing modes further includes a prohibited processing mode; determining a target processing mode among the plurality of processing modes based on the type of the restricted area and / or the positional relationship between the restricted area and the object to be processed includes: when it is determined that the object to be processed is located within the restricted area, determining the target processing mode as a prohibited processing mode.

[0017] In one exemplary embodiment of this disclosure, determining a target processing mode among multiple processing modes based on the type of the restricted area and / or the positional relationship between the restricted area and the object to be processed includes: determining a first processing mode as the target processing mode when the type of the restricted area is determined to be a non-safe restricted area and the object to be processed is located within the restricted area; and determining a second processing mode as the target processing mode when the type of the restricted area is determined to be a safe restricted area and the object to be processed is located within the restricted area.

[0018] In one exemplary embodiment of this disclosure, controlling the smart device to process the object to be processed includes: controlling a mechanical gripping structure configured in the smart device to grip, move, or place the object to be processed.

[0019] In one exemplary embodiment of this disclosure, the method further includes: adjusting the control parameters of the mechanical gripping structure according to the object feature data of the object to be processed; the control parameters include control force and / or control speed.

[0020] Secondly, this disclosure provides a control device for an intelligent device, the device comprising: an environmental data acquisition module for acquiring environmental data of a work scene; a processing object identification module for identifying objects to be processed in the work scene based on the environmental data; a target mode determination module for determining a target processing mode among multiple processing modes according to a preset mode judgment strategy; the multiple processing modes include at least a first processing mode and a second processing mode, wherein the first processing mode is a passive object processing mode and the second processing mode is an active object processing mode; and an object processing control module for controlling the intelligent device to process the objects to be processed using the target processing mode.

[0021] In one exemplary embodiment of this disclosure, the object recognition module includes: a model processing unit, configured to input the environmental data into an object processing model for processing, obtain object recognition results, and determine the objects to be processed in the work scene based on the object recognition results.

[0022] In one exemplary embodiment of this disclosure, the object recognition result includes multiple candidate objects and object attribute information of each candidate object; the model processing unit includes: a processing object filtering subunit, used to determine the object to be processed in the work scene from the multiple candidate objects according to the object attribute information of each candidate object.

[0023] In one exemplary embodiment of this disclosure, the target mode determination module includes: a user location detection unit, configured to detect a user location in a work scenario; a first determination unit, configured to determine a first processing mode as a target processing mode when the user location is detected to be in the work scenario; and a second determination unit, configured to determine a second processing mode as a target processing mode when the user location is detected to be not in the work scenario.

[0024] In one exemplary embodiment of this disclosure, the target mode determination module includes: a preference information acquisition unit, used to acquire the user's historical preference information; the historical preference information includes the object type of the historical processing object corresponding to multiple processing modes; and a third determination unit, used to determine the target processing mode among multiple processing modes based on the historical preference information and the object type of the object to be processed.

[0025] In one exemplary embodiment of this disclosure, the target pattern determination module includes: a feature data determination unit, used to determine object feature data of an object to be processed; and a fourth determination unit, used to determine a target processing mode among multiple processing modes based on the matching result of the object feature data of the object to be processed and the pattern selection rule; wherein the pattern selection rule includes multiple processing modes and object feature data corresponding to each processing mode.

[0026] In one exemplary embodiment of this disclosure, when the target processing mode is determined to be a first processing mode, the object processing control module includes: a processing instruction receiving unit, configured to, in response to detecting that a smart device has moved to the user's location, send an inquiry prompt message about the processing object to the user, and receive a processing instruction from the user regarding the inquiry prompt message; and a first processing unit, configured to perform corresponding operations on the processing object according to the processing instruction.

[0027] In one exemplary embodiment of this disclosure, the processing instruction receiving unit includes: a first state determination subunit, configured to determine whether the user is in a busy state, and if the user is not in a busy state, to send the user an inquiry prompt message about the processing object.

[0028] In one exemplary embodiment of this disclosure, when the target processing mode is determined to be the second processing mode, the object processing control module includes: a second processing unit, used to control the smart device to perform automatic processing operations on the object to be processed.

[0029] In one exemplary embodiment of this disclosure, after controlling the smart device to perform automatic processing operations on the object to be processed, the device further includes: a processing result sending unit, used to send the object processing result to the user through voice information and / or prompt information.

[0030] In one exemplary embodiment of this disclosure, the processing result sending unit includes: a second state determination subunit, used to determine whether the user is in a busy state, and if the user is not in a busy state, to send the object processing result to the user via voice information and / or prompt information.

[0031] In one exemplary embodiment of this disclosure, the apparatus further includes: a restricted area identification module, configured to identify restricted areas in the work scene based on the environmental data; and a target mode determination module, including: a fifth determination unit, configured to determine a target processing mode among multiple processing modes based on the type of the restricted area and / or the positional relationship between the restricted area and the object to be processed.

[0032] In one exemplary embodiment of this disclosure, the fifth determining unit is configured to determine the first processing mode as the target processing mode when it is determined that the object to be processed is located within the restricted area, or the distance between the object to be processed and the restricted area is less than a first preset distance.

[0033] In one exemplary embodiment of this disclosure, the fifth determining unit is configured to determine the target processing mode as a prohibited processing mode when it is determined that the object to be processed is located within the restricted area.

[0034] In one exemplary embodiment of this disclosure, the fifth determining unit is configured to determine the first processing mode as the target processing mode when it is determined that the type of the restricted area is a non-safe restricted area and the object to be processed is located within the restricted area; and to determine the second processing mode as the target processing mode when it is determined that the type of the restricted area is a safe restricted area and the object to be processed is located within the restricted area.

[0035] In one exemplary embodiment of this disclosure, the object processing control module includes: a structure control unit, used to control a mechanical gripping structure configured in a smart device to grip, move, or place the object to be processed.

[0036] In one exemplary embodiment of this disclosure, the device further includes: a control parameter adjustment module, used to adjust the control parameters of the mechanical gripping structure according to the object feature data of the object to be processed; the control parameters include control force and / or control speed.

[0037] Thirdly, this disclosure provides a sorting robot, comprising: a body; and a controller disposed in the body; the controller being configured to execute the control method of the aforementioned intelligent device.

[0038] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described control method for an intelligent device.

[0039] Fifthly, this disclosure provides an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described control method of an intelligent device by executing the executable instructions.

[0040] The exemplary embodiments disclosed herein have the following beneficial effects: The system acquires environmental data of the work scenario; identifies objects to be processed in the work scenario based on the environmental data; determines a target processing mode from multiple processing modes according to a preset mode judgment strategy; the multiple processing modes include at least a first processing mode and a second processing mode, where the first processing mode is for passively processing objects and the second processing mode is for actively processing objects; and controls the intelligent device to process the objects to be processed using the target processing mode. On the one hand, this exemplary embodiment, by acquiring environmental data of the work scenario, can perceive scenario information and accurately identify objects to be processed, enabling the intelligent device to locate the objects to be processed according to the actual situation of different scenarios, enhancing its adaptability to diverse work scenarios. On the other hand, this exemplary embodiment, by determining a target processing mode from multiple processing modes based on a preset mode judgment strategy, avoids the problem of the intelligent device having a single processing method and requiring frequent user intervention, allowing the intelligent device to autonomously or on-demand select a processing method according to scenario needs, improving the level of automation and user convenience.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 This schematically illustrates a flowchart of a control method for a smart device in this exemplary embodiment; Figure 2 This schematically illustrates a flowchart of a control method for a smart device in this exemplary embodiment; Figure 3 This schematic diagram illustrates a structural block diagram of a control device for a smart device in this exemplary embodiment. Figure 4 An electronic device for implementing the above method is illustrated in this exemplary embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this disclosure clearer, the disclosure will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0044] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. The singular forms “a,” “the,” and “the” as used in the embodiments of this disclosure and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0045] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0046] It should be understood that although the terms first, second, third, etc., may be used to describe... in the embodiments of this disclosure, these... should not be limited to these terms. These terms are only used to distinguish... For example, first... may also be referred to as second... without departing from the scope of the embodiments of this disclosure, and similarly, second... may also be referred to as first...

[0047] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0048] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0049] With the development of smart homes, smart devices such as robot vacuums and robot mops have been widely used. However, existing smart devices can usually only respond to basic user commands such as "start cleaning" or "stop cleaning". They cannot generate personalized service strategies based on user needs, nor can they proactively sense special scenarios. Furthermore, they lack in-depth interaction with users to adjust strategies and convenient device maintenance guidance, making it difficult to meet users' needs for high-quality, user-friendly smart devices.

[0050] Based on this, the exemplary embodiments of this disclosure provide a smart device control method that can be applied to user equipment, in which machine learning models, such as multimodal large models, can be configured. LargeModels (MLMs) process data acquired by smart devices using multimodal large models. Leveraging the ability of MLMs to simultaneously process and understand multiple data types (such as text, images, audio, and video), data processing can be performed locally on the smart device or online. The MLMs can interact with the user to determine a target processing mode for the object being processed, allowing the smart device to execute corresponding processing procedures based on that mode. Alternatively, this method can be applied to smart devices such as robotic vacuum cleaners and storage robots. These devices can be configured with MLMs to process the data they acquire. The MLMs interact with the user to determine the target processing mode for the object being processed and execute the processing operation. The smart device can implement this method through hardware and / or software. Furthermore, this method can be applied to cloud processors or servers, where MLMs can be configured to interact with user devices and smart devices. They can acquire data collected by the smart device or receive commands input from the user device, process the data, generate corresponding target processing strategies, and send them to the smart device to control the smart device to execute corresponding operations according to the target processing mode. (See reference...) Figure 1 The flowchart illustrates an exemplary embodiment of a smart device control method provided in this disclosure, which may include the following steps S110-S140: Step S110: Obtain environmental data of the work scene.

[0051] Environmental data refers to various types of information that reflect the environmental state of the working scene of a smart device, including the spatial structure, object distribution, and ground conditions of the working scene. In this exemplary embodiment, various sensing modules can be integrated into the smart device, and the sensing modules can collect data from the working scene according to a preset collection frequency and range. After the data collection is completed, the environmental data can be preliminarily sorted and stored to ensure the integrity and availability of the data.

[0052] In practical applications, once the smart device is activated, the sensing module can automatically enter working mode and capture information about the work scene from all angles. For example, it can use LiDAR to scan the spatial layout to determine the room size, furniture position, etc.; and use the image acquisition module to obtain image information such as the ground material and the presence of obstacles. All the collected data will be aggregated to form a complete environmental data set.

[0053] In one exemplary embodiment, the acquisition of environmental data of the work scenario described above may include: Scene sensing data is collected by lidar sensors configured in the cleaning equipment, and map data of the work scene is constructed based on the scene sensing data; and / or Image data of the work scene is collected by a camera module configured in the cleaning equipment.

[0054] A lidar sensor is a sensing device that acquires spatial information about the surrounding environment by emitting laser beams and receiving reflected signals. Scene sensing data refers to the raw data collected by the lidar sensor that reflects the spatial structure, object positions, and distances of the work scene. Map data is map information that intuitively displays the spatial layout of the work scene after processing based on the scene sensing data. In this exemplary embodiment, the lidar sensor configured in the smart device emits a laser beam into the work scene according to a preset scanning angle and frequency. The laser beam reflects off an object, and the sensor receives the reflected signal. Based on the signal propagation time and angle, it calculates the distance and orientation of the object to the sensor, forming scene sensing data. Subsequently, the device's data processing unit performs filtering, stitching, and coordinate transformation on the scene sensing data to remove redundant and interfering data, integrating the effective data to construct map data of the work scene.

[0055] A camera module can be a modular component comprising a camera, an image sensor, and a lens, used to capture visual information of a work scene. Image data refers to image information captured by the camera module that reflects visual features such as the appearance, color, and ground conditions of objects in the work scene. In this exemplary embodiment, the camera module configured in the smart device can continuously capture images of the work scene according to preset shooting angles, resolutions, and frame rates. The image sensor in the camera module converts light signals into electrical signals, which are then processed through noise reduction, color correction, compression, and other methods to form standard format image data.

[0056] Step S120: Identify the objects to be processed in the work scene based on environmental data.

[0057] A typical work scenario may include various objects, such as data cables, trash cans, slippers, crumpled paper, and toys. Objects to be processed refer to those requiring specific operations, such as grasping, cleaning, organizing, or tidying up. For example, trash cans do not require processing and are not considered objects to be processed, while crumpled paper needs to be processed and is therefore considered an object to be processed. Users or the system can pre-configure a list of objects to be processed, or their characteristic information, to facilitate the subsequent determination of which objects can be used as objects to be processed.

[0058] In this exemplary embodiment, after receiving environmental data, the user device or smart device can analyze and process the environmental data through a pre-trained machine learning model, extract feature information from the data, such as the shape, color, and outline of objects, and the boundaries and ranges of regions; and determine objects or regions in the work scene that match the features of the object to be processed based on the extracted feature information. For example, a multimodal large model can compare the extracted feature information with a preset object feature library, or analyze and predict the features, and determine the object to be processed in the work scene based on the processing results.

[0059] Step S130: Based on the preset mode judgment strategy, determine the target processing mode among multiple processing modes; the multiple processing modes include at least a first processing mode and a second processing mode, the first processing mode is the mode of passively processing objects, and the second processing mode is the mode of actively processing objects.

[0060] The preset mode judgment strategy is a pre-defined rule used to select the method of processing the object to be processed. The processing mode is the way the smart device processes the object. The first processing mode can be a mode where the smart device only processes the object after receiving explicit instructions from the user. The second processing mode can be a mode where the smart device can process the object autonomously based on preset rules or historical data without real-time user instructions. It should be noted that the multiple processing modes may also include other processing modes besides the first and second processing modes, such as a prohibited processing mode, i.e., a mode that does not perform any processing.

[0061] In this exemplary embodiment, the user device or cleaning device can first obtain a preset pattern judgment strategy from the storage unit. For example, the multimodal large model configured in the user device or cleaning device can first obtain a judgment strategy based on user location, or a judgment strategy based on user historical preferences, or a combination of multiple judgment strategies. Next, it analyzes the identified object information to be processed, such as object type and location, as well as relevant information about the current working scene, such as whether the user is in the scene and whether there are user operation records. This information is then substituted into the preset pattern judgment strategy for logical judgment. Finally, based on the judgment result, the processing mode suitable for the current situation is selected from multiple processing modes as the target processing mode.

[0062] Step S140: Using the target processing mode, control the intelligent device to process the object to be processed.

[0063] After determining the target processing mode, the intelligent device can be controlled to process the object to be processed. Processing the object to be processed can include cleaning or storing it. For example, the robotic arm, cleaning component, and moving component of the intelligent device can be driven to perform corresponding operations on the identified object to be processed in order to complete the work task.

[0064] It should be noted that this exemplary embodiment can be executed by a smart device. After the smart device determines the target processing mode, it can be directly controlled to perform the corresponding operation. This exemplary embodiment can also be executed by a user device. After the user device determines the target processing mode, for example, a multimodal large model provides a target processing mode for the user. After the user confirms the target processing mode, a trigger instruction can be generated. The user device can send the trigger instruction to the smart device to control the smart device to process the object to be processed.

[0065] Based on the above description, in this exemplary embodiment, environmental data of the work scene is acquired; objects to be processed in the work scene are identified based on the environmental data; a target processing mode is determined from multiple processing modes according to a preset mode judgment strategy; the multiple processing modes include at least a first processing mode and a second processing mode, where the first processing mode is a mode for passively processing objects and the second processing mode is a mode for actively processing objects; the target processing mode is adopted to control the intelligent device to process the objects to be processed. On the one hand, by acquiring environmental data of the work scene, this exemplary embodiment can perceive scene information in real time and accurately identify objects to be processed, enabling the intelligent device to locate the objects to be processed according to the actual situation of different scenes, thus enhancing its adaptability to diverse work scenes; on the other hand, by determining the target processing mode from multiple processing modes based on a preset mode judgment strategy, this exemplary embodiment avoids the problem of the intelligent device having a single processing method and requiring frequent user intervention, allowing the intelligent device to autonomously or on-demand select the processing method according to scene requirements, thereby improving the level of automation and user convenience.

[0066] In an exemplary embodiment, the above-described identification of objects to be processed in a work scene based on environmental data may include: Environmental data is input into the object processing model for processing to obtain object recognition results. Based on the object recognition results, the objects to be processed in the work scene are determined.

[0067] Among them, the object processing model is a model built based on artificial intelligence algorithms, such as deep learning and machine learning, that can analyze input environmental data and output recognition results. The object recognition result refers to the data output by the object processing model after processing the environmental data, which includes information such as the category, attributes, and location of objects or regions in the environment.

[0068] This exemplary embodiment can configure an object processing model in a user device or a smart device to process environmental data. For example, when configuring an object processing model in a user device, environmental data can be collected by a smart device and sent to the user device, and then processed by the object processing model. When configuring an object processing model in a smart device, the environmental data collected by the smart device can be directly input into the object processing model as input data for processing to obtain object recognition results.

[0069] After obtaining the object recognition results, the object processing model can also match each recognized object in the object recognition results, such as building blocks, sofas, coffee tables, slippers, crumpled paper, toys, etc., with the work objectives and filtering rules according to preset work objectives, such as "cleaning the environment" for a robot vacuum cleaner and "tidying up the environment" for a storage robot, as well as the object screening rules, such as filtering out items that need to be cleaned or objects that need to be tidied up. The model determines whether the recognized object meets the conditions for processing, such as whether it belongs to the processing scope related to the work objective or whether it meets the requirements of the filtering rules. Finally, the recognized objects that meet the conditions are determined as the objects to be processed in the work scene.

[0070] In an exemplary embodiment, the object recognition result includes multiple candidate objects and object attribute information for each candidate object; the above-mentioned determination of the object to be processed in the work scene based on the object recognition result may include: Based on the object attribute information of each candidate object, the object to be processed in the work scenario is determined from multiple candidate objects.

[0071] The object recognition result includes multiple candidate objects and object attribute information for each candidate object.

[0072] Candidate objects refer to all possible objects or regions identified by the object processing model from the environmental data; they are the alternative set of objects to be processed. Object attribute information can be various types of data used to describe the characteristics of candidate objects, such as name, category, color, size, shape, material, location, and status, such as whether it is stationary or easily damaged.

[0073] When processing preprocessed environmental data, the object processing model first identifies all distinguishable objects or regions in the environment and marks them as candidate objects. Then, using algorithms such as feature extraction and attribute prediction, it analyzes each candidate object, extracting or predicting its category (e.g., building blocks, sofa, coffee table); color (e.g., red, gray, brown); size (e.g., building blocks with a side length of 5cm, sofa with a length of 2m); shape (e.g., building blocks are cubes, coffee tables are rectangles); and location coordinates. This information constitutes the object attribute information for each candidate object. Finally, the model integrates all candidate objects and their corresponding object attribute information to form a complete object recognition result. For example, the object processing model configured in a robot vacuum cleaner can analyze a living room image... After data processing, four candidate objects are identified: children's building blocks, sofa, coffee table, and rug. The attribute information of each candidate object is output at the same time: children's building blocks (category: objects that need to be organized and stored, color: red, blue, size: 5cm×5cm, shape: cube, position: (100, 200)); sofa (category: furniture, color: gray, size: 200cm×80cm, shape: cuboid, position: (300, 150)); coffee table (category: furniture, color: brown, size: 120cm×60cm, shape: rectangle, position: (200, 180)); rug (category: decorative item, color: off-white, size: 150cm×150cm, shape: square, position: (150, 250)).

[0074] Then, based on the attribute information of each candidate object, the object to be processed can be selected from all the candidate objects. During the screening, the working requirements of the smart device and the preset screening conditions can be combined. For example, the preset screening conditions for the robot vacuum cleaner are that the category belongs to the objects that need to be organized or the garbage that needs to be cleaned, and the location is within the cleaning path. The model compares the attribute information of the candidate objects (children's building blocks, sofa, coffee table, carpet). Among them, the children's building blocks are classified as objects that need to be organized and the location is within the cleaning path, which meets the conditions and can be identified as objects to be processed; the sofa and coffee table are classified as furniture, and the carpet is classified as decorative items, neither of which belong to the category to be processed.

[0075] In an exemplary embodiment, determining the target processing mode from multiple processing modes according to a preset mode determination strategy may include: Detecting user location in a work environment; When it is determined that the user's location is in a work environment, the first processing mode is set as the target processing mode. When it is determined that the user's location is not in a working environment, the second processing mode is set as the target processing mode.

[0076] User location refers to the specific location or coordinates of a user within the spatial area currently in operation of a smart device. Detecting user location can be the process by which a user device or smart device obtains the user's location information within the work environment through specific detection methods. For example, it can be used to locate the user by acquiring voice data of the user speaking in the work environment, or by capturing scene images or using infrared sensors, etc.

[0077] When a user is detected within the designated space, the first processing mode is selected, meaning the device waits for user instructions before processing the object. When a user is detected outside the smart device's current operating space, the second processing mode is selected, meaning the smart device autonomously processes the object without waiting for real-time user instructions.

[0078] In this exemplary embodiment, the determination of the mode can also be achieved through a machine learning model. For example, a multimodal large model can be configured in the user device or smart device, which can take the user's location result as input data and output a determined target processing mode. Alternatively, the user can be asked to tell the model their location through an interaction process, and then the target processing mode can be determined based on the user's location. The target processing mode can be presented to the user in an interactive form, such as prompting the user through voice or text, "Please confirm that the first processing mode / second processing mode is determined as the target processing mode."

[0079] In an exemplary embodiment, determining the target processing mode from multiple processing modes according to a preset mode determination strategy may include: Obtain the user's historical preference information; the historical preference information includes the object types of historical processing objects corresponding to multiple processing modes; Based on historical preference information and the object type of the object to be processed, the target processing mode is determined from multiple processing modes.

[0080] The user's historical preference information can be a data set reflecting the user's processing habits, recording the processing modes selected by the user for different types of objects in the past. Object types can be categorized based on the attributes of the objects to be processed, such as purpose, material, and category. After each processing of an object, the user device or smart device can automatically record relevant information, including the object type (e.g., building blocks, slippers, paper balls), the user's selected processing mode, the processing time, and feedback on the processing result, such as user satisfaction. This recorded information is stored in a preset format to form a historical preference information database. When historical preference information is needed, the historical records related to the current object can be retrieved from the storage area, filtering out information containing the processing mode and corresponding object type.

[0081] By comparing the object type of the current object to be processed with the processing modes corresponding to different object types in the historical preference information, the system can select the processing mode that matches the user's past habits as the target processing mode. For example, if the object type of the current object to be processed is building blocks, and the historical preference information shows that the second processing mode was used when processing building blocks in the past, then the second processing mode can be determined as the target processing mode based on the historical preference. Alternatively, if the first and second processing modes have been used in the historical records, but the second processing mode was selected most frequently, then the second processing mode can also be determined as the target processing mode. If the current object to be processed is a plush toy, and there is no record of this type in the historical preference information, the robot vacuum can default to the first processing mode and ask the user for processing methods, etc.

[0082] In this exemplary embodiment, the processing model configured in the user device or smart device can directly process based on pre-stored historical preference information and the object type of the current object to be processed obtained from analysis, and determine the target processing mode among multiple processing modes.

[0083] In an exemplary embodiment, determining the target processing mode from multiple processing modes according to a preset mode determination strategy may include: Determine the object characteristic data of the object to be processed; Based on the matching results between the object feature data of the object to be processed and the pattern selection rules, the target processing mode is determined from multiple processing modes. The mode selection rules include multiple processing modes and object feature data corresponding to each processing mode.

[0084] The object characteristic data of the object to be processed refers to attribute data that can identify the object and is related to the selection of the processing mode, such as the object's size, weight, material, hazard, fragility, and value. The mode selection rule refers to the preset rules that associate different processing modes with specific object characteristic data. These rules are used to determine the corresponding processing mode based on the characteristics of the object to be processed. The mode selection rule can include the correspondence between processing modes and object characteristic data. For example, the second processing mode (active processing) corresponds to characteristic data such as weight < 100g, material is plastic / fabric, low fragility, and size < 10cm; the first processing mode (passive processing) corresponds to characteristic data such as weight ≥ 100g, material is glass / metal, high fragility, and size ≥ 10cm. The matching result refers to the degree of fit between the characteristic data of the object to be processed and the characteristic data corresponding to each processing mode in the rules. When the characteristic data of the object to be processed matches the object characteristic data corresponding to the first processing mode, it indicates a successful match with the first processing mode; when the characteristic data of the object to be processed matches the object characteristic data corresponding to the second processing mode, it indicates a successful match with the second processing mode.

[0085] In this exemplary embodiment, a multimodal large model can be configured in the user equipment or smart terminal to determine the target processing mode. This model can analyze environmental data to identify the object to be processed and parse the object feature data of the object. It can also retrieve pre-configured mode selection rules and match them with the object feature data of the object to be processed to obtain a matching result. Based on the matching result, the target processing mode is determined from multiple processing modes. For example, the object feature data of a building block is a weight of 30g (<100g), a plastic material, low fragility, and a size of 4cm (<10cm), which perfectly matches the feature data corresponding to the second processing mode in the mode selection rules. Therefore, the second processing mode can be determined as the target processing mode. If the object to be processed is a glass water cup with feature data of a weight of 200g (≥100g), a glass material, high fragility, and a size of 15cm (≥10cm), which matches the feature data of the first processing mode, the first processing mode is determined as the target processing mode. It should be noted that when the processing mode in the mode selection rule corresponds to multiple object feature data, such as the first processing mode corresponding to multiple conditions such as size, material, and fragility, the object feature data of the object to be processed can be set to match the mode selection rule completely or partially, and both can be considered as a successful match.

[0086] In an exemplary embodiment, when the target processing mode is determined to be a first processing mode, the above-mentioned use of the target processing mode to control the smart device to process the object to be processed may include: In response to detecting that a smart device has moved to the user's location, the system sends a prompt message to the user about the object to be processed and receives the user's processing instructions in response to the prompt message. Perform corresponding operations on the object to be processed according to the processing instructions.

[0087] When the target processing mode is passive processing mode, the smart device can be moved to the user's location first, facilitating interactive communication and inquiry between the user and the smart device. The user's location can be determined in various ways, such as voice data analysis or sensor data analysis, and this disclosure does not specifically limit this. Inquiry prompts refer to messages sent by the smart device to the user in the form of voice, text, light, etc., informing the user of the existence of the object to be processed and inquiring about the processing method. Processing instructions refer to instructions issued by the user to the device in response to the prompts, using voice, buttons, touchscreen, etc., instructing the device how to process the object to be processed. For example, after detecting that the smart device has reached the user's location, a voice inquiry prompt can be sent to the user, or a text inquiry prompt can be displayed on the user's device display terminal, such as "Children's building blocks detected, located next to the coffee table in the living room. Do you need to use a robotic arm to organize and store them?" After receiving the inquiry prompt, the user can input processing instructions, such as the user saying "Yes" or "Organize in the toy box" via voice, or pressing a confirmation option via a button, or clicking the storage icon via touchscreen operation, etc., to issue processing instructions. The processing instructions can be transmitted to smart devices, which can then control the smart devices to perform corresponding operations on the objects to be processed, such as grabbing, storing, cleaning, and printing.

[0088] It should be noted that the interaction process with the user can be achieved through a multimodal big data model configured in the user's device or smart device. For example, when the smart device is detected to have moved to the user's location, the multimodal big data model can send a prompt message to the user. Then, the user can input processing instructions to the multimodal big data model through voice or text. After receiving the instructions, the multimodal big data model can send the processing instructions or other control instructions to the smart device so that the smart device can perform corresponding operations.

[0089] In an exemplary embodiment, sending the above-mentioned query prompt information to the user regarding the processing object may include: Determine if the user is in a busy state. If the user is not in a busy state, send the user an inquiry prompt message about the processing object.

[0090] The "busy" state refers to a state where the user is unable to receive device prompts or interact in a timely manner. This includes both actively engaged, focused scenarios and passively unresponsive scenarios. Examples of focused scenarios include users studying (e.g., listening to lectures), reading (e.g., keeping a fixed view on a page for an extended period while browsing), or handling complex household chores. Passively unresponsive scenarios include situations where the device fails to respond after a preset time following an initial prompt, such as no response for more than 15 seconds or no physical or verbal feedback for more than 30 seconds.

[0091] Determining whether a user is in a busy state can be achieved by acquiring and analyzing user status data in various ways. For example, data can be collected from the device's sensors and combined with a multimodal model for judgment. Specifically, the user's camera can capture the user's body posture (such as whether they are hunched over or have maintained a fixed sitting posture for a long time), facial expression (such as whether they are staring intently at a book / screen or have a furrowed brow or focused expression), or the microphone can collect ambient sounds (such as whether there is audio of learning courses or whether there is no everyday conversation). The above image and audio data are input into the multimodal model, and the model uses a preset focus behavior recognition algorithm to output a judgment result on whether the user is in a busy state.

[0092] This exemplary embodiment can first determine whether the user is busy, and only send an inquiry prompt about the processing object when the user is idle. For example, before sending an inquiry prompt to the user, or some time beforehand, the smart device can activate its camera and microphone to collect the user's real-time posture and ambient sound, and input the data into a multimodal large model. After model analysis, if it determines that the user is not in a busy state such as studying or doing complex housework, it will then send an inquiry prompt about the processing object to the user. If it determines that the user is in a busy state, the inquiry prompt will not be sent temporarily, or it will be detected again after a preset interval, and the decision to resend the prompt will be made based on the detection result.

[0093] In an exemplary embodiment, when the target processing mode is determined to be the second processing mode, the above-mentioned use of the target processing mode to control the smart device to process the object to be processed may include: Automatic processing refers to the autonomous actions performed by a smart device on an object to be processed, based on preset processing procedures and parameters, without real-time user intervention or commands. Examples of such actions include automatic grasping, storage, cleaning, watering, and heating. In this exemplary embodiment, after determining that the target processing mode is the second processing mode, the device can perform the operation of "locating, grasping, moving to the specified storage position, and placing" on the object according to a preset library of automatic processing procedures corresponding to the type of the object to be processed. For example, when the object to be processed is a building block, the device can perform the operation of "locating, grasping, moving to the specified storage position, and placing" according to the corresponding processing procedure.

[0094] In an exemplary embodiment, after controlling the smart device to perform automatic processing operations on the object to be processed, the above method may further include: The results of object processing are sent to the user via voice messages and / or prompts.

[0095] After a smart device performs an automatic processing operation on an object, it can also send the processing result to the user, informing them of the processing process or completion. The processing result can include the type of operation (e.g., storage, cleaning, heating), object information (e.g., object type, quantity), processing status (e.g., success, failure), and processing location. Voice messages refer to delivering results to the user via audio announcements, while prompts refer to delivering results via non-voice means such as text, icons, lights, or vibrations. For example, after a robotic vacuum cleaner completes the storage of building blocks, it can send a voice notification to the user, "Successfully picked up 1 children's building block and placed it in the living room toy box. Please confirm." It can also send this text prompt to the user's device to display the information in the user's application.

[0096] In this exemplary embodiment, after performing automatic processing on the object to be processed, the smart device can directly send the object processing result to the user, or it can first control the smart device to move to the user's location and then send the object processing result to the user. For example, after the smart device performs storage processing on the toys in the living room, it moves to the vicinity of the user's location in the bedroom and announces the tidying result to ensure that the user can clearly and promptly hear the object processing result.

[0097] In an exemplary embodiment, sending the object processing result to the user via voice information and / or prompt information may include: Determine if the user is busy. If the user is not busy, send the object processing result to the user via voice message and / or prompt message.

[0098] As mentioned above, in this exemplary embodiment, when it is necessary to send the object processing result to the user via voice information and / or prompt information, it is also possible to first determine whether the user is in a busy state. If the user is not in a busy state, the object processing result is sent to the user via voice information and / or prompt information. For example, before sending the object processing result to the user, or for a period of time beforehand, the smart device can first activate its camera and microphone to collect the user's real-time posture and ambient sound, and input the data into a multimodal large model. After model analysis, if it is determined that the user is not in a busy state such as studying or doing complex housework, the object processing result is then sent to the user. If it is determined that the user is in a busy state, the object processing result is not sent for the time being.

[0099] In one exemplary embodiment, the method may further include: Identify restricted areas in the work environment based on environmental data; The above-mentioned strategy for determining the target processing mode from multiple processing modes based on a preset mode can include: Based on the type of restricted area and / or the positional relationship between the restricted area and the object to be processed, the target processing mode is determined from multiple processing modes.

[0100] A restricted area is a spatial region that is pre-defined or identified in real time based on work objectives and environmental characteristics, prohibiting equipment from entering or handling objects within it. Examples include areas containing fragile items, private user areas, and areas where equipment cannot function properly.

[0101] After acquiring environmental data, user devices or smart devices, or configured multimodal large models, can identify restricted areas in the work scene in addition to identifying objects to be processed. This can be done through restricted area identification algorithms, such as image recognition-based region segmentation algorithms or preset coordinate-based region matching algorithms. For example, user-preset restricted area data, such as the location range of the preset restricted areas, can be obtained and compared with the environmental data to identify the preset restricted areas. Alternatively, the environmental data obtained in real time can be analyzed for object features and spatial structure. For example, if glass display cases (fragile item storage areas), lockable drawers (private areas), or water accumulation areas (equipment failure-prone areas) are identified in the environment, these areas can be determined as real-time restricted areas based on preset restricted area characteristics, such as the presence of fragile items, privacy signs, or failure-prone environments.

[0102] The type of restricted area refers to the categories of restricted areas classified based on various factors, such as fragile items restricted areas, privacy restricted areas, water accumulation restricted areas, and easily tangled restricted areas. The positional relationship between the restricted area and the object to be processed refers to the spatial orientation and distance of the object to be processed relative to the restricted area, such as the object being processed being inside the restricted area, 10cm outside the restricted area, or at the boundary of the restricted area.

[0103] In determining the target processing mode, this exemplary embodiment considers factors such as the user's previous location and historical preferences, and can also use the identified restricted area information. For example, if the object to be processed is a children's building block and its location coordinates are within a restricted area, and the type of the restricted area is a fragile item restricted area, and the strategy for the restricted area corresponds to the object to be processed within the fragile item restricted area, then the first processing mode can be adopted. Even if the second processing mode is determined to be adopted for the building block based on historical preference information, the first processing mode can still be determined as the target processing mode after comprehensive judgment, and the user needs to be consulted before processing.

[0104] In an exemplary embodiment, determining the target processing mode among multiple processing modes based on the type of the restricted area and / or the positional relationship between the restricted area and the object to be processed includes: When it is determined that the object to be processed is located within a restricted area, or the distance between the object to be processed and the restricted area is less than a first preset distance, the first processing mode is determined as the target processing mode.

[0105] The first preset distance refers to a preset distance threshold used to determine whether the object to be processed is too close to the restricted area and requires a cautious processing mode. This value can be set as needed, such as 5cm, 10cm, etc. When the object to be processed is within the restricted area or is less than the threshold, to avoid damage to objects within the restricted area or equipment malfunction during processing, the first processing mode can be selected as the target processing mode. For example, if the restricted area is a glass display case area, the first preset distance is set to 8cm, and the object to be processed is a children's building block, when it is determined that the children's building block is located within the restricted area, and / or when it is determined that the shortest distance between the children's building block and the boundary of the restricted area is less than 8cm, the first processing mode is determined as the target processing mode, and the smart device will ask the user whether to proceed.

[0106] In one exemplary embodiment, the plurality of processing modes further includes a prohibited processing mode; the determination of the target processing mode among the plurality of processing modes based on the type of the restricted area and / or the positional relationship between the restricted area and the object to be processed may include: When it is determined that the object to be processed is located within a restricted area, the target processing mode is set to prohibited processing mode.

[0107] The prohibited processing mode refers to a preset mode of the smart device where, when an object to be processed is located within a specific restricted area, no processing operations on that object are permitted. This aims to maximize the safety of objects or equipment within the restricted area. Unlike the first processing mode, the prohibited processing mode has no query step and directly does not process the object. This exemplary embodiment can pre-store the types of restricted areas and corresponding processing rules in the work scenario in the user device, smart device, or a multimodal large model configured in the user device and smart device. It can also preset the rules for the prohibited processing mode when the object to be processed is located within a restricted area. These rules can apply to high-risk restricted areas, such as restricted areas storing hazardous materials or restricted areas where equipment is absolutely prohibited. When it is determined that the object to be processed is located within a restricted area, without considering other factors such as user location or historical preference information, the prohibited processing mode can be directly determined as the target processing mode. Simultaneously, a prompt message can be sent to the user indicating that the object to be processed is within a restricted area and processing is prohibited.

[0108] It should be noted that when the work environment includes multiple restricted areas, a prohibition mode can be set for individual restricted areas, while the first or second processing mode can be used for other restricted areas with lower safety requirements.

[0109] In an exemplary embodiment, determining the target processing mode among multiple processing modes based on the type of the restricted area and / or the positional relationship between the restricted area and the object to be processed may include: When the restricted area is determined to be a non-safe restricted area and the object to be processed is located within the restricted area, the first processing mode is determined as the target processing mode; When the restricted area is determined to be a safe restricted area and the object to be processed is located within the restricted area, the second processing mode is determined as the target processing mode.

[0110] Non-safety restricted areas refer to areas within restricted zones that may pose safety hazards. For example, areas with accumulated water can easily cause short circuits, areas with flammable debris can easily cause fires, and areas with tangled wires can easily cause equipment malfunctions. When it is identified that the object to be processed is within a restricted zone, and that restricted zone is a non-safety restricted area, the first processing mode can be set as the target processing mode. After consulting the user, the processing operation will be decided.

[0111] A safety restricted area refers to a restricted area where there are no high-risk factors, such as dangerous goods, fragile items, or precision equipment. The restriction on equipment handling is only due to user preferences or scenario requirements, such as a desk area where users temporarily place items or a pet's resting area. When an object to be processed is identified as being within a restricted area, and that area is a safety restricted area, the second processing mode can be designated as the target processing mode, and the object to be processed can be processed directly.

[0112] In an exemplary embodiment, the above-described control smart device processing the object to be processed may include: The mechanical gripping structure configured in the control smart device is used to grip, move, or place the object to be processed.

[0113] A mechanical gripping structure refers to a mechanical component on a smart device used for grasping, holding, moving, and placing objects. It can consist of a robotic arm, gripping claws, a drive motor, and sensors (such as force sensors and position sensors). The mechanical gripping structure can hold the object in place using claws or suction cups. After grasping the object, the mechanical structure can move the object to change its spatial position and release it to a designated location. Smart devices can use mechanical gripping structures to store and manage objects.

[0114] In one exemplary embodiment, the method may further include: Based on the object characteristic data of the object to be processed, adjust the control parameters of the mechanical gripping structure; the control parameters include control force and / or control speed.

[0115] The control parameters of a mechanical gripping structure refer to the quantitative indicators used to control its movements. Control force refers to the magnitude of the force applied by the gripping structure when grasping and holding the object to be processed, while control speed refers to the speed at which various components of the gripping structure, such as the robotic arm joints and gripping claws, execute their actions, such as the robotic arm's movement speed and the gripping claw's opening and closing speed. Adjusting the control parameters based on the object's characteristic data can prevent damage to the object or gripping failure due to inappropriate parameters. For example, if the object to be processed is a children's building block, and its characteristic data is: material plastic, low fragility, weight 30g (light), and smooth surface, then by referring to the correspondence table between the object's characteristic data and control parameters, the recommended control force for the building block can be determined to be Level 1, and the control speed 5cm / s.

[0116] In this exemplary embodiment, after determining the control parameters, the system can also interact with the user through a multimodal large model to ask the user whether the control parameters need to be updated and adjusted. After receiving the user's confirmation instruction, the determined control parameters and the object to be processed can be used for processing.

[0117] Figure 2 A flowchart illustrating another control method for a smart device in this exemplary embodiment is shown, which may specifically include the following steps: Step S210: Collect environmental data of the work scene through intelligent devices; Step S220: Input the environmental data into the first processing model for processing to determine the objects to be processed in the work scene; Step S230: Based on the preset mode judgment strategy, the target processing mode is determined from multiple processing modes through the second processing model; Step S240: Generate control instructions so that the smart device processes the object to be processed according to the target processing mode based on the control instructions.

[0118] The first processing model and the second processing model can be different models. For example, two models can be trained separately: the first model processes environmental data to identify the object to be processed, and the second model interacts with the user to determine the target processing mode. Alternatively, the first processing model and the second processing model can be the same model. For example, a large multimodal model can be trained that can both perform the object recognition process and interact with the user to determine the target processing mode.

[0119] In exemplary embodiments of this disclosure, a control device for a smart device is also provided, such as... Figure 3As shown, the device 300 may include: an environmental data acquisition module 310, used to acquire environmental data of the work scene; a processing object identification module 320, used to identify the object to be processed in the work scene based on the environmental data; a target mode determination module 330, used to determine a target processing mode among multiple processing modes according to a preset mode judgment strategy; the multiple processing modes include at least a first processing mode and a second processing mode, the first processing mode being a passive object processing mode and the second processing mode being an active object processing mode; and an object processing control module 340, used to control the intelligent device to process the object to be processed using the target processing mode.

[0120] In one exemplary embodiment of this disclosure, the object recognition processing module includes: The model processing unit is used to input environmental data into the object processing model for processing, obtain object recognition results, and determine the objects to be processed in the work scene based on the object recognition results.

[0121] In one exemplary embodiment of this disclosure, the object recognition result includes multiple candidate objects and object attribute information of each candidate object; the model processing unit includes: a processing object filtering subunit, used to determine the object to be processed in the work scene from multiple candidate objects according to the object attribute information of each candidate object.

[0122] In one exemplary embodiment of this disclosure, the target mode determination module includes: a user location detection unit, configured to detect the user's location in a work scenario; a first determination unit, configured to determine a first processing mode as the target processing mode when the user's location is detected to be in a work scenario; and a second determination unit, configured to determine a second processing mode as the target processing mode when the user's location is detected to be not in a work scenario.

[0123] In one exemplary embodiment of this disclosure, the target mode determination module includes: a preference information acquisition unit, used to acquire the user's historical preference information; the historical preference information includes the object type of the historical processing object corresponding to multiple processing modes; and a third determination unit, used to determine the target processing mode among multiple processing modes based on the historical preference information and the object type of the object to be processed.

[0124] In one exemplary embodiment of this disclosure, the target pattern determination module includes: a feature data determination unit, used to determine object feature data of the object to be processed; and a fourth determination unit, used to determine a target processing mode among multiple processing modes based on the matching result of the object feature data of the object to be processed and the pattern selection rule; wherein the pattern selection rule includes multiple processing modes and object feature data corresponding to each processing mode.

[0125] In one exemplary embodiment of this disclosure, when the target processing mode is determined to be the first processing mode, the object processing control module includes: a processing instruction receiving unit, configured to send an inquiry prompt message about the processing object to the user in response to detecting that the smart device has moved to the user's location, and receive the user's processing instruction in response to the inquiry prompt message; and a first processing unit, configured to perform corresponding operations on the processing object according to the processing instruction.

[0126] In one exemplary embodiment of this disclosure, the processing instruction receiving unit includes: a first state determination subunit, configured to determine whether the user is in a busy state, and if the user is not in a busy state, to send the user an inquiry prompt message about the processing object.

[0127] In one exemplary embodiment of this disclosure, when the target processing mode is determined to be the second processing mode, the object processing control module includes: a second processing unit, used to control the smart device to perform automatic processing operations on the object to be processed.

[0128] In one exemplary embodiment of this disclosure, after controlling the smart device to perform automatic processing operations on the object to be processed, the apparatus further includes: a processing result sending unit, used to send the object processing result to the user via voice information and / or prompt information.

[0129] In one exemplary embodiment of this disclosure, the processing result sending unit includes: a second state determination subunit, used to determine whether the user is in a busy state, and if the user is not in a busy state, to send the object processing result to the user via voice information and / or prompt information.

[0130] In one exemplary embodiment of this disclosure, the apparatus further includes: a restricted area identification module, used to identify restricted areas in the work scene based on environmental data; and a target mode determination module, including: a fifth determination unit, used to determine a target processing mode among multiple processing modes based on the type of the restricted area and / or the positional relationship between the restricted area and the object to be processed.

[0131] In one exemplary embodiment of this disclosure, the fifth determining unit is used to determine the first processing mode as the target processing mode when it is determined that the object to be processed is located in a restricted area, or the distance between the object to be processed and the restricted area is less than a first preset distance.

[0132] In one exemplary embodiment of this disclosure, the fifth determining unit is configured to determine the target processing mode as a prohibited processing mode when it is determined that the object to be processed is located in a restricted area.

[0133] In one exemplary embodiment of this disclosure, the fifth determining unit is configured to determine the first processing mode as the target processing mode when the type of the restricted area is determined to be a non-safe restricted area and the object to be processed is located within the restricted area; and to determine the second processing mode as the target processing mode when the type of the restricted area is determined to be a safe restricted area and the object to be processed is located within the restricted area.

[0134] In one exemplary embodiment of this disclosure, the object processing control module includes: a structure control unit, used to control the mechanical gripping structure configured in the smart device to grip, move, or place the object to be processed.

[0135] In one exemplary embodiment of this disclosure, the apparatus further includes: a control parameter adjustment module, used to adjust the control parameters of the mechanical gripping structure according to the object feature data of the object to be processed; the control parameters include control force and / or control speed.

[0136] The specific details of the control device modules of each of the above-mentioned smart devices have been described in detail in the corresponding control methods of the smart devices, so they will not be repeated here.

[0137] It should be noted that although several modules or units for the execution device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0138] In an exemplary embodiment of this disclosure, a sorting robot is also provided, comprising: a body; and a controller disposed in the body; the controller is configured to perform the control method of the intelligent device described above.

[0139] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided. For example, the electronic device may be a cleaning robot capable of implementing the above-described method.

[0140] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0141] The following reference Figure 4 To describe an electronic device 400 according to such an exemplary embodiment of the present disclosure. Figure 4The electronic device 400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0142] like Figure 4 As shown, the electronic device 400 is manifested in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including storage unit 420 and processing unit 410), and a display unit 440.

[0143] The storage unit stores program code, which can be executed by the processing unit 410 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 410 can execute... Figure 1 or Figure 2 The steps shown are as follows.

[0144] Storage unit 420 may include readable media in the form of volatile storage units, such as random access memory (RAM) 421 and / or cache memory 422, and may further include read-only memory (ROM) 423.

[0145] Storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0146] Bus 430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0147] Electronic device 400 can also communicate with one or more external devices 500 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 400, and / or with any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. As shown, network adapter 460 communicates with other modules of electronic device 400 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0148] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0149] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this disclosure described in the "Exemplary Methods" section above.

[0150] The program product for implementing the above-described method according to embodiments of the present disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0152] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0153] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0154] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0155] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0156] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0157] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. A control method for an intelligent device, characterized in that, The method comprises: acquiring environment data of a work scene; identifying an object to be processed in the work scene according to the environment data; determining a target processing mode from multiple processing modes according to a preset mode judgment strategy; the multiple processing modes at least include a first processing mode and a second processing mode, the first processing mode is a mode of passively processing the object, and the second processing mode is a mode of actively processing the object; controlling the intelligent device to process the object to be processed in the target processing mode.

2. The method of claim 1, wherein, The identification of the object to be processed in the work scene according to the environment data comprises: inputting the environment data into an object processing model for processing to obtain an object identification result, and determining the object to be processed in the work scene based on the object identification result.

3. The method of claim 2, wherein, The object identification result includes multiple candidate objects and object attribute information of each candidate object. The determination of the object to be processed in the work scene based on the object identification result comprises: determining the object to be processed in the work scene from the multiple candidate objects according to the object attribute information of each candidate object.

4. The method of claim 1, wherein, The determination of the target processing mode from the multiple processing modes according to the preset mode judgment strategy comprises: detecting a user position in the work scene; determining the first processing mode as the target processing mode when it is detected that the user position is in the work scene; determining the second processing mode as the target processing mode when it is detected that the user position is not in the work scene.

5. The method of claim 1, wherein, The determination of the target processing mode from the multiple processing modes according to the preset mode judgment strategy comprises: acquiring historical preference information of a user; the historical preference information includes object types of historical processing objects corresponding to multiple processing modes; determining the target processing mode from the multiple processing modes according to the historical preference information and an object type of the object to be processed.

6. The method of claim 1, wherein, The determination of the target processing mode from the multiple processing modes according to the preset mode judgment strategy comprises: determining object feature data of the object to be processed; determining the target processing mode from the multiple processing modes according to a matching result of the object feature data of the object to be processed and a mode selection rule; wherein the mode selection rule includes multiple processing modes and object feature data corresponding to each processing mode.

7. The method of claim 1, wherein, When the target processing mode is determined to be the first processing mode, the controlling of the intelligent device to process the object to be processed in the target processing mode comprises: sending inquiry prompt information about the object to be processed to the user and receiving a processing instruction of the user for the inquiry prompt information in response to detecting that the intelligent device moves to a position where the user is located; performing corresponding operations on the object to be processed according to the processing instruction.

8. The method of claim 7, wherein, The sending of the inquiry prompt information about the object to be processed to the user comprises: determining whether the user is in a busy state, and sending the inquiry prompt information about the object to be processed to the user if the user is not in the busy state.

9. The method of claim 1, wherein, When the target processing mode is determined to be the second processing mode, the controlling of the intelligent device to process the object to be processed in the target processing mode comprises: The method further comprises:

10. The method of claim 9, wherein, controlling the smart device to perform an automatic processing operation on the object to be processed. After controlling the smart device to perform an automatic processing operation on the object to be processed, the method further comprises:

11. The method of claim 10, wherein, sending an object processing result to the user in the form of voice information and / or prompt information. The method further comprises:

12. The method of claim 1, wherein, determining whether the user is in a busy state, and if the user is not in a busy state, sending an object processing result to the user in the form of voice information and / or prompt information. The method further comprises: identifying a forbidden area in the work scene according to the environment data; The method further comprises:

13. The method of claim 12, wherein, determining a target processing mode from a plurality of processing modes according to a preset mode judgment strategy. The method further comprises:

14. The method of claim 12, wherein, determining a target processing mode from a plurality of processing modes according to a type of the forbidden area and / or a positional relationship between the forbidden area and the object to be processed. The method further comprises:

15. The method of claim 12, wherein, determining a first processing mode as the target processing mode when it is determined that the object to be processed is located in the forbidden area or the distance between the object to be processed and the forbidden area is less than a first preset distance. The plurality of processing modes further comprises a prohibited processing mode; and the method further comprises: determining the target processing mode as the prohibited processing mode when it is determined that the object to be processed is located in the forbidden area.

16. The method of claim 1, wherein, The method further comprises: determining a first processing mode as the target processing mode when it is determined that the type of the forbidden area is a non-safe forbidden area and the object to be processed is located in the forbidden area; 17. The method of claim 16, wherein, determining a second processing mode as the target processing mode when it is determined that the type of the forbidden area is a safe forbidden area and the object to be processed is located in the forbidden area. The method further comprises:

18. A control device of a smart device, characterized by, controlling a mechanical grabbing structure configured in the smart device to grab, move or place the object to be processed. The method further comprises: adjusting a control parameter of the mechanical grabbing structure according to object characteristic data of the object to be processed; the control parameter comprises a control strength and / or a control speed. The device comprises: an environment data acquisition module configured to acquire environment data of a work scene; a processing object identification module configured to identify an object to be processed in the work scene according to the environment data; a target mode determination module configured to determine a target processing mode from a plurality of processing modes according to a preset mode judgment strategy; the plurality of processing modes at least comprises a first processing mode and a second processing mode, the first processing mode is a passive processing mode of the object to be processed, and the second processing mode is an active processing mode of the object to be processed; an object processing control module configured to control the smart device to process the object to be processed in the target processing mode.

19. The apparatus of claim 18, wherein, The processing object recognition module comprises: The model processing unit is configured to input the environment data into an object processing model to obtain an object recognition result, and determine a processing object in the working scenario based on the object recognition result.

20. The apparatus of claim 19, wherein, The object recognition result comprises a plurality of candidate objects and object attribute information of each candidate object. The model processing unit comprises: The processing object screening subunit is configured to determine the processing object in the working scenario from the plurality of candidate objects based on the object attribute information of each candidate object.

21. The apparatus of claim 18, wherein, The target mode determination module comprises: The user position detection unit is configured to detect a user position in the working scenario. The first determination unit is configured to determine the first processing mode as the target processing mode when the detected user position is in the working scenario. The second determination unit is configured to determine the second processing mode as the target processing mode when the detected user position is not in the working scenario.

22. The apparatus of claim 18, wherein, The target mode determination module comprises: The preference information acquisition unit is configured to acquire historical preference information of a user, wherein the historical preference information comprises object types of historical processing objects corresponding to a plurality of processing modes. The third determination unit is configured to determine the target processing mode from the plurality of processing modes based on the historical preference information and an object type of the processing object.

23. The apparatus of claim 18, wherein, The target mode determination module comprises: The feature data determination unit is configured to determine object feature data of the processing object. The fourth determination unit is configured to determine the target processing mode from the plurality of processing modes based on a matching result of the object feature data of the processing object and a mode selection rule. The mode selection rule comprises a plurality of processing modes and object feature data corresponding to each processing mode.

24. The apparatus of claim 18, wherein, When the target processing mode is determined to be the first processing mode, the object processing control module comprises: The processing instruction receiving unit is configured to send inquiry prompt information about the processing object to the user and receive a processing instruction of the user for the inquiry prompt information in response to detecting that the intelligent device moves to the position of the user. The first processing unit is configured to perform a corresponding operation on the processing object according to the processing instruction.

25. The apparatus of claim 24, wherein, The processing instruction receiving unit comprises: The first state determination subunit is configured to determine whether the user is in a busy state, and send the inquiry prompt information about the processing object to the user if the user is not in the busy state.

26. The apparatus of claim 18, wherein, When the target processing mode is determined to be the second processing mode, the object processing control module comprises: The second processing unit is configured to control the intelligent device to perform an automatic processing operation on the processing object.

27. The apparatus of claim 26, wherein, After controlling the intelligent device to perform the automatic processing operation on the processing object, the apparatus further comprises: The processing result sending unit is configured to send an object processing result to the user in the form of voice information and / or prompt information.

28. The apparatus of claim 27, wherein, The processing result sending unit comprises: The second state determination subunit is configured to determine whether the user is in a busy state, and send the object processing result to the user in the form of voice information and / or prompt information if the user is not in the busy state.

29. The apparatus of claim 18, wherein, The apparatus further comprises: The forbidden area recognition module is configured to recognize a forbidden area in the working scene according to the environment data. The target mode determination module comprises: The fifth determination unit is configured to determine a target processing mode from a plurality of processing modes according to the type of the forbidden area and / or the positional relationship between the forbidden area and the to-be-processed object.

30. The apparatus of claim 29, wherein, The fifth determination unit is configured to determine the first processing mode as the target processing mode when it is determined that the to-be-processed object is located in the forbidden area or the distance between the to-be-processed object and the forbidden area is less than a first preset distance.

31. The apparatus of claim 29, wherein, The fifth determination unit is configured to determine the target processing mode as the forbidden processing mode when it is determined that the to-be-processed object is located in the forbidden area.

32. The apparatus of claim 29, wherein, The fifth determination unit is configured to determine the first processing mode as the target processing mode when it is determined that the type of the forbidden area is a non-safe forbidden area and the to-be-processed object is located in the forbidden area, and determine the second processing mode as the target processing mode when it is determined that the type of the forbidden area is a safe forbidden area and the to-be-processed object is located in the forbidden area.

33. The apparatus of claim 18, wherein, The object processing control module comprises: The structural control unit is configured to control a mechanical grabbing structure configured in the intelligent device to grab, move or place the to-be-processed object.

34. The apparatus of claim 33, wherein, The apparatus further comprises: The control parameter adjustment module is configured to adjust control parameters of the mechanical grabbing structure according to object feature data of the to-be-processed object, wherein the control parameters comprise control strength and / or control speed.

35. A finishing robot characterized by Comprise: A body; And A controller arranged in the body; The controller is configured to execute the control method of the intelligent device according to any one of claims 1-17.

36. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the control method of the intelligent device according to any one of claims 1-17.

37. An electronic device, comprising: Comprise: A processor; And A memory for storing executable instructions of the processor; Wherein the processor is configured to execute the control method of the intelligent device according to any one of claims 1-17 by executing the executable instructions.

Images