Robot object searching method based on multi-modal detection
Through the robotic object search method of multimodal detection, using voice interaction and RGB/depth image combination, the problem of cumbersome operation and high expansion costs in the prior art is solved, and accurate and convenient item search is achieved.
Patent Information
- Application Number
- CN202510589039.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2025-08-08
AI Technical Summary
Existing robotic object search methods require pre-pasting labels or pre-defined item categories, which are cumbersome and costly to expand, making it difficult to accurately find items with specific attributes.
The multimodal detection method is used to obtain object properties through voice interaction, combine RGB and depth images for real-time detection, use multimodal models to judge the object position, and use 3D coordinates to control the robot's movement until the target object is found.
It realizes accurate object search without predefined item categories. Users can operate easily and can directly specify item attributes through voice interaction, which is suitable for the search of various common items.
Smart Images
Figure CN120447554A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robotics technology, and in particular to a robot object-finding method based on multimodal detection. Background Art
[0002] With the continuous advancement of robotics technology, robots have been widely used in high-risk, high-precision and highly repetitive work areas. In recent years, household robots have gradually entered thousands of households and become special robots serving humans, mainly undertaking household cleaning, repairs, maintenance, monitoring and other tasks, and gradually expanding to solving various household chores. With the development of sweeping robot technology, household robots have made significant progress in automatic cleaning and other aspects. However, in daily life, finding forgotten items is a common problem in daily life. For example, for items such as keys and mobile phones, people often spend a lot of time and energy to find them, especially for the elderly at home, who are more likely to forget where they put their personal items and may not be very mobile. At this time, an efficient object-finding robot can greatly reduce the time the owner spends looking for items;
[0003] There are two main existing technologies. The first is an RFID-based object search method that attaches a target tag to the target object, storing its attribute information. Reference tags are attached to fixed objects to accurately locate the target object. A fixed RFID system is installed in the room to monitor the target object's position changes. Target objects outside its coverage are detected using a mobile RFID system carried by the robot. The second is an object search method based on 2D object detection. A camera on the robot captures images in real time in front of the robot and uses deep learning neural network reasoning to determine whether the target object is in the image. The robot's movement allows for object search throughout the room. The first problem with both methods is that the RFID-based object search method requires pre-attaching specific tags to the objects and entering the corresponding item information into the system, making the entire operation process too cumbersome for users. The second problem with the object search method based on 2D object detection is that it limits the objects to be found and is costly to scale. Existing 2D object detection technology predefines object categories, collects a dataset containing all categories, and then trains a deep learning detection model to detect the presence of the predefined objects in a given image in real time. When my predefined categories include cups, I can find all cups the robot can see. However, when I need to find a more precise object, such as a red cup, existing methods cannot accurately locate it. To expand the categories of objects to be found, we need to collect new data and train the model. However, in deep learning, collecting large amounts of data, labeling them, and training models is very expensive. Therefore, we propose a robot object-finding method based on multimodal detection. Summary of the Invention
[0004] The main purpose of the present invention is to provide a robot object-finding method based on multimodal detection, which can effectively solve the problems in the background technology.
[0005] To achieve the above object, the technical solution adopted by the present invention is:
[0006] A robot object-finding method based on multimodal detection comprises the following steps:
[0007] S1. First, the user interacts with the robot through voice communication, allowing the robot to obtain the attributes of the object to be found;
[0008] S2. After human-machine interaction, the robot starts searching to obtain time-synchronized RGB and depth images;
[0009] S3: The human-machine uses the multimodal model to infer and obtain the inference results based on the acquired RGB and depth images;
[0010] S4: The robot determines whether it has detected the object based on the inference results. If not, it continues to move and search to obtain time-synchronized RGB and depth images, and uses the multimodal model to infer and obtain the inference results. If yes, it proceeds to the next step.
[0011] S5, the robot obtains the 3D coordinates of the object in combination with the depth;
[0012] S6. The robot determines whether the object is too far away from the robot according to the 3D coordinates. If so, the robot is controlled to move to a position closer to the object according to the currently acquired 3D coordinates of the object, and repeats S2-S5 at this position.
[0013] S7. When the robot determines whether the object is too far away from the robot based on the 3D coordinates of the object obtained in combination with the depth, it can be determined that the object has been found and the result is announced by voice.
[0014] Preferably, in said S1, the object attribute may be instruction information text such as the color and shape of the object, so that when there are multiple objects of the same category at the same time, it can be determined more accurately whether they are the items to be found.
[0015] Preferably, in S3, the multimodal object detection uses the current image and the user instruction as the input of the model, and detects in real time whether there is an object in the current image that is the same as the object described in the instruction.
[0016] Preferably, in S1, the robot can be a wheeled, humanoid or semi-humanoid robot, and the robot is provided with a front depth camera for acquiring RGB and depth images, a microphone for voice human-computer interaction to acquire object attribute text, and a processor for processing voice text and RGB and depth image processing.
[0017] Compared with the prior art, the present invention has the following beneficial effects:
[0018] In the present invention, the user operation of the robot object-finding method based on multimodal detection is simple and does not require any settings. The user can directly input the object through voice interaction to find any common object, and can specify the attributes of the object to be found, such as color, shape, etc., without the need to enter the item category through the APP. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 The figure is a flow chart of a robot object-finding method based on multimodal detection according to the present invention. DETAILED DESCRIPTION
[0020] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.
[0021] like Figure 1 As shown, a robot object-finding method based on multimodal detection includes the following steps:
[0022] S1. First, the user interacts with the robot through voice communication, allowing the robot to obtain the attributes of the object to be found;
[0023] S2. After human-machine interaction, the robot starts searching to obtain time-synchronized RGB and depth images;
[0024] S3: The human-machine uses the multimodal model to infer and obtain the inference results based on the acquired RGB and depth images;
[0025] S4: The robot determines whether it has detected the object based on the inference results. If not, it continues to move and search to obtain time-synchronized RGB and depth images, and uses the multimodal model to infer and obtain the inference results. If yes, it proceeds to the next step.
[0026] S5, the robot obtains the 3D coordinates of the object in combination with the depth;
[0027] S6. The robot determines whether the object is too far away from the robot according to the 3D coordinates. If so, the robot is controlled to move to a position closer to the object according to the currently acquired 3D coordinates of the object, and repeats S2-S5 at this position.
[0028] S7: When the robot determines whether the object is too far away from the robot based on the 3D coordinates of the object obtained in combination with the depth, it can be determined that the object has been found and the result is announced by voice;
[0029] In S1, the object attributes can be instruction information text such as the object's color and shape, so that when there are multiple objects of the same category at the same time, it can be more accurately determined whether it is the object to be found; in S3, multimodal target detection uses the current image and user instructions as inputs to the model to detect in real time whether there are objects in the current image that are the same as those described in the instructions; in S1, the robot can be a wheeled, humanoid or semi-humanoid robot, and the robot is provided with a front depth camera for acquiring RGB and depth images, and a microphone for voice human-computer interaction to obtain object attribute text, and a processor for processing voice text and RGB and depth image processing.
[0030] It should be noted that the present invention is a robot object-finding method based on multimodal detection. When in use, speech recognition: speech recognition is performed through microphone input. Speech recognition technology converts natural language into text information in real time to obtain the user's specific instructions. Multimodal target detection: Multimodal target detection uses the current image and user instructions as the input of the model, and detects in real time whether there are objects in the current image that are the same as those described in the instructions. The advantage of multimodal target detection is that there is no need to predefine the category of objects. At the same time, it can also find objects more finely, such as adding the attributes of the object (color, shape) to the instruction, so that when there are multiple objects of the same category at the same time, it can more accurately determine whether it is the object to be found. The multimodal target detection described in this patent only includes two modalities: image and text, and the detection is based on 2D target detection. In alternative solutions, more modalities can be added, such as point cloud, or the modality can be replaced, such as replacing the text input of the model with direct voice input; 2D detection can also be replaced with 3D target detection.
[0031] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.
Claims
1. A robot object-finding method based on multimodal detection, characterized in that: The following steps are involved: S1. First, the user interacts with the robot through voice communication, allowing the robot to obtain the attributes of the object to be found; S2. After human-machine interaction, the robot starts searching to obtain time-synchronized RGB and depth images; S3: The human-machine uses the multimodal model to infer and obtain the inference results based on the acquired RGB and depth images; S4: The robot determines whether it has detected the object based on the inference results. If not, it continues to move and search to obtain time-synchronized RGB and depth images, and uses the multimodal model to infer and obtain the inference results. If yes, it proceeds to the next step. S5, the robot obtains the 3D coordinates of the object in combination with the depth; S6. The robot determines whether the object is too far away from the robot according to the 3D coordinates. If so, the robot is controlled to move to a position closer to the object according to the currently acquired 3D coordinates of the object, and repeats S2-S5 at this position. S7. When the robot determines whether the object is too far away from the robot based on the 3D coordinates of the object obtained in combination with the depth, it can be determined that the object has been found and the result is announced by voice.
2. The robot object-finding method based on multimodal detection according to claim 1, characterized in that: In the above S1, the object attribute may be instruction information text such as the color and shape of the object, so that when there are multiple objects of the same category at the same time, it can be more accurately determined whether it is the item to be found.
3. The robot object-finding method based on multimodal detection according to claim 1, characterized in that: In S3, the multimodal object detection uses the current image and user instructions as the input of the model to detect in real time whether there is an object in the current image that is the same as the object described in the instruction.
4. The robot object-finding method based on multimodal detection according to claim 1, characterized in that: In S1, the robot can be a wheeled, humanoid or semi-humanoid robot, and the robot is provided with a front depth camera for acquiring RGB and depth images, a microphone for voice human-computer interaction to acquire object attribute text, and a processor for processing voice text and RGB and depth image processing.