Intelligent old-age service robot based on deep vision

Through deep vision technology and visual language model, the intelligent elderly care service robot system solves the problems of navigation and operation connection, improves task execution efficiency and flexibility, and ensures that the robot can correctly position and complete the task.

CN118915734AInactive Publication Date: 2024-11-08VIDEOSTRONG TECH CO LTD

Patent Information

Application Number
CN202410956637.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing intelligent elderly care service robot navigation methods have failed to effectively solve the problem of connection between navigation and operation, resulting in the robot being unable to correctly locate to perform tasks after reaching the target position.

Method used

Adopt an intelligent elderly care service robot system based on deep vision, and control the middle platform, navigation module, acquisition module, determination module and visual language model through the robot, and use preset navigation algorithms and point cloud data to determine the target area suitable for the task to reduce navigation Distance between destination and operation location.

Benefits of technology

The efficiency and flexibility of intelligent elderly care service robots to perform tasks within the preset range are improved, ensuring that the robot can correctly position and effectively perform target tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915734A_ABST
    Figure CN118915734A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an intelligent old-age service robot based on depth vision, which comprises a robot control middle table, a first navigation module, a first acquisition module, a first determination module, a second acquisition module, a second determination module and a second navigation module, the robot control middle table is connected with the first navigation module, the first acquisition module, the first determination module, the second acquisition module, the second determination module and the second navigation module and manages and controls all the modules. According to the invention, after the intelligent old-age service robot receives the target task executed for the target object, the intelligent old-age service robot can be guided to the area which is near the target object and is suitable for executing the target task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent elderly care service robot based on depth vision. Background Art

[0002] In the field of intelligent elderly care service robot technology, zero-shot object goal navigation (ZSON) is regarded as a key capability to achieve efficient operation. This technology allows intelligent elderly care service robots to navigate to the location of a specified target object and perform target tasks for the target object (such as opening the refrigerator door) through visual observation, semantic understanding and motion planning without any training data for a specific target object. Current navigation methods mainly rely on visual language models to identify objects and determine whether the object is a target or an obstacle. Then, a navigation path is generated using a traditional path planning algorithm to guide the intelligent elderly care service robot directly to the target location. However, this approach does not consider the connection between navigation and operation. In practical applications, intelligent elderly care service robots not only need to reach the target location, but also need to be positioned in the correct direction and alignment so that subsequent operation tasks can be performed smoothly. For example, when the navigation system guides the intelligent elderly care service robot to reach the refrigerator, if the intelligent elderly care service robot is positioned at an unrealistic side or rear, it will not be able to effectively open the refrigerator door for further operations. Similarly, for tasks such as sitting on the sofa, the intelligent elderly care service robot must correctly face the sofa and be positioned to sit comfortably, rather than just reaching near the sofa. Summary of the invention

[0003] The present invention provides an intelligent elderly care service robot based on depth vision, which is used to guide the intelligent elderly care service robot to an area suitable for performing a target task near a target object after the intelligent elderly care service robot receives a target task to be performed on a target object.

[0004] A deep vision-based intelligent elderly care service robot comprises a robot control center, a first navigation module, a first acquisition module, a first determination module, a second acquisition module, a second determination module and a second navigation module; the robot control center is respectively connected to the first navigation module, the first acquisition module, the first determination module, the second acquisition module, the second determination module and the second navigation module to manage and control each module;

[0005] The first navigation module is used to, in response to receiving a target task performed by a target user on a target object, guide the intelligent elderly care service robot to a first target area within a preset range of the target object using a preset navigation algorithm; wherein the coordinates of the first target area are determined according to a historical inspection record of the intelligent elderly care service robot;

[0006] The first acquisition module is used to acquire point cloud data information near the first target area and point cloud data information of the target object;

[0007] The first determination module is used to determine a plurality of candidate areas around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object;

[0008] A second acquisition module is used to project the multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, to obtain first image data carrying information of the multiple candidate areas;

[0009] The second determination module is used to determine a second target area from the multiple candidate areas according to the first image data and information of the target task by using a visual language model;

[0010] The second navigation module is used to guide the intelligent elderly care service robot to reach the second target area using a preset navigation algorithm, so that the intelligent elderly care service robot performs the target task based on the second target area.

[0011] The present invention also provides an intelligent elderly care service method based on depth vision, comprising:

[0012] In response to receiving a target task performed by a target user on a target object, using a preset navigation algorithm, guiding the intelligent elderly care service robot to a first target area within a preset range of the target object; the coordinates of the first target area are determined according to a historical inspection record of the intelligent elderly care service robot;

[0013] Acquire point cloud data information near the first target area and point cloud data information of the target object;

[0014] Determine a plurality of candidate areas around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object;

[0015] Projecting the multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, to obtain first image data carrying information of the multiple candidate areas;

[0016] Determine a second target region from the plurality of candidate regions using a visual language model according to the first image data and information about the target task;

[0017] The preset navigation algorithm is used to guide the intelligent elderly care service robot to reach the second target area, so that the intelligent elderly care service robot performs the target task based on the second target area.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the intelligent elderly care service method based on deep vision as described in any one of the above items is implemented.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned deep vision-based intelligent elderly care service methods.

[0020] The present invention also provides a computer product having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer product implements any of the above-mentioned deep vision-based intelligent elderly care service methods.

[0021] The intelligent elderly care service robot based on depth vision provided by the present invention comprises a robot control center, a first navigation module, a first acquisition module, a first determination module, a second acquisition module, a second determination module and a second navigation module; the robot control center is respectively connected to the first navigation module, the first acquisition module, the first determination module, the second acquisition module, the second determination module and the second navigation module to manage and control each module, and when the intelligent elderly care service robot receives a target task to be performed on a target object, the preset navigation algorithm is used to guide the intelligent elderly care service robot to a first target area within a preset range of the target object; multiple candidate areas are determined around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object. domain; projecting multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, to obtain first image data carrying information of multiple candidate areas; utilizing the powerful semantic understanding and image processing capabilities of the visual language model, and according to the first image data and information of the target task, determining a second target area suitable for performing the target task from multiple candidate areas; utilizing a preset navigation algorithm to guide the intelligent elderly care service robot to the second target area, so that the intelligent elderly care service robot performs the target task based on the second target area, thereby reducing the distance between the navigation destination of the intelligent elderly care service robot and the location where the target task is performed, and effectively improving the efficiency and flexibility of the intelligent elderly care service robot in performing tasks within a preset range. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a structural schematic diagram of the intelligent elderly care service robot based on depth vision provided by the present invention.

[0023] Figure 2 It is a flowchart of the intelligent elderly care service method based on deep vision provided by the present invention.

[0024] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0026] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0027] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0028] Reference Figure 1 , Figure 1 : is a structural diagram of the intelligent elderly care service robot based on deep vision provided by the present invention. The intelligent elderly care service robot based on deep vision of the embodiment of the present invention includes a robot control center, a first navigation module, a first acquisition module, a first determination module, a second acquisition module, a second determination module and a second navigation module. The robot control center is connected to the first navigation module, the first acquisition module, the first determination module, the second acquisition module, the second determination module and the second navigation module respectively to manage and control each module.

[0029] Optionally, when the user needs to perform a task, the user can send a target task to the intelligent elderly care service robot. Therefore, the first navigation module responds to receiving the target task performed by the target user for the target object, and uses a preset navigation algorithm to guide the intelligent elderly care service robot to the first target area within a preset range of the target object, wherein the coordinates of the first target area are determined based on the historical inspection records of the intelligent elderly care service robot.

[0030] Optionally, the first acquisition module acquires point cloud data information near the first target area and point cloud data information of the target object.

[0031] Optionally, the first determination module determines a plurality of candidate areas around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object.

[0032] Optionally, the second acquisition module projects the multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, to obtain first image data carrying information of the multiple candidate areas. Optionally, the second determination module uses a visual language model to determine the second target area from the multiple candidate areas based on the first image data and information of the target task.

[0033] Optionally, the second navigation module uses a preset navigation algorithm to guide the intelligent elderly care service robot to reach a second target area, so that the intelligent elderly care service robot performs a target task based on the second target area.

[0034] The intelligent elderly care service robot of the embodiment of the present invention comprises a robot control center, a first navigation module, a first acquisition module, a first determination module, a second acquisition module, a second determination module and a second navigation module; the robot control center is respectively connected to the first navigation module, the first acquisition module, the first determination module, the second acquisition module, the second determination module and the second navigation module to manage and control each module, and when the intelligent elderly care service robot receives a target task to be performed on a target object, the preset navigation algorithm is used to guide the intelligent elderly care service robot to a first target area within a preset range of the target object; multiple candidate areas are determined around the target object based on point cloud data information near the first target area and point cloud data information of the target object; Multiple candidate areas are projected onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, so as to obtain first image data carrying information of multiple candidate areas; the powerful semantic understanding ability and image processing ability of the visual language model are utilized to determine a second target area suitable for performing the target task from multiple candidate areas according to the first image data and information of the target task; a preset navigation algorithm is utilized to guide the intelligent elderly care service robot to reach the second target area, so that the intelligent elderly care service robot performs the target task based on the second target area, thereby reducing the distance between the navigation destination of the intelligent elderly care service robot and the location where the target task is performed, and effectively improving the efficiency and flexibility of the intelligent elderly care service robot in performing tasks within a preset range.

[0035] Reference Figure 2 , Figure 2 is a flow chart of the intelligent elderly care service method based on deep vision provided by the present invention, such as Figure 2 As shown, the method includes steps 10 to 60, wherein:

[0036] Step 10: In response to receiving a target task performed by a target user on a target object, using a preset navigation algorithm, guide the intelligent elderly care service robot to a first target area within a preset range of the target object.

[0037] The intelligent elderly care service robot may be a robot that works within a preset range, for example, a home service robot, an intelligent elderly care service robot, or other robots that work indoors.

[0038] As just an example, the target task to be performed on the target object may be: a task of opening a refrigerator, a task of sitting on a sofa, and the like.

[0039] The first target area is located within the preset range of the target object. After the intelligent elderly care service robot reaches this area, the target object is within the field of vision of the intelligent elderly care service robot. However, the first target area may be an area that is not suitable for performing the target task. For example, for the task of opening the refrigerator door, the position area along the direction of opening the refrigerator door is the best position for the intelligent elderly care service robot to operate. Although the remaining position areas are located within the preset range of the target object, they are not suitable for performing the task of opening the refrigerator. For another example, for the task of opening a drawer, the position area along the direction of opening the drawer is the best position for the intelligent elderly care service robot to operate. Although the remaining position areas are located within the preset range of the target object, they are not suitable for performing the task of opening the drawer.

[0040] In the specific implementation process, the coordinates of the first target area can be determined according to the historical inspection records of the intelligent elderly care service robot in various ways. An embodiment of determining the coordinates of the first target area according to the historical inspection records of the intelligent elderly care service robot.

[0041] The preset navigation algorithm may be a commonly used navigation algorithm executed by the intelligent elderly care service robot terminal, and is not limited by the description of this specification. For example, based on the coordinates of the first target area, a set of scene images obtained according to the historical inspection records of the intelligent elderly care service robot may be used to obtain a 2D occupancy map, and navigation may be performed to reach the first target area according to the 2D occupancy map.

[0042] Step 20: Acquire point cloud data information near the first target area and point cloud data information of the target object.

[0043] In the specific implementation process, after the intelligent elderly care service robot reaches the first target area, it can use the camera equipment it carries to collect nearby point cloud data information to obtain extensive and up-to-date spatial information around the target object. As an example only, the following method can be used to obtain rich point cloud data information near the target object: After the intelligent elderly care service robot reaches the first target area, it is positioned to directly face the target object. Subsequently, the intelligent elderly care service robot rotates to the left and right by an angle of α0=30°, during which the intelligent elderly care service robot captures images and depth data around the target object. Subsequently, the intelligent elderly care service robot tilts downward at a specified angle of α1=60° and makes an additional rotation of α0 to the left and right, during which the intelligent elderly care service robot captures images and depth data around the target object. The above-mentioned action sequence of the intelligent elderly care service robot enables the intelligent elderly care service robot to capture detailed images and depth data around the target object. Using the above-mentioned acquired images and depth data, a detailed point cloud P near the target object is constructed.

[0044] In the specific implementation process, the intelligent elderly care service robot can obtain the point cloud data information of the target object after arriving at the first target area. As an example only, after the intelligent elderly care service robot arrives at the first target area, it takes an image facing the target object, and can use image segmentation models (for example, SEEM (Segment-Everything-Everywhere-All-At-Once) and SAM (Segment Anything Model)) to divide the image into multiple regions based on semantic content. Then, each segmented image area is colored, and each image area is identified with a number. After the image carrying the color and digital identification is input into the visual language model together with the original image, the visual language model is guided to identify the target object from the image carrying the color and digital identification according to the original image, and the digital identification of the target object is output. Using the digital identification and related depth data, the precise coordinate sequence of the target object can be obtained; according to the coordinate sequence and the related depth data, a 3D point cloud P of the target object is constructed. q .

[0045] Step 30: Determine a plurality of candidate regions around the target object based on the point cloud data information near the first target region and the point cloud data information of the target object.

[0046] The shape of the candidate area may be a shape suitable for the intelligent elderly care service robot to stand, for example, a circle, an ellipse, etc.

[0047] In the specific implementation process, multiple candidate areas can be determined around the target object in a variety of ways based on the point cloud data information near the first target area and the point cloud data information of the target object, which is not limited to the description of this specification.

[0048] After obtaining multiple candidate regions, subsequent steps can be used to select a location region suitable for performing the target task.

[0049] Step 40: Project the multiple candidate areas onto an image captured when the intelligent elderly care service robot reaches the first target area and faces the target object, to obtain first image data carrying information of the multiple candidate areas.

[0050] In the specific implementation process, when the intelligent elderly care service robot moves to the first target area, it can use the camera device it carries to capture an image of the current position. Then, multiple candidate areas are projected onto the image captured when the intelligent elderly care service robot reaches the first target area and faces the target object, to obtain the first image data. The first image data can fully reflect the positional relationship between the intelligent elderly care service robot and the target object, and carry information on multiple candidate areas at the same time.

[0051] Step 50: Determine a second target region from multiple candidate regions using a visual language model according to the first image data and information about the target task.

[0052] During the specific implementation process, when the computing resources of the intelligent elderly care service robot terminal are abundant, the visual language model can be deployed in the intelligent elderly care service robot terminal; when the computing resources of the intelligent elderly care service robot terminal are limited, the visual language model can be deployed on a cloud server.

[0053] In a specific implementation process, the first image data and information of the target task may be input into a visual language model, and the powerful semantic understanding capability of the visual language model may be utilized to determine the second target area from multiple candidate areas according to the information of the target task.

[0054] As an example only, multiple candidate regions in the first image data can be digitally labeled, and the labeled first image data can be input into the visual language model, and the information of the target task can be input into the visual language model. The visual language model uses its powerful semantic understanding ability to score multiple candidate regions according to the information of the target task (for example, opening the refrigerator door), and outputs the digital identification of the candidate region that is most suitable for performing the target task, thereby obtaining the second target region.

[0055] Step 60: Use a preset navigation algorithm to guide the intelligent elderly care service robot to reach a second target area, so that the intelligent elderly care service robot performs a target task based on the second target area.

[0056] During the specific implementation process, a preset navigation algorithm can be used to generate a path to the center of the second target area, and the intelligent elderly care service robot can be driven along the path to make the intelligent elderly care service robot reach the center position of the second target area.

[0057] In the specific implementation process, due to various reasons, the second target area is not the best location area suitable for performing the target task. As an example only, the target task is "open the refrigerator door", and the first target position is located on the right side of the refrigerator. Due to the occlusion of the refrigerator, the first image data cannot fully display the best candidate area on the front of the refrigerator, resulting in the visual language model not selecting the best location area for performing the target task from multiple candidate areas. For example, the candidate area on the right side of the front of the refrigerator is selected as the second target area, while the ideal second target area is located in the middle of the front of the refrigerator.

[0058] In one embodiment, in order to obtain a more accurate location area suitable for performing a target task, information of multiple candidate areas can be projected onto an image captured by the intelligent elderly care service robot after it reaches the second target area and faces the target object, and the candidate areas are marked with numbers to obtain second image data carrying information of multiple candidate areas; the second image data and information of the target task are input into a visual language model so that the visual language model determines a third target area from multiple candidate areas (such as outputting a digital identifier of the third target area) based on the information of the target task; and a preset navigation algorithm is used to guide the intelligent elderly care service robot from the second target area to the third target area, and then the intelligent elderly care service robot is driven to rotate to face the target object, so that the intelligent elderly care service robot performs the target task in the third target area.

[0059] Just as an example, after the intelligent elderly care service robot receives the target task of "turning on the table lamp on the bedside table", in the first stage, the visual language model is used to obtain the coordinates of the first target area according to a set of scene images obtained from the inspection record, and the 2D occupancy map to the first target area is obtained according to the set of scene images. In the second stage, after reaching the first target area using the 2D occupancy map, the intelligent elderly care service robot collects point cloud data information near the first target area from various angles, and constructs a point cloud near the target object. In the third stage, based on the constructed point cloud, multiple candidate areas around the target object are determined, and the visual language model is used to determine the second target area from multiple candidate areas, and the intelligent elderly care service robot is guided to the second target area. In the fourth stage, the third target area is re-determined based on the image collected in the second target area, and after the intelligent elderly care service robot is guided to reach the third target area, it stares at the table lamp, and then the intelligent elderly care service robot can perform the task of turning on the table lamp in the best position and posture.

[0060] In an embodiment of the present invention, when an intelligent elderly care service robot receives a target task to be performed on a target object, a preset navigation algorithm is used to guide the intelligent elderly care service robot to a first target area within a preset range of the target object; multiple candidate areas are determined around the target object based on point cloud data information near the first target area and point cloud data information of the target object; the multiple candidate areas are projected onto an image taken when the intelligent elderly care service robot faces the target object after reaching the first target area, so as to obtain first image data carrying information of multiple candidate areas; the powerful semantic understanding ability and image processing ability of the visual language model are used to determine a second target area suitable for performing the target task from multiple candidate areas based on the first image data and information of the target task; the preset navigation algorithm is used to guide the intelligent elderly care service robot to the second target area, so that the intelligent elderly care service robot performs the target task based on the second target area, thereby reducing the distance between the navigation destination of the intelligent elderly care service robot and the location where the target task is performed, and effectively improving the efficiency and flexibility of the intelligent elderly care service robot in performing tasks within a preset range.

[0061] In one embodiment, a method for determining a plurality of candidate regions around a target object includes the following steps:

[0062] Step 201: Project the point cloud data information near the first target area and the point cloud data information of the target object into a two-dimensional grid map centered on the projection center of the target object to obtain a two-dimensional grid map carrying the point cloud data information.

[0063] As an example, a two-dimensional grid map M can be constructed on the ground. s , each grid side length is r q +1, resolution 1 cm, in c q Centered. q is the radius of the target object, c q is the point cloud P of the target object q The 2D projection center of each grid reserves a space for the intelligent elderly care service robot to stand. The plane extraction algorithm RANSAC is used to extract the ground point cloud data information P from the constructed point cloud P g Then, the non-ground point cloud and the ground point cloud P g , and the point cloud P of the target object q Projecting it into a two-dimensional grid map, we get a two-dimensional grid map carrying point cloud data information. Each grid can take one of four possible values: 0 indicates no projection point; 0 indicates an obstacle point; 1 indicates an available ground point; 2 indicates an object point of the target object (a special type of obstacle). The hierarchy of these values ​​(from highest priority to lowest priority) is: 2, 1, 0, The precedence here implies replaceability - for example, if a raster that was originally 0 receives a projection of 2, it will be updated to 2; otherwise, it will remain unchanged.

[0064] Step 202: using the two-dimensional grid map carrying point cloud data information, determine multiple candidate areas around the target object.

[0065] In the specific implementation process, the two-dimensional grid map can be re-divided into multiple grids of preset sizes. With the vertex of each grid as the initial center, a circle of preset radius is generated to obtain multiple first areas. Among them, the preset radius of the first area can be determined according to the circumference of the intelligent elderly care service robot, so that the intelligent elderly care service robot can stand in the circular area.

[0066] A first area between a first preset range and a second preset range is selected from the plurality of first areas to obtain a plurality of second areas, wherein the first preset range and the second preset range are both centered on the target object, and the first preset range is smaller than the second preset range.

[0067] The positions of the plurality of second areas are adjusted so that the minimum distance between the centers of the plurality of second areas and the target object is equal to a preset distance threshold. The preset distance threshold is determined according to the target task. For example, for the target task of taking an item from a table cabinet, the preset distance threshold can be determined according to the circumference of the intelligent elderly care service robot. For another example, for the target task of opening a refrigerator door, the preset distance threshold can be determined according to the length of the refrigerator door after it is opened and the circumference of the intelligent elderly care service robot.

[0068] Finally, from the multiple second areas after the positions are adjusted, multiple second areas whose grids of the two-dimensional grid map are non-intersecting and corresponding to the grids of the two-dimensional grid map are available ground points are selected to obtain multiple candidate areas. Whether the grid on the two-dimensional grid map is an available ground point is determined based on the point cloud data information it carries.

[0069] As an example only, in order to ensure that the intelligent elderly care service robot can correctly face the target object, the two-dimensional grid map M established in step 201 s Generate k circular candidate regions {R i}1 K , the radius of each candidate region is r r (The radius of the intelligent elderly care service robot with a circular base.) The specific steps of obtaining the candidate area are as follows.

[0070] First, in the two-dimensional grid map M s Create a new grid map M s1 , whose resolution is r r / 3, use the mesh vertices as the initial centers of the circles to generate the first regions of multiple circles.

[0071] The minimum distance between the center and the projection center of the target object is less than r r / 2 or more than 3r r / 2, the first region is deleted to obtain the second region. In a specific implementation, the center points of the remaining candidate regions of different types on the grid can be saved in different octrees to facilitate the subsequent rapid calculation of the nearest neighbor distance.

[0072] Then, the shortest distance from the center of each second area to the projection center of the target object and the direction from the center of the second area to the projection center of the target object corresponding to the shortest distance are calculated. Then, the second area is moved in the opposite direction of the direction to ensure that the shortest distance from the center of the second area to the projection point of the target object is equal to the preset distance threshold.

[0073] Finally, a second area is randomly selected from the second areas after the adjustment, and the areas intersecting with the second area are deleted. This step is repeated until there are no intersecting second areas. Finally, a second area whose grid of the corresponding two-dimensional grid map is an available ground point is selected from the multiple non-intersecting second areas as the final candidate area.

[0074] In the embodiment provided by the present invention, after the intelligent elderly care service robot reaches the first target area, the collected nearby point cloud data information is projected into a two-dimensional grid map centered on the projection center of the target object to obtain a two-dimensional grid map carrying the point cloud data information; using the two-dimensional grid map carrying the point cloud data information, multiple candidate areas are determined around the target object. Since the candidate areas are determined using the point cloud data information collected near the target object, information that can accurately reflect the candidate areas around the target object that are suitable for performing the target task can be obtained.

[0075] In one embodiment, the method for determining the coordinates of the first target area includes the following steps:

[0076] Step 301: obtain multiple sets of images containing coordinate information of the target object from the historical inspection records of the intelligent elderly care service robot; wherein the multiple sets of images are taken at preset distance intervals along the path from the current location of the intelligent elderly care service robot to the location of the target object.

[0077] Just as an example, the intelligent elderly care service robot is a home intelligent elderly care service robot. Before starting home work, the home intelligent elderly care service robot needs to patrol the indoor environment, collect scene images and coordinate information of various indoor locations in the process, and record the collected information.

[0078] Step 302: Using the visual language model, select an image that is closest to the target object from multiple groups of images as the target image.

[0079] In some embodiments, a scoring function can be developed using a visual language model, where the scoring function is used to calculate the image set S = {I i}1 N Each image I in i The probability p(I i ,q,S), the specific implementation method is as follows.

[0080] In the specific implementation process, in order to obtain a more accurate recognition effect, each image can be marked with a number of a preset color (e.g., red) at a preset position (e.g., the upper left corner of the image). The image with the digital mark is then input into the visual language model.

[0081] After the image is input into the visual language model, the visual language model is guided to identify the image containing the target object from the input multiple sets of images, and then the clearest image is identified from the images containing the target object, and the digital identification of the clearest image is output. Finally, the target image is determined using the digital identification.

[0082] Step 303: Obtain coordinate information of the first target area according to the coordinate information carried by the target image.

[0083] In a specific implementation process, after obtaining the identifier of the target image output by the visual language model, the coordinate information carried by the target image is used as the coordinate information of the first target area.

[0084] In the embodiment provided by the present invention, multiple groups of images carrying coordinate information of the target object are obtained from the historical inspection records of the intelligent elderly care service robot; the powerful image processing capability of the visual language model is utilized to accurately identify the image closest to the target object from the multiple groups of images as the target image, thereby obtaining the accurate coordinates of the first target area.

[0085] See also Figure 3 , Figure 3 Schematic diagram of an embodiment of the electronic device provided by the present invention. Figure 3As shown, an embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, the following steps are implemented:

[0086] In response to receiving a target task performed by a target user on a target object, using a preset navigation algorithm, guiding the intelligent elderly care service robot to a first target area within a preset range of the target object; the coordinates of the first target area are determined according to a historical inspection record of the intelligent elderly care service robot;

[0087] Acquire point cloud data information near the first target area and point cloud data information of the target object;

[0088] Determine a plurality of candidate areas around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object;

[0089] Projecting the multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, to obtain first image data carrying information of the multiple candidate areas;

[0090] Determine a second target region from the plurality of candidate regions using a visual language model according to the first image data and information about the target task;

[0091] The preset navigation algorithm is used to guide the intelligent elderly care service robot to reach the second target area, so that the intelligent elderly care service robot performs the target task based on the second target area.

[0092] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the intelligent elderly care service method based on deep vision provided by the above methods, and the method includes:

[0093] In response to receiving a target task performed by a target user on a target object, using a preset navigation algorithm, guiding the intelligent elderly care service robot to a first target area within a preset range of the target object; the coordinates of the first target area are determined according to a historical inspection record of the intelligent elderly care service robot;

[0094] Acquire point cloud data information near the first target area and point cloud data information of the target object;

[0095] Determine a plurality of candidate areas around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object;

[0096] Projecting the multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, to obtain first image data carrying information of the multiple candidate areas;

[0097] Determine a second target region from the plurality of candidate regions using a visual language model according to the first image data and information about the target task;

[0098] The preset navigation algorithm is used to guide the intelligent elderly care service robot to reach the second target area, so that the intelligent elderly care service robot performs the target task based on the second target area.

[0099] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the intelligent elderly care service method based on deep vision provided by the above methods, the method comprising:

[0100] In response to receiving a target task performed by a target user on a target object, using a preset navigation algorithm, guiding the intelligent elderly care service robot to a first target area within a preset range of the target object; the coordinates of the first target area are determined according to a historical inspection record of the intelligent elderly care service robot;

[0101] Acquire point cloud data information near the first target area and point cloud data information of the target object;

[0102] Determine a plurality of candidate areas around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object;

[0103] Projecting the multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, to obtain first image data carrying information of the multiple candidate areas;

[0104] Determine a second target region from the plurality of candidate regions using a visual language model according to the first image data and information about the target task;

[0105] The preset navigation algorithm is used to guide the intelligent elderly care service robot to reach the second target area, so that the intelligent elderly care service robot performs the target task based on the second target area.

[0106] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0107] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0111] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0112] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An intelligent elderly care service robot based on deep vision, It is characterized in that The intelligent elderly care service robot includes a robot control center, a first navigation module, a first acquisition module, a first determination module, a second acquisition module, a second determination module and a second navigation module; the robot control center is respectively connected to the first navigation module, the first acquisition module, the first determination module, the second acquisition module, the second determination module and the second navigation module to manage and control each module; The first navigation module is used to, in response to receiving a target task performed by a target user on a target object, guide the intelligent elderly care service robot to a first target area within a preset range of the target object using a preset navigation algorithm; wherein the coordinates of the first target area are determined according to a historical inspection record of the intelligent elderly care service robot; The first acquisition module is used to acquire point cloud data information near the first target area and point cloud data information of the target object; The first determination module is used to determine a plurality of candidate areas around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object; A second acquisition module is used to project the multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, to obtain first image data carrying information of the multiple candidate areas; The second determination module is used to determine a second target area from the multiple candidate areas according to the first image data and information of the target task by using a visual language model; The second navigation module is used to guide the intelligent elderly care service robot to reach the second target area using a preset navigation algorithm, so that the intelligent elderly care service robot performs the target task based on the second target area.

2. A smart elderly care service method based on deep vision, It is characterized in that include: In response to receiving a target task performed by a target user on a target object, guiding the intelligent elderly care service robot to a first target area within a preset range of the target object using a preset navigation algorithm; The coordinates of the first target area are determined according to the historical inspection records of the intelligent elderly care service robot; Acquire point cloud data information near the first target area and point cloud data information of the target object; Determine a plurality of candidate areas around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object; Projecting the multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the first target area and faces the target object, to obtain first image data carrying information of the multiple candidate areas; Determine a second target region from the plurality of candidate regions using a visual language model according to the first image data and information about the target task; The preset navigation algorithm is used to guide the intelligent elderly care service robot to reach the second target area, so that the intelligent elderly care service robot performs the target task based on the second target area.

3. According to the deep vision-based intelligent elderly care service method of claim 2, It is characterized in that The step of determining a plurality of candidate areas around the target object according to the point cloud data information near the first target area and the point cloud data information of the target object comprises: Projecting the point cloud data information near the first target area and the point cloud data information of the target object into a two-dimensional grid map centered on the projection center of the target object to obtain a two-dimensional grid map carrying the point cloud data information; Using the two-dimensional grid map carrying the point cloud data information, a plurality of candidate areas are determined around the target object.

4. According to claim 3, the intelligent elderly care service method based on deep vision, It is characterized in that The method of using the two-dimensional grid map carrying the point cloud data information to determine a plurality of candidate areas around the target object includes: Dividing the two-dimensional grid map into a plurality of grids of preset sizes; Taking each vertex of the grid as the initial center, a circle with a preset radius is generated to obtain a plurality of first areas; wherein the preset radius is determined according to the circumference of the intelligent elderly care service robot; Selecting a first area between a first preset range and a second preset range from the plurality of first areas to obtain a plurality of second areas; wherein the first preset range and the second preset range are both centered on the target object, and the first preset range is smaller than the second preset range; Adjusting the positions of the plurality of second areas so that the minimum distance between the centers of the plurality of second areas and the target object is equal to a preset distance threshold; wherein the preset distance threshold is determined according to the target task; From the multiple second areas after adjusting the position, multiple second areas that are non-intersecting and correspond to the grids of the two-dimensional grid map as available ground points are selected as the multiple candidate areas; wherein whether the grid on the two-dimensional grid map is an available ground point is determined based on the point cloud data information it carries.

5. According to claim 2, the intelligent elderly care service method based on deep vision, It is characterized in that The using the visual language model to determine the second target area from the multiple candidate areas according to the first image data and the information of the target task includes: The first image data and the information of the target task are input into the visual language model, so that the visual language model determines a second target region from the multiple candidate regions according to the information of the target task.

6. According to claim 5, the intelligent elderly care service method based on deep vision, It is characterized in that The method further comprises: Projecting the information of the multiple candidate areas onto an image captured by the intelligent elderly care service robot when it reaches the second target area and faces the target object, to obtain second image data carrying the information of the multiple candidate areas; Inputting the second image data and the information of the target task into the visual language model, so that the visual language model determines a third target region from the multiple candidate regions according to the information of the target task; The preset navigation algorithm is used to guide the intelligent elderly care service robot from the second target area to the third target area, so that the intelligent elderly care service robot performs the target task in the third target area.

7. The intelligent elderly care service method based on deep vision according to claim 2, It is characterized in that The coordinates of the first target area are determined using the following method: Acquire multiple sets of images containing coordinate information of the target object from the historical inspection records of the intelligent elderly care service robot; wherein the multiple sets of images are taken at preset distance intervals along the path from the current location of the intelligent elderly care service robot to the location of the target object; Using the visual language model, selecting an image closest to the target object from the multiple groups of images as the target image; The coordinate information of the first target area is obtained according to the coordinate information carried by the target image.

8. An electronic device, It is characterized in that It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the intelligent elderly care service method based on deep vision as described in any one of claims 2 to 7 is implemented.

9. A non-transitory computer-readable storage medium, It is characterized in that The storage medium stores a computer software program, which, when executed by a processor, implements the intelligent elderly care service method based on deep vision as described in any one of claims 2 to 7.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by the processor, the intelligent elderly care service method based on deep vision as described in any one of claims 2 to 7 is implemented.

Citation Information

Patent Citations

  • Intelligent old-age service robot based on ROS and deep vision

    CN115107057A

  • Route navigation method, related device, and computer readable storage medium

    WO2020024234A1

Cited By

  • Robot positioning and navigation method and device based on large language model thinking chain

    CN119533474A