Manipulator control method and related device for object picking

Abstract

The present application provides a manipulator control method and related device for object picking, the method comprising: obtaining input data including voice data and / or gesture data of a user wearing the manipulator, the gesture data including the joint state of the manipulator, the joint state including the joint angle and the joint direction; based on the input data, identifying the object picking intention; obtaining a first environmental image through an image acquisition device arranged on the manipulator; identifying the first environmental image to determine whether the user's first picking range includes a target object corresponding to the object picking intention; in response to the recognition result that the first picking range includes the target object, adjusting the joint state of the manipulator so that the manipulator presents a target hand shape, wherein the joint state of the manipulator under the target hand shape is determined according to the position of the target object, the position of the manipulator and the object shape of the target object. The present application is conducive to improving the safety and efficiency of picking up objects by visually impaired persons based on manipulator control.

Description

Technical Field

[0001] The present application relates to the field of robot control, and in particular to a robot control method and related devices for object picking. Background Art

[0002] Due to limited vision, visually impaired people face many inconveniences in their daily lives. With the development of science and technology, there are already some special tools developed for the visually impaired on the market, such as navigation tools for the visually impaired, which have facilitated the lives of the visually impaired to a certain extent. However, the development of special tools is currently mainly focused on travel issues, while little attention is paid to the problem of picking up items in the daily activities of the visually impaired. Currently, the visually impaired mainly rely on touch to pick up items, which has problems of insufficient safety and low efficiency. Summary of the invention

[0003] The embodiments of the present application provide a robot control method and related devices for picking up objects, in order to improve the safety and efficiency of visually impaired people in picking up objects based on robot control.

[0004] In a first aspect, an embodiment of the present application provides a manipulator control method for picking up an object, comprising:

[0005] Acquire input data, where the input data includes voice data and / or gesture data of a user, where the user is a user wearing the manipulator, and the gesture data includes a joint state of the manipulator, where the joint state includes a joint angle and a joint direction;

[0006] Based on the input data, identifying an object picking intention;

[0007] Acquire a first environment image by an image acquisition device arranged on the manipulator;

[0008] Identify the first environment image to determine whether the first picking range of the user includes a target item corresponding to the item picking intention;

[0009] In response to the recognition result that the first picking range includes the target object, the joint state of the manipulator is adjusted so that the manipulator presents a target hand shape, wherein the joint state of the manipulator under the target hand shape is determined according to the position of the target object, the position of the manipulator and the object shape of the target object.

[0010] In a second aspect, an embodiment of the present application provides a manipulator control device for picking up an object, comprising:

[0011] an acquisition unit, configured to acquire input data, wherein the input data includes voice data and / or gesture data of a user, wherein the user is a user wearing the manipulator, and the gesture data includes a joint state of the manipulator, wherein the joint state includes a joint angle and a joint direction;

[0012] A first recognition unit, configured to recognize an object picking intention based on the input data;

[0013] An acquisition unit, configured to acquire a first environment image through an image acquisition device disposed on the manipulator;

[0014] A second recognition unit, configured to recognize the first environment image to determine whether the first picking range of the user includes a target item corresponding to the item picking intention;

[0015] An adjustment unit is used to adjust the joint state of the manipulator in response to the recognition result that the first picking range includes the target object, so that the manipulator presents a target hand shape, wherein the joint state of the manipulator under the target hand shape is determined according to the position of the target object, the position of the manipulator and the object shape of the target object.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a communication interface, wherein the processor, the memory, and the communication interface are interconnected, wherein the communication interface is used to receive or send data, the memory is used to store application code for a server to execute the method of the first aspect, and the processor is configured to execute the method of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium that stores a computer program for electronic data exchange, wherein the above-mentioned computer program enables a computer to execute part or all of the steps described in the method of the first aspect of the embodiment of the present application.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0019] It can be seen that in the embodiment of the present application, the electronic device recognizes the intention to pick up the item based on the acquired voice data of the user wearing the manipulator and / or the gesture data including the joint state of the manipulator, and then acquires the first environmental image through the image acquisition device set on the manipulator, and then recognizes the first environmental image to determine whether the user's first picking range includes the target item corresponding to the intention to pick up the item, and in response to the recognition result that the first picking range includes the target item, adjusts the joint state of the manipulator so that the manipulator presents the target hand shape, wherein the joint state includes the joint angle and the joint direction, and the joint state of the manipulator under the target hand shape is determined according to the position of the target item, the position of the manipulator, and the item shape of the target item. That is to say, the electronic device can recognize the user's intention to pick up the item based on the user's voice data and / or the gesture data of the user using the manipulator, and adjust the joint state of the manipulator by collecting the environmental image through the manipulator, so that the manipulator presents the target hand shape corresponding to the position of the target item in the environmental image, the position of the manipulator, and the item shape of the target item, and assists the user to understand which gesture to use to pick up the item without multiple touches, which is conducive to improving the safety and efficiency of the visually impaired person picking up the item. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a structural schematic diagram of a manipulator control system provided in an embodiment of the present application;

[0022] Figure 2 It is a structural schematic diagram of a manipulator provided in an embodiment of the present application;

[0023] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0024] Figure 4 It is a flow chart of a manipulator control method for picking up objects provided in an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of an object picking gesture provided in an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of another object picking gesture provided in an embodiment of the present application;

[0027] Figure 7 This is a schematic diagram of information prompts under a location condition provided by an embodiment of the present application;

[0028] Figure 8 This is a schematic diagram of information prompts under another location condition provided by an embodiment of the present application;

[0029] Fig. 9 It is a structural schematic diagram of a manipulator control device for picking up objects provided in an embodiment of the present application;

[0030] Fig.10 It is a schematic diagram of the structure of another manipulator control device for picking up objects provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] The embodiments of the present application are described below with reference to the accompanying drawings.

[0035] Reference Figure 1, the embodiment of the present application provides a manipulator control system 10, which may include a manipulator 102 and an electronic device 101. The electronic device 101 may specifically be a personal computer or a server, and the personal computer may be, for example, a smart phone, a tablet computer, a laptop computer, a desktop computer, a wearable device, a head-mounted device, etc., which is installed with an application (i.e., an application client) for controlling the manipulator 102. No specific restrictions are made here. In actual applications, the electronic device 101 may also be directly set on the manipulator 102 as a part of the manipulator 102. The manipulator 102 and the electronic device 101 may be connected to each other by wired or wireless communication to achieve data interaction between them.

[0036] Specifically, the electronic device 101 can obtain input data, which includes voice data and / or gesture data of a user, and the user is a user wearing the manipulator 102, and the gesture data includes a joint state of the manipulator, and the joint state includes a joint angle and a joint direction; based on the input data, an intention to pick up an object is identified; a first environmental image is acquired by an image acquisition device provided on the manipulator; the first environmental image is identified to determine whether a first picking range of the user includes a target object corresponding to the intention to pick up the object; in response to an identification result that the first picking range includes the target object, the joint state of the manipulator is adjusted so that the manipulator presents a target hand shape, wherein the joint state of the manipulator under the target hand shape is determined according to the position of the target object, the position of the manipulator and the object shape of the target object.

[0037] The electronic device 101 adjusts the joint state of the manipulator 102 by sending the joint angle parameters and joint direction parameters corresponding to the joints of the manipulator under the target hand shape to the manipulator 102. The manipulator 102 adjusts the joint state according to the received joint angle parameters and joint direction parameters, so that the manipulator presents the target hand shape. At this time, the user wearing the manipulator can accurately perceive which hand shape should be used to pick up items, thereby improving the safety and efficiency of picking up items.

[0038] It should be noted that Figure 1 The number of manipulators 102 is only for illustrative purposes. In actual applications, the number of manipulators 102 to which the electronic device 101 is communicatively connected may be one or more, and no specific limitation is made here.

[0039] Reference Figure 2 The manipulator 102 in the present application may specifically include a finger joint driver 01, a battery and a battery cover 02, a power adapter interface 03, a power switch 04, a palm fixing strap 05 and a finger fixing strap 06.

[0040] The manipulator 102 can specifically adjust the joint state of each joint through the finger joint driver 01 corresponding to each joint.

[0041] It should be noted that Figure 2 The structure of the manipulator 102 is only an example. In actual applications, the specific structures included in the manipulator 102 may be more or less, and the appearance of the manipulator 102, the type of each specific structure included in the manipulator 102, and the location of each specific structure may also be different. Figure 2 For example, an auxiliary switch button for picking up an item may be provided on the manipulator 102 (the electronic device 101 obtains the user's voice data and / or gesture data and adjusts the joints of the manipulator 102 to the target hand shape only when the auxiliary switch button is detected to be in an on state; when the auxiliary switch button is detected to be in an off state, it is assumed that the user does not use the manipulator 102 for auxiliary picking operations, and the electronic device 101 may not obtain the user's voice data and / or gesture data), or a camera may be provided on the manipulator 102, etc. No specific restrictions are made here.

[0042] The composition structure of the electronic device in this application can be as follows Figure 3 As shown, the electronic device may include a processor 110, a memory 120, a communication interface 130, and one or more programs 121, wherein the one or more programs 121 are stored in the above-mentioned memory 120 and are configured to be executed by the above-mentioned processor 110, and the one or more programs 121 include instructions for executing any step in the following method embodiment. Among them, the communication interface 130 is used to support communication between the electronic device and other devices. In a specific implementation, the processor 110 is used to execute any step performed by the electronic device in the following method embodiment, and when performing data transmission such as sending, the communication interface 130 may be selectively called to complete the corresponding operation. It should be noted that the structural diagram of the above-mentioned electronic device is only an example, and the specific devices included may be more or less, and are not uniquely limited here.

[0043] See also Figure 4 , Figure 4 is a flow chart of a manipulator control method for picking up objects provided in an embodiment of the present application, such as Figure 4 As shown, the robot control method for object picking includes the following steps:

[0044] S201, the electronic device obtains input data.

[0045] The input data includes voice data and / or gesture data of the user, the user is a user wearing the robotic arm, the gesture data includes the joint state of the robotic arm, and the joint state includes the joint angle and joint direction.

[0046] In a specific implementation, the triggering condition for the electronic device to obtain input data may be any of the following, for example: detecting that the target application is started; detecting that the target application is started, and determining that the user is a preset type of user based on the login information of the target application; detecting that the manipulator establishes a communication connection with the electronic device; detecting that the target button on the manipulator (such as the aforementioned auxiliary switch button) is in the on state; obtaining user voice including preset keywords; detecting that the joint state of the manipulator is the preset state. Based on the above triggering conditions, the manipulator can be conveniently used to assist in picking up items without the user performing complex operations, which can better improve the user experience of visually impaired users.

[0047] The electronic device can obtain the user's voice data through an audio collection device such as a microphone, and the audio collection device can be set on the electronic device or the manipulator, or the audio collection device can be set on other devices connected to the electronic device, such as Bluetooth headphones, etc. When obtaining gesture data, the electronic device can directly obtain the joint state data of the manipulator.

[0048] S202: The electronic device identifies the intention to pick up the item based on the input data.

[0049] The electronic device may identify the intention to pick up an object by, for example, identifying voice data, determining text information corresponding to the voice data, and performing keyword recognition on the text information to determine the target object to be picked up. Alternatively, the electronic device may store a correspondence between preset gesture data and target objects in a local or cloud database, and the electronic device may determine the target object corresponding to the gesture data by querying the database.

[0050] For example, taking the input data as voice data, the electronic device may determine the water cup as the target object when recognizing that the text information corresponding to the voice data includes the preset keyword "water cup".

[0051] In a specific implementation, after the electronic device determines the target object according to the input data, it can also output the characteristic information of the target object through voice, such as the name and color of the target object, and end the identification of the object picking intention when the user's confirmation operation is detected. If the user does not confirm after the target object is output, the electronic device can re-determine the target object and output the characteristic information of the target object again until the user confirms the target object. For example, when the electronic device confirms that there are multiple water cups in the environmental image based on the preset keyword "water cup" in the text information corresponding to the user's voice data, it can also identify the color of the water cup, and then output specific characteristic information such as "yellow water cup" and "red water cup" through voice, so as to provide the user with the differences between different objects that cannot be perceived by their own sense of touch, so as to facilitate the user to distinguish and confirm, thereby improving the accuracy of the determination of the object picking intention.

[0052] Among them, the confirmation operation can, for example, be to recognize that the text information corresponding to the user's voice data contains words with positive semantics after outputting the characteristic information of the target object, or to determine that the user has made a preset confirmation gesture based on the joint state of the robot, or to detect that the user has pressed a confirmation button set on the robot, etc.

[0053] S203: The electronic device acquires a first environment image through an image acquisition device disposed on the manipulator.

[0054] S204: The electronic device identifies the first environment image to determine whether the first picking range of the user includes a target item corresponding to the item picking intention.

[0055] The first picking range of the user may specifically be the position area that the robot can cover in the first environment image, and specifically may be the projection area of ​​the robot projected onto the plane where the target object is located in the first environment image. On this basis, if it is determined that the projection area, i.e., the first picking range, includes the target object, then when the user's palm falls onto the plane where the target object is located, the palm may touch the target object.

[0056] S205: In response to the recognition result that the first picking range includes the target object, the electronic device adjusts the joint state of the manipulator so that the manipulator presents a target hand shape.

[0057] The joint state of the manipulator under the target hand shape is determined according to the position of the target object, the position of the manipulator and the object shape of the target object.

[0058] In a specific implementation, if the electronic device recognizes that the first environmental image does not include the target object, it can also prompt the user to move the manipulator through voice or by using the manipulator joints to drive the user's joints to move in a specific direction, and continuously collect new environmental images through the manipulator during the movement until it recognizes that the newly collected environmental image includes the target object.

[0059] Specifically, the electronic device can determine the direction to prompt the user to move based on the object information recognized in the first environmental image. For example, the target object that the user needs to pick up is a water cup placed in the upper right corner of the desktop. Since the user does not know the location of the water cup, he may grope for it in the lower left corner of the desktop. At this time, the water cup cannot be recognized in the first environmental image collected by the manipulator, and only a partial incomplete desktop image can be recognized. This means that the manipulator and the desktop are too close. The electronic device can prompt the user to move the manipulator away from the desktop, and then the manipulator can obtain a new environmental image including a more complete desktop, until the target object is recognized from the new environmental image and the target object is within the user's picking range. When executing step S205, the joint state of the manipulator can be adjusted according to the new environmental image to reduce the time the user spends groping for the object.

[0060] Alternatively, when prompting the user to move the robot arm, the direction in which the user is prompted to move can be determined based on the item type of the identified item and the item type of the target item. Taking the target item as a water cup as an example, if the water cup is not identified in the first environmental image, but two items, a kettle and a socket, are identified, since the types of the kettle and the water cup match, the user can be prompted to move the robot arm toward the kettle to obtain a new environmental image, until the target item is identified as being within the user's picking range based on the environmental image.

[0061] In particular, when a target object is identified in an environmental image but is not within the user's picking range, the electronic device can also determine the distance between the target object and the robot arm based on the environmental image, and prompt the user to move their own position or only move the arm wearing the robot arm based on the distance.

[0062] In particular, when determining the direction in which to prompt the user to move, the electronic device can also identify dangerous information in the environmental image (such as sharp parts of objects, depressions in the road surface, etc.), and then avoid the direction of the dangerous information when determining the direction of movement.

[0063] Among them, when determining the joint state of the manipulator according to the position of the target object, the position of the manipulator and the object shape of the target object, specifically, the relative position relationship between the target object and the manipulator can be determined according to the position of the target object and the position of the manipulator, and then at least one alternative gesture can be determined according to the object shape and the relative position relationship of the target object, and the dangerous part of the target object can be determined according to the object shape of the target object, and dangerous gestures that will touch the dangerous part are excluded from the at least one alternative gesture, and the target hand shape is determined from the remaining alternative gestures after excluding the dangerous gestures, and the joint state of the manipulator corresponding to the target hand shape is determined.

[0064] For example, if there is a sharp object placed vertically, and the user's palm is currently above the sharp object, there are two alternative gestures, one of which is as follows: Figure 5As shown in the figure, the user grabs the item directly with the palm facing down, and the other is as follows Figure 6 As shown in FIG. 1 , the user tilts his palm to a vertical position to grab an object from the side. At this time, since the palm is directly above the sharp object, if the user Figure 5 If you use the gesture to pick up an item, you will directly touch the sharp part above the item, so you need to Figure 5 If the gesture is excluded, Figure 6 The gesture is determined as the target hand shape.

[0065] In addition, in actual applications, since the user's hand is actually still some distance away from the target object when the target hand shape is determined in step S205, the user needs to move to grasp the object. After the robot presents the target hand shape for the user to perceive, the normal hand shape can be restored and the robot will no longer be driven to actively adjust the joint shape, so as to avoid the robot fixing the user's hand shape during the movement, causing the user to be unable to pick up the object correctly or colliding with the object.

[0066] It can be seen that in the embodiment of the present application, the electronic device recognizes the intention to pick up the item based on the acquired voice data of the user wearing the manipulator and / or the gesture data including the joint state of the manipulator, and then acquires the first environmental image through the image acquisition device set on the manipulator, and then recognizes the first environmental image to determine whether the user's first picking range includes the target item corresponding to the intention to pick up the item, and in response to the recognition result that the first picking range includes the target item, adjusts the joint state of the manipulator so that the manipulator presents the target hand shape, wherein the joint state includes the joint angle and the joint direction, and the joint state of the manipulator under the target hand shape is determined according to the position of the target item, the position of the manipulator, and the item shape of the target item. That is to say, the electronic device can recognize the user's intention to pick up the item based on the user's voice data and / or the gesture data of the user using the manipulator, and adjust the joint state of the manipulator by collecting the environmental image through the manipulator, so that the manipulator presents the target hand shape corresponding to the position of the target item in the environmental image, the position of the manipulator, and the item shape of the target item, and assists the user to understand which gesture to use to pick up the item without multiple touches, which is conducive to improving the safety and efficiency of the visually impaired person picking up the item.

[0067] In one possible example, the manipulator is provided with a tilt sensor, and the first pickup range is determined by the following steps: obtaining the overall tilt of the manipulator acquired by the tilt sensor at a first moment, the first moment being the acquisition moment of the first environmental image; obtaining the joint state of the manipulator at the first moment; determining the gesture of the manipulator at the first moment based on the joint state at the first moment; determining the coverage area of ​​the manipulator from the first environmental image based on the tilt and the gesture at the first moment, and determining the coverage area as the first pickup range.

[0068] Alternatively, in other embodiments, the manipulator may not be provided with a tilt sensor, and the electronic device may determine the overall tilt of the manipulator according to the relative positional relationship between the manipulator and the plane where the target object is located in the first environment image, and the rotation direction and rotation angle of the wrist joint of the electronic device manipulator.

[0069] Specifically, the coverage area is a projection area of ​​the gesture of the manipulator on the plane where the target object is located, which is determined according to the overall inclination of the manipulator.

[0070] It can be seen that in this example, the electronic device determines the coverage area of ​​the manipulator from the first environmental image as the user's first picking range based on the inclination and gesture of the manipulator at the first moment of capturing the first environmental image, which is conducive to improving the real-time and accuracy of determining the user's picking range.

[0071] In one possible example, after adjusting the joint state of the manipulator, the method further includes: outputting a first prompt information, the first prompt information being used to indicate a moving direction and a placement method of the target object, the moving direction being determined according to a position of the manipulator and a position of the target object, the placement method of the target object being determined according to an object shape of the target object, and the placement method of the target object including a relative position relationship between a dangerous part of the target object and the manipulator.

[0072] In this possible example, the prompt information includes voice information and / or joint tilt direction information, and the joint tilt direction information is used to control the target joint of the manipulator to rotate in the moving direction.

[0073] For example, taking the example of prompt information including both voice information and joint tilt direction information, assuming that the electronic device determines based on the shape of the target object, such as a pair of scissors, that the scissors are placed with the tip facing upwards under the left hand of the user wearing the robotic arm, it can output a voice prompt message such as "The scissors are placed with the tip facing upwards under your left hand, and can be moved to the lower left to pick up." At the same time, the joint corresponding to the little finger in the robotic arm can be tilted and rotated to the lower left, prompting the user with the exact direction of movement.

[0074] The joint tilt direction information can be determined based on the relative position relationship between the dangerous part of the target object and the manipulator. Under the joint tilt direction, the path for the manipulator to move to the position of the target object does not pass through the dangerous part of the target object.

[0075] It should be noted that the first prompt information combining voice information and joint tilt direction information is only an example. In actual applications, the prompt information may also include only voice information or only joint tilt direction information, or the electronic device may determine which prompt information to use based on environmental information or the connection relationship between the electronic device and the headphone device, such as using joint tilt direction information for prompting when the environmental volume is greater than the preset volume, and using voice prompt information when the electronic device is connected to a headphone device, etc., without specific restrictions here. When the prompt information includes voice information, the voice information may be output by the electronic device or the manipulator, without specific restrictions here.

[0076] It can be seen that in this example, after adjusting the joint state of the manipulator, the electronic device outputs the first prompt information for indicating the moving direction and the placement of the target object, and the placement includes the relative position relationship between the dangerous part of the target object and the manipulator, which is conducive to further improving the accuracy and safety of picking up the object. Furthermore, the first prompt information includes voice data and / or joint tilt direction data, which is convenient for the visually impaired to accurately determine the moving direction, which is conducive to improving the accuracy of the information prompt.

[0077] In one possible example, after adjusting the joint state of the manipulator, the method further includes: outputting a second prompt information, the second prompt information being used to indicate a moving direction and a target picking position of the target object, the moving direction being determined according to a position of the manipulator and a position of the target object, and the target picking position including a safe picking position of the target object determined according to an object shape of the target object.

[0078] In this possible example, the prompt information includes voice information and / or joint tilt direction information, and the joint tilt direction information is used to control the target joint of the manipulator to rotate in the moving direction.

[0079] Specifically, the electronic device can determine the safe picking position of the target object according to the object type and the object shape of the target object, and then determine the moving direction according to the safe picking position and the position of the manipulator.

[0080] Alternatively, when the electronic device recognizes the first environmental image and determines that the user's first picking range includes the target item, it can also first determine whether the target item has a dangerous part based on the item type and the item shape of the target item. If there is a dangerous part, determine whether the dangerous part is within the first picking range. If it is within the first picking range, determine the moving direction based on the position of the dangerous part and the first picking range, and prompt the user to move, so that after the movement, the user's picking range still includes the target item, but does not include the dangerous part of the item. At this time, the part of the target item within the user's picking range is the safe picking part.

[0081] For example, if the target object is a curling iron, Figure 7 and Figure 8 The part a shown in the figure is the dangerous part of the curling iron, that is, the heating part, and the part b is the safe picking part of the curling iron, that is, the handle part. Figure 7 When the user moves to the position shown in the figure, the electronic device can detect that the dangerous part of the curling iron is within the user's picking range, and can prompt the user to move to the left through a voice prompt message such as "Please move to the left to pick up the handle of the curling iron". Figure 8 When the user is in the position shown, the electronic device can detect that the dangerous part of the curling iron is not within the user's picking range, while the safe picking part of the curling iron is within the user's picking range, and can prompt the user to safely pick up the curling iron by, for example, displaying a prompt message such as "the handle of the curling iron is directly below your left hand".

[0082] It should be noted that the voice information included in the second prompt information here is only for illustrative purposes. In actual applications, the prompt information may also be joint tilt direction information, or the prompt information may also be a combination of voice information and joint tilt direction information. Alternatively, the electronic device may also determine which prompt information to use based on environmental information or the connection relationship between the electronic device and the headphone device. For example, when the environmental volume is greater than the preset volume, the joint tilt direction information is used for prompting, and when the electronic device is connected to a headphone device, voice prompt information is used, etc., and no specific restrictions are made here. When the prompt information includes voice information, the voice information may be output by the electronic device or the manipulator, and no specific restrictions are made here.

[0083] It can be seen that in this example, after adjusting the joint state of the manipulator, the electronic device outputs the second prompt information for indicating the moving direction and the target picking position, and the target picking position includes a safe picking position of the target object determined according to the object shape of the target object, which is conducive to further improving the accuracy and safety of object picking. Furthermore, the second prompt information includes voice data and / or joint tilt direction data, which is convenient for the visually impaired to accurately determine the moving direction, which is conducive to improving the accuracy of the information prompt.

[0084] In one possible example, the target joint is determined based on the position of the manipulator, the position of the target object and a target mapping relationship, the target mapping relationship includes a mapping relationship between multiple preset distances and multiple joints, the multiple preset distances include a first distance, and the multiple joints include a first joint. The method also includes: outputting first voice data, the first voice data is used to prompt the user to move along the second direction; when it is detected that the moving distance reaches the first distance, outputting second voice data, the second voice data is used to prompt the user to stop moving; outputting third voice data, the third voice data is used to prompt the user to select a joint; detecting that the angle of the first joint of the manipulator changes, and establishing a mapping relationship between the first joint and the first distance.

[0085] The values ​​of the multiple preset distances can be set by the user, or the electronic device can determine the size of the space of the user's current environment based on the environment image captured by the manipulator, and then determine the values ​​of the multiple preset distances based on the space size, so that the preset distances match the possible movement distances of the user when picking up an item in the current environment. The first direction can be determined by the electronic device based on the environment image captured by the manipulator, and there are no obstacles on the path of the user moving the first distance along the first direction.

[0086] Specifically, multiple preset distances may include multiple distance values ​​of different sizes. By encouraging the user to perceive different distance values ​​and select the joints corresponding to the distance values, a mapping relationship between different joints and different distance values ​​is established. When the user is prompted to move through the joint tilt direction information, the user can also determine the moving distance by rotating the target joint, thereby avoiding the problem of low efficiency in picking up items or collision with items due to moving distances that are too close or too far.

[0087] For example, after the electronic device prompts the user to move 10 cm and prompts the user to select a joint, it detects that the angle of the index finger joint changes, then the correspondence between 10 cm and the index finger can be determined. After the correspondence between 25 cm and the thumb is determined based on the same method, assuming that when prompting the user to pick up the target item, it is detected that it is necessary to move 25 cm to the left to pick up the target item (that is, the distance between the position of the manipulator and the position of the target item is 25 cm), then when prompting the user to move, the thumb joint of the manipulator can be controlled to rotate to the left. Assuming that it is necessary to move 10 cm to the left to pick up the target item (that is, the distance between the position of the manipulator and the position of the target item is 10 cm), then when prompting the user to move, the index finger joint of the manipulator can be controlled to rotate to the left.

[0088] It can be seen that in this example, the electronic device determines the target joint to rotate in the prompt information according to the position of the manipulator, the position of the target object and the mapping relationship, which is conducive to improving the flexibility of the information prompt and improving the accuracy of object picking. At the same time, the electronic device prompts the user to select the joint corresponding to each preset distance in the mapping relationship through voice prompts, realizing convenient and personalized settings, which is conducive to further improving the user's object picking experience.

[0089] In one possible example, after outputting the first prompt information, or after outputting the second prompt information, the method further includes: acquiring a second environmental image by an image acquisition device provided on the manipulator; identifying the second environmental image to determine whether the second picking range of the user under the target hand shape includes the target picking part of the target object; in response to the recognition result that the second picking range includes the target picking part, outputting a third prompt information, wherein the third prompt information is used to instruct the user to stop moving.

[0090] In a specific implementation, the manner in which the electronic device determines the second pickup range based on the second environment image may refer to the aforementioned manner in which the first pickup range is determined based on the first environment image.

[0091] Among them, the third prompt information may also include voice information and / or joint tilt direction information. For example, after the electronic device detects that the second picking range includes the target picking part, it can output a voice prompt information of "Please pick up the item at the current position", prompting the user to pick up the item with the target hand shape at the current position, or, after the electronic device detects that the second picking range includes the target picking part, it can rotate the target joint in the direction opposite to the moving direction to prompt the user to stop moving and pick up the item, or, the electronic device can also prompt the user to move by vibrating the manipulator, and no specific restrictions are made here. It should also be noted that the specific content included in the voice information in the embodiment of the present application, and the specific joints that move in the joint tilt direction are all exemplary descriptions. In actual applications, the specific content of the voice information and the specific joints that move can be selected as different from the examples in the embodiment of the present application as needed, and no specific restrictions are made here.

[0092] It can be seen that in this example, after outputting the prompt information instructing the user to move, the electronic device continues to collect environmental images through the manipulator, and when it is determined that the picking range includes the target picking part based on the environmental image, it outputs the first three prompt information to instruct the user to stop moving, which is conducive to further improving the accuracy of object picking.

[0093] See also Fig. 9 , Fig. 9 : is a structural schematic diagram of a manipulator control device for picking up an object provided in an embodiment of the present application. The manipulator control device 30 for picking up an object includes:

[0094] An acquisition unit 301 is used to acquire input data, wherein the input data includes voice data and / or gesture data of a user, wherein the user is a user wearing the manipulator, and the gesture data includes a joint state of the manipulator, wherein the joint state includes a joint angle and a joint direction;

[0095] A first recognition unit 302, configured to recognize an object picking intention based on the input data;

[0096] An acquisition unit 303 is used to acquire a first environment image through an image acquisition device provided on the manipulator;

[0097] A second recognition unit 304 is used to recognize the first environment image to determine whether the first picking range of the user includes a target item corresponding to the item picking intention;

[0098] The adjustment unit 305 is used to adjust the joint state of the manipulator in response to the recognition result that the first picking range includes the target object, so that the manipulator presents a target hand shape, wherein the joint state of the manipulator under the target hand shape is determined according to the position of the target object, the position of the manipulator and the object shape of the target object.

[0099] In one possible example, the manipulator is provided with a tilt sensor, and the first pickup range is determined by the following steps: obtaining the overall tilt of the manipulator acquired by the tilt sensor at a first moment, the first moment being the acquisition moment of the first environmental image; obtaining the joint state of the manipulator at the first moment; determining the gesture of the manipulator at the first moment based on the joint state at the first moment; determining the coverage area of ​​the manipulator from the first environmental image based on the tilt and the gesture at the first moment, and determining the coverage area as the first pickup range.

[0100] In a possible example, the manipulator control device 30 for picking up objects is also used to output a first prompt message after adjusting the joint state of the manipulator, wherein the first prompt message is used to indicate a moving direction and a placement method of the target object, wherein the moving direction is determined according to the position of the manipulator and the position of the target object, and the placement method of the target object is determined according to the object shape of the target object, and the placement method of the target object includes the relative position relationship between the dangerous parts of the target object and the manipulator.

[0101] In one possible example, the manipulator control device 30 for picking up objects is also used to output a second prompt message after adjusting the joint state of the manipulator, wherein the second prompt message is used to indicate a moving direction and a target picking position of the target object, wherein the moving direction is determined according to the position of the manipulator and the position of the target object, and the target picking position includes a safe picking position of the target object determined according to the object shape of the target object.

[0102] In a possible example, the prompt information includes voice information and / or joint tilt direction information, and the joint tilt direction information is used to control the target joint of the manipulator to rotate in the moving direction.

[0103] In one possible example, the target joint is determined based on the position of the manipulator, the position of the target object and a target mapping relationship, the target mapping relationship includes a mapping relationship between multiple preset distances and multiple joints, the multiple preset distances include a first distance, and the multiple joints include a first joint. The manipulator control device 30 for picking up objects is also used to: output first voice data, and the first voice data is used to prompt the user to move along the second direction; when it is detected that the moving distance reaches the first distance, output second voice data, and the second voice data is used to prompt the user to stop moving; output third voice data, and the third voice data is used to prompt the user to select a joint; when it is detected that the angle of the first joint of the manipulator changes, a mapping relationship between the first joint and the first distance is established.

[0104] In one possible example, the robot control device 30 for object picking is also used to, after outputting the first prompt information, or after outputting the second prompt information, acquire a second environmental image through an image acquisition device arranged on the robot; identify the second environmental image to determine whether the second picking range of the user under the target hand type includes the target picking part of the target object; in response to the recognition result that the second picking range includes the target picking part, output a third prompt information, and the third prompt information is used to instruct the user to stop moving.

[0105] In the case of using an integrated unit, a structural schematic diagram of another manipulator control device for object picking provided in an embodiment of the present application is as follows: Fig.10 As shown. Fig.10 In the embodiment, the robot control device for picking up items includes: a processing module 310 and a communication module 311. The processing module 310 is used to control and manage the actions of the robot control device for picking up items, for example, the steps performed by the acquisition unit 301, the first recognition unit 302, the collection unit 303, the second recognition unit 304 and the adjustment unit 305, and / or other processes for performing the technology described in this application. The communication module 311 is used to support the interaction between the robot control device 30 for picking up items and other devices. Fig.10 As shown, the robot control device for object picking may further include a storage module 312, and the storage module 312 is used to store program codes and data of the robot control device for object picking.

[0106] Among them, the processing module 310 can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 311 can be a transceiver, an RF circuit or a communication interface, and the like. The storage module 312 can be a memory.

[0107] Among them, all relevant contents of each scenario involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. The above-mentioned manipulator control device for picking up objects can execute the above-mentioned Figure 4 The steps performed by the electronic device in the robot control method for object picking are shown.

[0108] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0109] The present application also provides a computer program product, including a computer program, which implements part or all of the steps of any method described in the above method embodiment when executed by a processor. The computer program product may be a software installation package.

[0110] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and units involved are not necessarily required by the present application.

[0111] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0112] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0113] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0116] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable memory, which may include a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0117] The embodiments of the present application are introduced in detail above. Specific examples are used in the present application to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in the field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A robot control method for picking up objects, characterized in that: include: Acquire input data, where the input data includes voice data and / or gesture data of a user, where the user is a user wearing the manipulator, and the gesture data includes a joint state of the manipulator, where the joint state includes a joint angle and a joint direction; Based on the input data, identifying an object picking intention; Acquire a first environment image by an image acquisition device arranged on the manipulator; Identify the first environment image to determine whether the first picking range of the user includes a target item corresponding to the item picking intention; In response to the recognition result that the first picking range includes the target object, adjusting the joint state of the manipulator so that the manipulator presents a target hand shape, wherein the joint state of the manipulator under the target hand shape is determined according to the position of the target object, the position of the manipulator, and the object shape of the target object; Output second prompt information, the second prompt information is used to indicate the moving direction and the target picking position of the target object, the moving direction is determined according to the position of the manipulator and the position of the target object, the target picking position includes a safe picking position of the target object determined according to the object shape of the target object, the prompt information includes voice information and / or joint tilt direction information, the joint tilt direction information is used to control the target joint of the manipulator to rotate in the moving direction, the target joint is determined according to the position of the manipulator, the position of the target object and a target mapping relationship, the target mapping relationship includes a mapping relationship between multiple preset distances and multiple joints, the multiple The preset distance includes a first distance, and the multiple joints include a first joint. The method also includes: outputting first voice data, and the first voice data is used to prompt the user to move along a second direction; when it is detected that the moving distance reaches the first distance, outputting second voice data, and the second voice data is used to prompt the user to stop moving; outputting third voice data, and the third voice data is used to prompt the user to select a joint; detecting that the angle of the first joint of the manipulator changes, and establishing a mapping relationship between the first joint and the first distance, wherein the multiple preset distances are determined according to the space size of the user's environment, and the space size is determined according to the environment image captured by the manipulator.

2. The method according to claim 1, characterized in that The manipulator is provided with an inclination sensor, and the first picking range is determined by the following steps: Acquire the overall inclination of the manipulator collected by the inclination sensor at a first moment, where the first moment is a moment of collecting the first environment image; Acquire the joint state of the manipulator at the first moment; determining a gesture of the manipulator at the first moment according to the joint state at the first moment; According to the inclination and the gesture at the first moment, a coverage area of ​​the manipulator is determined from the first environment image, and the coverage area is determined as the first picking range.

3. The method according to claim 1, characterized in that After adjusting the joint state of the manipulator, the method further comprises: Output first prompt information, wherein the first prompt information is used to indicate a moving direction and a placement method of the target object, wherein the moving direction is determined according to a position of the manipulator and a position of the target object, wherein the placement method of the target object is determined according to an object shape of the target object, and wherein the placement method of the target object includes a relative position relationship between a dangerous part of the target object and the manipulator.

4. The method according to claim 1 or 3, characterized in that: After outputting the first prompt information, or after outputting the second prompt information, the method further includes: Acquire a second environment image by using an image acquisition device disposed on the manipulator; Identify the second environment image to determine whether the second picking range of the user under the target hand shape includes the target picking position of the target object; In response to the recognition result that the second pickup range includes the target pickup position, third prompt information is output, where the third prompt information is used to instruct the user to stop moving.

5. A robot control device for picking up objects, characterized in that: include: an acquisition unit, configured to acquire input data, wherein the input data includes voice data and / or gesture data of a user, wherein the user is a user wearing the manipulator, and the gesture data includes a joint state of the manipulator, wherein the joint state includes a joint angle and a joint direction; A first recognition unit, configured to recognize an object picking intention based on the input data; An acquisition unit, configured to acquire a first environment image through an image acquisition device disposed on the manipulator; A second recognition unit, configured to recognize the first environment image to determine whether the first picking range of the user includes a target item corresponding to the item picking intention; an adjusting unit, configured to adjust the joint state of the manipulator in response to the recognition result that the first picking range includes the target object, so that the manipulator presents a target hand shape, wherein the joint state of the manipulator in the target hand shape is determined according to the position of the target object, the position of the manipulator, and the object shape of the target object; The manipulator control device is also used to: output second prompt information, the second prompt information is used to indicate the moving direction and the target picking position of the target object, the moving direction is determined according to the position of the manipulator and the position of the target object, the target picking position includes a safe picking position of the target object determined according to the object shape of the target object, the prompt information includes voice information and / or joint tilt direction information, the joint tilt direction information is used to control the target joint of the manipulator to rotate in the moving direction, the target joint is determined according to the position of the manipulator, the position of the target object and the target mapping relationship, the target mapping relationship includes a mapping relationship between multiple preset distances and multiple joints, The multiple preset distances include a first distance, the multiple joints include a first joint, and the manipulator control device is also used to: output first voice data, the first voice data is used to prompt the user to move along a second direction; when it is detected that the moving distance reaches the first distance, output second voice data, the second voice data is used to prompt the user to stop moving; output third voice data, the third voice data is used to prompt the user to select a joint; detect that the angle of the first joint of the manipulator changes, and establish a mapping relationship between the first joint and the first distance, wherein the multiple preset distances are determined according to the space size of the user's environment, and the space size is determined according to the environment image captured by the manipulator.

6. An electronic device, characterized in that: The electronic device comprises: a processor, a memory and a communication interface, wherein the processor, the memory and the communication interface are interconnected, wherein the communication interface is used to receive or send data, the memory is used to store application code for the electronic device to execute the method as described in any one of claims 1-4, and the processor is configured to execute the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps in the method according to any one of claims 1 to 4.

Images