Object moving method and device, electronic equipment and robot

Abstract

The embodiment of the invention provides an object moving method and device, electronic equipment and a robot, and relates to the technical field of robotics.The method comprises the steps that target voice data, collected by a voice receiving device, of a target person is obtained, and target image data, collected by an image collecting device, of the target person is obtained; performing content identification on the target voice data and the target image data to obtain an initial position and a destination position of the target object; and controlling the robot to move the target object at the initial position to the target position. According to the scheme, the time and manpower consumed by moving the object when the initial position and / or the destination position of the object are / is changed can be reduced.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to an object moving method, device, electronic equipment and robot. Background Art

[0002] In the robot-assisted handling solution, the starting position and destination position of the stacked objects are fixed. The robot moves to the starting position according to the pre-planned movement trajectory, and uses the robotic arm to move the objects to the transport vehicle; after the transport vehicle and the robot move to the destination position, the robot uses the robotic arm to unload the objects on the transport vehicle and stack the objects at the destination position.

[0003] When the starting position and / or destination position of the stacked objects changes, the user needs to plan the robot's movement trajectory according to the new starting position and destination position of the stacked objects, and configure the movement trajectory into the robot to move the objects. This object moving method is time-consuming and labor-intensive and does not have generalizability. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and robot for moving an object, so as to reduce the time and manpower consumed in moving the object when the starting position and / or the destination position of the object changes. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for moving an object, the method comprising:

[0006] Acquire target voice data of the target person collected by the voice receiving device, and acquire target image data of the target person collected by the image collecting device;

[0007] Performing content recognition on the target voice data and the target image data to obtain a starting position and a destination position of the target object;

[0008] The robot is controlled to move the target object at the starting position to the destination position.

[0009] In some embodiments, the step of acquiring the target voice data of the target person collected by the voice receiving device includes:

[0010] Acquiring voice data collected by the voice receiving device;

[0011] extracting speech features of the acquired speech data;

[0012] If the voice feature matches the voice feature that triggers the migration, the voice data collected by the voice receiving device continues to be obtained as the target voice data of the target person.

[0013] In some embodiments, the voice feature includes at least one of a person's voice biometric feature, a voice spectrogram that triggers key information, text data corresponding to the trigger key information, and voice data corresponding to the trigger key information.

[0014] In some embodiments, the step of acquiring the target voice data of the target person collected by the voice receiving device includes:

[0015] Get the wake-up command;

[0016] According to the wake-up instruction, the voice data collected by the voice receiving device is obtained as the target voice data of the target person.

[0017] In some embodiments, the step of obtaining the wake-up instruction includes:

[0018] When a click operation of the wake-up button is detected, a wake-up instruction is generated; or

[0019] In case of receiving a task instruction including triggering critical information, obtaining a wake-up instruction based on the task instruction; or

[0020] When the image data of the target person is acquired, a wake-up instruction is generated.

[0021] In some embodiments, the step of acquiring the target image data of the target person acquired by the image acquisition device includes:

[0022] Locate the current location of the target person;

[0023] The image acquisition device is instructed to track the target person at the current position to obtain target image data of the target person.

[0024] In some embodiments, the step of locating the current position of the target person includes:

[0025] Acquire first image data acquired by an image acquisition device;

[0026] extracting image biometric features of the person included in the first image data;

[0027] If the extracted image biometric feature matches the image biometric feature of the target person, the position of the person included in the first image data is located as the current position of the target person.

[0028] In some embodiments, the method further comprises:

[0029] When the image acquisition device is instructed to track the target person for a preset time period, or when the starting position and the destination position are obtained, the image acquisition device is instructed to stop tracking the target person.

[0030] In some embodiments, the step of acquiring the target image data of the target person acquired by the image acquisition device is performed under any of the following conditions:

[0031] Receiving a task instruction including triggering key information;

[0032] It is detected that the stacking time of the objects is longer than the preset time;

[0033] Detect the presence of people in a designated area;

[0034] The voice data of the target person is obtained.

[0035] In some embodiments, the step of performing content recognition on the target voice data and the target image data to obtain a starting position and a destination position for placement of the target object includes:

[0036] Performing content recognition on the target voice data and the target image data to obtain the intention of the target person and the action of the target person;

[0037] If the intention indicates moving an object, a first starting position and a first destination position of the object indicated by the action are identified to obtain a starting position and a destination position of the target object.

[0038] In some embodiments, the method further comprises:

[0039] If the intention indicates moving an object, and the intention indicates a second starting position and a second destination position for placing the object, then identify the starting direction and the destination direction for placing the object indicated by the action; determine a first distance between the second starting position and the starting direction, and determine a second distance between the second destination position and the destination direction; if both the first distance and the second distance are less than a threshold distance, then use the second starting position and the second destination position as the starting position and the destination position for placing the target object, respectively.

[0040] In some embodiments, the step of identifying a first starting position and a first destination position of the object indicated by the action, comprises:

[0041] Identify the starting direction and the destination direction of the object placed as indicated by the action;

[0042] Determine the object closest to the starting direction, and determine the destination area closest to the destination direction; or determine the object closest to the target person in the starting direction, and determine the destination area closest to the target person in the destination direction;

[0043] The determined position of the object is used as a first starting position, and the determined position of the destination area is used as a first target position.

[0044] In some embodiments, before acquiring the target voice data and acquiring the target image data, the method further includes:

[0045] Based on the working scene of the robot, obtaining the type of obstacle;

[0046] If the type of the obstacle is the type of the object to be moved, the position of the obstacle is located, so that when the obstacle is identified as the target object, the position of the obstacle is used as the starting position of the target object.

[0047] In some embodiments, the step of obtaining the type of obstacle based on the working scene of the robot includes:

[0048] Acquire second image data of an obstacle in the working scene of the robot acquired by an image acquisition device; identify the second image data to obtain the type of the obstacle; or

[0049] Read a tag carried by an obstacle in the working scene of the robot; and determine the type of the obstacle according to the tag.

[0050] In some embodiments, the step of obtaining the type of obstacle based on the working scene of the robot includes:

[0051] Detect obstacles in the working scene of the robot using a detection device, wherein the detection device includes at least one of the image acquisition device and a millimeter wave antenna;

[0052] When the obstacle is detected, the type of the obstacle is identified.

[0053] In some embodiments, the method further comprises:

[0054] When a task instruction including triggering key information is received, the step of obtaining the type of obstacle based on the working scene of the robot is executed.

[0055] In a second aspect, an embodiment of the present application provides an object moving device, the device comprising:

[0056] An acquisition module, used to acquire the target voice data of the target person acquired by the voice receiving device, and acquire the target image data of the target person acquired by the image acquisition device;

[0057] A recognition module, used to perform content recognition on the target voice data and the target image data to obtain a starting position and a destination position of the target object;

[0058] The control module is used to control the robot to move the target object at the starting position to the destination position.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0060] The memory is used to store computer programs;

[0061] The processor is used to implement any of the above-mentioned object moving methods when executing the program stored in the memory.

[0062] In a fourth aspect, an embodiment of the present application provides a robot, comprising a voice receiving device, an image acquisition device, a processor, a communication interface, a memory, and a communication bus, wherein the voice receiving device, the image acquisition device, the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0063] The voice receiving device is used to collect target voice data of the target person;

[0064] The image acquisition device is used to acquire target image data of the target person;

[0065] The memory is used to store computer programs;

[0066] The processor is used to implement any of the above-mentioned object moving methods when executing the program stored in the memory.

[0067] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned object moving methods is implemented.

[0068] In a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described object moving methods.

[0069] Beneficial effects of the embodiments of the present application:

[0070] In the technical case provided by the embodiment of the present application, when executing the object moving, the starting position and the destination position of the target object are determined by combining the voice data and the image data, that is, the starting position and the destination position of this moving operation are determined, so as to control the robot to move the target object at the starting position to the destination position. In this object moving method, even if the starting position and / or the destination position of the object placement changes, the device can actively analyze the voice data and the image data to obtain the starting position and the destination position of this moving operation, without the need to manually plan and configure the robot's moving trajectory, thereby reducing the time and manpower consumed in object moving and improving the generalization of the object moving method.

[0071] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0073] Figure 1a This is a first schematic diagram of an object moving scene in the related art;

[0074] Figure 1b It is a second schematic diagram of an object moving scene in the related art;

[0075] Figure 1c This is a third schematic diagram of an object moving scene in the related art;

[0076] Figure 2 A schematic diagram of a first flow chart of a method for moving an object provided in an embodiment of the present application;

[0077] Figure 3a A schematic diagram of a first deployment of an image acquisition device and a voice receiving device provided in an embodiment of the present application;

[0078] Figure 3b A schematic diagram of a second deployment of the image acquisition device and the voice receiving device provided in an embodiment of the present application;

[0079] Figure 4a A schematic diagram of a mobile device provided in an embodiment of the present application;

[0080] Figure 4b A schematic diagram of a robot provided in an embodiment of the present application;

[0081] Figure 5A schematic diagram of a command input provided in an embodiment of the present application;

[0082] Figure 6 A schematic diagram of a prompt information output scenario provided in an embodiment of the present application;

[0083] Figure 7 A first schematic diagram of a robot working scenario provided in an embodiment of the present application;

[0084] Figure 8 A second flow chart of the object moving method provided in the embodiment of the present application;

[0085] Figure 9a A second schematic diagram of a robot working scenario provided in an embodiment of the present application;

[0086] Figure 9b A third schematic diagram of a robot working scenario provided in an embodiment of the present application;

[0087] Fig.10 A third flow chart of the object moving method provided in the embodiment of the present application;

[0088] Fig.11 A fourth schematic diagram of a robot working scenario provided in an embodiment of the present application;

[0089] Fig.12 A fourth flow chart of the object moving method provided in the embodiment of the present application;

[0090] Fig.13 A signaling schematic diagram of an object moving system provided in an embodiment of the present application;

[0091] Fig.14 A schematic diagram of the structure of an object moving device provided in an embodiment of the present application;

[0092] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0093] Fig.16 A schematic diagram of the structure of a robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0095] In the robot-assisted handling solution, the starting and destination positions of the stacked objects are fixed. Figure 1a and Figure 1b As shown in FIG. 1 , the robot moves to the starting position A according to the pre-planned moving trajectory, and uses the robotic arm to move the object to the transport vehicle. The transport vehicle can be a small car independent of the robot, such as Figure 1a As shown; the transport vehicle can also be the robot itself, that is, the robot itself has storage space, such as Figure 1b As shown. Figure 1b For example, after the object is moved to the transporter, the transporter and the robot move to the destination position B according to the planned moving trajectory. After the transporter and the robot move to the destination position B, the robot uses the robotic arm to unload the object on the transporter and stack the objects at the destination position B.

[0096] In the object moving scheme, if the starting position and / or the destination position of the stacked objects change, such as Figure 1c As shown in the figure, the destination position changes to position C. The user needs to re-plan the robot's moving trajectory according to the starting position A and the destination position C where the objects are stacked, and configure the moving trajectory into the robot to move the objects. This object moving method is time-consuming and labor-intensive and does not have generalizability.

[0097] To solve the above problems, the present application provides a method for moving an object. Figure 2 As shown, the method comprises the following steps:

[0098] Step S201, obtaining target voice data of a target person collected by a voice receiving device, and obtaining target image data of a target person collected by an image collecting device;

[0099] Step S202, performing content recognition on the target voice data and the target image data to obtain the starting position and the destination position of the target object;

[0100] Step S203, controlling the robot to move the target object at the starting position to the destination position.

[0101] In the technical case provided by the embodiment of the present application, when executing the object moving, the starting position and the destination position of the target object are determined by combining the voice data and the image data, that is, the starting position and the destination position of this moving operation are determined, so as to control the robot to move the target object at the starting position to the destination position. In this object moving method, even if the starting position and / or the destination position of the object placement changes, the device can actively analyze the voice data and the image data to obtain the starting position and the destination position of this moving operation, without the need to manually plan and configure the robot's moving trajectory, thereby reducing the time and manpower consumed in object moving and improving the generalization of the object moving method.

[0102] The object moving method provided in the embodiment of the present application can be applied to a robot or an electronic device such as a server, a cluster, a mobile terminal, etc. connected to the robot. For ease of description, the following description is based on the electronic device as the execution subject, which does not serve as a limitation. Among them, the electronic device can be integrated on the robot or can be an independent physical machine.

[0103] In the above step S201, the voice receiving device is used to collect voice data in the robot working scene, and the image collecting device is used to collect image data in the robot working scene. The number of the voice receiving device and the image collecting device can be one or more.

[0104] The voice receiving device and the image acquisition device can be independent physical machines, that is, they are independently deployed in the robot working scene, such as Figure 3a The camera and microphone shown; the voice receiving device and the image acquisition device can also be integrated into a robot or electronic device, such as Figure 3b As shown, four cameras are deployed on the robot (the camera on the back of the robot is not shown in the figure) to collect image data in all directions, and a microphone (i.e., a voice receiving device) is deployed on the top of the robot to collect voice data.

[0105] The target person is the person who directs the robot to move objects. It can be a specific person or any person in the robot's working scene.

[0106] When moving an object, the voice receiving device collects the voice data of the target person, i.e., the target voice data; in addition, the image acquisition device collects the image data of the target person, i.e., the target image data. The electronic device acquires the target voice data and the target image data. The number of the target image data can be one or more. The one or more target image data can come from the same image acquisition device or from different image acquisition devices.

[0107] In the embodiment of the present application, the voice receiving device can collect voice data emitted by all personnel and equipment in the robot working scene. In order to improve the accuracy of object movement, the electronic device obtains the voice data collected by the voice receiving device, performs noise reduction and filtering on the obtained voice data, and obtains the target voice data of the target person.

[0108] In addition, the image acquisition device can collect image data from all directions in the robot's working scene, and will continue to collect image data. In order to improve the accuracy of object movement, the electronic device obtains multiple image data collected by the image acquisition device, and selects target image data including the target person from the multiple image data obtained. For example, the electronic device performs image recognition on the obtained image data to obtain image biometric features of the person included in each image data; obtains image data whose image biometric features match the image biometric features of the target person as the target image data.

[0109] In the above step S202, the target object is the object that needs to be moved this time, the starting position is the position where the target object is currently stacked and placed, and the destination position is the position to which the target object needs to be moved.

[0110] After acquiring the target voice data, the electronic device performs content recognition on the target voice data to obtain the content recognition result of the voice, such as recognizing key information such as keywords and key words included in the target voice data, as well as the user intention indicated by the target voice data, the starting position of the target object stacking, and the destination position of the target object stacking. The key information may include but is not limited to preset words, words / terms representing entities, words / terms representing actions, etc., such as words representing objects to be moved, words representing the number or name of the robot, words representing geographic locations, words representing moving operations, etc. The user intention represents the purpose of the target person, such as the purpose of the target person is to move the object, or the object is moved from a certain position (i.e., the starting position) to another position (i.e., the destination position).

[0111] After acquiring the target image data, the electronic device performs content recognition on the target image data to obtain a content recognition result of the image, such as recognizing the action of the target person, the starting direction and starting position indicated by the action, the destination direction and destination position indicated by the action, etc.

[0112] Based on the content recognition results of the target voice data and the target image data, the electronic device can determine the starting position and the destination position of the stacked target objects.

[0113] For example, the target voice data is "Move the cartons from position A to position B". The electronic device performs content recognition on the target voice data, and obtains the content recognition result: the starting position of the stacked cartons is position A, and the destination position of the stacked cartons is position B. In addition, the electronic device performs content recognition on the target image data, and obtains the content recognition result: the target person performs the specified object moving action. Combining the content recognition results of the target voice data and the target image data, the electronic device can determine that an object is to be moved, and the target object is a carton, the starting position of the target object is position A, and the destination position of the stacked target object is position B.

[0114] In the embodiment of the present application, the electronic device may also use other methods to determine the starting position and the destination position of the target object, which is not limited to this.

[0115] In the above step S203, after determining the starting position and the destination position of the target object, the electronic device controls the robot to move to the starting position, and uses the robot arm to move the target object at the starting position to the destination position. Figure 1a The transporter shown or Figure 1b Then, the electronic device controls the robot to move to the target location. If the target object is moved to the storage space of the robot as shown in FIG. Figure 1a On the transport vehicle shown, the electronic device also controls the transport vehicle to move to the destination location. The electronic device controls the robot to use the mechanical arm to unload the target object at the destination location, thereby achieving the object movement.

[0116] In the embodiment of the present application, when the electronic device is integrated on the robot, the electronic device can directly control itself to move the target object at the starting position to the destination position. When the electronic device is an independent physical machine, the electronic device sends a control instruction to the robot, and the control instruction includes the starting position and the destination position; the robot receives the control instruction, and controls itself to move the target object at the starting position to the destination position according to the starting position and the destination position included in the control instruction.

[0117] In the embodiment of the present application, the electronic device integrates voice data and image data to determine whether to move and the starting position and destination position of the move, which can effectively avoid the problem of false triggering and improve the accuracy of object movement.

[0118] In some embodiments, the electronic device may use the following two methods to implement the operation of acquiring the target voice data in the above step S201.

[0119] Mode a1, active acquisition mode. Specifically, the electronic device acquires the voice data collected by the voice receiving device; extracts the voice features of the acquired voice data; if the voice features match the voice features that trigger the transfer, the voice data collected by the voice receiving device is continuously acquired as the target voice data of the target person.

[0120] The voice feature may include at least one of the voice biometrics of the person, the voice spectrogram of the triggering key information, the text data corresponding to the triggering key information, and the voice data corresponding to the triggering key information. The triggering key information is information that triggers the object moving operation, such as words indicating the object to be moved, words indicating the number or name of the robot, words indicating the geographical location, words indicating the moving operation, and sentences indicating the moving intention. The voice biometrics may include but are not limited to timbre, pitch, sound intensity, etc. Using the voice biometrics, the electronic device can distinguish the person who speaks.

[0121] Voice feature matching means that the deviation of two voice features is less than a preset deviation threshold, or the two voice features are the same.

[0122] In an embodiment of the present application, the electronic device pre-records the voice features that trigger the relocation. The voice receiving device collects voice data in real time. The electronic device obtains the voice data collected by the voice receiving device in real time, and extracts the voice features of the voice data. The electronic device matches the extracted voice features with the voice features that trigger the relocation; if the two match, the subsequently acquired voice data is used as the target voice data of the target person, and the target voice data is subjected to content recognition processing such as in the above step S202; if the two do not match, the voice data is not subjected to subsequent processing, such as the content recognition processing in the above step S202.

[0123] Take voice features including voice biometrics as an example. The electronic device pre-records the voice biometrics of the target person. The electronic device extracts the voice biometrics of the voice data. If the extracted voice biometrics are the same as the pre-recorded voice biometrics of the target person, it means that the current target person is speaking, and the subsequent voice data collected by the voice receiving device is the voice data of the target person. Then the electronic device continues to obtain the voice data collected by the voice receiving device as the target voice data of the target person.

[0124] Take the trigger key information as the robot number, and the voice feature includes the voice data of the robot number as an example. The electronic device pre-records the voice data of the robot number. The electronic device extracts the voice data of the robot number. If the voice data of the robot number is extracted, it means that the robot may need to perform a moving operation, and then the electronic device continues to obtain the voice data collected by the voice receiving device, and uses the obtained voice data as the target voice data of the target person, so as to obtain the complete target voice data and accurately determine the starting position and destination position of the target object.

[0125] For example, the robot number is 0001. The electronic device pre-records the voice data of "0001". After acquiring the voice data of "0001" collected by the voice receiving device, the electronic device uses the voice data collected by the voice receiving device as the target voice data of the target person.

[0126] In the embodiment of the present application, the voice receiving device collects voice data in real time to ensure the integrity of the target voice data. The electronic device will only perform subsequent migration operations, such as the processing operations of step S202 and step S203, when it obtains voice data whose voice features match the voice features that trigger the migration, thereby reducing the resource consumption of the electronic device.

[0127] In addition, in order to reduce the energy consumption of electronic devices, the electronic devices are in standby mode; when voice data whose voice features match the voice features that trigger the move is obtained, the electronic devices will be awakened and perform subsequent move operation processing.

[0128] Mode a2, passive acquisition mode. Specifically, the electronic device obtains a wake-up instruction; according to the wake-up instruction, the voice data collected by the voice receiving device is obtained as the target voice data of the target person. The wake-up instruction is used to wake up the electronic device so that the electronic device can normally process voice data and image data.

[0129] In the embodiment of the present application, the electronic device can obtain the wake-up instruction in any of the following ways.

[0130] Mode a21, the electronic device is equipped with a wake-up button. The electronic device is in standby mode. The user (such as the target person) clicks the wake-up button to wake up the electronic device. After the electronic device detects that the user clicks the wake-up button, that is, detects that the wake-up button generates a click operation, it generates a wake-up instruction. According to the wake-up instruction, the electronic device is awakened, and then performs subsequent moving operation processing, such as the processing operations of the above-mentioned steps S202 and S203.

[0131] For example, electronic devices are Figure 4a The mobile device such as a mobile phone shown is connected to the robot, and the power button is a wake-up button. The user clicks the power button. After that, the mobile device is awakened, and then starts to collect voice data through its own microphone (i.e., voice receiving device); and the voice data collected by the microphone is used as the target voice data of the target person.

[0132] For another example, an electronic device is Figure 4bIn the robot shown, the power button of the robot is a wake-up button. The user clicks the power button of the robot. After that, the robot is awakened, and then the microphone of the robot (i.e., the voice receiving device) starts to collect voice data; the robot uses the voice data collected by the microphone as the target voice data of the target person.

[0133] In mode a22, the electronic device receives the task instruction and detects the information carried by the task instruction; if it is determined that the task instruction carries the trigger key information, a wake-up instruction is obtained based on the task instruction. That is, when a task instruction including the trigger key information is received, the electronic device obtains the wake-up instruction based on the task instruction. According to the wake-up instruction, the electronic device is awakened, and then performs subsequent moving operation processing, such as the processing operations of the above-mentioned steps S202 and S203.

[0134] In the embodiment of the present application, the task instruction may be in text form or in voice form. The source of the task instruction may be an electronic device sent by other devices or directly input into the electronic device by a user. When receiving a task instruction including trigger key information, the electronic device may directly use the task instruction as a wake-up instruction or may use the task instruction to generate a specified wake-up instruction, without limitation.

[0135] For example, Figure 4b The robot shown in the figure has a display screen for human-machine interaction. The user can input the task instructions for moving on the display screen, such as Figure 5 In the command input box shown, enter "object move" and click the "OK" button. The robot obtains the task command carrying "object move". "Object move" is the trigger key information, and then the robot determines that the task command is a wake-up command.

[0136] Mode a23: When the image data of the target person is acquired, the electronic device generates a wake-up instruction. According to the wake-up instruction, the electronic device is awakened and then performs subsequent moving operation processing, such as the processing operations of the above steps S202 and S203.

[0137] In the embodiment of the present application, the image acquisition device acquires image data in real time. The electronic device acquires the image data acquired by the image acquisition device, and generates a wake-up instruction when determining that the image data is image data of a target person.

[0138] In the embodiment of the present application, the voice receiving device can start collecting voice data after the electronic device is awakened, so as to reduce the energy consumption of the voice receiving device. The voice receiving device can also collect voice data in real time; when the electronic device is in standby mode, the voice data collected by the voice receiving device will not be obtained. When the electronic device is awakened, the electronic device will obtain the voice data collected by the voice receiving device again, so as to reduce the delay in obtaining voice data.

[0139] In addition, after the electronic device is awakened, the voice data collected by the voice receiving device is obtained as the target voice data, which reduces the voice data processed by the electronic device and saves the computing resources of the device.

[0140] In some embodiments, the electronic device may use the following two methods to implement the operation of acquiring the target image data in the above step S201.

[0141] Mode b1, active acquisition mode. Specifically, the electronic device locates the current position of the target person; instructs the image acquisition device to track the target person at the current position, and obtains the target image data of the target person. Then, the subsequent moving operation processing is performed, such as the processing operations of the above steps S202 and S203.

[0142] In the embodiment of the present application, the user can input the current location of the target person into the electronic device, and then the electronic device obtains the current location of the target person input by the user. The electronic device can also locate the current location of the target person using the image data including the target person collected by the image acquisition device.

[0143] In some embodiments, the electronic device may acquire first image data acquired by an image acquisition device; extract image biometric features of a person included in the first image data; if the extracted image biometric features match the image biometric features of a target person, it indicates that the person included in the first image data is the target person, and locate the position of the person included in the first image data as the current position of the target person. The image biometric features include facial features and body features, etc.

[0144] In the embodiment of the present application, the first image data may be any image data acquired by the image acquisition device. When the electronic device determines that the first image data includes the target person, it may determine the current position of the target person by using the calibration relationship between the image coordinate system and the world coordinate system and the pixel coordinates of the target person included in the first image data.

[0145] When the electronic device determines that the first image data includes the target person, it can also use the first image data to determine the direction of the target person relative to the electronic device, and use the millimeter wave antenna to send a first signal to the direction of the target person relative to the electronic device, and receive a second signal reflected by the target person, where the second signal is an echo signal of the first signal; the electronic device uses the first signal and the second signal to determine the current position of the target person.

[0146] In the embodiment of the present application, the electronic device may also use other methods to locate the current position of the target person, which is not limited to this.

[0147] When the current position of the target person is located, the electronic device instructs the image acquisition device to track the target person at the current position; accordingly, the image acquisition device tracks the target person, collects target image data, and then the electronic device obtains the target image data.

[0148] In an embodiment of the present application, after the electronic device instructs the image acquisition device to track the target person at the current position, the image acquisition device can continue to track the target person and collect target image data until the target person leaves the work scene.

[0149] In some embodiments, in order to save energy and avoid the accidental triggering of the object moving operation, when the image acquisition device is instructed to track the target person for a preset time, the electronic device can instruct the image acquisition device to stop tracking the target person. In this way, the subsequent image acquisition device no longer tracks the target person, but normally collects image data of the working scene, which saves energy consumption of the image acquisition device and avoids the problem of the accidental triggering of the object moving operation.

[0150] In some embodiments, in order to save energy, after obtaining the starting position and the destination position of the target object, the electronic device can instruct the image acquisition device to stop tracking the target person.

[0151] In the embodiment of the present application, the electronic device performs content recognition on each acquired target voice data and each acquired target image data to obtain the starting position and the destination position of the target object. When the starting position and the destination position of the target object are obtained, the electronic device instructs the image acquisition device to stop tracking the target person. Subsequently, the image acquisition device no longer tracks the target person, but normally acquires image data of the working scene, which saves energy consumption of the image acquisition device and avoids the problem of erroneously triggering the object moving operation.

[0152] In an embodiment of the present application, if the extracted image biometric features do not match the image biometric features of the target person, the electronic device may not perform subsequent processing on the image data, that is, no subsequent move operation processing, such as the processing operations of steps S202 and S203 above.

[0153] In some embodiments, the image acquisition device acquires image data in real time. The electronic device acquires the image data acquired by the image acquisition device in real time, and extracts the image biometric features of the person included in each image data acquired by the image acquisition device. If the extracted image biometric features match the image biometric features of the target person, the image data is used as the target image data of the target person. In this way, the electronic device can select one or more target image data of the target person from the multiple image data acquired by the image acquisition device, and then perform subsequent moving operation processing, such as the processing operations of the above-mentioned steps S202 and S203.

[0154] Mode b2: Passive acquisition mode: Specifically, the electronic device may adopt any of the following passive acquisition modes to acquire the target image data.

[0155] In mode b21, the electronic device receives a task instruction and detects the information carried by the task instruction; if it is determined that the task instruction carries the trigger key information, the target image data of the target person collected by the image acquisition device is obtained. That is, when receiving the task instruction including the trigger key information, the electronic device executes the step of obtaining the target image data of the target person collected by the image acquisition device.

[0156] In the embodiment of the present application, the acquisition of the task instruction including triggering the key information can be seen in the above Figure 5 In the case of receiving a task instruction including triggering key information, the electronic device can adopt the above-mentioned method b1 to obtain target image data.

[0157] In some embodiments, the task instruction including the triggering key information may also include the current location of the target person; based on the current location of the target person included in the task instruction, the electronic device adopts the above-mentioned method b1 to obtain the target image data.

[0158] In mode b22, the electronic device can detect the stacking time of objects in the work scene. When it is detected that the stacking time of the objects is greater than the preset time, the electronic device executes the step of acquiring the target image data of the target person acquired by the image acquisition device. The electronic device can adopt the above-mentioned mode b1 to acquire the target image data.

[0159] In an embodiment of the present application, the electronic device can pre-acquire the location of objects placed in the work scene. Based on the acquired object location, the electronic device can detect the corresponding object and then detect the stacking time of the object.

[0160] The location where the object is placed can be manually input into the electronic device by the user, such as the user can enter the location of the object manually. Figure 5The position of the object can also be detected by the electronic device using an image acquisition device or a millimeter wave antenna or other detection device.

[0161] For example, the image acquisition device acquires image data in real time. The electronic device recognizes the image data acquired by the image acquisition device, obtains the placed object, and determines the position of the object by using the calibration relationship between the image coordinate system and the world coordinate system, and the image data including the pixel coordinates of the object.

[0162] For another example, the electronic device identifies the image data collected by the image acquisition device, obtains the placed object, and the direction of the object relative to the electronic device, and uses the millimeter wave antenna to send a first signal to the direction of the object relative to the electronic device, and receives a second signal reflected by the target person, where the second signal is an echo signal of the first signal; the electronic device uses the first signal and the second signal to determine the position of the object.

[0163] After determining the location of the object, the electronic device can detect how long the object has been stacked and decide whether to acquire target image data.

[0164] In the embodiment of the present application, in order to facilitate the acquisition of target image data, when it is detected that the stacking time of the object is greater than the preset time, the electronic device can output a prompt message indicating the object image is moved through a voice playback device (such as a speaker) or a display screen, such as Figure 6 In one possible scenario shown, the robot's chest display screen displays the message "the object at position X needs to be moved" so that the target person can perform the moving action in time, ensuring that the electronic device can quickly obtain the target image data and quickly realize the object moving.

[0165] Method b23, a designated area for moving is pre-planned in the work scene, and the electronic device will execute the target image data acquisition only when the target person is located in the designated area to perform the moving action. That is, when it is detected that there is a person in the designated area, the electronic device regards the person in the designated area as the target person and executes the step of acquiring the target image data of the target person captured by the image acquisition device.

[0166] In the embodiment of the present application, the image acquisition device can acquire image data in real time. The electronic device recognizes the image data and obtains a recognition result. If the recognition result indicates that a person is located in a specified area, the image data of the specified area is continuously acquired, and the image data is the target image data.

[0167] For example, robots are electronic devices, and people are Figure 7 In the working scenario shown, area 1 is the designated area. Figure 7 In the case where the personnel are outside of area 1, Figure 7As shown in (1), the robot will not obtain the target image data; when the person moves into area 1, Figure 7 As shown in (2), the robot continuously collects image data of area 1, and executes the above steps S202 and S203 according to the collected image data.

[0168] In mode b24, the voice receiving device can collect voice data in real time. The electronic device obtains the voice data collected by the voice receiving device, and when it is determined that the voice data is the voice data of the target person, the electronic device executes the step of obtaining the target image data of the target person collected by the image acquisition device. That is, when the voice data of the target person is obtained, the electronic device executes the step of obtaining the target image data of the target person collected by the image acquisition device.

[0169] In the embodiment of the present application, the image acquisition device may start to acquire image data only after the electronic device acquires the voice data of the target person, so as to reduce the energy consumption of the image acquisition device. The image acquisition device may also acquire image data in real time; the electronic device will not acquire the image data acquired by the image acquisition device, and after the electronic device acquires the voice data of the target person, the electronic device acquires the image data acquired by the image acquisition device, so as to reduce the delay in acquiring the image data.

[0170] In some embodiments, Figure 8 As shown, a method for moving an object is also provided, which may include the following steps:

[0171] Step S801, obtaining the target voice data of the target person collected by the voice receiving device, and obtaining the target image data of the target person collected by the image collecting device; it is the same as the above step S201 and will not be repeated here.

[0172] Step S802, performing content recognition on the target voice data and the target image data to obtain the intention and action of the target person;

[0173] Step S803, if the intention is to indicate moving the object, then identifying the first starting position and the first destination position of the object indicated by the action, and obtaining the starting position and the destination position of the target object;

[0174] Step S804, controlling the robot to move the target object at the starting position to the destination position, which is the same as the above step S203 and will not be described again.

[0175] In the technical solution provided in the embodiment of the present application, the electronic device determines the starting position and destination position of the target object stacking placement based on the intention indicated by the target voice data and the position indicated by the target image data, thereby reducing the error rate of determining the starting position and destination position by a single factor and improving the accuracy of object moving.

[0176] In the above step S802, the electronic device performs content recognition on the target voice data to obtain the intention of the target person, and performs content recognition on the target image data to obtain the action of the target person.

[0177] After obtaining the intention and action of the target person, the electronic device can determine whether the object needs to be moved, as well as the starting position and destination position of the object.

[0178] If the electronic device recognizes that the intention is to instruct to move an object, the electronic device executes step S803 to further identify the action of the target person to determine whether the action of the target person is an action to move the object.

[0179] In the above step S803, when the electronic device determines that the target person's intention is to move the object, it identifies the starting position of the object placed indicated by the target person's action (ie, the first starting position) and the destination position of the object placed indicated (ie, the first destination position).

[0180] If the first starting position and the first destination position are identified, it means that the target person has performed the object moving action, and then the first starting position and the first destination position are used as the starting position and the destination position for stacking the target object.

[0181] For example Figure 9a In the work scene shown, the target person said "Move the things from there to there", and at the same time, the target person's waving arms pointed in two directions, that is, from one direction to another, such as Figure 9a In the direction a and direction b, direction a points to position 1 where the object is placed, and direction b points to position 2, which is the destination area for placing the object. The robot obtains the voice data of "Move things from there to there" and obtains the image data of the target person waving his arms and pointing in two directions. The robot performs content recognition on the voice data "Move things from there to there" and can obtain that the target person's intention is to move the object; in addition, the robot performs content recognition on the image data of the target person waving his arms and pointing in two directions and can obtain that the target person's action is an object moving action, and the starting position is position 1 and the destination position is position 2. Combining the target person's intention and the target person's action, the robot determines that the object needs to be moved, and the starting position of this object movement is position 1, and the destination position of this object movement is position 2.

[0182] In some embodiments, the steps for an electronic device to identify the first starting position and the first destination position of the object placement indicated by the action may be: identifying the starting direction and the destination direction of the object placement indicated by the action; determining the object closest to the starting direction and determining the destination area closest to the destination direction; taking the position of the determined object as the first starting position and taking the position of the determined destination area as the first destination position. Herein, the destination area may be identified by a wireframe drawn on the ground, or by a sign, or by an obstacle such as a shelf, and there is no limitation thereto.

[0183] For example Figure 9b In the working scenario shown, the target person said "Move the thing from there to there", and at the same time, the target person waved his arm pointing to two directions, such as Figure 9b the direction a and the direction b in. There are 3 positions where objects are placed in the direction a, such as Figure 9b the positions 1 to 3 in; there are 3 destination areas in the direction b, such as Figure 9b the positions 4 to 6 in. The robot obtains the image data of the target person waving his arm pointing to two directions. The robot performs content recognition on the image data of the target person waving his arm pointing to two directions, and can obtain that the action of the target person is an object moving action, and the starting direction of the object placement is the direction a, and the destination direction of the object placement is the direction b; wherein, the distances between the positions 1 to 3 and the direction a are d1 to d3 respectively, d1 < d2 < d3, and the distances between the positions 4 to 5 and the direction b are d4 to d6 respectively, d4 < d5 < d6. Furthermore, the robot determines the position 1 as the first starting position, that is, the starting position of the target object placement, and determines the position 4 as the first destination position, that is, the destination position of the target object placement.

[0184] In some other embodiments, the steps for an electronic device to identify the first starting position and the first destination position of the object placement indicated by the action may be: identifying the starting direction and the destination direction of the object placement indicated by the action; determining the object closest to the target person in the starting direction and determining the destination area closest to the target person in the destination direction; taking the position of the determined object as the first starting position and taking the position of the determined destination area as the first destination position.

[0185] For example Figure 9b In the working scenario shown, the target person said "Move the thing from there to there", and at the same time, the target person waved his arm pointing to two directions, such as Figure 9b the direction a and the direction b in. There are 3 positions where objects are placed in the direction a, such as Figure 9b the positions 1 to 3 in; there are 3 destination areas in the direction b, such as Figure 9bPositions 4 to 6 in the middle. The robot obtains the image data of the target person waving the arm and pointing to two directions. The robot performs content recognition on the image data of the target person waving the arm and pointing to two directions, and can obtain that the action of the target person is an object moving action, and the starting direction of the object placement is direction a, and the target direction of the object placement is direction b; among them, the distances between positions 1 to 3 in direction a and the robot are d11 to d13 respectively, d11 < d12 < d13, and the distances between positions 4 to 5 in direction b and direction b are d14 to d16 respectively, d14 < d15 < d16. Furthermore, the robot determines position 1 as the first starting position, that is, the starting position of the target object placement, and determines position 4 as the first target position, that is, the target position of the target object placement.

[0186] In the embodiments of the present application, the electronic device can uniquely identify a starting position and a target position by using the intention and action of the target person, without manually configuring the starting position and the target position, reducing the human consumption of object moving.

[0187] In some embodiments, as Fig.10 shown, there is also provided an object moving method, which may include the following steps:

[0188] Step S1001, obtaining the target voice data of the target person collected by the voice receiving device, and obtaining the target image data of the target person collected by the image collecting device; the same as step S201 above, which will not be elaborated here.

[0189] Step S1002, performing content recognition on the target voice data and the target image data to obtain the intention of the target person and the action of the target person; the same as step S802 above, which will not be elaborated here.

[0190] Step S1003, if the intention indicates moving an object, then identifying the first starting position and the first target position of the object placement indicated by the action to obtain the starting position and the target position of the target object placement; the same as step S803 above, which will not be elaborated here.

[0191] Step S1004, if the intention indicates moving an object, and the intention indicates the second starting position and the second target position of the object placement, then identifying the starting direction and the target direction of the object placement indicated by the action;

[0192] Step S1005, determining the first distance between the second starting position and the starting direction, and determining the second distance between the second target position and the target direction;

[0193] Step S1006, if both the first distance and the second distance are less than the threshold distance, then using the second starting position and the second target position as the starting position and the target position of the target object placement respectively.

[0194] Step S1007, controlling the robot to move the target object at the starting position to the destination position, which is the same as the above step S203 and will not be described again.

[0195] In the technical solution provided in the embodiment of the present application, the electronic device determines the starting position and destination position of the target object by combining the intention and position indicated by the target voice data and the direction indicated by the target image data, thereby reducing the error rate of determining the starting position and destination position by a single factor and improving the accuracy of object movement.

[0196] In the above step S1002, the electronic device performs content recognition on the target voice data to obtain the intention of the target person, and performs content recognition on the target image data to obtain the action of the target person.

[0197] After obtaining the intention and action of the target person, the electronic device can determine whether the object needs to be moved, as well as the starting position and destination position of the object.

[0198] If the electronic device recognizes that the intention is to instruct the moving of an object, and the intention indicates the starting position (i.e., the second starting position) and the destination position (i.e., the second destination position) of the object, the electronic device executes step S1004 to further identify the action of the target person to determine whether the action of the target person is to move the object.

[0199] In the above step S1004, when the electronic device determines that the target person intends to move the object and intends to indicate the second starting position and the second destination position of the object, it identifies the starting direction of the object and the destination direction of the object indicated by the target person's action.

[0200] If the starting direction and the destination direction are identified, it means that the target person has performed an object moving action. The electronic device calculates the distance between the second starting position and the starting direction (ie, the first distance), and calculates the distance between the second destination position and the destination direction (ie, the second distance).

[0201] If the starting direction and the destination direction are not identified, it means that the target person has not performed the object moving action, and the electronic device can end the object moving process.

[0202] In the above step S1005, the magnitude of the threshold distance can be set according to actual requirements. The electronic device compares the first distance and the second distance with the threshold distance respectively. If the first distance is greater than or equal to the threshold distance, or, the second distance is greater than or equal to the threshold distance, or, both the first distance and the second distance are greater than or equal to the threshold distance, it indicates that an error has occurred in the recognition, and the electronic device can end the current object moving process. If both the first distance and the second distance are less than the threshold distance, it indicates that the recognition is correct. Furthermore, the electronic device uses the second starting position and the second destination position as the starting position and the destination position where the target object is placed respectively.

[0203] For example Fig.11 In the working scenario shown, the target person said "Move the thing from position 1 to position 2". At the same time, the target person waved their arm in two directions, such as Fig.11 the directions a and b in. The robot obtains the voice data of "Move the thing from position 1 to position 2" and obtains the image data of the target person waving their arm in two directions. The robot performs content recognition on the voice data of "Move the thing from position 1 to position 2" and can obtain that the intention of the target person is to move an object, and the second starting position where the object is placed is position 1, and the second destination position where the object is placed is position 2. The robot performs content recognition on the image data of the target person waving their arm in two directions and can obtain that the action of the target person is an object moving action, and the starting direction where the object is placed is direction a, and the destination direction where the object is placed is direction b; among them, the distance d21 between position 1 and direction a, the distance d22 between position 2 and direction b, and the threshold distance is d0. If d21 < d0 and d22 < d0, the robot determines position 1 as the second starting position, that is, the starting position where the target object is placed, and determines position 2 as the second destination position, that is, the destination position where the target object is placed.

[0204] In the embodiments of the present application, the electronic device can uniquely identify a starting position and a destination position by using the intention and action of the target person, without manually configuring the starting position and the destination position, reducing the human consumption of object moving.

[0205] In some embodiments, as Fig.12 shown, there is also provided an object moving method, which may include the following steps:

[0206] Step S1201, based on the working scenario of the robot, obtain the type of the obstacle;

[0207] When performing object movement, the electronic device pre-identifies the types of obstacles in the work scene to determine the obstacles that need to be moved in the work scene (such as the target object mentioned above). Among them, the types of obstacles can be divided into stacked object types, single object types, and object names, etc. The types of obstacles can be divided according to the work scene of the robot. For example, in a work scene where stacked objects need to be moved, the types of obstacles can be divided into stacked placement types and non-stacked placement types; in a work scene where specific objects need to be moved, the types of obstacles can be divided into specific object names.

[0208] In the implementation of this application, the electronic device can obtain the type of obstacle in any of the following ways.

[0209] In mode c1, a user inputs a task instruction to an electronic device, wherein the task instruction carries the type of obstacles in the working scene. The electronic device determines the type of obstacles in the working scene using the task instruction.

[0210] Mode c2: The image acquisition device acquires image data of obstacles in the working scene of the robot, namely, second image data; the electronic device obtains the second image data acquired by the image acquisition device; and identifies the second image data to obtain the type of the obstacle.

[0211] In the embodiment of the present application, the user can input a task instruction to the electronic device, and the task instruction carries the location of the obstacle in the working scene. The electronic device instructs the image acquisition device to collect image data at the location, and then the image acquisition device collects the image data of the obstacle at the location to obtain the second image data; the electronic device identifies the second image data to obtain the type of the obstacle.

[0212] In the embodiment of the present application, the electronic device may also use the detection device to detect obstacles in the working scene of the robot, wherein the detection device may include at least one of an image acquisition device and a millimeter wave antenna; when an obstacle is detected, the type of the obstacle is identified, such as the above-mentioned instruction image acquisition device to collect the second image data at the position of the obstacle, and the second image data is identified to obtain the type of the obstacle. The method of detecting obstacles is the same as the method of detecting target objects, and the details can be referred to the above-mentioned related description, which will not be repeated here.

[0213] Method c3: The obstacle carries a tag, such as an electronic tag such as a Radio Frequency Identification (RFID) tag. The electronic device reads the tag carried by the obstacle in the robot's working scene and determines the type of the obstacle based on the read tag.

[0214] In the embodiment of the present application, the user can input a task instruction to the electronic device, and the task instruction carries the location of the obstacle in the working scene. The electronic device reads the RFID tag carried by the obstacle at the location, and then determines the type of the obstacle.

[0215] In the embodiment of the present application, the electronic device may also use detection devices such as an image acquisition device and / or a millimeter wave antenna to detect obstacles in the robot's working scene; when an obstacle is detected, the type of obstacle is identified, such as reading the RFID tag at the obstacle position, and then determining the type of obstacle based on the read tag. The method of detecting obstacles is the same as the method of detecting target objects, and the details can be found in the above-mentioned related description, which will not be repeated here.

[0216] In the embodiments of the present application, the electronic device may also detect the position of the obstacle in other ways, for example, using a sensor installed on the robot chassis to detect the position of the obstacle, which is not limited to this.

[0217] In the embodiment of the present application, the electronic device may obtain the position or image data of the obstacle in advance, but does not identify the type of the obstacle; when receiving a task instruction including triggering key information, the type of obstacle is identified, that is, when receiving a task instruction including triggering key information, the step of obtaining the type of obstacle is performed. If the task instruction including triggering key information is not received, the electronic device may not identify the type of obstacle to reduce unnecessary energy consumption of the electronic device.

[0218] For example, the electronic device is a robot, and the image acquisition device is installed on the robot. The robot moves continuously in the working scene; the image acquisition device collects image data of the working scene as the robot moves. The robot recognizes the image data and obtains image data of the obstacle. The robot can record the image data of the obstacle to obtain an image data set. When the robot receives a task instruction including trigger key information, the robot identifies the type of obstacle in each image data in the image data set.

[0219] Step S1202, if the type of the obstacle is the type of the object to be moved, the position of the obstacle is located, so that when the obstacle is identified as a target object, the position of the obstacle is used as the starting position of the target object;

[0220] In the embodiment of the present application, the type of the object to be moved can be set according to actual needs. For example, the type of the object to be moved is a stacking type, and multiple objects are stacked and placed in one place, then the type of these multiple objects is a stacking type. For another example, the type of the object to be moved is a specified item type, such as cartons, food, daily necessities, etc.

[0221] The electronic device can record the type of the object to be moved in advance. After identifying the type of obstacle, the electronic device can determine whether the type of obstacle is the type of the object to be moved. If so, it means that the obstacle is the object to be moved, such as the target object mentioned above, and then locate the position of the obstacle. Subsequently, based on the position of the obstacle, the electronic device can determine whether the obstacle is the target object, and if it is determined that the obstacle is the target object, the position of the obstacle is used as the starting position of the target object. For example, the electronic device can locate the position of one or more obstacles, and use the positions of these obstacles to adopt Figure 8 and Fig.10 The process shown in FIG. 1 determines the position of an obstacle as the first starting position or the second starting position.

[0222] The location of the obstacle may be input by a user into the electronic device, or may be obtained by the electronic device detecting the obstacle using an image acquisition device and / or a millimeter wave antenna or other detection device, and this is not limited.

[0223] Step S1203, obtaining the target voice data of the target person collected by the voice receiving device, and obtaining the target image data of the target person collected by the image collecting device; it is the same as the above step S201 and will not be repeated here.

[0224] Step S1204, performing content recognition on the target voice data and the target image data to obtain the starting position and the destination position of the target object; this is the same as the above step S202 and will not be repeated here.

[0225] Step S1205, controlling the robot to move the target object at the starting position to the destination position, which is the same as the above step S203 and will not be described again.

[0226] In the embodiment of the present application, in order to facilitate the positioning of the starting position and the destination position of the target object, the electronic device can pre-configure an electronic map. The electronic device marks the position of the object to be moved (such as the position of the above-mentioned obstacle) on the electronic map, and determines the starting position and the destination position of this moving operation in combination with the electronic map.

[0227] In the technical solution provided in the embodiment of the present application, the electronic device pre-locates the position of obstacles in the working scene, which facilitates the subsequent determination of the starting position and the destination position of the target object and the accurate movement of the object.

[0228] Combine the following Fig.13 The system signaling diagram shown in the figure describes in detail the object moving method provided by the embodiment of the present application. The electronic device, the image acquisition device and the voice receiving device can be configured on the robot or can be an independent physical machine. Fig.13In the embodiment, the electronic device, the image acquisition device and the voice receiving device are configured on the robot as an example for explanation. Here, the electronic device can be understood as the processor of the robot.

[0229] Step S1301: The image acquisition device acquires image data of the working scene in real time and sends the acquired image data to the electronic device.

[0230] In the embodiment of the present application, the robot moves in the working scene. As the robot moves, the image acquisition device can acquire image data of the entire working scene to form an image data set.

[0231] Step S1302: the electronic device identifies the image data collected by the image acquisition device to obtain image data of the obstacle.

[0232] In the embodiment of the present application, the electronic device filters out image data including obstacles from the image data set acquired by the image acquisition device to form an obstacle image data set.

[0233] Step S1303: the electronic device further identifies the image data of the obstacle, obtains image data indicating that the obstacle is an object to be moved, and locates the position of the object to be moved.

[0234] Step S1304: the voice receiving device collects voice data in real time and sends the collected voice data to the electronic device.

[0235] In step S1305, the electronic device recognizes the voice data collected by the voice receiving device; if the voice data includes trigger key information, step S1306 is executed for the subsequent image data, and step S1307 is executed for the subsequent image data and voice data; if the voice data does not include trigger key information, the relocation operation is terminated, that is, the subsequent steps S1306 and S1307 are not executed.

[0236] Step S1306: The electronic device recognizes the image data to obtain image data of the target person.

[0237] After determining that the voice data includes the trigger key information, the electronic device identifies the image data of the target person from the subsequently acquired image data.

[0238] Step S1307, the electronic device performs content recognition on the target person's voice data and image data to obtain the starting position and the destination position of this moving operation. The target person's voice data is the voice data including the trigger key information.

[0239] Step S1308: the electronic device controls the robot to move the target object at the starting position to the destination position.

[0240] In the embodiment of the present application, after obtaining the starting position and destination position of the moving operation, the electronic device can use the robot's display screen or voice playback device to output response information, such as the display screen showing "received the moving instruction, satisfied the moving object" etc. After completing the moving operation, the electronic device can also use the robot's display screen or voice playback device to output response information, such as the display screen showing "moving the object completed" etc.

[0241] By outputting response information, users can determine the status of various operations in a timely manner and adjust the status of the robot. For example, after the display shows "Moving object completed", users can promptly assign the next moving task.

[0242] Corresponding to the above-mentioned object moving method, the embodiment of the present application also provides an object moving device, such as Fig.14 As shown, the device comprises:

[0243] An acquisition module 1401 is used to acquire target voice data of a target person collected by a voice receiving device, and acquire target image data of a target person collected by an image acquisition device;

[0244] The recognition module 1402 is used to perform content recognition on the target voice data and the target image data to obtain the starting position and the destination position of the target object;

[0245] The control module 1403 is used to control the robot to move the target object at the starting position to the destination position.

[0246] In some embodiments, the acquisition module 1401 may be specifically used to:

[0247] Acquiring voice data collected by a voice receiving device;

[0248] extracting speech features of the acquired speech data;

[0249] If the voice feature matches the voice feature that triggers the transfer, the voice data collected by the voice receiving device continues to be obtained as the target voice data of the target person.

[0250] In some embodiments, the voice feature includes at least one of a person's voice biometric feature, a voice spectrogram that triggers key information, text data corresponding to the triggering key information, and voice data corresponding to the triggering key information.

[0251] In some embodiments, the acquisition module 1401 may be specifically used to:

[0252] Obtain a wake-up instruction; according to the wake-up instruction, obtain the voice data collected by the voice receiving device as the target voice data of the target person.

[0253] In some embodiments, the acquisition module 1401 may be specifically used to:

[0254] When a click operation of the wake-up button is detected, a wake-up instruction is generated; or

[0255] In case of receiving a task instruction including triggering critical information, obtaining a wake-up instruction based on the task instruction; or

[0256] When the image data of the target person is acquired, a wake-up instruction is generated.

[0257] In some embodiments, the acquisition module 1401 may be specifically used to:

[0258] Locate the current location of the target person;

[0259] The image acquisition device is instructed to track the target person at the current position to obtain the target image data of the target person.

[0260] In some embodiments, the acquisition module 1401 may be specifically used to:

[0261] Acquire first image data acquired by an image acquisition device;

[0262] Extracting image biometric features of a person included in the first image data;

[0263] If the extracted image biometric feature matches the image biometric feature of the target person, the position of the person included in the first image data is located as the current position of the target person.

[0264] In some embodiments, the acquisition module 1401 may be specifically used to:

[0265] When the image acquisition device is instructed to track the target person for a preset time period, or when the starting position and the destination position are obtained, the image acquisition device is instructed to stop tracking the target person.

[0266] In some embodiments, the acquisition module 1401 may also be used to:

[0267] Under any of the following conditions, the step of acquiring target image data of the target person acquired by the image acquisition device is performed:

[0268] Receiving a task instruction including triggering key information;

[0269] It is detected that the stacking time of the objects is longer than the preset time;

[0270] Detect the presence of people in a designated area;

[0271] The target person’s voice data is obtained.

[0272] In some embodiments, the identification module 1402 may be specifically configured to:

[0273] Perform content recognition on the target voice data and the target image data to obtain the target person's intention and action;

[0274] If the intention is to instruct to move the object, then the first starting position and the first destination position of the object indicated by the action are identified to obtain the starting position and the destination position of the target object.

[0275] In some embodiments, the identification module 1402 may be specifically configured to:

[0276] If the intention is to indicate moving the object, and the intention is to indicate a second starting position and a second destination position for placement of the object, then identifying the starting direction and the destination direction for placement of the object indicated by the action;

[0277] Determine a first distance between a second starting position and a starting direction, and determine a second distance between a second destination position and a destination direction;

[0278] If the first distance and the second distance are both smaller than the threshold distance, the second starting position and the second destination position are respectively used as the starting position and the destination position for placing the target object.

[0279] In some embodiments, the identification module 1402 may be specifically configured to:

[0280] Identify the starting and destination directions of the object placed as indicated by the action;

[0281] Determine the object closest to the starting direction and determine the destination area closest to the destination direction; or, determine the object closest to the target person in the starting direction and determine the destination area closest to the target person in the destination direction;

[0282] The determined position of the object is used as a first starting position, and the determined position of the destination area is used as a first target position.

[0283] In some embodiments, the acquisition module 1401 may also be used to:

[0284] Before acquiring the target voice data and the target image data, the type of obstacle is acquired based on the robot's working scene;

[0285] If the type of the obstacle is the type of the object to be moved, the position of the obstacle is located, so that when the obstacle is identified as a target object, the position of the obstacle is used as the starting position of the target object.

[0286] In some embodiments, the acquisition module 1401 may be specifically used to:

[0287] Acquire second image data of an obstacle in the working scene of the robot acquired by an image acquisition device; identify the second image data to obtain the type of the obstacle; or

[0288] Read the tags carried by obstacles in the robot's working scene; determine the type of obstacles based on the read tags.

[0289] In some embodiments, the acquisition module 1401 may be specifically used to:

[0290] Detect obstacles in the working scene of the robot using a detection device, the detection device including at least one of an image acquisition device and a millimeter wave antenna;

[0291] In the event that an obstacle is detected, the type of obstacle is identified.

[0292] In some embodiments, the acquisition module 1401 may also be used to:

[0293] When a task instruction including triggering key information is received, a step of obtaining the type of obstacle based on the robot's working scenario is executed.

[0294] In the technical case provided by the embodiment of the present application, when executing the object moving, the starting position and the destination position of the target object are determined by combining the voice data and the image data, that is, the starting position and the destination position of this moving operation are determined, so as to control the robot to move the target object at the starting position to the destination position. In this object moving method, even if the starting position and / or the destination position of the object placement changes, the device can actively analyze the voice data and the image data to obtain the starting position and the destination position of this moving operation, without the need to manually plan and configure the robot's moving trajectory, thereby reducing the time and manpower consumed in object moving and improving the generalization of the object moving method.

[0295] Corresponding to the above-mentioned object moving method, the embodiment of the present application further provides an electronic device, such as Fig.15 As shown, it includes a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0296] Memory 1503, used for storing computer programs;

[0297] The processor 1501 is used to implement any of the above-mentioned object moving methods when executing the program stored in the memory 1503.

[0298] Corresponding to the above-mentioned object moving method, the embodiment of the present application also provides a robot, such as Fig.16 As shown, it includes a voice receiving device 1605, an image acquisition device 1606, a processor 1601, a communication interface 1602, a memory 1603 and a communication bus 1604, wherein the processor 1601, the communication interface 1602, and the memory 1603 communicate with each other through the communication bus 1604.

[0299] The voice receiving device 1605 is used to collect the target voice data of the target person;

[0300] Image acquisition device 1606, used to acquire target image data of a target person;

[0301] Memory 1603, used for storing computer programs;

[0302] The processor 1601 is used to implement any of the above-mentioned object moving methods when executing the program stored in the memory 1603.

[0303] The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0304] The communication interface is used for communication between the above electronic device or robot and other devices.

[0305] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0306] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0307] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned object moving methods is implemented.

[0308] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned object moving methods.

[0309] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0310] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0311] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, robot, storage medium and program product embodiments, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0312] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for moving an object, characterized in that: The method comprises: Acquire target voice data of the target person collected by the voice receiving device, and acquire target image data of the target person collected by the image collecting device; Performing content recognition on the target voice data and the target image data to obtain a starting position and a destination position of the target object; The robot is controlled to move the target object at the starting position to the destination position.

2. The method according to claim 1, characterized in that: The step of obtaining the target voice data of the target person collected by the voice receiving device comprises: Acquiring voice data collected by the voice receiving device; Extracting speech features of the acquired speech data; If the voice feature matches the voice feature that triggers the transfer, then continue to obtain the voice data collected by the voice receiving device as the target voice data of the target person; and / or, The voice feature includes at least one of a voice biometric feature of a person, a voice spectrogram of triggering key information, text data corresponding to the triggering key information, and voice data corresponding to the triggering key information; and / or, Under any of the following conditions, the step of acquiring the target image data of the target person acquired by the image acquisition device is performed: Receiving a task instruction including triggering key information; It is detected that the stacking time of the objects is longer than the preset time; Detect the presence of people in a designated area; The voice data of the target person is obtained.

3. The method according to claim 1, characterized in that The step of acquiring the target voice data of the target person collected by the voice receiving device includes: acquiring a wake-up instruction; acquiring the voice data collected by the voice receiving device according to the wake-up instruction as the target voice data of the target person; and / or, The step of obtaining the wake-up instruction includes: generating a wake-up instruction when a click operation of a wake-up button is detected; or, obtaining a wake-up instruction based on a task instruction including triggering key information is received; or, generating a wake-up instruction when image data of the target person is obtained.

4. The method according to any one of claims 1 to 3, characterized in that: The step of acquiring the target image data of the target person acquired by the image acquisition device comprises: locating the current position of the target person; instructing the image acquisition device to track the target person at the current position to obtain the target image data of the target person; and / or, The step of locating the current position of the target person includes: acquiring first image data acquired by an image acquisition device; extracting image biometric features of the person included in the first image data; if the extracted image biometric features match the image biometric features of the target person, locating the position of the person included in the first image data as the current position of the target person; and / or, The method further includes: when the image acquisition device is instructing to track the target person for a preset time period, or when the starting position and the destination position are obtained, instructing the image acquisition device to stop tracking the target person.

5. The method according to any one of claims 1 to 4, characterized in that: The step of performing content recognition on the target voice data and the target image data to obtain the starting position and the destination position of the target object includes: Performing content recognition on the target voice data and the target image data to obtain the intention of the target person and the action of the target person; If the intention indicates moving an object, then identifying a first starting position and a first destination position of the object indicated by the action, and obtaining a starting position and a destination position of the target object; or, If the intention indicates moving an object, and the intention indicates a second starting position and a second destination position for placing the object, then identifying the starting direction and the destination direction for placing the object indicated by the action; determining a first distance between the second starting position and the starting direction, and determining a second distance between the second destination position and the destination direction; if both the first distance and the second distance are less than a threshold distance, then using the second starting position and the second destination position as the starting position and the destination position for placing the target object, respectively; and / or, The step of identifying a first starting position and a first destination position of the object indicated by the action, comprises: Identify the starting direction and destination direction of the object indicated by the action; determine the object closest to the starting direction, and determine the destination area closest to the destination direction; or, determine the object closest to the target person in the starting direction, and determine the destination area closest to the target person in the destination direction; use the determined position of the object as the first starting position, and use the determined position of the destination area as the first destination position.

6. The method according to any one of claims 1 to 5, characterized in that: Before acquiring the target voice data and acquiring the target image data, the method further includes: acquiring the type of obstacle based on the working scene of the robot; if the type of the obstacle is the type of the object to be moved, locating the position of the obstacle, so that when the obstacle is identified as the target object, the position of the obstacle is used as the starting position of the target object; and / or, The method further includes: upon receiving a task instruction including triggering key information, executing the step of obtaining the type of obstacle based on the working scene of the robot.

7. The method according to claim 6, characterized in that The step of obtaining the type of obstacle based on the working scene of the robot includes at least one of the following: Acquire second image data of an obstacle in the working scene of the robot acquired by an image acquisition device; identify the second image data to obtain the type of the obstacle; or read a label carried by the obstacle in the working scene of the robot; determine the type of the obstacle according to the label; Obstacles in the working scene of the robot are detected by using a detection device, the detection device including at least one of the image acquisition device and a millimeter wave antenna; when the obstacle is detected, the type of the obstacle is identified.

8. An object moving device, characterized in that: The device comprises: An acquisition module, used to acquire the target voice data of the target person acquired by the voice receiving device, and acquire the target image data of the target person acquired by the image acquisition device; A recognition module, used to perform content recognition on the target voice data and the target image data to obtain a starting position and a destination position of the target object; The control module is used to control the robot to move the target object at the starting position to the destination position.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A robot, characterized in that: It includes a voice receiving device, an image acquisition device, a processor, a communication interface, a memory and a communication bus, wherein the voice receiving device, the image acquisition device, the processor, the communication interface and the memory communicate with each other through the communication bus; The voice receiving device is used to collect target voice data of the target person; The image acquisition device is used to acquire target image data of the target person; The memory is used to store computer programs; The processor is used to implement the method steps described in any one of claims 1 to 7 when executing the program stored in the memory.

Images