Control method and device

Abstract

The present application discloses a control method and device, which includes: controlling a virtual object to perform an action; the virtual object is constructed based on a physical object; obtaining a target instruction according to the action of the virtual object; and sending the target instruction to the physical object to enable the physical object to perform the action.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a control method and device. Background Art

[0002] Currently, when controlling a robot, there may be erroneous or redundant operations, resulting in low accuracy of the robot's movements. Summary of the Invention

[0003] In view of this, the present application provides a control method and device as follows:

[0004] A control method, comprising:

[0005] Controlling a virtual object to perform an action; the virtual object is constructed based on a physical object;

[0006] Obtaining a target instruction according to the action of the virtual object;

[0007] The target instruction is sent to the entity object, so that the entity object performs an action.

[0008] The above method preferably obtains the target instruction according to the action of the virtual object, including:

[0009] When the action of the virtual object satisfies a control condition, a first instruction to which the virtual object responds is obtained as a target instruction.

[0010] The above method preferably obtains the target instruction according to the action of the virtual object, including:

[0011] In a case where the action of the virtual object satisfies a control condition, a second instruction is obtained as a target instruction according to the action of the virtual object.

[0012] The above method preferably obtains the target instruction according to the action of the virtual object, including:

[0013] Obtaining historical actions of the virtual object;

[0014] De-duplication processing is performed on the historical actions to obtain the target action;

[0015] According to the target action, a third instruction is obtained as a target instruction.

[0016] The above method, preferably, after controlling the virtual object to perform an action and before obtaining the historical action of the virtual object, further comprises:

[0017] The method includes: starting from the time when the virtual object performs an action, and executing, when the timed duration reaches a duration threshold, obtaining the historical actions of the virtual object.

[0018] The above method preferably controls the virtual object to perform an action, including:

[0019] At least one virtual sub-object in the virtual object is controlled to perform a corresponding action, wherein the virtual sub-object corresponds to a physical component in the physical object.

[0020] In the above method, preferably, the virtual object is output in a target image, and the target image is an image corresponding to the environment in which the physical object is located.

[0021] In the above method, preferably, the target image is obtained by the following method:

[0022] Obtaining environmental data corresponding to the entity object;

[0023] constructing a three-dimensional global map as a target image based on the environmental data, wherein the pose parameters of the virtual object in the target image match the pose parameters of the physical object;

[0024] The target image is superimposed with a first-perspective image corresponding to the physical object, and the first-perspective image is an image of the physical object collected in its environment.

[0025] The above method, preferably, further comprises:

[0026] Obtaining a perspective switching instruction, wherein the perspective switching instruction includes a second perspective parameter;

[0027] Obtaining a second perspective image in the target image at least according to the second perspective parameter, where the second perspective image includes at least a portion of the first perspective image and at least a portion of the target image;

[0028] Output the second perspective image.

[0029] A control device, comprising:

[0030] An object control unit, configured to control a virtual object to perform an action; the virtual object is constructed based on a physical object;

[0031] an instruction obtaining unit, configured to obtain a target instruction according to the action of the virtual object;

[0032] An instruction sending unit is used to send the target instruction to the entity object so that the entity object performs an action.

[0033] As can be seen from the above technical solution, in a control method and device disclosed in this application, a virtual object corresponding to a physical object is pre-constructed. Based on this, the virtual object is first controlled to perform an action. Then, a corresponding target instruction is obtained based on the action of the virtual object. The target instruction is then sent to the physical object to cause the physical object to perform the action. As can be seen, in this application, by constructing a virtual object to pre-execute an action, and then controlling the action of the physical object based on the action of the virtual object, the action of the physical object is made more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 A flow chart of a control method provided in Example 1 of the present application;

[0036] Figure 2 This is an example diagram of controlling a physical robot through a virtual robot in an embodiment of the present application;

[0037] Figure 3 A partial flow chart of a control method provided in Example 1 of the present application;

[0038] Figure 4 This is an example diagram of the target image in the embodiment of this application;

[0039] Figure 5 This is another partial flow chart of a control method provided in Example 1 of the present application;

[0040] Figure 6 This is an example diagram of a target image superimposed with a first-perspective image in an embodiment of the present application;

[0041] Figure 7 This is another partial flow chart of a control method provided in Example 1 of the present application;

[0042] Figure 8 and Figure 9 They are respectively example diagrams of the image display interface in the embodiments of the present application;

[0043] Figure 10 A schematic structural diagram of a control device provided in Example 2 of the present application;

[0044] Figure 11 This is another structural schematic diagram of a control device provided in Example 2 of the present application;

[0045] Figure 12 A schematic diagram of the structure of an electronic device provided in Example 3 of the present application;

[0046] Figure 13-16 They are respectively example diagrams of scenarios in which this application is applicable to remote-controlled robots. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] refer to Figure 1 The figure shows a flow chart for implementing a control method provided in the first embodiment of the present application. This method can be applied to an electronic device capable of controlling the motion of a physical object. The electronic device can be a device independent of the physical object and capable of transmitting data with the physical object, such as a mobile phone, computer, or server. The technical solution in this embodiment is primarily used to improve the accuracy of controlling the motion of the physical object.

[0049] Specifically, the method in this embodiment may include the following steps:

[0050] Step 101: Control the virtual object to perform an action.

[0051] Virtual objects are constructed based on physical objects. Physical objects are objects that can perform actions, such as physical robots, physical cars, and physical spacecraft. Virtual objects are objects constructed based on physical objects, such as virtual robots, virtual cars, and virtual spacecraft.

[0052] Specifically, in this embodiment, the virtual object may be controlled to perform a corresponding action in response to a first instruction. The first instruction is an instruction generated based on a received control operation.

[0053] For example, taking the virtual object as a virtual robot, the user operates the joystick of the physical robot to move forward, backward, left or right. Based on this, in this embodiment, a first instruction is generated for the control operation generated by the joystick, and the virtual robot is controlled to move forward, backward, left or right according to the first instruction.

[0054] For another example, taking the virtual object as a virtual spacecraft, the user operates the mobile control in the control interface of the physical spacecraft. Based on this, in this embodiment, the control operation generated by the mobile control generates a first instruction, and controls the virtual spacecraft to move according to the first instruction.

[0055] In a specific implementation, controlling the virtual object to perform an action in step 101 may specifically include controlling at least one virtual sub-object in the virtual object to perform a corresponding action.

[0056] The virtual sub-objects herein correspond to physical components within a physical object. For example, a physical object contains one or more physical components, and a virtual object constructed based on the physical object contains virtual sub-objects corresponding to each physical component. Therefore, in this embodiment, when controlling a virtual object to perform an action, the virtual sub-objects corresponding to the physical components within the virtual object are controlled to perform the corresponding action.

[0057] For example, taking the virtual object as a virtual robot, the user operates the physical robot's robotic arm operating lever to move to the left and operates the physical robot's robotic leg operating lever to move forward. Based on this, in this embodiment, the control operations generated by the robotic arm operating lever and the robotic leg operating lever respectively generate first instructions, and control the virtual robot's robotic arm to move to the left according to the first instruction corresponding to the robotic arm and control the virtual robot's robotic leg to move forward according to the first instruction corresponding to the robotic leg.

[0058] Step 102: Obtain target instructions according to the actions of the virtual object.

[0059] Specifically, in this embodiment, the action of the virtual object may be parsed, and then based on the parsing result, a target instruction may be obtained, where the target instruction is related to the action performed by the virtual object.

[0060] Step 103: Send the target instruction to the physical object so that the physical object performs an action.

[0061] For example, Figure 2 As shown in , taking the physical object as a physical robot as an example, the electronic device where the method in this embodiment is located is a mobile phone independent of the physical robot. The user performs touch operations on the mobile controls in the control interface of the physical robot on the mobile phone. Based on this, the control operation generated by the mobile control in this embodiment generates a first instruction, and controls the virtual robot to move according to the first instruction. Afterwards, according to the action performed by the virtual robot, the target instruction is obtained, and the target instruction is sent to the physical robot, so that the physical robot performs the corresponding action according to the target instruction.

[0062] As can be seen from the above technical solution, in the control method provided in Example 1 of this application, a virtual object corresponding to a physical object is pre-constructed. Based on this virtual object, the virtual object is first controlled to perform an action. Then, a corresponding target instruction is obtained based on the action of the virtual object, and the target instruction is sent to the physical object to cause the physical object to perform the action. It can be seen that in this embodiment, by constructing a virtual object to pre-execute an action, and then controlling the action of the physical object based on the action of the virtual object, the action of the physical object is more accurate.

[0063] In one implementation, in step 102, obtaining a target instruction according to the action of the virtual object can be specifically implemented in the following manner:

[0064] It is detected whether the action of the virtual object meets the control condition, and when the action of the virtual object meets the control condition, a first instruction responded by the virtual object is obtained as a target instruction.

[0065] The control condition is a condition that a confirmation operation is received for the action of the virtual object. The confirmation operation is an operation received through the operation interface from the user to confirm the action of the virtual object.

[0066] That is, in this embodiment, after controlling a virtual object to perform an action, if a user confirms the action, the virtual object's action can be determined to satisfy the control condition. Therefore, the virtual object's action can be determined to satisfy the user's needs. The first instruction to which the virtual object responds, i.e., the first instruction that causes the virtual object's action to satisfy the control condition, can be used as the target instruction. Based on this, after sending the target instruction to the physical object, the physical object can be caused to perform the action, and the action performed by the physical object can also satisfy the user's needs.

[0067] For example, taking the physical object as a physical robot, the electronic device where the method in this embodiment is located is a mobile phone independent of the physical robot. The user performs touch operations on the mobile controls in the control interface of the physical robot on the mobile phone. Based on this, the control operation generated by the mobile control in this embodiment generates a first instruction, and controls the virtual robot to move according to the first instruction. If a confirmation operation is received from the user to confirm the action of the virtual robot, the first instruction is determined as the target instruction and sent to the physical robot, so that the physical robot performs the corresponding action according to the first instruction like the virtual robot, so that the action actually performed by the physical robot meets the user's needs.

[0068] In one implementation, in step 102, obtaining a target instruction according to the action of the virtual object can be specifically implemented in the following manner:

[0069] When the action of the virtual object satisfies the control condition, a second instruction is obtained as a target instruction according to the action of the virtual object.

[0070] The second instruction is different from the first instruction, and is an instruction generated according to the action actually performed by the virtual object.

[0071] In one case, the action actually performed by the virtual object is completely consistent with the action indicated by the first instruction; for example, the virtual object does not encounter any obstacles during the movement according to the first instruction and continues to move. At this time, the action actually performed by the virtual object is consistent with the action of continuous movement indicated by the first instruction.

[0072] In another case, the actual action of the virtual object is not completely consistent with the action indicated by the first instruction; for example, the virtual object encounters an obstacle while moving according to the first instruction and stops moving. At this time, the actual action performed by the virtual object is to move to the obstacle, which is inconsistent with the action of continuous movement indicated by the first instruction.

[0073] Based on this, in this embodiment, after controlling a virtual object to perform an action, if a user confirms the action, the virtual object's action can be determined to satisfy the control condition. Therefore, the virtual object's action can be determined to satisfy the user's needs. A second instruction can then be generated based on the action actually performed by the virtual object, with the second instruction serving as the target instruction. Therefore, after sending the target instruction to the physical object, the physical object can be caused to perform the action, and the action actually performed by the physical object can also satisfy the user's needs.

[0074] For example, taking the physical object as a physical robot, the electronic device where the method in this embodiment is located is a mobile phone independent of the physical robot. The user performs touch operations on the mobile controls in the control interface of the physical robot on the mobile phone. Based on this, in this embodiment, the control operation generated by the mobile control generates a first instruction, and controls the virtual robot to move according to the first instruction. If a confirmation operation of the user to confirm the action of the virtual robot is received, a second instruction is generated according to the actual action performed by the virtual robot, and the second instruction is determined as the target instruction and sent to the physical robot, so that the physical robot performs the corresponding action according to the second instruction, so that the actual action performed by the physical robot meets the needs of the user.

[0075] In one implementation, when obtaining the target instruction according to the action of the virtual object in step 102, it can be specifically implemented by the following steps: Figure 3 As shown in:

[0076] Step 301: Obtain historical actions of a virtual object.

[0077] The historical actions of the virtual object are actions that the virtual object has already completed, such as moving forward 10 meters and then moving back 4 meters.

[0078] Step 302: De-redundancy processing is performed on historical actions to obtain target actions.

[0079] Specifically, deduplication processing of the historical actions executed by the virtual object may be: deduplication processing of the action path of the virtual object, for example, deleting overlapping parts in the round-trip action path, etc., thereby obtaining the valid action executed by the virtual object, that is, the target action.

[0080] Step 303: According to the target action, obtain a third instruction as the target instruction.

[0081] The third instruction corresponds to a portion of the virtual object's historical actions, i.e., valid actions. Based on this, after the third instruction is generated according to the virtual object's target action, it is sent as the target instruction to the physical object, causing it to execute the corresponding action. This prevents the physical object from executing invalid actions, reduces its power consumption, and improves control efficiency.

[0082] For example, after the virtual robot's robotic arm moves 10 centimeters to the left and then 5 centimeters to the right under the control of the operating lever, the effective action of the virtual robot's robotic arm is moving 5 centimeters to the left. Based on this, a third instruction is generated according to the effective action of the robotic arm moving 5 centimeters to the left, and the third instruction is determined as the target instruction and sent to the physical robot, so that the robotic arm of the physical robot moves 5 centimeters to the left, avoiding the robotic arm from performing invalid actions, reducing the power consumption of the physical robot, and improving control efficiency.

[0083] For another example, after the wheel structure of the virtual robot moves forward 10 meters, it moves backward 4 meters. Therefore, the effective action of the wheel structure of the virtual robot is to move forward 8 meters. Based on this, a third instruction is generated according to the effective action of the wheel structure moving forward 8 meters. The third instruction is determined as the target instruction and sent to the physical robot, so that the wheel structure of the physical robot moves forward 8 meters, avoiding the wheel structure from performing invalid actions, reducing the power consumption of the physical robot, and improving control efficiency.

[0084] Based on the above implementation scheme, in this embodiment, the motion control of the physical object can be delayed for a certain period of time relative to the motion control of the virtual object.

[0085] Specifically, after step 101, in this embodiment, timing is started from the time when the virtual object executes the action, and step 301 is executed until the timing reaches the time threshold. Therefore, in this embodiment, the action execution of the physical object is delayed by the time threshold relative to the virtual object, that is, before the physical object executes the action, the virtual object first provides a pre-action for the user. After the time threshold, a third instruction is generated according to the effective action of the virtual object, and it is sent to the physical object as a target instruction, so that the physical object executes the corresponding action, thereby avoiding the physical object from executing invalid actions, reducing the power consumption of the physical object, and improving control efficiency.

[0086] The duration threshold can be 20 seconds or 10 seconds.

[0087] For example, the timing starts when the robotic arm of the virtual robot starts to move to the left under the control of the joystick. After 10 seconds, the historical action of the robotic arm is obtained. For example, after the robotic arm moves 10 centimeters to the left, it moves 5 centimeters to the right. Then the effective action of the robotic arm of the virtual robot is to move 5 centimeters to the left. Based on this, a third instruction is generated according to the effective action of the robotic arm moving 5 centimeters to the left. The third instruction is determined as the target instruction and sent to the physical robot, so that the robotic arm of the physical robot moves 5 centimeters to the left, avoiding the robotic arm from performing invalid actions, reducing the power consumption of the physical robot, and improving control efficiency.

[0088] For another example, timing starts when the wheel structure of the virtual robot starts to move forward. After 20 seconds of timing, the historical action of the wheel structure is obtained. For example, after the wheel structure moves forward 10 meters, it moves backward 4 meters. Then, the effective action of the wheel structure of the virtual robot is to move forward 8 meters. Based on this, a third instruction is generated according to the effective action of the wheel structure moving forward 8 meters. The third instruction is determined as the target instruction and sent to the physical robot, so that the wheel structure of the physical robot moves forward 8 meters, avoiding the wheel structure from performing invalid actions, reducing the power consumption of the physical robot, and improving control efficiency.

[0089] In one implementation, the virtual object is output in a target image, which is an image corresponding to the environment in which the physical object is located.

[0090] For example, Figure 4 As shown in , a virtual object is output in the target image, and the virtual object performs an action in the target image and is provided to the user as a reference for controlling the physical object.

[0091] Specifically, the target image can be obtained by the following methods: Figure 5 As shown in:

[0092] Step 501: Obtain environmental data corresponding to the physical object.

[0093] Specifically, in this embodiment, environmental data of the environment in which the entity is located can be collected by devices such as cameras and scanners. The environmental data includes images, videos, point cloud data, and other content.

[0094] Step 502: Based on the environmental data, a three-dimensional global map is constructed as a target image, and the pose parameters of the virtual object in the target image are matched with the pose parameters of the physical object.

[0095] Among them, the posture parameters include position parameters and attitude parameters, and the posture parameters can be expressed by six-degree-of-freedom parameters.

[0096] Specifically, in this embodiment, a three-dimensional global map can be constructed using a three-dimensional scene construction algorithm based on environmental data, and virtual objects with posture parameters matching those of physical objects can be created in the three-dimensional global map.

[0097] Furthermore, in this embodiment, a first-perspective image corresponding to the physical object is superimposed on the target image. The first-perspective image is an image collected of the physical object in its environment.

[0098] Specifically, the first-perspective image may be a real-scene image captured by an image capture device on the physical object in a corresponding capture direction, that is, at the first perspective.

[0099] For example, Figure 6 As shown in , the target image includes a three-dimensional global map of the space where the physical robot is located, and the real scene image collected from the first perspective of the physical robot is superimposed on the target image.

[0100] Based on the above solution, the method in this embodiment may further include the following steps: Figure 7 As shown in:

[0101] Step 104: Obtain a perspective switching instruction, where the perspective switching instruction includes a second perspective parameter.

[0102] The perspective switching instruction may be generated according to the received perspective switching operation.

[0103] For example, Figure 8 As shown in the figure, the first perspective image of the target image is output on the image display interface, and a perspective switching control is also output. After the user clicks the perspective switching control to the right, a perspective switching instruction is generated, which includes the second perspective parameters after the perspective is switched to the right.

[0104] Step 105: Obtain a second-viewing angle image in the target image at least according to the second viewing angle parameter.

[0105] The second perspective image includes at least a portion of the first perspective image and at least a portion of the target image.

[0106] Specifically, in this embodiment, the corresponding three-dimensional local map can be obtained in the target image according to the second viewing angle parameters, and then the overlapping part of the image that overlaps with the three-dimensional local image in the viewing angle range is cut out in the first viewing angle image, and finally the overlapping part of the image is superimposed on the three-dimensional local image according to the viewing angle range to obtain the second viewing angle image.

[0107] For example, Figure 9 As shown in , a three-dimensional local map corresponding to the second perspective parameter is intercepted in the target image, and then an overlapping portion of the image overlapping with the second perspective parameter is intercepted in the first perspective image. Finally, the overlapping portion of the image is superimposed on the three-dimensional local image to obtain a second perspective image.

[0108] Step 106: Output the second perspective image.

[0109] For example, in this embodiment, the second viewing angle image is output on the image display interface.

[0110] refer to Figure 10 , is a schematic diagram of the structure of a control device provided in Example 2 of this application. This control device can be configured in an electronic device capable of controlling the motion of a physical object. The electronic device can be a device independent of the physical object and capable of transmitting data with the physical object, such as a mobile phone, computer, or server. The technical solution in this embodiment is primarily used to improve the accuracy of controlling the motion of the physical object.

[0111] Specifically, the device in this embodiment may include the following units:

[0112] The object control unit 1001 is used to control a virtual object to perform an action; the virtual object is constructed based on a physical object;

[0113] An instruction obtaining unit 1002 is configured to obtain a target instruction according to the action of the virtual object;

[0114] The instruction sending unit 1003 is configured to send the target instruction to the entity object so that the entity object performs an action.

[0115] As can be seen from the above technical solution, in the control device provided in Example 2 of this application, a virtual object corresponding to a physical object is pre-constructed. Based on this virtual object, the virtual object is first controlled to perform an action. Then, a corresponding target instruction is obtained based on the action of the virtual object. The target instruction is then sent to the physical object to cause the physical object to perform the action. It can be seen that in this embodiment, by constructing a virtual object to pre-execute an action, and then controlling the action of the physical object based on the action of the virtual object, the action of the physical object is more accurate.

[0116] In one implementation, the instruction obtaining unit 1002 is specifically configured to obtain, when the action of the virtual object satisfies a control condition, a first instruction to which the virtual object responds as a target instruction.

[0117] In one implementation, the instruction obtaining unit 1002 is specifically configured to obtain a second instruction as a target instruction according to the action of the virtual object when the action of the virtual object satisfies a control condition.

[0118] In one implementation, the instruction obtaining unit 1002 is specifically configured to: obtain historical actions of the virtual object; perform redundancy processing on the historical actions to obtain a target action; and obtain a third instruction as a target instruction according to the target action.

[0119] Furthermore, the instruction obtaining unit 1002 is used to start timing from the time when the virtual object performs the action, after controlling the virtual object to perform the action, and before obtaining the historical action of the virtual object, until the timing reaches a duration threshold, and then execute the step of obtaining the historical action of the virtual object.

[0120] In one implementation, the object control unit 1001 is specifically configured to control at least one virtual sub-object in the virtual object to perform a corresponding action, where the virtual sub-object corresponds to a physical component in the physical object.

[0121] In one implementation, the virtual object is output in a target image, and the target image is an image corresponding to the environment in which the physical object is located.

[0122] Specifically, the device in this embodiment may further include the following units: Figure 11 As shown in:

[0123] The image acquisition unit 1004 is used to obtain environmental data corresponding to the physical object; based on the environmental data, a three-dimensional global map is constructed as a target image, and the posture parameters of the virtual object in the target image match the posture parameters of the physical object; wherein the target image is superimposed with a first-perspective image corresponding to the physical object, and the first-perspective image is an image of the physical object collected in its environment.

[0124] In one implementation, the image acquisition unit 1004 is further used to: obtain a perspective switching instruction, the perspective switching instruction including a second perspective parameter; obtain a second perspective image in the target image based at least on the second perspective parameter, the second perspective image including at least a portion of the image in the first perspective image and at least a portion of the image in the target image; and output the second perspective image.

[0125] It should be noted that the specific implementation of each unit in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0126] refer to Figure 12 , is a schematic diagram of the structure of an electronic device provided in Example 3 of the present application, and the electronic device may include the following structure:

[0127] Memory 1201, used to store computer programs and data generated by the execution of the computer programs;

[0128] Processor 1202 is used to execute a calculation program to achieve: controlling a virtual object to perform an action; the virtual object is constructed based on a physical object; obtaining a target instruction according to the action of the virtual object; and sending the target instruction to the physical object so that the physical object performs an action.

[0129] As can be seen from the above technical solution, in an electronic device provided by Example 3 of the present application, a virtual object corresponding to a physical object is pre-constructed. Based on this, the virtual object is first controlled to perform an action. Then, a corresponding target instruction is obtained based on the action of the virtual object. The target instruction is then sent to the physical object to cause the physical object to perform the action. It can be seen that in this embodiment, by constructing a virtual object to pre-execute an action, and then controlling the action of the physical object based on the action of the virtual object, the action of the physical object is more accurate.

[0130] For example, in current remote robot control scenarios, operating from the robot's first-person perspective prevents a better understanding of the surrounding environment. For example, changes in the environment below the robot's feet cannot be seen, making remote operation in complex terrain difficult and making it difficult to control the range of operation. Furthermore, if multiple cameras are installed on the robot to transmit real-time images, the user will need to pay attention to multiple angles simultaneously during operation.

[0131] To address the above-mentioned shortcomings, this application proposes the following implementation solutions:

[0132] 1. Collect the environmental data required by the physical robot to build a three-dimensional virtual scene, which is the three-dimensional global map mentioned above.

[0133] 2. When the physical robot enters a complex road section for operation, a three-dimensional global map of the current section is called up based on the physical robot's map position.

[0134] 3. Generate a virtual 3D image of the actual robot, i.e., the virtual robot, on the 3D global map using the current real-time posture of the physical robot.

[0135] 4. Superimpose the virtual image of the virtual robot on the real-time image that the current user is viewing remotely (the user is not aware of the image at this time).

[0136] 5. The user rotates the perspective on the first-person real-time picture transmitted by the physical robot (i.e., the first-person image mentioned above). Based on this, the three-dimensional global map is supplemented with content beyond the first-person real-time picture in real time based on real-time data and modeling information.

[0137] 6. The user begins an operation, for example: pressing and holding the arm movement button on the remote display terminal and sliding it forward, a virtual arm extends from the robot's arm and follows the movement. The arm's position adjusts in real time based on the user's sliding and is displayed in the current composite image. The movement of the real arm lags behind the virtual arm by X seconds. After the user releases the button, the operations performed in the previous X seconds will not be synchronized with the real robot. For another example, if the user presses and holds the forward button area to make the robot move forward, the virtual robot appears in the current composite image. The virtual robot advances for X seconds before the real robot begins to move. If the user releases the forward button area, the operations performed in the previous X seconds will not be synchronized with the real robot.

[0138] 7. The virtual robot prioritizes the operation process for X seconds and provides pre-feedback intelligent suggestions on the user interface (UI) (i.e., the image display interface mentioned above). For example, if the arm moves like this, it may hit the device, so it is recommended to move it 10 degrees to the right.

[0139] The value of X can be defined by the user.

[0140] The advantages of adopting this solution are as follows:

[0141] 1. Display the robot perspective required by the user in a virtual and real picture.

[0142] 2. The virtual robot intuitively displays the results of operations in the next X seconds. The intelligent suggestions displayed in the UI are continuously optimized and refined based on the robot's experience in the same scenario. This can also solve the problem of poor user experience caused by the delay of remote robot operation without the user's awareness.

[0143] The details are as follows:

[0144] 1. The original first-person perspective image is as follows Figure 13 The image shown in is a real scene image.

[0145] 2. Switch to any third perspective (here is a 3D virtual map combined with a 3D virtual robot with synchronized coordinates), such as Figure 14 As shown in .

[0146] Among them, the user adjusts the appropriate angle of the third perspective on the current virtual screen through touch screen gestures / remote control, etc., and starts to operate the robot (adjust the robotic arm, move forward, backward, rise and fall, etc.) after the adjustment is completed.

[0147] 3. The user's operation on the robot will be pre-feeded back to the user in the current virtual 3D scene, such as Figure 15 As shown in .

[0148] 4. After switching to the first-person perspective, the virtual robot's pre-operation trajectory and feedback will also be superimposed on the first-person perspective in AR form, such as Figure 16 As shown in .

[0149] 5. After confirmation, the real robot will execute the remote operation.

[0150] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0151] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0153] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A control method, comprising: Control virtual objects to perform actions; The virtual object is constructed based on the physical object, and the control of the virtual object to perform an action is to generate a control operation on a mobile control in the control interface for the virtual object; Obtaining a target instruction according to the action of the virtual object; Sending the target instruction to the entity object so that the entity object performs an action; Wherein, obtaining a target instruction according to the action of the virtual object includes: Obtaining historical actions of the virtual object; De-duplication processing is performed on the historical actions to obtain the target action; According to the target action, a third instruction is obtained as a target instruction; Wherein, after controlling the virtual object to perform an action and before obtaining the historical action of the virtual object, the method further includes: Starting from the time when the virtual object performs an action, and when the timed duration reaches a duration threshold, performing the following steps: obtaining a historical action of the virtual object; The virtual object is output in a target image, which is an image corresponding to the environment in which the physical object is located. The virtual object is output in the target image, and the virtual object performs actions in the target image, which is provided to the user as a reference for controlling the physical object.

2. The method according to claim 1, obtaining a target instruction according to the action of the virtual object, comprising: When the action of the virtual object satisfies a control condition, a first instruction to which the virtual object responds is obtained as a target instruction.

3. The method according to claim 1, wherein obtaining a target instruction according to the action of the virtual object comprises: In a case where the action of the virtual object satisfies a control condition, a second instruction is obtained as a target instruction according to the action of the virtual object.

4. The method according to claim 1, controlling the virtual object to perform an action, comprising: At least one virtual sub-object in the virtual object is controlled to perform a corresponding action, wherein the virtual sub-object corresponds to a physical component in the physical object.

5. The method according to claim 1, wherein the target image is obtained by: Obtaining environmental data corresponding to the entity object; constructing a three-dimensional global map as a target image based on the environmental data, wherein the pose parameters of the virtual object in the target image match the pose parameters of the physical object; in, The target image is superimposed with a first-perspective image corresponding to the physical object, where the first-perspective image is an image of the physical object collected in its environment.

6. The method according to claim 5, further comprising: Obtaining a perspective switching instruction, wherein the perspective switching instruction includes a second perspective parameter; Obtaining a second perspective image in the target image at least according to the second perspective parameter, where the second perspective image includes at least a portion of the first perspective image and at least a portion of the target image; Output the second perspective image.

7. A control device comprising: An object control unit, used to control the virtual object to perform actions; The virtual object is constructed based on the physical object, and the control of the virtual object to perform an action is to generate a control operation on a mobile control in the control interface for the virtual object; an instruction obtaining unit, configured to obtain a target instruction according to the action of the virtual object; an instruction sending unit, configured to send the target instruction to the entity object so that the entity object performs an action; Wherein, obtaining a target instruction according to the action of the virtual object includes: Obtaining historical actions of the virtual object; De-duplication processing is performed on the historical actions to obtain the target action; According to the target action, a third instruction is obtained as a target instruction; Wherein, after controlling the virtual object to perform an action and before obtaining the historical action of the virtual object, the device further includes: Starting from the time when the virtual object performs an action, and when the timed duration reaches a duration threshold, performing the following steps: obtaining a historical action of the virtual object; The virtual object is output in a target image, which is an image corresponding to the environment in which the physical object is located. The virtual object is output in the target image, and the virtual object performs actions in the target image, which is provided to the user as a reference for controlling the physical object.

Images