Method for operating target object, electronic device, medium and computer program product
Patent Information
- Application Number
- CN202380101241.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-18
- Publication Date
- 2026-03-24
AI Technical Summary
The operation method of traditional distribution cabinets requires the participation of electrical engineers or technicians, and is inefficient and insecure. Especially when the engineer is not on site, it may lead to overload or short circuit failure, resulting in power outages of the distribution cabinet and financial losses.
By creating a digital twin of scenes corresponding to the actual environment in the virtual game engine, the user generates control signals in the preset meta-universe environment, and the robot receives and convertsses into specific operations of the target object in the real environment, realizing remote control.
The accurate remote operation of the target object is achieved, the capabilities of the robot assist system are enhanced, effective cooperation between operators and automation technology is promoted, and the safe and efficient operation of the target object is ensured.
Smart Images
Figure CN121729666A_ABST
Abstract
Description
Method, electronic device, medium and computer program product for operating a target object Technical Field
[0001] The embodiments of the present application mainly relate to the field of remote control, and in particular to a method, electronic device, medium, and computer program product for operating a target object. Background Art
[0002] A power distribution cabinet, a general term for motor control centers, receives energy from the power supply and distributes it to different areas or equipment. Typically serving as backbone equipment in manufacturing environments, it is located in a designated workspace, such as a control room or electrical room. Traditional methods for monitoring, maintaining, and operating power distribution cabinets typically require the involvement of electrical engineers or technicians. However, this traditional manual approach is not only inefficient but also unsafe. Furthermore, if the engineer is not on-site, complex situations such as overloads or short circuits can occur, potentially leading to a power outage in the distribution cabinet, resulting in significant financial consequences.
[0003] Summary of the Invention
[0004] The embodiments of the present application provide a method for operating a target object, an electronic device, a medium, and a computer program product, which can help users achieve accurate remote operation of the target object.
[0005] In a first aspect, a method for operating a target object is provided, comprising: receiving a control signal from a user to operate a target object in a preset metaverse environment; wherein the preset metaverse environment includes a digital twin of a real scene; the control signal includes: an action source, time information corresponding to when the action occurs, the name of the target object, and action position information; according to the name of the target object, moving to a preset position of the corresponding target object in the real scene; according to the action position information, adjusting the posture of a robotic arm; and operating the target object in the real scene through the adjusted robotic arm.
[0006] In a second aspect, a device for operating a target object is provided, comprising: a receiving module, configured to receive a control signal of a user operating a target object in a preset metaverse environment; wherein the preset metaverse environment includes a digital twin of a real scene; the control signal includes an action source, time information corresponding to when the action occurs, the name of the target object, and action position information; a moving module, configured to move to a preset position of the corresponding target object in the real scene according to the name of the target object; an adjusting module, configured to adjust the posture of the robotic arm according to the action position information; and an operating module, configured to operate the target object in the real scene through the adjusted robotic arm.
[0007] In a third aspect, an electronic device is provided, comprising: at least one memory configured to store computer-readable code; and at least one processor configured to call the computer-readable code and execute each step of the method provided in the first aspect.
[0008] In a fourth aspect, a computer-readable medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the processor executes each step in the method provided in the first aspect.
[0009] In a fifth aspect, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes computer-executable instructions, which, when executed, cause at least one processor to perform the steps in the method provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The following figures are intended only to illustrate and explain the embodiments of the present application and are not intended to limit the scope of the embodiments of the present application.
[0011] FIG1 is a schematic diagram of a digital twin of a real scene according to an embodiment of the present application;
[0012] FIG2 is a flowchart of a method for operating a target object according to an embodiment of the present application;
[0013] FIG3 is a schematic diagram of an apparatus for operating a target object according to an embodiment of the present application;
[0014] FIG4 is a schematic diagram of an electronic device according to an embodiment of the present application.
[0015] Explanation of reference numerals 200: method for operating a target object 201-204: method steps 30: apparatus for operating a target object 31: receiving module 32: moving module 33: adjusting module 34: operating module 400: electronic device 401: processor 402: communication interface 403: memory 404: communication bus 405: program DETAILED DESCRIPTION
[0016] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that discussing these embodiments is merely to enable those skilled in the art to better understand and implement the subject matter described herein, and is not intended to limit the scope of protection, applicability, or examples set forth in the claims. The functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the embodiments of the present application. Various examples may omit, replace, or add various processes or components as needed. For example, the described method may be performed in an order different from the described order, and various steps may be added, omitted, or combined. In addition, the features described relative to some examples may also be combined in other examples.
[0017] As used herein, the term "including" and its variations are open terms meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.
[0018] The embodiment of the present application creates a digital twin of a scene corresponding to the actual environment in a virtual game engine, as shown in FIG1 , to create a corresponding metaverse. The metaverse can be expressed using various mixed reality technologies (including but not limited to augmented reality (AR), virtual reality (VR), mixed reality (MR), etc.), enabling users to perform interactive operations on related devices (such as head-mounted devices, portable devices, etc.). Specifically, users can use related devices to simulate the manipulation of target objects such as power equipment in the metaverse and generate control signals, which will be received by the robot and converted into specific operations of the target objects in the real environment, thereby achieving the purpose of remote control.
[0019] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0020] FIG2 is a flow chart of a method for operating a target object according to an embodiment of the present application. As shown in FIG2 , the method 200 for operating a target object includes:
[0021] Step 201: Receive a control signal from a user manipulating a target object in a pre-set metaverse environment. The pre-set metaverse environment includes a digital twin of a real-world scene. The control signal includes the source of the action, the time when the action occurred, the name of the target object, and the location of the action.
[0022] Step 201 can be applied to the robot side. Optionally, before step 201, a simulator can also be used to synchronously simulate the user's operation on the target object in the preset metaverse to ensure that the robot's movement in the real scene does not conflict with the surrounding environment, thereby further improving the safety of the machine equipment and protecting the overall environment. If a relevant conflict occurs, such as the robot's mechanical arm may collide with the surrounding equipment, a notification of operation failure is immediately sent to the user through the head-mounted device or other receiving device, and the forwarding of the control signal is terminated. On the contrary, if the simulation shows that the operation can be carried out successfully, the control signal of the user operating the target object in the preset metaverse is sent to the server, and forwarded by the server to the robot to perform the actual target object operation. Optionally, the conflict between the robot's movement and the surrounding environment may include but is not limited to: (1) physical collision: the robot or mechanical arm may collide with the surrounding equipment or mechanism. This conflict may be due to the mismatch between the robot's position or movement path and the spatial layout of the surrounding environment; (2) safety conflict: the robot's movement may cause the equipment to over-operate, resulting in the inability to achieve the action.
[0023] Optionally, when the server receives a control signal of the user operating the target object in the preset metaverse and a signal to activate the robot for remote operation, the server sends the control signal of the user operating the target object in the preset metaverse to the robot.
[0024] Optionally, the robot can consist of an automated guided vehicle (AGV) and a robotic arm. The AGV is responsible for positioning and movement within the site, while the robotic arm is responsible for fine manipulation. Alternatively, the server can be located in the cloud or at the edge to meet different network environments and real-time requirements.
[0025] Step 202: Move to a preset position of a corresponding target object in a real scene according to the name of the target object.
[0026] In one embodiment, before step 202 , the name of the target object in the real scene and the corresponding preset position information are stored.
[0027] In one embodiment, before step 202, a navigation control system is preset in the real scene to guide the robot's movement. Next, guidance information in the preset navigation control system is determined based on the name of the target object. Based on the guidance information, the robot moves to the preset position of the corresponding target object in the real scene.
[0028] Optionally, the navigation control system preset in the real scene may be composed of different navigation identifiers or devices, such as a QR (Quick Response) code, a laser or a magnetic tape.
[0029] QR Code: QR codes can be printed and affixed to specific locations to serve as navigation markers for robots. Each QR code encodes a specific location. When the robot scans the QR code with its camera, it can identify the location corresponding to the QR code and thus understand its specific position in the scene.
[0030] Lasers are commonly used in laser navigation systems. In these systems, a laser emits multiple laser beams into the surrounding environment. By measuring the time or angle of reflected laser light, the robot can determine its own position and the positions of surrounding objects. This method is commonly used for obstacle avoidance, path planning, and precise positioning.
[0031] Magnetic tape: This tape is typically laid on the ground to indicate the robot's path. The robot is often equipped with sensors that sense magnetic fields. As the robot travels on the tape, it can track and adjust its path by sensing changes in the magnetic field.
[0032] Step 203: Adjust the posture of the robotic arm according to the action position information.
[0033] Optionally, the 6D pose of the robotic arm is calculated based on the action position information. The robotic arm is adjusted based on the calculated 6D pose. The visual system on the adjusted robotic arm determines whether the robotic arm is pointing to the object corresponding to the first object pointed to by the user in the preset metaverse environment in the real scene. When it is determined that the object corresponds to the first object pointed to by the user in the preset metaverse environment in the real scene, the center position of the object is detected by the visual system. Based on the detected center position, the 6D pose of the robotic arm is adjusted again.
[0034] Alternatively, the corresponding 6D poses can be preset based on the positions of several buttons in the real scene. In this way, after receiving the action position information, the robotic arm can be directly adjusted according to the preset 6D pose, thereby simplifying the operation process and improving control efficiency.
[0035] Step 204 : operate the target object in the real scene through the adjusted robotic arm.
[0036] In one scenario, after adjusting the posture of the robotic arm, the robotic arm can press the real button corresponding to the button pressed by a user of the power equipment in the real scene in the metaverse environment, thereby completing the operation in the metaverse.
[0037] The embodiments of this application allow operators to remotely control a target object in a real-world scenario in the metaverse with the assistance of a robot. This embodiment not only enhances the capabilities of robotic-assisted systems but also promotes more effective collaboration between operators and automated technology, thereby ensuring the safe and efficient operation of the target object.
[0038] FIG3 is a schematic diagram of a device for operating a target object according to an embodiment of the present application. As shown in FIG3 , the device 30 for operating a target object includes:
[0039] Receiving module 31 is configured to receive control signals from a user manipulating a target object in a pre-set metaverse environment. The pre-set metaverse environment includes a digital twin of a real-world scene. The control signals include the source of the action, the time corresponding to the action, the name of the target object, and the location of the action.
[0040] The moving module 32 is configured to move to a preset position of a corresponding target object in a real scene according to the name of the target object.
[0041] The adjustment module 33 is configured to adjust the posture of the robotic arm according to the action position information.
[0042] The operation module 34 is configured to operate the target object in the real scene through the adjusted robotic arm.
[0043] The embodiments of the present application not only enhance the capabilities of robotic-assisted systems, but also promote more effective collaboration between operators and automation technology, thereby ensuring the safe and efficient operation of target objects.
[0044] FIG4 is a schematic diagram of an electronic device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device. As shown in FIG4 , the electronic device 400 may include: a processor 401, a communications interface 402, a memory 403, and a communication bus 404.
[0045] The processor 401 , the communication interface 402 , and the memory 403 communicate with each other via the communication bus 404 .
[0046] The communication interface 402 is used to communicate with other electronic devices or servers.
[0047] The processor 401 is configured to execute the program 402 , and specifically may execute the relevant steps in any one of the aforementioned method embodiments.
[0048] Specifically, the program 405 may include program codes, which include computer operation instructions.
[0049] Processor 401 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.
[0050] The memory 403 is used to store the program 405. The memory 403 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0051] The program 405 can be specifically used to enable the processor 401 to execute any one of the multiple method embodiments in the aforementioned embodiments.
[0052] The specific implementation of each step in program 405 can be found in the corresponding descriptions of the corresponding steps and units in the aforementioned method embodiment for operating the target object, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the aforementioned method embodiment, and will not be repeated here.
[0053] The present application also provides a computer-readable storage medium storing instructions for causing a machine to perform any of the multiple method embodiments described herein. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.
[0054] In this case, the program code read from the storage medium itself can realize the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of this application.
[0055] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.
[0056] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform any corresponding operation in the above-mentioned multiple method embodiments.
[0057] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.
[0058] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or can be implemented as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.
[0059] It should be noted that not all steps and modules in the above processes and system structure diagrams are required. Certain steps or modules may be omitted based on actual needs. The execution order of the steps is not fixed and may be adjusted as needed. The system structure described in the above embodiments may be a physical structure or a logical structure. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or may be implemented by certain components in multiple independent devices.
[0060] In the above embodiments, the hardware module can be implemented mechanically or electrically. For example, a hardware module can include a permanent dedicated circuit or logic (such as a dedicated processor, FPGA or ASIC) to complete the corresponding operation. The hardware module can also include programmable logic or circuits (such as a general-purpose processor or other programmable processors), which can be temporarily set by software to complete the corresponding operation. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.
[0061] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.
[0062] Nouns and pronouns referring to persons in this patent application are not limited to a specific gender.
Claims
1. A method for operating a target object, comprising: - receiving (201) a control signal of a user operating a target object in a preset metaverse environment; wherein the preset metaverse environment includes a digital twin of a real scene; the control signal includes: an action source, time information corresponding to when the action occurs, a name of the target object, and action location information; - according to the name of the target object, moving (202) to a preset position of a corresponding target object in the real scene; - adjusting (203) the position of the robot arm according to the action position information; - Using the adjusted robotic arm to operate the target object in the real scene (204).
2. The method according to claim 1, wherein: Before receiving (201) a control signal from a user operating a target object in a preset metaverse, the method further comprises: -When the server receives a control signal of a user operating a target object in a preset metaverse and a signal to activate a robot for remote operation, the server sends the control signal of the user operating the target object in the preset metaverse to the robot.
3. The method according to claim 1, wherein: Before moving (202) to a preset position of a corresponding target object in the real scene according to the name of the target object, the method further comprises: -Storing the name of the target object in the real scene and the corresponding preset position information.
4. The method according to claim 1, wherein: - before moving (202) to a preset position of a corresponding target object in the real scene according to the name of the target object, the method further comprises: -presetting a navigation control system in the real scene to guide the robot to move; - the step of moving (202) to a preset position of a corresponding target object in the real scene according to the name of the target object comprises: -determining guidance information in the preset navigation control system according to the name of the target object; - According to the guidance information, move to a preset position of a corresponding target object in the real scene.
5. The method according to claim 1, wherein: The adjusting (203) the posture of the robot arm according to the action position information comprises: -Calculate the 6D pose of the robot arm according to the motion position information; -Adjust the robot arm according to the calculated 6D pose; - determining, through the visual system on the adjusted robotic arm, whether the robotic arm is pointing to an object corresponding to the first object pointed to by the user in the preset metaverse environment in the real scene; -When it is determined that the first object pointed to by the user in the preset metaverse environment is the object corresponding to the real scene, - detecting the center position of the object by means of the visual system; - According to the detected center position, adjust the 6D pose of the robot arm again.
6. A device for operating a target object, comprising: - A receiving module (31), configured to: receive a control signal of a user operating a target object in a preset metaverse environment; wherein the preset metaverse environment includes a digital twin of a real scene; the control signal includes: an action source, time information corresponding to when the action occurs, a name of the target object, and action location information; - a moving module (32), configured to: move to a preset position of a corresponding target object in the real scene according to the name of the target object; - an adjustment module (33), configured to: adjust the posture of the robot arm according to the action position information; - an operation module (34), configured to operate the target object in the real scene through the adjusted mechanical arm.
7. An electronic device (400), comprising: A processor (401), a communication interface (402), a memory (403) and a communication bus (404), wherein the processor (401), the memory (403) and the communication interface (402) communicate with each other via the communication bus (404); The memory (403) is used to store at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the method for operating a target object as described in any one of claims 1 to 5.
8. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for operating a target object according to any one of claims 1 to 5 is implemented.
9. A computer program product, the computer program product being tangibly stored on a computer-readable medium and comprising computer-executable instructions, which when executed cause at least one processor to perform the method of operating a target object according to any one of claims 1-5.