Virtual-real fusion robot control method and system

Abstract

The invention provides a virtual-real fusion robot control method and system, and the method comprises the steps: obtaining observation information, the observation information is obtained through the observation of a virtual-real fusion space through a virtual-real fusion robot, and the virtual-real fusion space is obtained based on the fusion of a virtual space and a physical space; performing decision prediction on the virtual-real fusion space according to the observation information to obtain a decision prediction result; and controlling the virtual-real fusion robot to execute corresponding actions according to the decision prediction result. Decision prediction is carried out on the virtual-real fusion space according to the obtained observation information of the virtual-real fusion space, task allocation is carried out on the virtual space and the physical space, the decision prediction result is obtained, the virtual-real fusion robot is controlled to execute corresponding actions according to the decision prediction result, and cooperative control of the virtual-real fusion robot is achieved. The actual application of the virtual-real fusion robot is supported, services are provided in the virtual-real fusion space, and closed loops of physical and virtual spaces are formed.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a control method and system for a virtual-real fusion robot. Background Technology

[0002] Digital twins can be used to train robots in virtual spaces, such as smart cars, humanoid robots, and robotic arms, and then transfer the trained skills to real-world applications. This approach still ultimately targets physical robots; the AI ​​algorithm ultimately drives a real robot in a physical space. Besides physical robots, intelligent virtual humans are also widely used in games, live streaming, and virtual social networking, where the AI ​​algorithm drives the virtual human in a virtual space. Both virtual humans and physical robots can be seen as different manifestations of artificial intelligence.

[0003] With the continuous integration of virtual reality, augmented reality, and artificial intelligence, the virtual-real fusion space is becoming increasingly widespread. However, how intelligent agents can provide services to humans in the virtual-real fusion space remains an unexplored question. Furthermore, given the different natures of virtual and physical spaces, there are still no effective solutions for how virtual and physical embodied structures can effectively cooperate and how they can collaboratively perform tasks in the virtual-real fusion space. Summary of the Invention

[0004] This invention provides a control method and system for a virtual-real fusion robot, which solves the shortcomings of existing technologies that cannot effectively coordinate virtual and physical avatar structures in a virtual-real fusion space, realizes collaborative control of virtual and physical avatars, supports the practical application of virtual-real fusion robots, and enables the provision of services in a virtual-real fusion space.

[0005] This invention provides a control method for a virtual-real fusion robot, comprising: acquiring observation information, wherein the observation information is obtained by observing a virtual-real fusion space using the virtual-real fusion robot, and the virtual-real fusion space is obtained by fusing virtual space and physical space; making decision predictions on the virtual-real fusion space based on the observation information, and obtaining decision prediction results; and controlling the virtual-real fusion robot to perform corresponding actions based on the decision prediction results.

[0006] According to the control method of the virtual-real fusion robot provided by the present invention, the method performs decision prediction on the virtual-real fusion space based on observation information to obtain decision prediction results, including: inputting observation information into a decision prediction model to obtain the decision prediction results output by the decision prediction model; wherein, the decision prediction model is trained based on observation training information and decision labels corresponding to the observation training information; the decision prediction model is used to extract features from the input observation information and perform decision prediction based on the extracted state features.

[0007] According to the present invention, a control method for a virtual-real fusion robot includes a decision prediction model comprising: a feature extraction layer for extracting features from input observation information to obtain state features; an event classification layer for classifying the state features into virtual space events and physical space events; and a decision prediction layer for making decision predictions based on the virtual space events and physical space events to obtain decision prediction results, wherein the decision prediction results include physical space tasks and virtual space tasks.

[0008] According to a control method for a virtual-real fusion robot provided by the present invention, before inputting observation information into a decision prediction model, the method includes: acquiring observation training information and decision labels corresponding to the observation training information; using the observation training information as input data for training and the decision labels corresponding to the observation training information as labels for training, training the model to be trained to obtain a decision prediction model for predicting decisions.

[0009] According to the control method of the virtual-real fusion robot provided by the present invention, the decision prediction model adopts a deep learning Transformer model based on self-attention mechanism; or, the decision prediction model adopts a long short-term memory LSTM network; or, the decision prediction model adopts a large multimodal GPT-4 model.

[0010] According to a control method for a virtual-real fusion robot provided by the present invention, the method controls the virtual-real fusion robot to perform corresponding actions based on decision prediction results, including: controlling the virtual-real fusion robot to perform corresponding actions based on decision prediction results; during the process of controlling the virtual-real fusion robot to perform corresponding actions based on decision prediction results, acquiring observation information of the virtual-real fusion space again according to a preset period, and determining whether the reacquired observation information of the virtual-real fusion space is updated; based on the updated observation information of the reacquired virtual-real fusion space, using the updated observation information, re-making decision predictions for the virtual-real fusion space, and optimizing the decision prediction results.

[0011] According to a control method for a virtual-real fusion robot provided by the present invention, after controlling the virtual-real fusion robot to perform corresponding actions based on decision prediction results, the method includes: based on the completion of the virtual-real fusion robot performing corresponding actions, re-acquiring observation information of the virtual-real fusion space, and determining whether the re-acquiring observation information of the virtual-real fusion space is updated; based on the updated observation information of the re-acquiring virtual-real fusion space, using the updated observation information, re-making decision predictions for the virtual-real fusion space, and optimizing the decision prediction results.

[0012] The present invention also provides a control system for a virtual-real fusion robot, comprising: an information acquisition module for acquiring observation information, wherein the observation information is obtained by observing the virtual-real fusion space using the virtual-real fusion robot, and the virtual-real fusion space is obtained by fusion of virtual space and physical space; a decision prediction module for making decision predictions on the virtual-real fusion space based on the observation information, and obtaining decision prediction results; and a decision control module for controlling the virtual-real fusion robot to perform corresponding actions based on the decision prediction results.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the control method of the virtual-real fusion robot as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method of the virtual-real fusion robot as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the control method for the virtual-real fusion robot as described above.

[0016] The control method and system for virtual-real fusion robots provided by this invention make decisions and predictions about the virtual-real fusion space by acquiring observation information of the virtual-real fusion space, so as to allocate tasks to the virtual space and the physical space, obtain the decision prediction results, and control the virtual-real fusion robot to perform corresponding actions according to the decision prediction results, so as to realize the collaborative control of virtual and real integration, support the practical application of virtual-real fusion robots, realize the provision of services in the virtual-real fusion space, and form a closed loop of physical and virtual spaces. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the control method for the virtual-real fusion robot provided by the present invention.

[0019] Figure 2 This is a flowchart illustrating the training decision prediction model provided by the present invention.

[0020] Figure 3 This is the second flowchart illustrating the control method for the virtual-real fusion robot provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the control system of the virtual-real fusion robot provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the architecture of the control system for the virtual-real fusion robot provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Figure 1 A flowchart illustrating a control method for a virtual-real fusion robot according to the present invention is shown below. The method includes: S11, Obtain observation information. The observation information is obtained by using a virtual-real fusion robot to observe the virtual-real fusion space, which is based on the fusion of virtual space and physical space. S12, Based on the observation information, make decisions and predictions for the virtual-real fusion space to obtain the decision prediction results; S13, based on the decision prediction results, control the corresponding virtual-real fusion robot to perform the corresponding actions.

[0026] It should be noted that the step number "S1N" in this manual does not represent the order of the control methods for the virtual-real fusion robot. The following details will explain further. Figures 2-4 The present invention describes the control method for the virtual-real fusion robot.

[0027] Step S11: Obtain observation information. The observation information is obtained by using a virtual-real fusion robot to observe the virtual-real fusion space, which is obtained by fusing virtual space and physical space.

[0028] In this embodiment, the observation information includes virtual space observation information and physical space observation information, and the virtual-real fusion robot includes a physical space embodied intelligent agent and a virtual space embodied intelligent agent. Accordingly, observing the virtual-real fusion space using the virtual-real fusion robot includes: observing the physical space using the physical space embodied intelligent agent, and observing the virtual space using the virtual space embodied intelligent agent.

[0029] It should be noted that embodied intelligent agents in physical and virtual spaces can adopt the same or different embodied forms, without further limitations here. This allows for the deployment of diverse embodied forms of hybrid virtual-real robots in different spaces to better adapt to task requirements, expanding the scope of application of embodied intelligent agents. By relying on hybrid virtual-real robots to provide services in both physical and virtual spaces, a closed loop between the physical and virtual spaces is formed. Furthermore, embodied intelligent agents in both physical and virtual spaces can utilize correspondingly configured imaging or sensing devices for observation. For example, embodied intelligent agents in physical spaces can acquire physical spatial information such as images, sounds, and pressure through physical sensors, such as cameras, microphones, and tactile devices; similarly, embodied intelligent agents in virtual spaces can acquire virtual spatial information through virtual sensors, such as virtual cameras and virtual microphones. Again, without further limitations here.

[0030] It should be noted that in physical space, embodied intelligent agents can coexist with ordinary physical robots and biological humans (i.e., humans); in virtual space, embodied intelligent agents can coexist with ordinary virtual embodied forms and virtual humans.

[0031] In one alternative embodiment, the virtual-real fusion machine is driven by artificial intelligence (AI model). Accordingly, before using the virtual-real fusion robot to observe the virtual-real fusion space, the AI ​​model reads in a configuration file and obtains the controllable embodied form and control method from the configuration file.

[0032] For example, if a physical space contains a robotic arm, one can obtain the robotic arm's state and control methods; it is also possible to obtain information in virtual space, such as controlling several virtual people or virtual objects and their control methods.

[0033] Step S12: Based on the observation information, make a decision prediction on the virtual-real fusion space and obtain the decision prediction result.

[0034] In this embodiment, decision prediction is performed on the virtual-real fusion space based on observation information to obtain decision prediction results, including: inputting observation information into the decision prediction model to obtain the decision prediction results output by the decision prediction model; wherein, the decision prediction model is trained based on observation training information and the decision labels corresponding to the observation training information; the decision prediction model is used to extract features from the input observation information and make decision predictions based on the extracted state features.

[0035] It is worth noting that the decision prediction results are obtained by classifying events based on the state features extracted from the observation information in order to predict the decision. The decision prediction results include the task allocation results of the virtual space embodied intelligent agent and the physical space embodied intelligent agent. For example, the task can be to not make any reaction and continue to acquire observation information, or to perform some kind of behavior, such as helping people who are working to build a more efficient interactive environment in the virtual space, or to deliver a few drinks to children who are playing in the physical space. The specifics can be determined based on the actual observation information acquired, which will not be further explained here.

[0036] Furthermore, the decision prediction model includes: a feature extraction layer, which extracts features from the input observation information to obtain state features; an event classification layer, which classifies the state features into virtual space events and physical space events; and a decision prediction layer, which makes decision predictions based on the virtual space events and physical space events to obtain decision prediction results, which include physical space tasks and virtual space tasks.

[0037] Accordingly, the observation information is input into the decision prediction model to obtain the decision prediction results output by the decision prediction model, including: inputting the observation information into the feature extraction layer for feature extraction to obtain the state features output by the feature extraction layer; inputting the state features into the event classification layer for event classification to obtain the virtual space events and physical space events output by the event classification layer; inputting the virtual space events and physical space events into the decision prediction layer for decision prediction to obtain the decision prediction results output by the decision prediction layer.

[0038] In one alternative embodiment, the decision prediction model employs a deep learning Transformer model based on a self-attention mechanism; or, the decision prediction model employs a Long Short-Term Memory (LSTM) network; or, the decision prediction model employs a large-scale multimodal GPT-4 model.

[0039] It should be added that when using the Transformer model or LSTM network, the corresponding network can be trained in an end-to-end manner, directly from the observation information to the specific tasks of each structure, so as to assign tasks to the physical space embodied intelligent agent and the virtual space embodied intelligent agent.

[0040] Additionally, when using GPT-4, prompt words need to be provided to GPT-4 to output the task allocation results of the physical space embodied agent and the virtual space embodied agent in the given scenario. This facilitates the execution of corresponding tasks in the virtual-real fusion space obtained based on the task allocation results and the collaboration between the physical and virtual embodied agents.

[0041] In one alternative embodiment, training the decision prediction model is included before inputting the observation information into the decision prediction model. Specifically, refer to Figure 2 Training the decision prediction model includes: S21, Obtain the observation training information and the decision labels corresponding to the observation training information; S22, the observed training information is used as the input data for training, and the decision labels corresponding to the observed training information are used as the labels for training. The model to be trained is trained to obtain a decision prediction model for predicting decisions.

[0042] In one optional embodiment, after obtaining the observation training information and the corresponding decision labels, the method includes: performing data augmentation on the observation training information using data augmentation strategies. Specifically, the data augmentation strategies include at least one of flipping, rotating, cropping, deforming, and scaling; and / or, the data augmentation strategies include at least one of noise, blurring, color transformation, erasing, and filling.

[0043] It should be noted that the data augmentation strategy selected in this embodiment is applicable to augmenting the above-mentioned observation and training information, so as to increase the amount of observation and training information data and improve the accuracy of the model's decision prediction in physical and virtual spaces during subsequent model training.

[0044] In addition, training the model to be trained includes: inputting the observed training information into the model to be trained to obtain the decision training result output by the model to be trained; constructing a loss function based on the decision training result and the decision label corresponding to the observed training information; and ending the training based on the convergence of the loss function.

[0045] Step S13: Based on the decision prediction results, control the virtual-real fusion robot to perform corresponding actions.

[0046] It should be noted that since the virtual-real fusion robot includes physical space embodied intelligent agents and virtual space embodied intelligent agents, the virtual-real fusion robot is controlled to perform corresponding actions based on the decision prediction results. This includes controlling the physical space embodied intelligent agent and the virtual space embodied intelligent agent to perform corresponding tasks in their respective spaces based on the virtual-real linkage control, thereby realizing the collaborative control of embodied intelligent agents in different spaces to perform tasks in the virtual-real fusion space.

[0047] In an optional embodiment, controlling the virtual-real fusion robot to perform corresponding actions based on the decision prediction results further includes: during the process of controlling the virtual-real fusion robot to perform corresponding actions based on the decision prediction results, acquiring observation information of the virtual-real fusion space again according to a preset period, and determining whether the reacquired observation information of the virtual-real fusion space has been updated; based on the updated observation information of the reacquired virtual-real fusion space, making decision predictions for the virtual-real fusion space again using the updated observation information, and optimizing the decision prediction results.

[0048] It should be noted that during the task execution phase of the virtual-real fusion robot, real-time feedback on the task execution process is obtained to facilitate further optimization of virtual and physical task allocation based on instantaneous situations during execution. Furthermore, since the scene does not change significantly or the scene changes do not affect task redistribution during the task execution phase, a preset period is set to periodically acquire new observation data to confirm whether the decision prediction results need optimization.

[0049] In another alternative embodiment, refer to Figure 3 After controlling the virtual-real fusion robot to perform corresponding actions based on the decision prediction results, the process includes: based on the completion of the virtual-real fusion robot's corresponding actions, reacquiring the observation information of the virtual-real fusion space, and determining whether the reacquiring observation information of the virtual-real fusion space has been updated; based on the updated observation information of the reacquiring virtual-real fusion space, using the updated observation information, re-making decisions and predictions for the virtual-real fusion space, and optimizing the decision prediction results.

[0050] It should be noted that since task execution has an expected outcome, if the virtual-real fusion robot fails to observe the expected outcome within the expected completion time, the execution is considered a failure and needs to be replanned. Furthermore, when re-making predictions, both the task execution strategies of each embodied agent and the overall task allocation of the virtual-real fusion robot in virtual and physical spaces are considered. For example, if the initial plan is to bring a glass of water to a user in the physical space, but the virtual-real fusion robot drops the glass, it will be impossible to bring the glass to the user after a period of time, resulting in task failure. When re-making predictions, based on the newly acquired observation information, a different beverage might be substituted, and the virtual or physical embodied agent performing the task might be replaced with a more suitable embodied agent of a different form for the task.

[0051] In summary, the embodiments of the present invention make decisions and predictions on the virtual-real fusion space by acquiring observation information of the virtual-real fusion space, so as to allocate tasks to the virtual space and the physical space, obtain decision prediction results, and control the virtual-real fusion robot to perform corresponding actions according to the decision prediction results, so as to realize the collaborative control of virtual and real integration, support the practical application of virtual-real fusion robot, realize the provision of services in the virtual-real fusion space, and form a closed loop of physical and virtual spaces.

[0052] The control system of the virtual-real fusion robot provided by the present invention will be described below. The control system of the virtual-real fusion robot described below can be referred to in correspondence with the control method of the virtual-real fusion robot described above.

[0053] Figure 4 A schematic diagram of the control system for a virtual-real fusion robot is shown. The system includes: Information acquisition module 41 acquires observation information, which is obtained by using the virtual-real fusion robot to observe the virtual-real fusion space, which is based on the fusion of virtual space and physical space; The decision prediction module 42 performs decision prediction on the virtual-real fusion space based on observation information and obtains the decision prediction results; The decision control module 43 controls the virtual-real fusion robot to perform corresponding actions based on the decision prediction results.

[0054] In this embodiment, the observation information includes virtual space observation information and physical space observation information, and the virtual-real fusion robot includes a physical space embodied intelligent agent and a virtual space embodied intelligent agent. Accordingly, the information acquisition module 41 includes: a first information acquisition unit, which uses the physical space embodied intelligent agent to observe the physical space; and a second information acquisition unit, which uses the virtual space embodied intelligent agent to observe the virtual space.

[0055] In one alternative embodiment, the virtual-real fusion robot is driven by artificial intelligence (AI model). Accordingly, the system also includes a configuration module, in which the AI ​​model reads a configuration file and obtains controllable embodied forms and control methods from the configuration file before using the virtual-real fusion robot to observe the virtual-real fusion space.

[0056] In this embodiment, the decision prediction module 42 is used to: input observation information into the decision prediction model to obtain the decision prediction result output by the decision prediction model; wherein, the decision prediction model is trained based on observation training information and the decision labels corresponding to the observation training information; the decision prediction model is used to extract features from the input observation information and make decision predictions based on the extracted state features.

[0057] Furthermore, the decision prediction model includes: a feature extraction layer, which extracts features from the input observation information to obtain state features; an event classification layer, which classifies the state features into virtual space events and physical space events; and a decision prediction layer, which makes decision predictions based on the virtual space events and physical space events to obtain decision prediction results, which include physical space tasks and virtual space tasks.

[0058] Accordingly, the decision prediction module 42 includes: a feature extraction unit, which inputs the observation information into the feature extraction layer to extract features and obtains the state features output by the feature extraction layer; an event classification unit, which inputs the state features into the event classification layer to classify events and obtains the virtual space events and physical space events output by the event classification layer; and a decision prediction unit, which inputs the virtual space events and physical space events into the decision prediction layer to make a decision prediction and obtains the decision prediction result output by the decision prediction layer.

[0059] In an optional embodiment, the system further includes a training module for training the decision prediction model before inputting the observation information into the model. Specifically, the training module includes a training data acquisition unit for acquiring observation training information and decision labels corresponding to the observation training information; and a training unit for training the model to be trained by using the observation training information as input data for training and the decision labels corresponding to the observation training information as labels for training, thereby obtaining a decision prediction model for predicting decisions.

[0060] In this embodiment, the network to be trained can be an existing network built into the training module. This existing network typically includes a network structure, or it can be other networks specified by the user, such as the Transformer algorithm, LSTM network, or GPT-4. The network to be trained typically includes a feature extraction layer for extracting corresponding state features, an event classification layer for classifying events based on the state features, a decision prediction layer for making decisions about the classified events, and a loss function. According to a preset iteration rule, the observed training information or the observed training information after data augmentation is input into the model to be trained for training, resulting in a trained decision prediction model.

[0061] Furthermore, the training module also includes a data augmentation module, which, after acquiring the observed training information and the corresponding decision labels, uses data augmentation strategies to augment the observed training information. Specifically, the data augmentation strategies include at least one of flipping, rotating, cropping, deforming, and scaling; and / or, the data augmentation strategies include at least one of noise, blurring, color transformation, erasing, and filling.

[0062] In addition, the training unit includes: a training result acquisition subunit, which inputs the observed training information into the model to be trained and obtains the decision training result output by the model to be trained; and a training subunit, which constructs a loss function based on the decision training result and the decision label corresponding to the observed training information, and terminates the training based on the convergence of the loss function.

[0063] In this embodiment, the decision control module 43 includes: a control unit, which, based on the decision prediction result and virtual-real linkage control, controls the physical space embodied intelligent agent and the virtual space embodied intelligent agent to perform corresponding tasks in their respective spaces, thereby realizing the collaborative control of embodied intelligent agents in different spaces to perform tasks in the virtual-real fusion space.

[0064] In one alternative embodiment, reference Figure 5 When controlling the virtual-real fusion robot to perform corresponding actions based on the decision prediction results, the virtual embodied intelligent agent controller can be used to control the virtual space embodied intelligent agent to perform corresponding tasks, and the physical embodied intelligent agent controller can be used to control the physical space embodied intelligent agent to perform corresponding tasks.

[0065] Furthermore, the virtual embodied intelligent agent controller and the physical embodied intelligent agent controller are connected to different types of terminal devices. The terminal devices connected to the virtual embodied intelligent agent controller may include VR head-mounted displays, AR head-mounted displays, projectors, and web clients, etc., to present virtual robot images (i.e., virtual embodiments) to users; the terminal devices connected to the physical embodied intelligent agent controller may include humanoid robots, robotic arms, intelligent vehicles, and drones, etc., to work together with the virtual space embodied intelligent agent to form a virtual-real integrated intelligent agent.

[0066] In an optional embodiment, the decision control module 43 further includes: a decision optimization unit, which, during the process of controlling the virtual-real fusion robot to perform corresponding actions according to the decision prediction results, reacquires the observation information of the virtual-real fusion space according to a preset period, and determines whether the reacquired observation information of the virtual-real fusion space is updated; based on the updated observation information of the reacquired virtual-real fusion space, uses the updated observation information to re-make decision predictions for the virtual-real fusion space, and optimizes the decision prediction results.

[0067] In another optional embodiment, the decision optimization unit is further configured to: after controlling the virtual-real fusion robot to perform corresponding actions according to the decision prediction results, based on the completion of the virtual-real fusion robot performing the corresponding actions, reacquire the observation information of the virtual-real fusion space, and determine whether the reacquired observation information of the virtual-real fusion space is updated; based on the updated observation information of the reacquired virtual-real fusion space, use the updated observation information to re-make decision predictions for the virtual-real fusion space, and optimize the decision prediction results.

[0068] In summary, this embodiment of the invention uses a decision prediction module to perform decision prediction on the observation information of the virtual-real fusion space acquired by the information acquisition module within the virtual-real fusion space. This allows for task allocation between the virtual and physical spaces, resulting in decision prediction results. The decision control module then controls the virtual-real fusion robot to perform corresponding actions based on these results. This achieves collaborative control of the virtual and real spaces, supports the practical application of the virtual-real fusion robot, and enables the provision of services within the virtual-real fusion space, forming a closed loop between the physical and virtual spaces.

[0069] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640. The processor 610, communication interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a control method for the virtual-real fusion robot. This method includes: acquiring observation information, which is obtained by observing a virtual-real fusion space using the virtual-real fusion robot; performing decision prediction on the virtual-real fusion space based on the observation information to obtain a decision prediction result; and controlling the virtual-real fusion robot to perform corresponding actions based on the decision prediction result.

[0070] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the control method for the virtual-real fusion robot provided by the above methods. The method includes: acquiring observation information, which is obtained by observing the virtual-real fusion space using the virtual-real fusion robot, the virtual-real fusion space being obtained based on the fusion of virtual space and physical space; making decision predictions on the virtual-real fusion space based on the observation information to obtain decision prediction results; and controlling the virtual-real fusion robot to perform corresponding actions based on the decision prediction results.

[0072] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a control method for a virtual-real fusion robot provided by the above methods. The method includes: acquiring observation information, the observation information being obtained by observing a virtual-real fusion space using the virtual-real fusion robot, the virtual-real fusion space being obtained based on the fusion of virtual space and physical space; making a decision prediction on the virtual-real fusion space based on the observation information, and obtaining a decision prediction result; and controlling the virtual-real fusion robot to perform corresponding actions based on the decision prediction result.

[0073] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A control method of a cyber-physical fusion robot, characterized by, The method comprises the following steps: acquiring observation information, the observation information being obtained by observing a virtual-real fusion space by a virtual-real fusion robot, the virtual-real fusion space being obtained by fusing a virtual space and a physical space; performing decision prediction on the virtual-real fusion space according to the observation information to obtain a decision prediction result; controlling the virtual-real fusion robot to perform a corresponding action according to the decision prediction result. 2.The control method of the real-fusion robot according to claim 1, characterized in that, The method of performing decision prediction on the virtual-real fusion space according to the observation information to obtain a decision prediction result comprises the following steps: inputting the observation information into a decision prediction model to obtain a decision prediction result output by the decision prediction model; wherein the decision prediction model is trained based on observation training information and a decision label corresponding to the observation training information; the decision prediction model is used for feature extraction on input observation information and decision prediction based on the extracted state features. 3.The control method of the real-fusion robot according to claim 2, characterized in that, The decision prediction model comprises: a feature extraction layer, which extracts features from input observation information to obtain state features; an event classification layer, which classifies events based on the state features to obtain virtual space events and physical space events; a decision prediction layer, which performs decision prediction based on the virtual space events and the physical space events to obtain a decision prediction result, the decision prediction result comprising a physical space task and a virtual space task. 4.The control method of the real-fusion robot according to claim 2, characterized in that, Before inputting the observation information into the decision prediction model, the method comprises the following steps: acquiring observation training information and a decision label corresponding to the observation training information; training a to-be-trained model by taking the observation training information as input data for training and taking the decision label corresponding to the observation training information as a label for training to obtain a decision prediction model for predicting decisions. 5.The control method of the real-fusion robot according to claim 2, characterized in that, The decision prediction model adopts a deep learning Transformer model based on a self-attention mechanism; or The decision prediction model adopts a long short-term memory (LSTM) network; or The decision prediction model adopts a large multi-modal GPT-4 model. 6.The control method of the real-fusion robot according to claim 1, characterized in that, The method of controlling the virtual-real fusion robot to perform a corresponding action according to the decision prediction result comprises the following steps: controlling the virtual-real fusion robot to perform a corresponding action according to the decision prediction result; during the process of controlling the virtual-real fusion robot to perform a corresponding action according to the decision prediction result, reacquiring observation information of the virtual-real fusion space according to a preset period and determining whether the reacquired observation information of the virtual-real fusion space is updated; based on the reacquired observation information update of the virtual-real fusion space, reperforming decision prediction on the virtual-real fusion space by using the updated observation information to optimize the decision prediction result. 7.The control method of the real-fusion robot according to claim 1, characterized in that, After controlling the virtual-real fusion robot to perform a corresponding action according to the decision prediction result, the method comprises the following steps: based on the completion of the virtual-real fusion robot performing a corresponding action, reacquiring observation information of the virtual-real fusion space and determining whether the reacquired observation information of the virtual-real fusion space is updated; based on the reacquired observation information update of the virtual-real fusion space, reperforming decision prediction on the virtual-real fusion space by using the updated observation information to optimize the decision prediction result.

8. A control system of a cyber-physical fusion robot, characterized in that, The method comprises the following steps: An information acquisition module acquires observation information obtained by observing a virtual-real fusion space using a virtual-real fusion robot, the virtual-real fusion space being obtained by fusing a virtual space and a physical space; A decision prediction module performs decision prediction on the virtual-real fusion space according to the observation information to obtain a decision prediction result; and a decision control module controls the virtual-real fusion robot to perform a corresponding action according to the decision prediction result.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the control method of the virtual-real fusion robot according to any one of claims 1 to 7 when executing the program.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the control method of the virtual-real fusion robot according to any one of claims 1 to 7 when executed by the processor.

Images