Robot gesture enhancement visualization method and device based on projection interaction

Abstract

The invention provides a robot gesture enhancement visualization method and device based on projection interaction, and relates to the technical field of robot interaction.The robot gesture enhancement visualization method comprises the steps that multi-modal action data in the process that a robot executes a current action is collected; through a pre-trained action prediction model, an action intention corresponding to the current action executed by the robot is predicted based on the multi-modal action data, and the action intention is used for representing the target orientation and the target moving position of the current action; a mark image corresponding to the action intention is generated, the mark image is projected to a ground display area through projection equipment, and the distance between the ground display area and the area where the robot is located meets the preset distance requirement. According to the method, the problems of poor real-time performance and slow response in the robot gesture interaction process can be effectively solved.

Description

Technical Field

[0001] This invention relates to the field of robot interaction technology, and in particular to a method and apparatus for enhancing the visualization of robot gestures based on projection interaction. Background Technology

[0002] Robot interaction technology bridges the communication gap between humans and robots by enabling robots to understand and respond to human needs. However, when robots provide information using gestures, they often face challenges such as insufficient real-time performance and slow movement, significantly impacting the human-robot interaction experience. Typically, the speed at which a robot executes gesture commands is limited by its mechanical structure and motion control system. This is especially true for humanoid robots, where limitations in joint structure and drive systems often result in lengthy adjustments and execution times for gestures. This slow response speed, particularly in dynamic interaction scenarios, can lead to robots failing to respond to user needs instantly, affecting the smoothness and naturalness of the interaction. This not only impacts interaction efficiency but also reduces the user experience. In practical applications, this limitation is particularly pronounced in scenarios requiring efficient and smooth interaction. Therefore, improving the accuracy and response speed of gesture recognition and optimizing the smoothness of robot movements have become core issues that urgently need to be addressed in current robot interaction technology. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method and device for enhancing the visualization of robot gestures based on projection interaction, which can effectively solve the problems of poor real-time performance and slow response in robot gesture interaction.

[0004] In a first aspect, the present invention provides a robot gesture enhancement visualization method based on projection interaction. The method is applied to a visualization system, the visualization system including a projection device mounted on a robot, and the method includes: Collect multimodal motion data during the robot's current action execution process; By using a pre-trained motion prediction model, the robot's motion intention is predicted based on multimodal motion data. The motion intention is used to characterize the target orientation and target movement position of the current motion. Generate a marker image corresponding to the action intent, and use a projection device to project the marker image onto the ground display area. The distance between the ground display area and the area where the robot is located meets the preset distance requirements.

[0005] In one implementation, the visualization system further includes a color camera, a depth camera, and a depth sensor; the current action is an upper limb action; and it collects multimodal motion data during the robot's execution of the current action, including: Using color camera equipment and depth camera equipment, gesture image information and gesture depth information are collected respectively during the robot's upper limb movements; And / or, using depth sensors, acquire upper limb joint pose sequence data during the robot's upper limb movements.

[0006] In one implementation, a pre-trained motion prediction model is used to predict the robot's intention to perform the current action based on multimodal motion data, including: By using a pre-trained motion prediction model, the robot can predict the intention of performing upper limb movements based on one or more of the following: gesture image information, gesture depth information, and upper limb joint pose sequence data.

[0007] In one implementation, generating a marker image corresponding to an action intent includes: Based on the pre-configured mapping relationship between action intentions and marker images, a marker image corresponding to the action intention is generated; wherein, the marker image adopts an arrow image, and the arrow displayed in the arrow image points to the target orientation used to represent the current action.

[0008] In one embodiment, projecting a sign image onto a ground display area using a projection device includes: Geometric calibration of the projection equipment is performed to establish a mapping relationship between pixel coordinates in the sign image and actual direct coordinates. Based on this mapping relationship, the sign image is projected onto the ground display area.

[0009] In one implementation, the size of the ground display area is larger than the size of the sign image; projecting the sign image onto the ground display area based on a mapping relationship includes: Based on the target's movement position in the action intent, determine the center offset of the marker image relative to the ground display area; The mapping relationship is adjusted based on the center offset so that the sign image is projected onto the ground display area according to the adjusted mapping relationship.

[0010] Secondly, the present invention also provides a robot gesture enhancement visualization device based on projection interaction. The device is applied to a visualization system, the visualization system including a projection device mounted on a robot, and the device includes: The data acquisition module is used to collect multimodal motion data of the robot during the current action. The intent prediction module is used to predict the action intent corresponding to the robot's current action based on multimodal action data using a pre-trained action prediction model. The action intent is used to characterize the target orientation and target movement position of the current action. The projection module is used to generate a marker image corresponding to the action intention, and to project the marker image onto the ground display area using a projection device. The distance between the ground display area and the area where the robot is located meets the preset distance requirements.

[0011] In one embodiment, the visualization system further includes a color camera device, a depth camera device, and a depth sensor, wherein the current action is an upper limb action; the data acquisition module is specifically used for: Using color camera equipment and depth camera equipment, gesture image information and gesture depth information are collected respectively during the robot's upper limb movements; And / or, using depth sensors, acquire upper limb joint pose sequence data during the robot's upper limb movements.

[0012] Thirdly, the present invention also provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement any of the methods provided in the first aspect.

[0013] Fourthly, the present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement any of the methods provided in the first aspect.

[0014] This invention provides a robot gesture enhancement visualization method and device based on projection interaction, applied to a visualization system. The visualization system includes a projection device mounted on a robot. First, multimodal motion data of the robot during its current action is collected. Then, a pre-trained motion prediction model predicts the robot's action intent based on the multimodal motion data. The action intent represents the target orientation and target movement position of the current action. Finally, a marker image corresponding to the action intent is generated and projected onto a ground display area using the projection device. The distance between the ground display area and the robot's location meets a preset distance requirement. This method predicts the robot's action intent based on corresponding multimodal motion data while the robot is performing its current action, and displays the marker image corresponding to the action intent on the ground display area before the robot finishes performing the current action. This effectively solves the problems of poor real-time performance and slow response in robot gesture interaction.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a robot gesture enhancement visualization method based on projection interaction, provided in an embodiment of the present invention; Figure 2 A technical framework diagram of a robot gesture enhancement visualization method based on projection interaction provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the positional relationship between an object point, an imaging point, and a coordinate system, provided in an embodiment of the present invention. Figure 4 A schematic diagram illustrating the transformation between the world coordinate system and the camera coordinate system provided in an embodiment of the present invention; Figure 5 A diagram illustrating the transformation relationship between the camera coordinate system and the image coordinate system provided in an embodiment of the present invention; Figure 6 A diagram showing the relationship between points in the camera coordinate system and points in the image coordinate system, provided as an embodiment of the present invention; Figure 7 A diagram showing the relationship between an image plane coordinate system and a pixel coordinate system provided in an embodiment of the present invention; Figure 8 A schematic diagram of the structure of a robot gesture enhancement visualization device based on projection interaction provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Currently, improving the accuracy and response speed of gesture recognition and optimizing the smoothness of robot movements have become core issues that need to be addressed in current robot interaction technology. Based on this, this invention provides a robot gesture enhancement visualization method and device based on projection interaction, which can effectively solve the problems of poor real-time performance and slow response in robot gesture interaction.

[0021] To facilitate understanding of this embodiment, a detailed description of a robot gesture enhancement visualization method based on projection interaction disclosed in this embodiment of the invention will be provided first. This method is applied to a visualization system, which includes a projection device mounted on the robot. See [link to relevant documentation]. Figure 1 The diagram shows a flowchart of a robot gesture enhancement visualization method based on projection interaction. The method mainly includes the following steps S102 to S106: Step S102: Collect multimodal motion data during the robot's current action execution process.

[0022] In one example, the visualization system also includes a color camera device, a depth camera device, and a depth sensor. The current action is an upper limb action. Based on this, the above devices are used to collect corresponding multimodal motion data when the robot performs the upper limb action. The multimodal motion data may include gesture image information, gesture depth information, and upper limb joint pose sequence data.

[0023] Step S104: Using a pre-trained motion prediction model, predict the robot's intention to perform the current action based on multimodal motion data.

[0024] Action intent is used to characterize the target orientation and target movement position of the current action. In one example, action intent can be predicted using a first action prediction model based on gesture image information and gesture depth information; or using a second action prediction model based on upper limb joint pose sequence data; or by combining the action intents output by the first action prediction model and the action intents output by the second action prediction model; or by using a third action prediction model based on gesture image information, gesture depth information, and upper limb joint pose sequence data.

[0025] Step S106: Generate a sign image corresponding to the action intent, and project the sign image onto the ground display area using a projection device.

[0026] The marker image is an arrow image, with the arrow pointing to the target orientation representing the current action. The distance between the ground display area and the robot's location meets a preset distance requirement, which can be a fixed distance threshold. For example, the ground display area can be the ground area near the robot's location or a distance range. In one example, a marker image corresponding to the action intention can be generated based on a preset mapping relationship between the action intention and the marker image. After geometric correction of the projection device, the marker image is projected onto the ground display area. The size of the ground display area can be the same as or larger than the size of the marker image. Furthermore, if the size of the ground display area is larger than the size of the marker image, a first effect (e.g., white light) can be used to project the marker image onto the ground display area to distinguish the entire ground display area from other ground surfaces. A second effect can then be used to project the marker image, and the degree of offset between the center of the marker image and the center of the ground display area indicates the target movement position of the current action.

[0027] The robot gesture enhancement visualization method based on projection interaction provided in this invention predicts the robot's action intent based on corresponding multimodal action data when the robot is performing the current action, so as to display the symbol image corresponding to the action intent on the ground display area before the robot finishes performing the current action. This can effectively solve the problems of poor real-time performance and slow response in robot gesture interaction.

[0028] Based on the foregoing embodiments, this invention provides a specific implementation of a robot gesture enhancement visualization method based on projection interaction. See [link to implementation details]. Figure 2 The diagram illustrates a technical framework for a robot gesture enhancement visualization method based on projection interaction. It involves acquiring gesture image information using an RGB camera, acquiring gesture depth information using a depth camera, and performing robot motion prediction using an image classification algorithm-based action prediction model to predict the target's orientation and movement position, among other action intentions. Graphics processing software generates arrow graphics corresponding to these action intentions, which are then projected onto a ground display area using a projector. The target orientation can be categorized into four types: left, right, forward, and backward, and can be further subdivided; this embodiment of the invention does not impose such limitations.

[0029] Based on this, the present invention provides an implementation of the aforementioned step S102. Specifically, the visualization system further includes a color camera device, a depth camera device, and a depth sensor. Since the current action is an upper limb action, the color camera device and the depth camera device can be used to collect gesture image information and gesture depth information during the robot's upper limb action, respectively; and / or, the depth sensor can be used to collect upper limb joint pose sequence data during the robot's upper limb action. The upper limb joint pose sequence data can be a sequence of joint rotation angles within the interval from the start time of the robot's current action to the current time.

[0030] This invention further provides an implementation of step S104, namely, using a pre-trained motion prediction model to predict the robot's intention to perform an upper limb movement based on one or more of gesture image information, gesture depth information, and upper limb joint pose sequence data. Specifically, this can be achieved through the following methods: Method 1: Predict the action intent based on gesture image information and gesture depth information using a first action prediction model. In one example, the first action prediction model can be a classification model based on an image classification algorithm. By inputting the gesture image information and gesture depth information into the first action prediction model, the model can output the action intent, such as the target's orientation and the target's movement position.

[0031] Method 2: Predict the intended action based on upper limb joint pose sequence data using a second action prediction model. In one example, the second action prediction model can be a deep learning model. The joint sequence data from the start of the action to the current moment is input into the deep learning model, which can then predict the robot's final target orientation and target movement position at future moments.

[0032] Method 3: The final action intention is determined by combining the action intentions output by the first and second models. The first model predicts the action intention based on gesture image and depth information, while the second model predicts the action intention based on upper limb joint pose sequence data. In one example, the action intentions output by the two models are combined to determine the final action intention. Figure 1 In cases where the two models output the same action intention, the final action intention can be obtained directly. In cases where the action intentions output by the two models are inconsistent, the weight of the second action prediction model is higher than that of the first action prediction model, that is, the action intention output by the second action prediction model is taken as the final action intention.

[0033] Method 4: Using a third motion prediction model, the intention of a movement is predicted based on gesture image information, gesture depth information, and upper limb joint pose sequence data. In one example, gesture image information, gesture depth information, and upper limb joint pose sequence data are input into the third motion prediction model. The third motion prediction model performs the following operations on the three types of input data: a feature pyramid network is applied to the gesture image information to extract multi-scale spatial features; multi-resolution point cloud processing is performed on the gesture depth information to obtain point cloud features; spatiotemporal attention processing is performed on the upper limb joint pose sequence data to extract spatiotemporal features; and the target orientation and target movement position are determined based on the multi-scale spatial features, point cloud features, and spatiotemporal features.

[0034] The embodiments of the present invention provide specific implementation methods for the aforementioned step S106.

[0035] In one example, the process of generating the marker image is as follows: Based on a pre-configured mapping relationship between the action intent and the marker image, a marker image corresponding to the action intent is generated; wherein the marker image is an arrow image, and the arrow displayed in the arrow image points to the target orientation used to represent the current action. In one implementation, graphics processing software (such as OpenGL, Unity3D, etc.) can be used to render left, right, forward, and backward arrows from the target orientation in the gesture action intent to obtain arrow images.

[0036] In one example, placing a projector on the robot's shoulder displays a bright area on the ground. This area moves with the robot, and after coordinate transformation and relative position calibration of the camera and projector, 3D modeling is used to map the arrow graphic onto that bright area on the ground before the robot's action is completed. Specifically, the projection process is as follows: (1) Perform geometric calibration on the projection device to establish the mapping relationship between the pixel coordinates in the mark image and the real direct coordinates.

[0037] First, the definitions of the four coordinate systems will be explained, such as... Figure 3 The diagram shows the positional relationship between an object point, an imaging point, and a coordinate system. World coordinate system: Normally, the world coordinate system corresponds to the camera coordinate system. However, to better describe the camera's position in the physical world, the definition of the world coordinate system is not fixed and is set according to different requirements. In three-dimensional space, the coordinates of a point P on the surface of a color image are defined as Pw(Xw, Yw, Zw). The unit is meters (m).

[0038] Camera coordinate system: The origin Oc of the camera coordinate system is the center of the camera's geometric optics. A rectangular coordinate system is established with the optical axis as the z-axis. The coordinates of a point P on the color image surface in the camera are defined as Pc(Xc, Yc, Zc). The unit is meters (m).

[0039] Image coordinate system: The image coordinate system is a two-dimensional coordinate system. The plane formed by its X-axis and Y-axis is the imaging plane, and the origin of the image coordinate system is the midpoint of the imaging plane. The coordinates of a point P on the surface of a color image in the image coordinate system are defined as P(X, Y). The unit is mm.

[0040] Pixel coordinate system: The pixel coordinate system has its origin at the top left or bottom left corner of the imaging plane. The coordinates of point P in the pixel coordinate system are defined as P(u, v). The unit is pixels.

[0041] Based on this, the embodiments of the present invention further explain the relationship and transformation between coordinates: (a) Transformation between world coordinate system and camera coordinate system: The transformation from the world coordinate system to the camera coordinate system only involves rotation and translation. It can be represented by a rotation matrix R and a translation variable T. These R and T are the camera's extrinsic parameter matrices, such as... Figure 4 The diagram illustrates a transformation between the world coordinate system and the camera coordinate system. The rotation operation can be decomposed into rotations of the three world coordinate axes (XW, YW, ZW) by angles α, β, and γ, respectively. The total rotation matrix can be expressed as the product of these three rotation matrices. ; Translation operations can be represented by vector addition or subtraction t. Therefore, the transformation relationship from point Pw in the world coordinate system to point Pc in the camera coordinate system can be expressed as: ; Based on the above description, the homogeneous coordinate transformation matrix can be used to represent it as follows: .

[0042] (b) For the transformation between the camera coordinate system and the image coordinate system, see [link to documentation]. Figure 5 The diagram shown illustrates the transformation relationship between the camera coordinate system and the image coordinate system: Figure 5 This diagram illustrates the transformation relationship between the camera coordinate system and the image coordinate system. Figure 5 It is known that the physical imaging plane is parallel to the plane of the camera coordinate system, and is separated by a focal length f. The coordinates of a point Pc(Xc, Yc, Zc) in the camera coordinate system projected onto a point P in the image coordinate system are P(x, y). For example... Figure 6 As shown, Figure 6 This diagram illustrates the relationship between points in a camera coordinate system and points in an image coordinate system. Based on the similarity relationship of triangles, specifically the similarity between triangle OCOP and triangle OCPCN, the following equation can be derived: ; You will then receive: ; This can be represented by matrix transformations as follows: ; The above only shows the transformation relationship between the camera coordinate system and the image coordinate system. The next step is to transform it to the pixel coordinate system.

[0043] (c) Relationship between image coordinate system and pixel coordinate system: From the definitions of these two coordinate systems, we know that the image coordinate system and the pixel coordinate system lie on the same plane. The origin O of the image coordinate system has coordinates O(u0, v0) in the pixel coordinate system. Based on the above relationship, the relationship between point P in the image coordinate system and the pixel coordinate system is as follows: Figure 7 As shown, Figure 7 A diagram showing the relationship between an image plane coordinate system and a pixel coordinate system: Let the coordinates of point P(x,y) in the image coordinate system correspond to P(u,v) in the pixel coordinate system. The transformation formula is as follows: ; Where dx and dy represent the width of one pixel in the x-axis and y-axis directions, respectively, with the unit being mm / pixel.

[0044] Represented using homogeneous coordinates and matrices: ; Multiplying both sides of the above equation by Zc yields: ; Substituting the transformation matrix between the camera coordinate system and the image coordinate system into the above equation, we get: ; The product of the first and second matrices on the right side of the above equation is the camera's intrinsic parameter matrix. Multiplying them together yields: ; Through the above coordinate system relationships and transformations, the transformation between the world coordinate system and the pixel coordinate system can be obtained. Substituting the transformation matrix between the world coordinate system and the camera coordinate system into the above formula, this transformation relationship can be constructed: ; Where fx = f / dx, fy = f / dy.

[0045] (2) Project the sign image onto the ground display area based on the mapping relationship. Optionally, the size of the ground display area is the same as the size of the sign image, and the sign image is projected completely onto the ground display area. Optionally, the size of the ground display area is larger than the size of the sign image. The center offset of the sign image relative to the ground display area is determined according to the target movement position in the action intention. The mapping relationship is adjusted based on the center offset so that the sign image is projected onto the ground display area according to the adjusted mapping relationship.

[0046] For example, since the size of the ground display area is larger than the size of the marker image, the movable distance of the marker image within the ground display area can be determined. By combining the distance between the target movement position in the action intention and the current position of the robot gesture, the center offset of the marker image relative to the ground display area can be determined. The larger the offset, the farther the target movement distance.

[0047] In summary, this embodiment of the invention, based on the directionality and action type of the robot's gestures, renders directional arrows (e.g., left, right, forward, or backward arrows) in a specific area on the ground before the robot completes its action. These arrows represent the robot's upcoming action or intention, intuitively guiding the user to understand the robot's intent and direction of action. This method effectively solves the problems of poor real-time performance and slow response during robot gesture interaction. The projected directional arrows provide clear visual cues, improving the interaction efficiency and user experience between the robot and the user. Users no longer need to wait for the robot to complete its action before providing feedback; instead, they can obtain the expected behavior of the robot in advance through the arrow cues on the ground, thus achieving more efficient interaction.

[0048] Based on the foregoing embodiments, this invention provides a robot gesture enhancement visualization device based on projection interaction. This device is applied to a visualization system, which includes a projection device mounted on a robot. (See also...) Figure 8 The diagram shows a structural schematic of a robot gesture enhancement visualization device based on projection interaction. The device mainly includes the following parts: The data acquisition module 802 is used to collect multimodal motion data of the robot during the current action process; The intent prediction module 804 is used to predict the action intent corresponding to the robot's current action based on multimodal action data through a pre-trained action prediction model. The action intent is used to characterize the target orientation and target movement position of the current action. The projection module 806 is used to generate a marker image corresponding to the action intention, and to project the marker image onto the ground display area using a projection device. The distance between the ground display area and the area where the robot is located meets the preset distance requirement.

[0049] The robot gesture enhancement visualization device based on projection interaction provided in this invention predicts the robot's action intent based on corresponding multimodal action data when the robot is performing the current action, so as to display the symbol image corresponding to the action intent on the ground display area before the robot finishes performing the current action. This can effectively solve the problems of poor real-time performance and slow response in robot gesture interaction.

[0050] In one embodiment, the visualization system further includes a color camera device, a depth camera device, and a depth sensor; the current action is an upper limb action; data acquisition module 802: Using color camera equipment and depth camera equipment, gesture image information and gesture depth information are collected respectively during the robot's upper limb movements; And / or, using depth sensors, acquire upper limb joint pose sequence data during the robot's upper limb movements.

[0051] In one implementation, the intent prediction module 804 is specifically used for: By using a pre-trained motion prediction model, the robot can predict the intention of performing upper limb movements based on one or more of the following: gesture image information, gesture depth information, and upper limb joint pose sequence data.

[0052] In one embodiment, the projection module 806 is specifically used for: Based on the pre-configured mapping relationship between action intentions and marker images, a marker image corresponding to the action intention is generated; wherein, the marker image adopts an arrow image, and the arrow displayed in the arrow image points to the target orientation used to represent the current action.

[0053] In one embodiment, the projection module 806 is specifically used for: Geometric calibration of the projection equipment is performed to establish a mapping relationship between pixel coordinates in the sign image and actual direct coordinates. Based on this mapping relationship, the sign image is projected onto the ground display area.

[0054] In one embodiment, the size of the ground display area is larger than the size of the sign image; the projection module 806 is specifically used for: Based on the target's movement position in the action intent, determine the center offset of the marker image relative to the ground display area; The mapping relationship is adjusted based on the center offset so that the sign image is projected onto the ground display area according to the adjusted mapping relationship.

[0055] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0056] This invention provides an electronic device, specifically, the electronic device includes a processor and a memory; the memory stores a computer program, which, when run by the processor, executes the method described in any of the above embodiments.

[0057] Figure 9This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes: a processor 90, a memory 91, a bus 92, and a communication interface 93. The processor 90, the communication interface 93, and the memory 91 are connected through the bus 92. The processor 90 is used to execute executable modules, such as computer programs, stored in the memory 91.

[0058] The memory 91 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 93 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0059] Bus 92 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0060] The memory 91 is used to store programs. After receiving an execution instruction, the processor 90 executes the programs. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 90 or implemented by the processor 90.

[0061] The processor 90 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 90 or by instructions in software form. The processor 90 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 91. Processor 90 reads the information in memory 91 and, in conjunction with its hardware, completes the steps of the above method.

[0062] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for robot gesture augmented visualization based on projection interaction, characterized in that, The method is applied to a visualization system, the visualization system comprising a projection device arranged on a robot, and the method comprises: acquiring multi-modal action data of the robot during execution of a current action; predicting, by a pre-trained action prediction model, an action intention corresponding to the current action of the robot based on the multi-modal action data, the action intention being used to represent a target orientation and a target movement position of the current action; generating a landmark image corresponding to the action intention, and projecting the landmark image to a ground display area by the projection device, a distance between the ground display area and a region where the robot is located satisfying a preset distance requirement.

2. The method of claim 1, wherein, The visualization system further comprises a color camera device, a depth camera device and a depth sensor, and the current action is an upper limb action; acquiring multi-modal action data of the robot during execution of a current action, comprising: acquiring gesture image information and gesture depth information of the robot during execution of the upper limb action by the color camera device and the depth camera device, respectively; and / or, acquiring upper limb joint pose sequence data of the robot during execution of the upper limb action by the depth sensor.

3. The method of claim 2, wherein, predicting, by a pre-trained action prediction model, an action intention corresponding to the current action of the robot based on the multi-modal action data, comprising: predicting, by the pre-trained action prediction model, the action intention corresponding to the upper limb action of the robot based on one or more of the gesture image information, the gesture depth information and the upper limb joint pose sequence data.

4. The method of claim 1, wherein, generating a landmark image corresponding to the action intention, comprising: generating the landmark image corresponding to the action intention according to a pre-configured mapping relationship between action intentions and landmark images; wherein the landmark image adopts an arrow image, and an arrow displayed in the arrow image points to a target orientation representing the current action.

5. The method of claim 1, wherein, projecting the landmark image to a ground display area by the projection device, comprising: geometrically calibrating the projection device to establish a mapping relationship between pixel coordinates in the landmark image and real direct coordinates, and projecting the landmark image to the ground display area based on the mapping relationship.

6. The method of claim 5, wherein, The size of the ground display area is greater than the size of the landmark image. projecting the landmark image to the ground display area based on the mapping relationship, comprising: determining a center offset of the landmark image relative to the ground display area according to the target movement position in the action intention; adjusting the mapping relationship based on the center offset, so as to project the landmark image to the ground display area according to the adjusted mapping relationship.

7. A robot gesture augmented visualization device based on projection interaction, characterized in that, The device is applied to a visualization system, the visualization system comprising a projection device arranged on a robot, and the device comprises: a data acquisition module, configured to acquire multi-modal action data of the robot during execution of a current action; An intention prediction module is configured to predict, by a pre-trained action prediction model, an action intention corresponding to the current action of the robot based on the multi-modal action data, the action intention being used to represent a target orientation and a target moving position of the current action. A projection module is configured to generate a landmark image corresponding to the action intention, and project the landmark image to a ground display area by using the projection device, a distance between the ground display area and an area where the robot is located satisfying a preset distance requirement.

8. The robot gesture-enhanced visualization device based on projected interaction of claim 7, wherein, The visualization system further includes a color camera device, a depth camera device, and a depth sensor, and the current action is an upper limb action. The data acquisition module is specifically configured to: acquire gesture image information and gesture depth information in a process in which the robot performs the upper limb action by using the color camera device and the depth camera device, respectively; and / or acquire upper limb joint pose sequence data in the process in which the robot performs the upper limb action by using the depth sensor.

9. An electronic device, comprising: The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the method in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the method in any one of claims 1 to 6.

Images