Target virtual object control method and electronic equipment

By identifying and correlating the user's gestures and operations of physical objects, using cameras and inertial sensors to achieve synchronous operation of virtual objects and physical objects, the problem of inaccurate manipulation of virtual objects is solved and the realism and accuracy of operations is improved.

CN119088207APending Publication Date: 2024-12-06LENOVO (BEIJING) LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410954064.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the prior art, when a user manipulates a virtual object, it is difficult for the user to accurately control the rotation and movement of the virtual object due to the lack of grip and touch and space operation, resulting in the actual operation not meeting expectations.

Method used

By identifying the user's gesture actions with the predetermined actions of the physical object and establishing a relationship under the satisfaction of time conditions, the user's operation behavior of the physical object drives the target virtual object to move accordingly, and using the camera and inertial sensor to monitor and collect data in real time. Implement mapping and synchronization operations between virtual objects and physical objects.

Benefits of technology

The realism and accuracy of the user's manipulation of virtual objects is improved, so that the target virtual object can move with the movement of the physical object, conforming to the user's expected operations, and simplifying the operation mode.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088207A_ABST
    Figure CN119088207A_ABST
Patent Text Reader

Abstract

The invention provides a target virtual object control method and electronic equipment, and the method comprises the steps: responding to a target virtual object in a target interaction interface, and recognizing a gesture action of a user; when the gesture action is a preset action formed between the hand of the user and the entity object and meets a preset condition in time, associating the target virtual object with the entity object; and driving the target virtual object according to the association result and the operation behavior of the user on the entity object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer virtual reality technology, and in particular to a target virtual object control method and electronic equipment. Background Art

[0002] Manipulating 3D objects in space is a basic task. In the naked-eye 3D red dot project, one of the tasks is to demonstrate the user's ability to manipulate models with gestures. The existing solution is to rotate the target virtual object by pinching the hands in the air. However, due to the remote operation and the lack of gripping touch, one can only rely on spatial imagination to construct a rotation anchor point in the mind, resulting in the actual rotation position not meeting one's expectations. Summary of the invention

[0003] The present disclosure provides a target virtual object control method and electronic device to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a target virtual object control method is provided, the method comprising:

[0005] In response to the target virtual object existing in the target interactive interface, recognizing a user's gesture action;

[0006] When the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies a predetermined time condition, associating the target virtual object with the physical object;

[0007] The target virtual object is driven according to the association result and the user's operation behavior on the physical object.

[0008] In one possible implementation, the identifying the user's gesture action includes at least one of the following:

[0009] Analyze the motion information between the user's hand and the physical object in the acquired video data to determine the user's gesture action; the video data is collected by a camera;

[0010] The position and posture change data of the user's hand is obtained, and the user's gesture action is determined based on the position and posture change data; the position and posture change data is collected by an inertial sensor.

[0011] In one possible implementation, the predetermined action includes at least one of the following:

[0012] Grab, hold, lift, slide, dangle, push and strike;

[0013] The predetermined time condition is met, including at least one of the following:

[0014] The predetermined action recognition results in a plurality of consecutive video frame images within a preset time are the same;

[0015] The inertial signal obtained within the preset time remains stable.

[0016] In one possible implementation manner, associating the target virtual object with the physical object includes:

[0017] Respectively obtaining first pose data of the physical object and second pose data of the target virtual object;

[0018] Establishing a mapping relationship between a physical object and a target virtual object based on the first posture data and the second posture data;

[0019] Based on the mapping relationship, the target virtual object is mapped as an associated object of a physical object in a virtual environment.

[0020] In one possible implementation manner, driving the target virtual object according to the association result and the user's operation behavior on the physical object includes:

[0021] Obtaining a motion offset caused by a user's operation on the physical object;

[0022] Based on the association result, the target virtual object is driven to move and change accordingly according to the motion offset.

[0023] In one possible implementation, the user's operation behavior on the physical object includes at least one of the following:

[0024] Rotate and slide;

[0025] The motion offset includes at least one of the following:

[0026] Three-axis rotation angle information of user gesture;

[0027] The three-axis rotation angle information of the physical object;

[0028] The rotation angle change information of the target virtual object relative to the initial moment.

[0029] In one possible implementation manner, the user's operation behavior on the physical object further includes:

[0030] Touch the inertial sensor; to generate a preset instruction, wherein the preset instruction is used to take the current moment as the starting moment and determine the rotation angle of the current target virtual object as the initial rotation angle.

[0031] In one possible implementation, after the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies a predetermined time condition, the method further includes:

[0032] The user is prompted to adjust the holding posture between the hand and the physical object, and the target virtual object is associated with the adjusted physical object.

[0033] In one possible implementation manner, after driving the target virtual object according to the association result and the user's operation behavior on the physical object, the method further includes:

[0034] When no contact point is detected between the user and the physical object, ending the control of the target virtual object; or

[0035] When it is detected that the displacement value of the user's hand in space exceeds a preset threshold, the control of the target virtual object is terminated.

[0036] According to a second aspect of the present disclosure, a target virtual object control device is provided, the device comprising:

[0037] A response module, used for identifying a user's gesture action in response to the presence of a target virtual object in the target interactive interface;

[0038] an associating module, configured to associate the target virtual object with the physical object when the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies a predetermined time condition;

[0039] The driving module is used to drive the target virtual object according to the association result and the user's operation behavior on the physical object.

[0040] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0041] at least one processor; and

[0042] a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0044] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0045] The target virtual object control method and electronic device disclosed in the present invention, the electronic device of the present application presents the target virtual object through a target interaction interface, and simultaneously monitors and identifies gesture actions existing in the target interaction interface in real time. When it is detected that the gesture action is a predetermined action between the user's hand and the physical object, the target virtual object is associated with the physical object, so that the user's operation action on the physical object can be applied to the target virtual object, so that the target virtual object moves with the movement of the physical object.

[0046] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0048] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0049] Figure 1 The schematic diagram shows the implementation process of the target virtual object control method of the embodiment of the present disclosure. Figure 1 ;

[0050] Figure 2 The schematic diagram shows the implementation process of the target virtual object control method of the embodiment of the present disclosure. Figure 2 ;

[0051] Figure 3 The schematic diagram shows the implementation process of the target virtual object control method of the embodiment of the present disclosure. Figure 3 ;

[0052] Figure 4 A schematic diagram showing the structure of a target virtual object control device according to an embodiment of the present disclosure is shown;

[0053] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0054] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0055] The following describes a target virtual object control method provided by the present application in conjunction with the accompanying drawings, the method comprising:

[0056] S101, in response to the presence of a target virtual object in a target interactive interface, recognizing a user's gesture action;

[0057] It is understandable that the embodiment of the present application first describes the electronic device used to implement the target virtual object control method. The electronic device provided in the embodiment of the present application may be an intelligent terminal or a server. For example, the intelligent terminal may be a laptop, a tablet computer, a desktop computer, a smart phone, a dedicated messaging device, a portable gaming device, a smart speaker, a smart watch, etc., which is not limited in the present application. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution network (CDN, Content Delivery Network) services, and basic cloud computing services such as big data and artificial intelligence platforms.

[0058] It is understandable that when the electronic device recognizes the user's gesture, the user's gesture is within the target interaction interface, that is, the electronic device only recognizes the gesture within the target interaction interface to avoid wasting resources. In this application, the electronic device collects gesture information within the target interaction interface in real time to avoid omissions when gestures occur. Among them, the target virtual object refers to a virtual object in a virtual reality scene, and the gesture is any movement of the user's hand after the user's hand is detected.

[0059] Take the electronic device as a computer as an example; for example, when the computer is turned on and receives the user's operation instructions, it presents the target virtual object in the target interaction interface according to the operation instructions, and then monitors in real time whether there are user gestures in the target interaction interface.

[0060] S102, when the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies a predetermined time condition, associating the target virtual object with the physical object;

[0061] It can be understood that the present application pre-sets a predetermined action between the user's hand and the physical object in the electronic device, and if the predetermined action can be maintained for a certain period of time, the electronic device generates a control signal to associate the target virtual object with the physical object. After the association, the target virtual object is mapped as an associated object of the physical object in the virtual environment.

[0062] For example, the predetermined action between the hand and the physical object may be fingers holding the physical object, the palm holding the physical object, or fingers draping the physical object, or other predetermined actions, which can be set according to user needs.

[0063] The physical object in the present application can be any physical object, for example, a cup, a mouse, a bottle, a book, etc.; for example, when the electronic device recognizes that the user's hand is grabbing a cup, holding a mouse, hanging a bottle, or other predetermined actions set, it determines whether the predetermined action reaches a certain time. If it reaches a certain time, it means that the predetermined action meets the predetermined time conditions, and the target virtual object is mapped as an associated object of the physical object in the virtual environment.

[0064] S103: driving the target virtual object according to the association result and the user's operation behavior on the physical object.

[0065] Specifically, after the target virtual object is associated with the physical object, the electronic device drives the target virtual object to move accordingly according to the user's operation on the physical object.

[0066] In the target interaction interface, taking the physical object as a water cup and the predetermined action as grabbing as an example, if the user grabs the water cup and rotates it at a certain angle, the target virtual object also rotates at the same angle. If the user grabs the water cup and moves it up and down a certain distance, the target virtual object also moves the same distance. It is understandable that the speed at which the target virtual object rotates or moves is the same as the speed at which the physical object rotates or moves. This makes the user's operation of the target virtual object more realistic and in line with expectations.

[0067] The target virtual object control method provided by the present application presents the target virtual object through a target interaction interface, and simultaneously monitors and identifies gesture actions existing in the target interaction interface in real time. When it is detected that the gesture action is a predetermined action between the user's hand and the physical object, the target virtual object is associated with the physical object, so that the user's operation action on the physical object can be applied to the target virtual object, so that the target virtual object moves with the movement of the physical object, and the user can drive the target virtual object by operating the physical object, so that the movement of the target virtual object meets the user's expectations, and the operation method is simple and convenient.

[0068] In addition, the physical object in the present application can be any physical object, and the user can operate the target virtual object by operating any physical object around him, which can not only achieve simplicity of operation but also give the user a sense of reality in operating the target virtual object.

[0069] In some optional embodiments, the identifying the user's gesture action includes at least one of the following:

[0070] Analyze the motion information between the user's hand and the physical object in the acquired video data to determine the user's gesture action; the video data is collected by a camera;

[0071] The position and posture change data of the user's hand is obtained, and the user's gesture action is determined based on the position and posture change data; the position and posture change data is collected by an inertial sensor.

[0072] It is understandable that in the present application, the user's gestures can be identified by visual recognition, or by collecting gesture position data by an inertial sensor.

[0073] Among them, the visual recognition method can be realized by a camera, and the camera can be an electronic device of the target virtual object control method of the present application, such as a camera built into the electronic device; or it can be an electronic device that does not belong to the target virtual object control method of the present application, such as an electronic device connected to the camera via a data cable. The camera collects video data of the target interaction interface and sends it to the electronic device. The built-in processing program of the electronic device analyzes the motion information of the user's hand and the physical object in the acquired video data, thereby identifying the user's gesture action.

[0074] For example, when there is a target virtual object in the target interactive interface, the processing program obtains the video data collected by the camera and then analyzes the movement information of the user's hand and the physical object in the video data.

[0075] Alternatively, the user sets the inertial sensor on the ornament and then wears the ornament on the finger, so that the inertial sensor can obtain the acceleration change of the hand in real time, thereby obtaining the position change of the user's hand, and the user's gesture action can be recognized according to the position change. For example, in this application, the inertial sensor can be installed on a ring, and the user wears the ring on the hand.

[0076] It is understandable that the present application can also use a camera and an inertial sensor. When the camera cannot capture a gesture due to spatial position, the inertial sensor can be used to collect the position change of the hand, thereby identifying the user's gesture. Alternatively, when the inertial sensor signal is unstable, the camera can be used to collect video data, so that the electronic device can recognize the user's gesture through the video data.

[0077] For example, when a user grabs a cup, the inertial sensor worn on the hand will recognize the user's grabbing action based on the change in acceleration.

[0078] In some optional embodiments, the predetermined action includes at least one of the following:

[0079] Grab, hold, lift, slide, dangle, push and strike;

[0080] The predetermined time condition is met, including at least one of the following:

[0081] The predetermined action recognition results in a plurality of consecutive video frame images within a preset time are the same;

[0082] The inertial signal obtained within the preset time remains stable.

[0083] Specifically, in the present application, a predetermined action can be pre-set in the electronic device according to the needs of the user or the volume, weight, etc. of the physical object.

[0084] For example, when the user can only find objects of relatively small weight and volume around him, such as a mouse, the user can set the predetermined action to lifting or grabbing, then lifting the mouse or grabbing the mouse forms the predetermined action; if the user can find an object of relatively moderate volume and weight to be placed in his hand, such as a cup, the user can set the predetermined action to grabbing, then grabbing the cup forms the predetermined action; if the user can only find objects of relatively heavy weight and volume around him, such as a kettle, the user can set the predetermined action to pushing, then pushing the kettle forms the predetermined action.

[0085] In the present application, it is determined that the user wants to associate the target virtual object with the physical object after a certain period of time has passed since the predetermined action was performed.

[0086] For example, when a camera is used for visual recognition, if the recognition results of a predetermined action in multiple consecutive video frame images are the same within a preset time, it is considered that the predetermined action meets the predetermined time condition; or when the inertial signal obtained within the preset time remains stable, it is determined that the predetermined time condition is met.

[0087] In some optional embodiments, associating the target virtual object with the physical object includes:

[0088] Respectively obtaining first pose data of the physical object and second pose data of the target virtual object;

[0089] Establishing a mapping relationship between a physical object and a target virtual object based on the first posture data and the second posture data;

[0090] Based on the mapping relationship, the target virtual object is mapped as an associated object of a physical object in a virtual environment.

[0091] Specifically, posture data refers to the position and posture information of an object, which is usually expressed as posture (position and rotation angle), that is, (x, y, θ), which includes the position information of the object in three-dimensional space and the orientation or rotation angle information. In this application, the position information of the physical object includes the first spatial coordinates of the physical object, and the second posture coordinates of the target virtual object include the second spatial coordinates of the target virtual object; in this application, the first spatial coordinates of the outer contour of the physical object are determined, the second spatial coordinates of the outer contour of the target virtual object are determined, and then a mapping relationship between the first spatial coordinates of the physical object and the second spatial coordinates of the target virtual object is established, as well as a mapping relationship between the rotation angle of the first spatial coordinate and the rotation angle of the second spatial coordinate, so that the target virtual object is mapped as an associated object of the physical object in the virtual environment.

[0092] For example, determine the first spatial coordinates of the feature points of the outer contour of the physical object, and determine the second spatial coordinates of the feature points of the outer contour of the target virtual object. The feature points may be the center point of the leftmost contour of the outer contour, the center point of the rightmost contour of the outer contour, or other feature points. For example, establish a mapping relationship between the first spatial coordinates of the center point of the leftmost contour of the outer contour of the physical object and the second spatial coordinates of the center point of the leftmost contour of the outer contour of the target virtual object, as well as a mapping relationship between the first spatial coordinate rotation angle and the second spatial coordinate rotation angle, so that the target virtual object can be mapped as an associated object of the physical object in the virtual environment. It is also possible to establish a mapping relationship between multiple feature points of the physical object and multiple feature points at corresponding positions of the target virtual object, as well as a mapping relationship between the rotation angles of multiple feature points, so as to make the mapping relationship between the physical object and the target virtual object more accurate.

[0093] In some optional embodiments, the step of driving the target virtual object according to the association result and the user's operation behavior on the physical object includes:

[0094] Obtaining a motion offset caused by a user's operation on the physical object;

[0095] Based on the association result, the target virtual object is driven to move and change accordingly according to the motion offset.

[0096] Specifically, after the physical object is associated with the virtual object, the user can operate the physical object to make the physical object move, so that the target virtual object also moves in the same way due to the association.

[0097] For example, if the user rotates the cup in one direction by a certain angle, the target virtual object will also rotate in the same direction by the same angle; or if the user grabs the cup and moves it upward by a certain distance, the target virtual object will also move upward by the same distance. In this way, when operating a real object, the behavior is mapped to the target virtual object, so that the user's manipulation of the target virtual object is more in line with the user's expectations.

[0098] In some optional embodiments, the user's operation on the physical object includes at least one of the following:

[0099] Rotate and slide;

[0100] The motion offset includes at least one of the following:

[0101] Three-axis rotation angle information of user gesture;

[0102] The three-axis rotation angle information of the physical object;

[0103] The rotation angle change information of the target virtual object relative to the initial moment.

[0104] In this application, the user can rotate the physical object, slide on the surface of the physical object, and other operations. If the user's operation behavior on the physical object is identified by visual recognition, when the user rotates the physical object, the electronic device system obtains the three-axis rotation angle information of the user gesture generated when the user rotates the physical object or the three-axis rotation angle information of the physical object. If the user's operation behavior on the physical object is collected by an inertial sensor, the system will record the current moment as the starting moment, and the rotation angle of the target virtual object at this time as the initial angle. In this way, after the operation of the physical object is rotated, the rotation angle change information of the target virtual object relative to the initial moment is recorded and applied to the target virtual object. For sliding operations, when the user slides on the surface of the physical object, whether it is visual recognition or inertial sensor recognition, it is determined that the finger has slid on the surface of the physical object by collecting the motion trajectory of the fingertip or discrete sliding events. In this application, the system pre-sets the operation change of the target virtual object corresponding to the sliding on the surface of the physical object. For example, sliding toward the left or lower side relative to the surface of the physical object shrinks the target virtual object, and sliding toward the right or upper side relative to the surface of the physical object enlarges the target virtual object.

[0105] For example, when an electronic device has a camera, the camera acquires video data, and the electronic device system analyzes the video data to acquire the three-axis rotation angle information of the user's gesture or the three-axis rotation angle information of a physical object such as a water cup. It is understandable that the three-axis rotation angle information of the user's gesture is the same as the three-axis rotation angle information of the water cup, and therefore, acquiring the three-axis rotation angle information of the user's gesture or the three-axis rotation angle information of the physical object has the same effect. If the user wears a ring with an inertial sensor, after recognizing the predetermined action, the electronic device system needs to record the current moment as the starting moment, and the rotation angle of the water cup from this moment as the initial angle. After the water cup rotates, the rotation angle change information of the water cup relative to the target virtual object at the initial moment can be determined, thereby driving the target virtual object according to the rotation angle change information.

[0106] When the user wants to enlarge or reduce the target virtual object, the user can slide the fingertip on the surface of the water cup to change the size of the target virtual object. For example, sliding toward the left or bottom side relative to the water cup surface will reduce the target virtual object, and sliding toward the right or top side relative to the water cup surface will enlarge the target virtual object. The user's sliding operation to achieve the size change of the target virtual object can be set accordingly according to the user's needs, and this application does not limit it here.

[0107] In some optional embodiments, the user's operation on the physical object further includes:

[0108] Touch the inertial sensor; to generate a preset instruction, wherein the preset instruction is used to take the current moment as the starting moment and determine the rotation angle of the current target virtual object as the initial rotation angle.

[0109] It can be understood that when using an inertial sensor to identify a user's gesture action, if the gesture action is already a predetermined action formed between the user's hand and a physical object, and the target virtual object is associated with the physical object, then as the user's action changes, the target virtual object may be driven to change without the user's knowledge. Therefore, in this application, after the gesture action is a predetermined action formed between the user's hand and a physical object and the predetermined time conditions are met, the electronic device will prompt the user whether to associate the target virtual object with the physical object. The user can touch the inertial sensor to generate a preset instruction, which is to associate the target virtual object with the physical object. At this time, the system will use the current moment as the starting moment and determine the rotation angle of the current target virtual object as the initial rotation angle.

[0110] For example, after the user grabs a water cup by wearing a ring with an inertial sensor, and determines that the inertial signal of the inertial sensor is stable within a certain period of time, the system will display a prompt message, such as whether to associate the physical object with the target virtual object. If the user is sure to operate the target virtual object, a preset instruction is generated by touching the inertial sensor once or twice. After receiving the preset instruction, the system associates the water cup with the target virtual object, and records the current time and the current rotation angle of the target virtual object. Later, when the user operates the water cup to rotate, the target virtual object rotates the water cup by the change angle based on the current rotation angle.

[0111] In some optional embodiments, Figure 2 As shown, after the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies the predetermined time condition, it also includes:

[0112] S201, prompting the user to adjust the holding posture between the hand and the physical object, and associating the target virtual object with the adjusted physical object.

[0113] Specifically, after the gesture action is a predetermined action formed between the user's hand and the physical object and meets the predetermined time conditions, the electronic device will prompt the user to adjust the holding posture. The user can adjust to a comfortable posture as the initial posture, and then generate a preset instruction by touching the inertial sensor. After receiving the preset instruction, the system associates the physical object with the target virtual object, or the electronic device automatically associates the physical object with the target virtual object after a certain period of time after prompting the user to adjust the holding posture.

[0114] For example, after the user grabs the water cup and a preset time has passed, the electronic device displays a message asking the user to adjust the holding posture. After receiving the prompt, the user adjusts the holding posture. After the adjustment, a preset instruction can be generated by touching the inertial sensor. The electronic device receives the preset specified association between the water cup and the target virtual object. After a certain period of time has passed, the electronic device directly associates the water cup with the target virtual object.

[0115] In some optional embodiments, Figure 3 As shown, after driving the target virtual object according to the association result and the user's operation behavior on the physical object, it also includes:

[0116] S301, when it is detected that there is no contact point between the user and the physical object, ending the control of the target virtual object; or

[0117] S302, when it is detected that the displacement value of the user's hand in space exceeds a preset threshold, the control of the target virtual object is terminated.

[0118] Specifically, in the present application, after the user finishes operating the target virtual object, the visual recognition can be performed by putting down the physical object; or when an inertial sensor is used, when the user's hand displacement in space exceeds a certain threshold and there is a hand-letting action, it is determined that the user has put down the object, at which time the association between the physical object and the target virtual object is ended, and the control of the target virtual object is ended.

[0119] For example, when the electronic device has a camera, the user can put down the cup in his hand if he wants to end the operation of the target virtual object. After analyzing that the user has put down the cup, the electronic device will disconnect the cup from the target virtual object. When the user wears a ring with an inertial sensor, the user can directly move the hand wearing the ring in the air for a certain distance. If the distance is greater than a preset threshold and there is a change in posture of letting go, the electronic device will determine that the user has put down the cup, and will disconnect the cup from the target virtual object.

[0120] like Figure 4 As shown, an embodiment of the present application provides a target virtual object control device, the device comprising:

[0121] The recognition module 401 is used to recognize the user's gesture action in response to the existence of the target virtual object in the target interactive interface;

[0122] An association module 402, configured to associate the target virtual object with the physical object when the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies a predetermined time condition;

[0123] The driving module 403 is used to drive the target virtual object according to the association result and the user's operation behavior on the physical object.

[0124] The target virtual object control device provided by the present application has an identification module 401 that responds to the target virtual object and identifies the user's gesture action. The association module 402 associates the target virtual object with the physical object when the user's gesture action is a predetermined action between the user's hand and the physical object and meets the time condition. Finally, the driving module 403 drives the target virtual object to perform corresponding actions according to the association result and the user's operation behavior on the physical object.

[0125] In some optional embodiments, the identification module 401 includes at least one of the following:

[0126] An analysis unit, used to analyze the motion information related to the user's hand and the physical object in the acquired video data, and determine the user's gesture action; the video data is collected by a camera;

[0127] The first acquisition unit is used to acquire the posture change data of the user's hand, and determine the user's gesture action based on the posture change data; the posture change data is collected by an inertial sensor.

[0128] In some optional embodiments, the predetermined action includes at least one of the following:

[0129] Grab, hold, lift, slide, dangle, push and strike;

[0130] The predetermined time condition is met, including at least one of the following:

[0131] The predetermined action recognition results in a plurality of consecutive video frame images within a preset time are the same;

[0132] The inertial signal obtained within the preset time remains stable

[0133] In some optional embodiments, the association module 402 includes:

[0134] A second acquisition unit, used to respectively acquire the first pose data of the physical object and the second pose data of the target virtual object;

[0135] An establishing unit, configured to establish a mapping relationship between a physical object and a target virtual object based on the first posture data and the second posture data;

[0136] A mapping unit is used to map the target virtual object as a physical object to an associated object in a virtual environment based on the mapping relationship.

[0137] In some optional embodiments, the driving module 403 includes:

[0138] A third acquisition unit, used to acquire a motion offset generated by the user's operation behavior on the physical object;

[0139] A driving unit is used to drive the target virtual object to move and change accordingly according to the motion offset based on the association result.

[0140] In some optional embodiments, the user's operation on the physical object includes at least one of the following:

[0141] Rotate and slide;

[0142] The motion offset includes at least one of the following:

[0143] Three-axis rotation angle information of user gesture;

[0144] The three-axis rotation angle information of the physical object;

[0145] The rotation angle change information of the target virtual object relative to the initial moment.

[0146] In some optional embodiments, the user's operation on the physical object further includes:

[0147] The determination unit is used to touch the inertial sensor to generate a preset instruction, wherein the preset instruction is used to take the current moment as the starting moment and determine the rotation angle of the current target virtual object as the initial rotation angle.

[0148] In some optional embodiments, after the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies a predetermined time condition, the method further includes:

[0149] The prompt module is used to prompt the user to adjust the holding posture between the hand and the physical object, and associate the target virtual object with the adjusted physical object.

[0150] In some optional embodiments, after driving the target virtual object according to the association result and the user's operation behavior on the physical object, the method further includes:

[0151] an ending module, configured to end the control of the target virtual object when detecting that there is no contact point between the user and the physical object; or

[0152] When it is detected that the displacement value of the user's hand in space exceeds a preset threshold, the control of the target virtual object is terminated.

[0153] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0154] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0155] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0156] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0157] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the target virtual object control method. For example, in some embodiments, the target virtual object control method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the target virtual object control method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the target virtual object control method in any other appropriate manner (e.g., by means of firmware).

[0158] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0160] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0162] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0163] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0164] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0165] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0166] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for controlling a target virtual object, wherein: The method comprises: In response to the target virtual object existing in the target interactive interface, recognizing a user's gesture action; When the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies a predetermined time condition, associating the target virtual object with the physical object; The target virtual object is driven according to the association result and the user's operation behavior on the physical object.

2. The method according to claim 1, wherein: The identifying of the user's gesture action includes at least one of the following: Analyze the motion information between the user's hand and the physical object in the acquired video data to determine the user's gesture action; the video data is collected by a camera; The position and posture change data of the user's hand is obtained, and the user's gesture action is determined based on the position and posture change data; the position and posture change data is collected by an inertial sensor.

3. The method according to claim 1, wherein: The predetermined action includes at least one of the following: Grab, hold, lift, slide, dangle, push and strike; The predetermined time condition is met, including at least one of the following: The predetermined action recognition results in a plurality of consecutive video frame images within a preset time are the same; The inertial signal obtained within the preset time remains stable.

4. The method according to claim 1, wherein: The associating the target virtual object with the physical object comprises: Respectively obtaining first pose data of the physical object and second pose data of the target virtual object; Establishing a mapping relationship between a physical object and a target virtual object based on the first posture data and the second posture data; Based on the mapping relationship, the target virtual object is mapped as an associated object of a physical object in a virtual environment.

5. The method according to claim 1, wherein: The driving of the target virtual object according to the association result and the user's operation behavior on the physical object includes: Obtaining a motion offset caused by a user's operation on the physical object; Based on the association result, the target virtual object is driven to move and change accordingly according to the motion offset.

6. The method according to claim 5, wherein: The user's operation behavior on the physical object includes at least one of the following: Rotate and slide; The motion offset includes at least one of the following: Three-axis rotation angle information of user gesture; The three-axis rotation angle information of the physical object; The rotation angle change information of the target virtual object relative to the initial moment.

7. The method according to claim 1, wherein: The user's operation behavior on the physical object also includes: Touch the inertial sensor; to generate a preset instruction, wherein the preset instruction is used to take the current moment as the starting moment and determine the rotation angle of the current target virtual object as the initial rotation angle.

8. The method according to claim 1, wherein: After the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies a predetermined time condition, the method further includes: The user is prompted to adjust the holding posture between the hand and the physical object, and the target virtual object is associated with the adjusted physical object.

9. The method according to claim 1, wherein: After driving the target virtual object according to the association result and the user's operation behavior on the physical object, the method further includes: When no contact point is detected between the user and the physical object, ending the control of the target virtual object; or When it is detected that the displacement value of the user's hand in space exceeds a preset threshold, the control of the target virtual object is terminated.

10. An electronic device, wherein: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform In response to the target virtual object existing in the target interactive interface, recognizing a user's gesture action; When the gesture action is a predetermined action formed between the user's hand and the physical object and satisfies a predetermined time condition, associating the target virtual object with the physical object; The target virtual object is driven according to the association result and the user's operation behavior on the physical object.