Control methods, devices, storage media and electronic devices for wearable devices

Abstract

This application provides a control method, apparatus, storage medium, and electronic device for wearable devices, relating to the field of computer technology. The method includes: acquiring a currently playing video frame of a target video and corresponding plot-related parameters; determining the three-dimensional posture information of a target character and the motion information of an interactive character in the video frame, wherein the target character is a character pre-bound to a user; detecting spatial interaction events between the interactive character and the target character based on the three-dimensional posture information of the target character and the motion information of the interactive character; generating a first haptic feedback control signal based on the spatial interaction events and the plot-related parameters; wherein the plot-related parameters include a relationship coefficient between the target character and the interactive character; and sending the first haptic feedback control signal to the user's wearable device to drive the wearable device. This method enables haptic feedback that is synchronized with the video feed, accurately matches the character's physical experience, and has emotional differentiation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a control method, apparatus, storage medium and electronic device for a wearable device. Background Technology

[0002] Immersive viewing experiences have become a significant research direction in multimedia technology in recent years. Traditional film and television viewing primarily relies on visual and auditory senses, with users passively receiving information and lacking a sense of physical participation. While existing technologies include haptic seats and vibrating vests, their haptic feedback is mostly based on audio intensity or preset timecodes, failing to adapt to real-time dynamic changes in characters on screen. When multiple characters appear on screen and interact, existing devices struggle to effectively associate the user with a specific character. Summary of the Invention

[0003] In view of this, this application provides a control method, apparatus, storage medium, and electronic device for wearable devices, capable of achieving haptic feedback that is synchronized with the screen, accurately matches the character's physical experience, and has emotional differentiation. The specific solution is as follows:

[0004] A method for controlling a wearable device, comprising:

[0005] Obtain the currently playing video frame of the target video and the plot-related parameters corresponding to the video frame;

[0006] The three-dimensional pose information of the target character and the motion information of the interactive character in the video frame are determined, wherein the target character is a character pre-bound to the user;

[0007] Based on the three-dimensional pose information of the target character and the motion information of the interactive character, detect spatial interaction events between the interactive character and the target character;

[0008] Based on the spatial interaction event and the plot association parameters, a first tactile feedback control signal is generated; wherein, the plot association parameters include a preset relationship coefficient between the target character and the interactive character;

[0009] The first haptic feedback control signal is sent to the user's wearable device to drive the wearable device to perform haptic feedback.

[0010] Optionally, in the above method, determining the three-dimensional pose information of the target character and the motion information of the interactive character in the video frame includes:

[0011] A depth map is generated based on the video frame images;

[0012] A scene model is constructed using the depth map;

[0013] Spatial matching is performed between the character model of the target character and the scene model to establish the initial position of the character model in the scene model;

[0014] Based on the initial position, the optimal pose parameters and global position of the character model in the video frame are determined, and the three-dimensional pose information of the target character is obtained.

[0015] The motion information of the interactive character in the scene model is determined based on the scene model.

[0016] Optionally, the above method involves determining the three-dimensional pose information of the target character and the motion information of the interactive character in the video frame, including:

[0017] Identify the interactive character from the video frame and obtain the outline mask of the interactive character in the video frame;

[0018] The contour mask is mapped onto the scene model to obtain the spatial position sequence of the interactive character in the scene model;

[0019] Based on the spatial location sequence, the motion trajectory of the interactive character in the scene model is calculated as the motion information of the interactive character.

[0020] Optionally, in the above method, detecting spatial interaction events between the interactive character and the target character based on the three-dimensional pose information of the target character and the motion information of the interactive character includes:

[0021] Obtain the surface of the target character model corresponding to the three-dimensional pose information of the target character;

[0022] Obtain the motion trajectory corresponding to the motion information of the interactive character;

[0023] Based on the surface of the target character model and the motion trajectory, calculate the relative positional relationship between the interactive character and the surface of the target character model;

[0024] When the relative positional relationship meets the preset interaction conditions, a spatial interaction event is determined to have occurred.

[0025] Optionally, in the above method, determining that a spatial interaction event has occurred when the relative positional relationship meets preset interaction conditions includes:

[0026] Based on the relative positional relationship, determine whether the interactive character is visible in the video frame;

[0027] If the interactive character is visible in the video frame, then if the motion trajectory of the interactive character points to the back of the target character, the motion state of the interactive character in the continuous multiple frames before disappearing is obtained.

[0028] Based on the continuous multi-frame motion states, predict the motion trajectory of the interactive character at future moments;

[0029] Based on the motion trajectory of the interactive character at a future moment, calculate the earliest collision time between the interactive character and the surface of the target character model.

[0030] If the earliest collision time is less than a preset time threshold, a spatial interaction event is determined to have occurred.

[0031] Optionally, in the above method, generating haptic feedback control signals based on the spatial interaction events and the plot-related parameters includes:

[0032] Obtain the interaction parameters of the spatial interaction event, wherein the interaction parameters include at least the collision position;

[0033] The body part to which it belongs on the target character is determined based on the collision location;

[0034] Obtain the preset relationship coefficient between the target character and the interactive character in the plot association parameters, wherein the preset relationship coefficient represents the emotional intimacy between the target character and the interactive character;

[0035] The intensity of emotional resonance is calculated based on the interaction parameters, the preset perception sensitivity weights of the body parts, and the preset relationship coefficients.

[0036] The intensity of the emotional resonance is converted into a first tactile feedback control signal.

[0037] The above methods may also include:

[0038] In response to detecting a spatial interaction event between an interactive object and the target character in the video frame, the physical parameters of the spatial interaction event are obtained, and the physical parameters include at least the collision intensity.

[0039] A second tactile feedback control signal is generated based on the physical parameters;

[0040] The second haptic feedback control signal is sent to the user's wearable device to drive the wearable device to perform haptic feedback.

[0041] A control device for a wearable device, comprising:

[0042] The acquisition unit is used to acquire the currently playing video frame of the target video and the plot-related parameters corresponding to the video frame.

[0043] The determining unit is used to determine the three-dimensional pose information of the target character and the motion information of the interactive character in the video frame, wherein the target character is a character pre-bound to the user;

[0044] The detection unit is used to detect spatial interaction events between the interactive character and the target character based on the three-dimensional pose information of the target character and the motion information of the interactive character.

[0045] The generation unit is used to generate a first haptic feedback control signal based on the spatial interaction event and the plot association parameters; wherein, the plot association parameters include a preset relationship coefficient between the target character and the interactive character;

[0046] The transmitting unit is used to send the first haptic feedback control signal to the user's wearable device to drive the wearable device to perform haptic feedback.

[0047] A storage medium comprising stored instructions, wherein, when the instructions are executed, the device in which the storage medium resides is controlled to perform the control method for a wearable device as described above.

[0048] An electronic device includes a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described above for the control method of a wearable device.

[0049] This application provides a control method, apparatus, storage medium, and electronic device for a wearable device. The method includes: acquiring a currently playing video frame of a target video and plot-related parameters corresponding to the video frame; determining the three-dimensional posture information of a target character and the motion information of an interactive character in the video frame, wherein the target character is a character pre-bound to a user; detecting spatial interaction events between the interactive character and the target character based on the three-dimensional posture information of the target character and the motion information of the interactive character; generating a first haptic feedback control signal based on the spatial interaction events and the plot-related parameters; wherein the plot-related parameters include a preset relationship coefficient between the target character and the interactive character; and sending the first haptic feedback control signal to the user's wearable device to drive the wearable device to perform haptic feedback. By applying the method provided in this application, by pre-binding the user to a target character in the video frame, detecting spatial interaction events based on the character's three-dimensional posture and the motion information of the interactive object, and simultaneously generating haptic feedback by integrating plot parameters including character relationship coefficients, this application can achieve immersive viewing haptic feedback that is synchronized with the screen content in real time, accurately matches the character's physical experience, and has emotional differentiation. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 A flowchart of a control method for a wearable device provided in this application;

[0052] Figure 2 A flowchart illustrating the process of determining the three-dimensional pose information of a target character and the motion information of an interactive character in a video frame, as provided in this application;

[0053] Figure 3 A schematic diagram of the structure of a control device for a wearable device provided in this application;

[0054] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0057] This application provides a control method for a wearable device. This method can be applied to electronic devices, such as smartphones, tablets, smart wearable devices, etc. The method flowchart is shown below. Figure 1 As shown, it specifically includes:

[0058] S101: Obtain the currently playing video frame of the target video and the plot-related parameters corresponding to the video frame.

[0059] In this embodiment, the target video can refer to video content containing a storyline that the user is currently watching, such as movies, TV series, or animations. The target video can be pre-stored locally on the electronic device or obtained through real-time streaming over a network. The currently playing video frame refers to a single frame image presented at a certain moment during the playback of the target video. This image can be a two-dimensional color image containing RGB three primary color channel information.

[0060] Optionally, the plot association parameters can refer to data that describes the storyline state of the current video frame in terms of time. These parameters can be obtained in advance through offline analysis of the target video and stored in conjunction with the timestamps corresponding to the video frames. In one optional embodiment, the plot association parameters include at least a preset relationship coefficient between the target character and the interacting characters. Furthermore, the plot association parameters may also include current plot tension, surprise level, spatial distance attenuation coefficient, etc., which can be specifically set according to actual needs.

[0061] In practice, electronic devices can obtain the currently playing video frame in real time through the video decoding module, and at the same time read the plot-related parameters that match the timestamp of the current video frame from the pre-built plot parameter database.

[0062] S102: Determine the 3D pose information of the target character and the motion information of the interactive character in the video frame, wherein the target character is a character pre-bound to the user.

[0063] Specifically, the target character is a character pre-bound to the user within a video frame. This binding relationship can be set via the user interface before viewing the film; for example, if the user chooses to bind to the protagonist (character A) in the film, then that protagonist becomes the target character. The target character's 3D pose information refers to its positional parameters in 3D space, including the rotation angles of its joints (pose parameter θ) and its global position in the scene (translation vector t). This 3D pose information can be obtained through visual analysis of the video frames. Interactive characters can be other characters in the video frames that spatially interact with the target character. Unlike the target character, interactive characters can be supporting characters, opponents, or any character with physical contact with the target character. The interactive character's motion information refers to its motion state in 3D space, including its spatial position sequence, velocity, and acceleration. This motion information can be obtained by tracking and calculating the interactive character's position across multiple consecutive frames.

[0064] It should be noted that both the 3D pose information of the target character and the motion information of the interactive character must be expressed in a 3D spatial coordinate system. In this embodiment, the 3D spatial coordinate system can be constructed based on the depth map and scene model generated from video frames.

[0065] S103: Based on the 3D pose information of the target character and the motion information of the interactive character, detect spatial interaction events between the interactive character and the target character.

[0066] In this embodiment, a spatial interaction event can be a physical contact or collision between an interactive character and a target character. For example, an interactive character punches a target character in the chest, or an interactive character pushes a target character from behind. Detecting a spatial interaction event requires calculating the relative positional relationship between the target character's 3D pose information and the interactive character's movement trajectory, based on the character model surface. When the relative positional relationship meets preset interaction conditions, such as a spatial distance less than a threshold or the movement direction pointing towards the target character's surface, a spatial interaction event is determined to have occurred.

[0067] In one optional embodiment, spatial interaction events can be divided into frontal interaction events and back-facing interaction events. A frontal interaction event refers to a collision between the visible side (facing the camera) of the interactive character and the target character; a back-facing interaction event refers to a collision between the invisible side (facing away from the camera) of the interactive character and the target character. For back-facing interaction events, the interactive character may disappear from the screen at the moment of collision due to being obscured by the target character itself.

[0068] S104: Generate the first tactile feedback control signal based on spatial interaction events and plot-related parameters; wherein, the plot-related parameters include the preset relationship coefficient between the target character and the interactive character.

[0069] In this embodiment, the first haptic feedback control signal can be a control command used to drive a force feedback device worn by the user to execute haptic feedback. This signal may include information such as the magnitude of the force, its location, duration, and pulse frequency. Generating the first haptic feedback control signal requires combining the physical parameters of the spatial interaction event itself (such as collision location and intensity) with plot-related parameters and preset relationship coefficients between the target character and the interacting character.

[0070] Specifically, a preset relationship coefficient is used to characterize the emotional intimacy between the target character and the interactive character. The relationship coefficient can range from -1 to 1, where 1 represents intimacy, 0 represents neutrality, and -1 represents hostility. The relationship coefficient can be preset by the filmmaker or automatically generated by analyzing the film's plot. When generating haptic feedback control signals, the system adjusts the intensity of the haptic feedback based on the relationship coefficient: when the relationship coefficient is positive and has a large absolute value, collisions caused by the interactive character will trigger stronger haptic feedback, simulating the user's empathetic response to injury or attack from an intimate character; when the relationship coefficient is negative, the haptic feedback can be weakened or its mode changed.

[0071] For example, if a target character witnesses an intimate interactive character being hit, the base force can be calculated based on the physical collision intensity and multiplied by a high relationship coefficient, such as 0.9, to generate a strong, heart-pounding haptic feedback that evokes emotional resonance in the user. If the interactive character is an enemy, the relationship coefficient is -0.8, which can reduce the intensity of the haptic feedback or produce different vibration patterns.

[0072] S105: Send the first haptic feedback control signal to the user's wearable device to drive the wearable device to perform haptic feedback.

[0073] In this embodiment, the wearable device can be a force feedback device worn by the user on their body, such as a force feedback vest, haptic gloves, or force feedback mask. The wearable device may include multiple discrete force feedback actuators, such as linear resonant actuators, eccentric rotary mass motors, or airbags, distributed across different parts of the user's body. The wearable device can communicate with the electronic device via wired or wireless means.

[0074] After generating the first haptic feedback control signal, the electronic device encodes the signal into an instruction format that the wearable device can recognize and sends it to the wearable device via a communication interface. Upon receiving the signal, the wearable device drives the corresponding actuator to produce haptic effects such as vibration, pressure, or pulses according to the instructions in the signal. For example, if the collision location corresponds to the target character's chest, the actuator of the wearable device located in the user's chest area is driven to produce the corresponding vibration; if the collision intensity is large, the vibration amplitude or frequency is increased.

[0075] As is easily understood, through the steps described above, users can experience haptic feedback synchronized with the characters in the film via wearable devices while watching videos, thus gaining an immersive viewing experience. In particular, due to the introduction of plot-related parameters, the haptic feedback not only reflects physical collisions but also the emotional relationships between characters, achieving a leap from physical touch to emotional touch.

[0076] In an optional embodiment, steps S101 to S105 in the method provided in this application can be executed cyclically, that is, the above processing is repeated for each frame of the target video, thereby generating a tactile feedback signal synchronized with the screen in real time during continuous playback.

[0077] In another optional embodiment, the electronic device can also perform offline analysis on the target video in advance, pre-generating 3D pose information, motion information, and plot-related parameters for each frame, and storing them as metadata files. During online viewing, the electronic device only needs to read the metadata files and combine them with the real-time playback progress to quickly generate haptic feedback control signals, reducing real-time computing overhead.

[0078] By applying the method provided in the embodiments of this application, by pre-binding the user with the target character in the video screen, and detecting spatial interaction events based on the three-dimensional posture of the character and the motion information of the interactive object, and at the same time integrating plot parameters including the character relationship coefficient to generate haptic feedback, this application can achieve immersive movie-watching haptic feedback that is synchronized with the screen content in real time, accurately matches the character's physical experience, and has emotional differentiation.

[0079] In one embodiment provided in this application, based on the above-described scheme, optionally, the process of determining the three-dimensional pose information of the target character and the motion information of the interactive character in the video frame is as follows: Figure 2 As shown, it includes:

[0080] S201: Generate a depth map based on video frame images.

[0081] In this embodiment, the depth map is a grayscale image corresponding to a video frame, where the grayscale value of each pixel represents the distance of the corresponding scene point from the virtual camera. The depth map can be obtained by inferring from the video frame using a monocular depth estimation network, or by estimating using traditional computer vision algorithms combined with multi-view geometry principles, or by being acquired by a depth sensor. This application embodiment does not impose any limitations on this.

[0082] S202: Construct a scene model using depth maps.

[0083] Optionally, the scene model can be a pseudo-3D model (also known as a 2.5D model) with spatial coordinate information, constructed based on video frames and their corresponding depth maps. Specifically, during construction, firstly, based on the depth values ​​of each pixel in the depth map, the pixels in the video frame are mapped from a 2D image coordinate system to a 3D spatial coordinate system, forming a 3D point cloud; then, the point cloud is meshed to generate a planar mesh with height; finally, the texture of the original video frame is mapped onto the mesh surface to obtain the scene model. It should be noted that this model only retains the visible geometric information facing the camera's viewpoint; the geometry of the back view is missing. However, since the user always uses the original viewpoint, this model is sufficient to provide an accurate spatial position reference from that perspective.

[0084] S203: Spatial matching of the target character's model with the scene model to establish the initial position of the character model in the scene model.

[0085] S204: Based on the initial position, determine the optimal pose parameters and global position of the character model in the video frame to obtain the three-dimensional pose information of the target character.

[0086] The optimal pose parameters are the set of joint rotation angles that best match the projection of the character model with the observed result of the target character in the video frame. Optionally, the global position can be the translation vector of the character model in the 3D space of the scene model. The 3D pose information is composed of the optimal pose parameters and the global position. Based on the initial position, the optimal pose parameters and global position of the character model in the video frame can be determined by a multi-objective optimization algorithm to obtain the 3D pose information of the target character. The multi-objective optimization algorithm is used to simultaneously optimize three objective functions: depth consistency error, contour consistency error, and pose prior constraints. Among them, the depth consistency error quantifies the pixel-level difference between the depth map provided by the scene model and the depth map obtained by rendering the character model in the current pose; the contour consistency error measures the degree of overlap between the projected contour of the character model and the visible contour of the target character detected in the video frame, usually measured by the intersection-over-union (IoU) index; the pose prior constraints are used to penalize unnatural poses that do not conform to the laws of human kinematics, and to constrain the optimization results within a reasonable pose space.

[0087] In implementation, iterative optimization algorithms (such as gradient descent, Gauss-Newton, or Levenberg-Marquardt methods) can be used to continuously adjust the pose parameters and global position of the character model. The weighted sum of the three errors mentioned above is calculated, and the parameters are updated in the direction of reducing the sum of errors. Optimization stops when the sum of errors converges to its minimum or reaches a preset number of iterations. The pose parameters and global position at this point are the optimal values, which, when assigned to the character model, provide the 3D pose information of the target character in the current frame. It's easy to understand that, due to the use of depth consistency errors, even if the target character is partially occluded in the image, the reasonable pose of the occluded part can still be inferred using the depth information of the visible part.

[0088] The motion information of the interactive character in the scene model includes spatial position sequence, velocity vector, and acceleration vector. Determining motion information requires obtaining the spatial position sequence of the interactive character in the scene model. Specifically, firstly, the interactive character is identified from video frames, and its position in a two-dimensional image (such as a bounding box or contour mask) is obtained. Then, using the constructed scene model, the two-dimensional position is mapped to three-dimensional space to obtain the spatial position of the interactive character in the scene model. This process is repeated for multiple consecutive frames to obtain the spatial position sequence. Finally, the velocity and acceleration of the interactive character are obtained by performing differential calculations on the position sequence.

[0089] In one embodiment provided in this application, based on the above-described scheme, optionally, the process of determining the three-dimensional pose information of the target character and the motion information of the interactive character in a video frame includes:

[0090] Identify interactive characters from video frames and obtain the outline mask of the interactive characters in the video frames;

[0091] Based on the scene model, the contour mask is mapped onto the scene model to obtain the spatial position sequence of the interactive character in the scene model;

[0092] Based on the spatial location sequence, the motion trajectory of the interactive character in the scene model is calculated as the motion information of the interactive character.

[0093] In this embodiment, the interactive character can be another character in the video frame that spatially interacts with the target character. Identifying the interactive character can be done using deep learning-based object detection algorithms (such as the YOLO series, Faster R-CNN) or instance segmentation algorithms (such as Mask R-CNN) to locate the region where the interactive character is located in the video frame and output its corresponding contour mask. The contour mask can be a binary image of the same size as the video frame, where pixels with a value of 1 belong to the visible part of the interactive character, and pixels with a value of 0 belong to the background or occluded parts.

[0094] For example, video frames can be input into a pre-trained instance segmentation model, which outputs the bounding box of the interactive character and its corresponding segmentation mask. This segmentation mask is the required contour mask. It should be noted that if there are multiple candidate characters in the video frame, the interactive character to be processed can be selected from them according to preset interaction rules (such as closest spatial distance to the target character, movement trajectory pointing to the target character, etc.).

[0095] In this embodiment, mapping the contour mask to the scene model means using the depth information provided by the scene model to back-project each pixel in the two-dimensional contour mask to three-dimensional space to obtain the three-dimensional coordinates of the scene point corresponding to that pixel.

[0096] Specifically, for each pixel with a value of 1 in the contour mask, its depth value is read from the depth map. The depth value represents the distance of the scene point corresponding to that pixel from the virtual camera. Combined with the intrinsic parameters obtained from camera calibration (such as focal length and principal point coordinates), the pixel position and its depth value can be converted into a 3D spatial point in the camera coordinate system. Performing the above conversion on all pixels within the contour mask yields a set of 3D point clouds, which reflect the spatial distribution of the interactive characters in the scene model.

[0097] Furthermore, the point cloud can be centered (e.g., by taking the centroid of the point cloud) or the geometric center of the bounding box can be used as the representative spatial position of the interactive character in a single frame. Repeat the above operation for multiple consecutive frames, and arrange the spatial positions obtained in each frame in chronological order to obtain the spatial position sequence.

[0098] A motion trajectory refers to the path of an interactive character's spatial position change over a continuous period of time. Motion information further includes position sequence, velocity vector, and acceleration vector.

[0099] Specifically, the motion trajectory can be directly obtained from the spatial position sequence, that is, the curve formed by connecting the spatial positions at each moment in sequence. The velocity vector can be obtained by dividing the change in position between adjacent frames by the inter-frame time interval. The acceleration vector can be obtained by dividing the change in velocity between adjacent frames by the inter-frame time interval. For example, if the video frame rate is 30 frames / second, then the inter-frame time interval is approximately 0.0333 seconds.

[0100] In addition, if the interactive character is occluded in some frames, resulting in incomplete or missing contour masks, Kalman filtering or interpolation algorithms can be used to estimate the missing positions to ensure the continuity of the motion trajectory.

[0101] In one embodiment provided in this application, based on the above-described scheme, optionally, spatial interaction events occurring between the interactive character and the target character are detected according to the three-dimensional pose information of the target character and the motion information of the interactive character, including:

[0102] Obtain the surface of the target character model corresponding to the three-dimensional pose information of the target character;

[0103] Obtain the motion trajectory corresponding to the motion information of the interactive character;

[0104] Based on the surface of the target character model and its motion trajectory, calculate the relative positional relationship between the interactive character and the surface of the target character model;

[0105] When the relative positional relationship meets the preset interaction conditions, a spatial interaction event is determined to have occurred.

[0106] In this embodiment, based on three-dimensional pose information, a pre-generated target character model with complete three-dimensional geometric information can be placed in the corresponding pose to obtain the surface of the target character model in that pose.

[0107] The surface of the target character model can be the geometric representation of the outer surface of the target character model in three-dimensional space, typically composed of multiple triangular facets or vertex meshes. Each point on this surface has three-dimensional spatial coordinates, and the normal direction of that point can be further calculated.

[0108] Specifically, based on the optimal pose parameters and global position obtained from the solution, skeletal transformation and vertex skinning calculations can be performed on the pre-built character model to generate the pose model for the current frame. All the outer surface vertices and triangles of this pose model constitute the surface of the target character model.

[0109] A motion trajectory refers to the path of an interactive character's spatial position change over a continuous period of time, which can be directly obtained by connecting the spatial position sequences sequentially.

[0110] It should be noted that the motion trajectory can be a record of positions that have occurred in the past few frames, or it can include future trajectory segments predicted based on the current motion state.

[0111] Relative positional relationship refers to the spatial proximity and directional relationship between the interactive character and the surface of the target character model during movement. Specifically, it may include: the minimum Euclidean distance between the interactive character's current or future spatial point and the target character model surface, the angle between the interactive character's movement direction and the normal to the target character's surface, and whether the interactive character is in the front or back area of ​​the target character, etc.

[0112] In practice, sampling points along the motion trajectory can be traversed, such as the spatial position of the interactive character in the current frame. The distance from that point to the nearest point on the surface of the target character model can be calculated, and the normal direction of that nearest point can be recorded. Simultaneously, the dot product between the velocity vector of the interactive character and the surface normal of the target character can be calculated to determine whether the interactive character is moving towards or away from the surface.

[0113] For example, if the minimum distance between the interactive character and the target character model surface is less than a preset threshold, and the velocity vector direction points inward to the surface, it indicates that the two are in an approaching state where they are about to collide.

[0114] Optionally, the preset interaction conditions refer to a pre-defined set of rules for determining whether spatial interaction has occurred. These conditions can vary depending on the interaction scenario, such as frontal interaction conditions and back-side interaction conditions. For example, for frontal interaction, the conditions might include: the spatial distance between the interactive character and the target character's model surface is less than a first threshold; the difference in depth values ​​between the two on the depth map is less than a second threshold; and the dot product of the interactive character's movement direction and the surface normal is negative (indicating movement towards the surface). When all three conditions are met, a frontal interaction event is determined to have occurred. For back-side interaction, since the interactive character may be occluded at the moment of collision, the conditions might include: the interactive character is visible in the current frame and its trajectory points towards the target character's back face; the predicted future trajectory based on the character's motion state before disappearing collides with the target character's model surface; and the predicted collision time is less than a preset time threshold. When these conditions are met, a back-side interaction event is determined to have occurred.

[0115] In one embodiment provided in this application, based on the above-described solution, optionally, when the relative positional relationship meets preset interaction conditions, a spatial interaction event is determined to have occurred, including:

[0116] Based on the relative positional relationship, determine whether the interactive character is visible in the video frame;

[0117] If the interactive character is visible in the video frame, then when the motion trajectory of the interactive character points to the back of the target character, the motion state of the interactive character in the continuous multi-frames before disappearing is obtained, and the continuous multi-frame motion state includes at least spatial position, velocity and acceleration.

[0118] Based on the motion states of multiple consecutive frames, predict the motion trajectory of the interactive character at future moments;

[0119] Calculate the earliest collision time between the interactive character and the surface of the target character model based on the movement trajectory of the interactive character at a future moment.

[0120] If the earliest collision time is less than a preset time threshold, a spatial interaction event is determined to have occurred.

[0121] Here, the back side refers to the side of the target character model's surface that faces away from the camera's viewpoint, i.e., the part that the user cannot directly observe from the screen. When the aforementioned relative positional relationship indicates that the interactive character is displayed in the current video frame, and the interactive character's movement trajectory points towards the back side of the target character, the back-side interaction detection process is triggered.

[0122] To determine whether a motion trajectory points towards the opposite side, we can analyze the relationship between the direction of the interactive character's velocity vector and the normal to the opposite side of the target character's model. If the projection of the velocity vector onto the normal to the opposite side is positive, and the interactive character is currently positioned behind the target character, then the trajectory can be considered to point towards the opposite side.

[0123] Before the interactive character disappears, it refers to a series of consecutive frames in which the interactive character has not yet been occluded by the target character. In these frames, the interactive character is visible in the frame, and the system can stably acquire its motion information. The motion state of the interactive character in the consecutive frames before disappearing is acquired, and this motion state includes at least spatial position, velocity, and acceleration. Velocity is obtained by dividing the position difference between adjacent frames by the inter-frame time interval, and acceleration is obtained by dividing the velocity difference between adjacent frames by the inter-frame time interval. For example, the motion state of the interactive character in the last N frames before disappearing (e.g., N=5) can be used as the basis for prediction. If the interactive character has never been occluded, then data from the current frame and several previous frames are used.

[0124] Based on the motion states of multiple consecutive frames, the trajectory of the interactive character at future moments is predicted. Predicting the trajectory at future moments refers to extrapolating the spatial position of the interactive character at subsequent moments using a kinematic model based on its motion state (position, velocity, acceleration) before disappearing. In this embodiment, a uniformly accelerated motion model can be used for prediction, assuming that the interactive character maintains its current acceleration for a short period in the future. This prediction process can be performed frame-by-frame on the time axis or at preset time steps to generate a sequence of motion trajectory points for future moments. Alternatively, if the motion of the interactive character is more complex (such as variable acceleration or being affected by external forces), more complex prediction models such as Kalman filtering can be used, but the uniformly accelerated model already has sufficient accuracy for short-term prediction.

[0125] Based on the predicted motion trajectory, the earliest collision time between the interactive character and the target character model surface is calculated. The earliest collision time refers to the moment when the interactive character, moving along the predicted trajectory, first comes into contact with the target character model surface, starting from the current moment. Calculating the earliest collision time requires intersecting the predicted trajectory with the target character model surface. Specifically, the predicted trajectory can be discretized into a series of spatial points at future moments, and each point can be sequentially determined whether it is located inside the target character model surface or intersects with the surface. Since the target character model surface is usually composed of triangular facets, the distance between the detection point and the triangular facets can be used to determine whether contact has occurred. When the predicted point is first detected to be inside the model surface or at a distance less than a preset contact threshold, the time corresponding to that point is the earliest collision time. Alternatively, a continuous collision detection method can be used to directly solve for the intersection between the motion trajectory parametric equation and the implicit function of the model surface to obtain a more accurate collision time. For example, suppose that at a certain moment, the position of a point in the predicted trajectory intersects exactly with the target character's back model surface; then that moment is the earliest collision time.

[0126] After obtaining the earliest collision time, it is determined whether this earliest collision time is less than a preset time threshold. The preset time threshold is used to limit the system to responding only to collisions that occur within a short period of time in the future, avoiding triggering false interactions due to excessive prediction errors. This threshold can be set according to the actual application scenario, for example, it can be set to 0.5 seconds or 1 second. When the earliest collision time is less than this threshold, the system believes that a back-to-back collision will indeed occur in the near future, thus determining that a spatial interaction event has occurred and recording the event as a back-to-back interaction event. After determining the interaction event, the collision location and collision intensity can be further determined. The collision location is the intersection of the predicted trajectory at the earliest collision time and the model surface; the collision intensity can be calculated based on factors such as the mass of the interactive character, the predicted collision velocity, and the material coefficient.

[0127] In one embodiment provided in this application, based on the above-described solution, optionally, a haptic feedback control signal is generated based on spatial interaction events and plot-related parameters, including:

[0128] Obtain the interaction parameters of spatial interaction events. The interaction parameters include at least the collision location and collision intensity.

[0129] The collision location determines the body part on the target character to which it belongs;

[0130] Obtain the preset relationship coefficient between the target character and the interactive character from the plot-related parameters, where the preset relationship coefficient represents the emotional intimacy between the target character and the interactive character;

[0131] The intensity of emotional resonance is calculated based on the collision intensity, the pre-set perception sensitivity weights of body parts, and the pre-set relationship coefficients.

[0132] The intensity of emotional resonance is converted into a pulsed tactile feedback signal, which serves as the first tactile feedback control signal.

[0133] In this embodiment, a spatial interaction event can refer to an interaction event determined by the aforementioned detection steps, and its type can be a frontal interaction event or a back-facing interaction event. Interaction parameters refer to key physical quantities describing the spatial interaction event, including at least the collision location and collision intensity. The collision location refers to the coordinates of the spatial point where the interactive character and the target character model surface come into contact; the collision intensity characterizes the magnitude of the mechanical impact of the collision, and can be calculated, for example, based on the estimated mass of the interactive character, the collision velocity, and the material coefficient.

[0134] The target character, a virtual avatar pre-bound to the user, has its body parts pre-divided into several regions, such as the head, chest, abdomen, back, left arm, right arm, left leg, and right leg. Each body part has a corresponding set of vertices or surface area on the 3D character model. Based on the spatial coordinates of the collision location, the body part corresponding to this collision can be determined by querying the pre-defined body part region to which those coordinates belong.

[0135] For example, if the collision location is within the vertex set of the chest region of the target character model, the corresponding body part is determined to be the chest; if it is within the surface range of the back region, the corresponding body part is determined to be the back. This determination of the body part will be used for the spatial mapping of subsequent haptic feedback.

[0136] It's important to note that the preset relationship coefficient is plot information independent of the physical parameters of the spatial interaction event. Even if two spatial interaction events have the exact same collision location and intensity, the generated tactile feedback will differ if the relationship coefficients between the interacting character and the target character are different. Emotional resonance intensity refers to the level of tactile feedback felt by the user through the wearable device, and its calculation integrates physical collision parameters and plot emotional parameters. Specifically, emotional resonance intensity can be expressed as a comprehensive function of collision intensity, body part perception sensitivity weights, and relationship coefficients. The preset perception sensitivity weights for body parts reflect the differences in sensitivity to tactile stimuli across different parts of the body. For example, the sensitivity weights for large muscle groups such as the chest and abdomen can be set to a medium value, while the sensitivity weights for parts like the back and limbs are relatively lower.

[0137] For example, the emotional resonance intensity is calculated as follows: First, the collision intensity is multiplied by the body part's perception sensitivity weight to obtain the basic perception intensity of that body part; then, this basic perception intensity is multiplied by a moderating factor mapped from the relationship coefficient. For example, a positive relationship coefficient enhances feedback, while a negative one weakens feedback, resulting in the final emotional resonance intensity. It's easy to understand that the higher the relationship coefficient, the greater the emotional resonance intensity triggered by the same physical collision; when the relationship coefficient is negative, the emotional resonance intensity may be suppressed or even produce different feedback patterns.

[0138] Pulsed haptic feedback signals can be drive signals with a specific time envelope (such as brief impacts or periodic pulsations) used to control force feedback actuators in wearable devices to produce corresponding haptic effects. The form of this signal can be correlated with the intensity of emotional resonance: the higher the intensity of emotional resonance, the larger the amplitude of the pulse signal, the longer its duration, or the higher the pulse frequency.

[0139] In one alternative embodiment, the intensity of emotional resonance can be mapped to the amplitude of the drive voltage of the vibration motor or the duty cycle of a pulse width modulation (PWM) circuit, thereby controlling the vibration intensity of the actuator. In another alternative embodiment, the intensity of emotional resonance can be input into a pulse generation function to generate a pulse sequence with a specific frequency and duty cycle, simulating the rhythm of a heartbeat or impact.

[0140] This pulsed tactile feedback signal is the first tactile feedback control signal, which will then be sent to the wearable device worn by the user to drive the actuators in the corresponding body parts to generate tactile feedback.

[0141] In one embodiment provided in this application, based on the above-described solution, optionally, it further includes:

[0142] In response to the detection of a spatial interaction event between an interactive object and a target character in a video frame, the physical parameters of the spatial interaction event are obtained, including at least the collision location and collision intensity.

[0143] A second tactile feedback control signal is generated based on physical parameters;

[0144] A second haptic feedback control signal is sent to the user's wearable device to drive the wearable device to perform haptic feedback synchronized with physical collisions.

[0145] In this embodiment, an interactive object refers to an entity in a video frame that engages in spatial interaction with a target character but does not possess character attributes, such as a flying projectile, a moving vehicle, or dynamic particles in the environment. The detection of spatial interaction events can employ the aforementioned frontal interaction detection or back-side interaction detection methods, specifically determined by the visibility of the interactive object in the frame and the direction of its motion trajectory. Physical parameters include at least the collision location and collision intensity. The collision location refers to the coordinates of the spatial point where the interactive object contacts the surface of the target character model; the collision intensity characterizes the magnitude of the mechanical impact of the collision and can be calculated based on the estimated mass, collision velocity, and material coefficients of the interactive object. For example, for a flying stone, its mass can be estimated based on its volume, its velocity can be calculated based on positional changes over multiple frames, and the collision intensity can be calculated by combining this with momentum changes. For high-speed objects such as bullets, a preset standard impact value can be used.

[0146] Specifically, the body part to which the collision belongs on the target character can be determined based on the collision location, and the tactile feedback intensity of the collision can be calculated based on the preset perception sensitivity weight of that body part and the collision intensity. Then, the tactile feedback intensity is converted into a driving signal, such as the driving voltage amplitude of a vibration motor or the pulse width modulation duty cycle, to obtain a second tactile feedback control signal.

[0147] Finally, a second haptic feedback control signal is sent to the wearable device worn by the user to drive the device to perform haptic feedback synchronized with the physical collision. Upon receiving the signal, the wearable device drives the actuators of the corresponding body parts to produce haptic effects such as vibration and pressure, according to the instructions in the signal. Because this signal is not adjusted by emotional parameters, the haptic feedback felt by the user directly corresponds to the physical collision occurring on the screen in terms of intensity, location, and timing.

[0148] The wearable device control method provided in this application can be applied to video viewing scenarios. Based on this method, an immersive motion-sensing viewing solution can be provided, which converts the physical experience of characters in a two-dimensional film into tactile feedback that the user can perceive in real time and accurately, thereby enabling the user to obtain a sense of immersion in the tactile dimension in addition to visual and auditory senses while watching the film. Specifically:

[0149] First, build a model and establish mapping relationships.

[0150] In this embodiment, the current video frame is first... Perform character detection and output the visible portion mask of the target character. The visible portion mask is a binary image of the same size as the video frame, where pixels with a value of 1 (u, v) belong to the visible area of ​​the target character, and pixels with a value of 0 belong to the background or occluded areas. This mask is used to indicate the pixels that need to participate in the optimization calculation during subsequent matching. Next, a depth map is generated for the current video frame. The depth map can be obtained by inferring from the video frame using a monocular depth estimation network (such as MiDaS, Depth Anything), or by using traditional computer vision algorithms (such as SIFT, optical flow) combined with multi-view geometric estimation, or by using a depth sensor. Direct data collection. Depth map. The grayscale value of each pixel in the depth map represents the distance of the corresponding scene point from the virtual camera. Specifically, the depth map is represented as follows: .

[0151] Given a pre-built complete 3D character model (with a fixed number of vertices and complete geometric information), along with the visible portion mask and depth map of the current frame, solve for the character model through multi-objective optimization. Given the optimal pose parameters θ (joint rotation angle) and global position t in the current frame, the optimization objective function is expressed as:

[0152]

[0153] in, The goal is to find the optimal attitude parameters (joint rotations) and global position. To find the set of parameters that minimizes the overall error in the subsequent steps, , Weighting coefficients are used to balance depth errors. Contour error and attitude prior The importance of these three factors in the total error.

[0154] Optional, balance depth error ,include:

[0155]

[0156] in, This is the depth error term, used to quantify the difference between the actual depth and the model's predicted depth; It can be the summation range of pixel coordinates (u,v) within the visible area; It can be the depth value at (u,v) in a real scene; It can be a pre-built 3D character model In the current attempt Render (project) onto a two-dimensional plane to obtain the depth map of the virtual model.

[0157] Optional, contour consistency error, including:

[0158]

[0159] in, This is the contour error term, used to measure the degree of matching between the actual visible area and the predicted area of ​​the character model. The intersection-union ratio (IUGR) is an indicator that measures the degree of overlap between regions. It is calculated by dividing the area of ​​the intersection of two regions by the area of ​​their union. The value ranges from [0,1], and the closer the value is to 1, the higher the degree of overlap. Indicates the actual outline of the character; This indicates the character model in the currently attempted pose. The obtained character outline region; since the optimization objective can be to minimize the error, and the larger the IoU, the better the matching degree, the larger the IoU after taking the negative, the smaller the error term.

[0160] Optional, prior attitude constraints include:

[0161]

[0162] In this embodiment, the posture prior constraint can be based on the posture distribution learned from massive human motion data, penalizing unnatural postures that do not conform to the laws of human kinematics, and constraining the optimization results within a reasonable posture space. When the character is severely occluded or visual evidence is insufficient, this item provides a strong regularization constraint to ensure the rationality and smoothness of the posture.

[0163] During the solution process, an iterative optimization algorithm (such as gradient descent, Gauss-Newton, or Levenberg-Marquardt method) is used to continuously adjust the pose parameters θ and global position t. The weighted sum of three errors is calculated, and the parameters are updated in the direction of reducing the sum of errors. Optimization stops when the sum of errors converges to its minimum or when a preset number of iterations is reached; at this point, the pose parameters and global position are considered optimal. This optimal pose is then applied to a pre-built complete 3D character model, resulting in a complete 3D digital avatar of the target character in the current frame. The details are as follows:

[0164]

[0165] Second: Physical interaction detection.

[0166] In one optional embodiment, physical interaction detection is divided into two scenarios: frontal interaction detection and back-side interaction detection. Frontal interaction detection is applicable to scenarios where dynamic objects interact with visible surfaces of the target character model's surface. It requires three conditions to be met simultaneously: the Euclidean distance between surface points is less than a preset distance threshold, the depth difference between matching points is less than a preset depth threshold, and the dot product of the relative velocity vector and the surface normal vector is negative. A negative value indicates that the object is moving towards the model surface rather than away from it. When all three conditions are met, the nearest point on the character model's surface is selected as the collision contact point. Back-side interaction detection is applicable to scenarios where dynamic objects interact with invisible surfaces of the target character model's surface. Since the object may be occluded by the character itself at the moment of collision and cannot be directly observed, the system acquires the object's motion state over several consecutive frames before it disappears, uses a uniformly accelerated motion model to predict its future trajectory, and defines the character model's surface as an implicit function. The earliest collision time is determined by the intersection of the predicted trajectory and the model surface. When this time is less than a preset time threshold, a back-side interaction event is determined, and the collision point is identified.

[0167] Third, interactive output.

[0168] Through the aforementioned frontal or back-side detection, the system outputs one or more physical interaction vectors. Each physical interaction vector includes at least: the three-dimensional spatial position of the collision point, the normal direction of the collision surface, and the relative velocity vector at the moment of collision. This vector will serve as input to subsequent force feedback calculation and mapping steps to generate haptic feedback control signals that drive the wearable device.

[0169] Fourth: Force feedback calculation and mapping.

[0170] Force feedback calculation includes basic force calculation, part classification, force distribution, and smoothing. Basic force calculation is based on a simplified model using the momentum theorem, determined by the product of the collision object's mass and velocity change, material coefficients, and collision validity criteria. The validity criteria ensure that force is generated only when the object moves towards the surface; the same formula is used for back-to-back collisions, with the velocity taken as a predicted value. Part classification determines the part to which the collision point belongs using a classification model. Force distribution employs a Gaussian kernel weighted model, where the weight of each actuator is determined by its spatial distance from the virtual reference point, with closer actuators receiving higher weights. Different parts are assigned perception weights and diffusion coefficients, and normalization ensures the total force distribution matches the basic force. Smoothing uses a first-order IIR filter to eliminate abrupt changes. Finally, frame-by-frame force values ​​are output to the wearable device, driving the actuators to generate haptic feedback.

[0171] Fifth: Through the director module, convey the physiological emotional experience that the director hopes users will feel.

[0172] In an optional embodiment, the wearable device control method provided in this application may further include a director module as an extended function of emotional resonance. The emotional resonance module may be placed in the user's heart area to enhance immersion by simulating physiological reactions such as accelerated heartbeat.

[0173] The plot weight in the director module is a weighted sum of the plot tension and the degree of surprise, used to reflect the intensity of emotions through the pulse frequency in the heart region; the intensity of emotional resonance is the product of the basic damage intensity (the product of the magnitude of the physical force and the sensitivity coefficient of the damage type), the relationship coefficient adjustment factor mapped by the Sigmoid function, the distance decay value (a combination of exponential decay and power-law decay), and the plot weight, integrating four dimensions: physical damage, character relationships, spatial distance, and narrative tension; after multiplying the intensity of emotional resonance by a scaling factor and inputting it into the pulse function, a rhythmic pulsating tactile sensation with an increasing frequency as the intensity increases can be generated in the heart region, and this feedback is independent of the physical tactile mapping; the force feedback device converts digital force vectors into physical stimuli, and generates force tracks on the skin surface through multi-point coordination and temporal control, thereby achieving tactile feedback synchronized with the film screen.

[0174] This application also provides a control device for a wearable device, the structural schematic of which is shown below. Figure 3 As shown, it includes:

[0175] The acquisition unit 301 is used to acquire the currently playing video frame of the target video and the plot association parameters corresponding to the video frame;

[0176] The determining unit 302 is used to determine the three-dimensional pose information of the target character and the motion information of the interactive character in the video frame, wherein the target character is a character pre-bound to the user;

[0177] The detection unit 303 is used to detect spatial interaction events between the interactive character and the target character based on the three-dimensional pose information of the target character and the motion information of the interactive character.

[0178] The generation unit 304 is used to generate a first tactile feedback control signal based on the spatial interaction event and the plot association parameters; wherein, the plot association parameters include a preset relationship coefficient between the target character and the interactive character;

[0179] The sending unit 305 is used to send the first haptic feedback control signal to the user's wearable device to drive the wearable device to perform haptic feedback.

[0180] The specific principles and execution processes of each unit and module in the control device of the wearable device disclosed in the above embodiments of this application are the same as those of the control method of the wearable device disclosed in the above embodiments of this application. Please refer to the corresponding parts of the control method of the wearable device provided in the above embodiments of this application, and they will not be repeated here.

[0181] This application also provides a storage medium, which includes stored instructions, wherein the device where the storage medium is located is controlled to execute the control method of the wearable device when the instructions are executed.

[0182] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 4 As shown, it specifically includes a memory 401 and one or more instructions 402, wherein one or more instructions 402 are stored in the memory 401 and configured to be executed by one or more processors 403 to perform the control method of the wearable device described above.

[0183] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0184] Finally, it should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0185] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0186] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0187] The present application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

Claims

1. A control method for a wearable device, characterized in that, include: Obtain the currently playing video frame of the target video and the plot-related parameters corresponding to the video frame; The three-dimensional pose information of the target character and the motion information of the interactive character in the video frame are determined, wherein the target character is a character pre-bound to the user; Based on the three-dimensional pose information of the target character and the motion information of the interactive character, detect spatial interaction events between the interactive character and the target character; Based on the spatial interaction event and the plot association parameters, a first tactile feedback control signal is generated; wherein, the plot association parameters include a preset relationship coefficient between the target character and the interactive character; The first haptic feedback control signal is sent to the user's wearable device to drive the wearable device to perform haptic feedback.

2. The method according to claim 1, characterized in that, Determining the three-dimensional pose information of the target character and the motion information of the interactive character in the video frame includes: A depth map is generated based on the video frame; A scene model is constructed using the depth map; Spatial matching is performed between the character model of the target character and the scene model to establish the initial position of the character model in the scene model; Based on the initial position, the optimal pose parameters and global position of the character model in the video frame are determined, and the three-dimensional pose information of the target character is obtained. The motion information of the interactive character in the scene model is determined based on the scene model.

3. The method according to claim 1, characterized in that, Determining the three-dimensional pose information of the target character and the motion information of the interactive character in the video frame includes: Identify the interactive character from the video frame and obtain the outline mask of the interactive character in the video frame; The contour mask is mapped onto the scene model to obtain the spatial position sequence of the interactive character in the scene model; Based on the spatial location sequence, the motion trajectory of the interactive character in the scene model is calculated as the motion information of the interactive character.

4. The method according to claim 1, characterized in that, The step of detecting spatial interaction events between the interactive character and the target character based on the three-dimensional pose information of the target character and the motion information of the interactive character includes: Obtain the surface of the target character model corresponding to the three-dimensional pose information of the target character; Obtain the motion trajectory corresponding to the motion information of the interactive character; Based on the surface of the target character model and the motion trajectory, calculate the relative positional relationship between the interactive character and the surface of the target character model; When the relative positional relationship meets the preset interaction conditions, a spatial interaction event is determined to have occurred.

5. The method according to claim 4, characterized in that, When the relative positional relationship satisfies the preset interaction conditions, a spatial interaction event is determined to have occurred, including: Based on the relative positional relationship, determine whether the interactive character is visible in the video frame; If the interactive character is visible in the video frame, then if the motion trajectory of the interactive character points to the back of the target character, the motion state of the interactive character in the continuous multiple frames before disappearing is obtained. Based on the continuous multi-frame motion states, predict the motion trajectory of the interactive character at future moments; Based on the motion trajectory of the interactive character at a future moment, calculate the earliest collision time between the interactive character and the surface of the target character model. If the earliest collision time is less than a preset time threshold, a spatial interaction event is determined to have occurred.

6. The method according to claim 1, characterized in that, The generation of haptic feedback control signals based on the spatial interaction events and the plot-related parameters includes: Obtain the interaction parameters of the spatial interaction event, wherein the interaction parameters include at least the collision position; The body part to which it belongs on the target character is determined based on the collision location; Obtain the preset relationship coefficient between the target character and the interactive character in the plot association parameters, wherein the preset relationship coefficient represents the emotional intimacy between the target character and the interactive character; The intensity of emotional resonance is calculated based on the interaction parameters, the preset perception sensitivity weights of the body parts, and the preset relationship coefficients. The intensity of the emotional resonance is converted into a first tactile feedback control signal.

7. The method according to claim 1, characterized in that, Also includes: In response to detecting a spatial interaction event between an interactive object and the target character in the video frame, the physical parameters of the spatial interaction event are obtained, and the physical parameters include at least the collision intensity. A second tactile feedback control signal is generated based on the physical parameters; The second haptic feedback control signal is sent to the user's wearable device to drive the wearable device to perform haptic feedback.

8. A control device for a wearable device, characterized in that, include: The acquisition unit is used to acquire the currently playing video frame of the target video and the plot-related parameters corresponding to the video frame. The determining unit is used to determine the three-dimensional pose information of the target character and the motion information of the interactive character in the video frame, wherein the target character is a character pre-bound to the user; The detection unit is used to detect spatial interaction events between the interactive character and the target character based on the three-dimensional pose information of the target character and the motion information of the interactive character. The generation unit is used to generate a first haptic feedback control signal based on the spatial interaction event and the plot association parameters; wherein, the plot association parameters include a preset relationship coefficient between the target character and the interactive character; The transmitting unit is used to send the first haptic feedback control signal to the user's wearable device to drive the wearable device to perform haptic feedback.

9. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device containing the storage medium is controlled to perform the control method of the wearable device as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a memory, and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any one of claims 1 to 7.

Images