Haptic interaction information processing method, device, and equipment

Abstract

A haptic interaction information processing method, device, and apparatus are disclosed, which belong to the field of haptic signal representation. The haptic interaction information processing method according to an embodiment of the present disclosure includes: counting the number of waveforms of haptic interaction signals in a target video screen, where the target video screen is one frame video screen in the target video, and one waveform of haptic interaction signal corresponds to the haptic interaction information of one video screen area in the target video screen (S101); and generating a grayscale information representation diagram corresponding to the target video screen based on the number of waveforms of the haptic interaction signals, where the grayscale information representation diagram is for representing the distribution position of each video screen area in the target video screen (S102).

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to a Chinese patent application filed with the China Patent Office on February 27, 2023, bearing application number 202310212530.1 and entitled "Tactile interaction information processing method, device and apparatus," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of haptic signal representation, and in particular to a haptic interaction information processing method, device and apparatus. [Background technology]

[0003] Haptics has been proposed as a primary media type, comparable to audio and video, in the ISO Base Media File Format (ISOBMFF) based on the ISO optical disc mirror file. Haptics has also been proposed as a supplement to the Motion Picture Experts Group Dynamic Adaptive Streaming over HTTP (MPEG-DASH) standard, allowing DASH stream clients to detect the presence of haptics in ISOBMFF segments. However, this method only meets the needs of single-screen, single-haptic interaction settings. Summary of the Invention [Means for solving the problem]

[0004] A first aspect provides a method for processing haptic interaction information, the method comprising: Counting the number of waveforms of haptic interaction signals in a target video screen, where the target video screen is one frame video screen in the target video, and one waveform of haptic interaction signal corresponds to the haptic interaction information of one video screen area in the target video screen; and generating a grayscale information representation diagram corresponding to the target video screen based on the number of waveforms of the haptic interaction signal, wherein the grayscale information representation diagram is for representing the distribution position of each video screen area on the target video screen.

[0005] A second aspect provides a method for processing haptic interaction information, the method comprising: Obtain and decode a haptic expression information stream corresponding to a target video, and obtain grayscale information representation diagrams corresponding to each of a plurality of video screens in the target video and encoded haptic interaction information corresponding to each of the plurality of video screens, wherein the grayscale information representation diagram corresponding to one video screen is for representing the distribution position of each video screen area in the video screen, and the waveform of each haptic interaction signal corresponding to one video screen corresponds to the haptic interaction information of each video screen area in the video screen; determining a grayscale information representation diagram corresponding to the target video screen from the grayscale information representation diagrams corresponding to the plurality of video screens in response to detecting a haptic interaction operation for the target video screen in the plurality of video screens; determining grayscale information corresponding to the haptic interaction operation based on a grayscale information representation diagram corresponding to the target video screen and a video screen area where the haptic interaction operation is located; determining haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of waveforms of haptic interaction signals corresponding to the target video screen based on grayscale information corresponding to the haptic interaction operation; and triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation.

[0006] A third form provides an apparatus for processing haptic interaction information, the apparatus comprising: a first statistics module for counting the number of waveforms of haptic interaction signals on a target video screen, the target video screen being one frame video screen in the target video, and one waveform of the haptic interaction signal corresponding to the haptic interaction information of one video screen area on the target video screen; and a first generation module for generating a grayscale information representation diagram corresponding to the target video screen based on the number of waveforms of the haptic interaction signal, the grayscale information representation diagram for representing the distribution position of each video screen area on the target video screen.

[0007] A fourth aspect provides an apparatus for processing haptic interaction information, the apparatus comprising: a first processing module for acquiring and decoding a haptic expression information stream corresponding to a target video, and obtaining grayscale information representation diagrams corresponding to each of a plurality of video screens in the target video and encoded haptic interaction information corresponding to each of the plurality of video screens, wherein the grayscale information representation diagram corresponding to one video screen is for representing the distribution position of each video screen area in the video screen, and the waveform of each haptic interaction signal corresponding to one video screen corresponds to the haptic interaction information of each video screen area in the video screen; a second processing module for determining, in response to detecting a haptic interaction operation with respect to a target video screen in the plurality of video screens, a grayscale information representation diagram corresponding to the target video screen from the grayscale information representation diagrams corresponding to the plurality of video screens; a third processing module for determining grayscale information corresponding to the haptic interaction operation based on a grayscale information representation diagram corresponding to the target video screen and a video screen area where the haptic interaction operation is located; a fourth processing module for determining, based on the grayscale information corresponding to the haptic interaction operation, haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of waveforms of haptic interaction signals corresponding to the target video screen; and a fifth processing module for triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation.

[0008] A fifth aspect provides a tactile interaction information processing device including a processor and a memory, the memory storing a program or instructions executable on the processor, and when the program or instructions are executed by the processor, realizing the steps of the tactile interaction information processing method described in the first aspect.

[0009] A sixth aspect provides a tactile interaction information processing device including a processor and a memory, the memory storing a program or instructions executable on the processor, and when the program or instructions are executed by the processor, realizing the steps of the tactile interaction information processing method described in the second aspect.

[0010] The seventh form is a memory having computer readable code stored therein; and one or more processors, wherein the computer-readable code, when executed by the one or more processors, implements the haptic interaction information processing method according to the first or second aspect.

[0011] An eighth aspect is a computer program comprising computer readable code that, when executed on a computing device, causes the computing device to perform the haptic interaction information processing method described in the first or second aspect.

[0012] A ninth aspect provides a computer-readable medium storing the computer program according to the eighth aspect.

[0013] A tenth aspect provides a tactile interaction information processing system including a video production device for performing the steps of the tactile interaction information processing method described in the first aspect above, and a video playback device for performing the steps of the tactile interaction information processing method described in the second aspect above.

[0014] An eleventh aspect provides a readable storage medium storing a program or instructions that, when executed by a processor, realizes the steps of the tactile interaction information processing method described in the first aspect or the steps of the tactile interaction information processing method described in the second aspect.

[0015] A twelfth form provides a chip including a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor executes a program or instruction to realize the tactile interaction information processing method described in the first form or the tactile interaction information processing method described in the second form.

[0016] A thirteenth aspect provides a computer program / program product stored on a storage medium, which, when executed by at least one processor, implements the steps of the tactile interaction information processing method described in the first or second aspect.

[0017] The above description is a summary of the present disclosure, and in order to more clearly understand the technical means of the present disclosure and to enable implementation in accordance with the content of the specification, and to more clearly and comprehend the above and other objects, features, and advantages of the present disclosure, specific embodiments of the present disclosure are listed below.

[0018] In order to more clearly describe the embodiments of the present disclosure, the following briefly describes the drawings that need to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings without creative work. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a flowchart of a haptic interaction information processing method according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram of a target video screen in an embodiment of the present disclosure. [Figure 3] FIG. 10 is a schematic diagram of another target video screen in accordance with an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of a haptic interaction index map according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a schematic diagram of an extended grayscale information representation diagram in an embodiment of the present disclosure. [Figure 6] 10 is a flowchart of another haptic interaction information processing method in accordance with the present disclosure. [Figure 7] FIG. 1 is a configuration block diagram of a haptic interaction information processing device according to an embodiment of the present disclosure. [Figure 8]FIG. 10 is a configuration block diagram of another haptic interaction information processing device according to an embodiment of the present disclosure. [Figure 9] 1 is a schematic diagram of a computing device for the method according to the present disclosure; [Figure 10] 1 illustrates schematically a storage unit for holding or carrying program code implementing the methods according to the present disclosure; DETAILED DESCRIPTION OF THE INVENTION

[0020] In order to clarify the purpose, technical means and advantages of the embodiments of the present disclosure, the technical means in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some embodiments of the present disclosure, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative labor belong to the scope of protection of the present disclosure.

[0021] The terms "first," "second," and the like in the specification and claims of the present disclosure are intended to distinguish between similar objects and not to describe a particular order or chronological order. Terms used in this manner are interchangeable where appropriate, so that embodiments of the present disclosure may be implemented in an order other than that illustrated or described herein. It should be understood that objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited; for example, the first object may be one or at least two. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the symbol " / " typically indicates that the related objects before and after are in an "or" relationship.

[0022] Haptic technology offers a new way of entertainment and sensory immersion beyond audio and video media. Take media streams such as streaming games or mobile advertisements as examples, adding haptics to the audio / video content of the media stream can greatly enhance the user's experience and enjoyment of the media content.

[0023] Therefore, haptics has been proposed as a primary media type in ISOBMFF, on par with audio and video, and as a supplement to the MPEG-DASH standard, allowing DASH stream clients to know that haptics are present in ISOBMFF segments. However, this method can only meet the needs of a single haptic interaction setting on a single screen, and cannot meet the needs of different haptic interaction settings in different areas of the screen.

[0024] To address the problems existing in the related art, the present disclosure designs a grayscale representation method for the position of a haptic interaction signal in a video screen area, using grayscale information to divide the target video screen into areas based on the waveform of the haptic interaction signal, thereby establishing an implementation basis for different haptic interaction settings in different areas of the screen (for example, regional haptic interaction settings of an array motor, or haptic interaction settings similar to mono / stereo transmission, etc.), so that a subsequent playback device can search for the haptic interaction signal corresponding to each area position based on the grayscale information in the grayscale information representation diagram, thereby improving the interactivity and entertainment value of the video screen.

[0025] A first embodiment is a flowchart of a haptic interaction information processing method according to an embodiment of the present disclosure, as shown in FIG. 1 , the method may include steps S101 to S102: Step S101: Count the number of waveforms of the haptic interaction signal on the target video screen.

[0026] Here, the target video screen is one frame video screen in the target video, and the waveform of one haptic interaction signal corresponds to the haptic interaction information of one video screen area in the target video screen.

[0027] In specific implementation, for a video containing haptic dimension information (i.e., a target video), the number of haptic interaction signals with different waveforms included in the video screen of each frame in the video and the screen area corresponding to the waveform of each haptic signal may vary depending on the designer's different haptic interaction settings. Therefore, by calculating the number of haptic interaction signal waveforms in the target video screen, it is advantageous for later resource allocation (e.g., grayscale value) for each video screen area in the target video screen, and the haptic dimension information of the target video screen can be feedback-described by area.

[0028] Step S102: Generate a grayscale information representation diagram corresponding to the target video screen based on the number of waveforms of the haptic interaction signal, and the grayscale information representation diagram is for representing the distribution position of each video screen area on the target video screen.

[0029] In a specific implementation, the video production device performs the above steps S101-S102, and uses grayscale information to represent the corresponding regions (i.e., video screen regions) of the target video screen for haptic interaction signals of different waveforms, thereby effectively reducing the coding cost while meeting the needs of different haptic interaction settings in different regions of the screen. As can be seen, the above-mentioned regional haptic dimensional information representation method is equally suitable for representing dimensional information such as audio (e.g., mono / stereo), video (e.g., interaction screen), and olfactory.

[0030] As can be seen from the above steps, by using the grayscale information representation diagram to represent the distribution position of each video screen area on the target video screen in the image grayscale manner, the location areas on the target video screen of haptic interaction signals with different waveforms can be accurately described, which is advantageous for realizing different haptic interaction settings for different areas of the screen. Embodiment 1

[0031] This embodiment describes an example of counting the number of waveforms of haptic interaction signals on a target video screen.

[0032] Haptic interaction information corresponding to each pixel point on the target video screen is determined, and an area formed by a plurality of pixel points having a mapping relationship between corresponding haptic interaction information is determined as one video screen area, and the number of waveforms of the haptic interaction signal is determined based on the number of determined video screen areas.

[0033] Here, the area formed by a plurality of pixel points that do not correspond to haptic interaction information is a non-video screen area, and the mapping relationship is for determining the haptic interaction signal amplitude of the second pixel point based on the attribute information between the first pixel point and the second pixel point and the haptic interaction signal amplitude of the first pixel point.

[0034] As can be understood, after a designer sets different haptic interaction settings for different areas of a target video screen, the different areas will correspond to haptic interaction signals with different waveforms, and there is no mapping relationship between the haptic interaction signals with different waveforms. In this case, the haptic interaction information corresponding to each pixel point in the same area is obtained by mapping from the haptic interaction signal with the waveform corresponding to the area, and there is no mapping relationship between the haptic interaction information corresponding to two pixel points in different areas. That is, the mapping relationship can describe the equality relationship that should be satisfied between the attribute information (e.g., color information, brightness information, position information, etc.) of two pixel points in the same area and the haptic interaction signal amplitude.

[0035] Therefore, when the video production equipment counts the number of waveforms of the haptic interaction signal for the target video screen, it can first determine the haptic interaction information corresponding to each pixel point on the target video screen, and then detect whether there is a mapping relationship between the haptic interaction information corresponding to two pixel points (for example, the first pixel point A and the second pixel point B shown in FIG. 2), that is, whether the first pixel point A and the second pixel point B are represented by the formula

number

number

number

number

[0036] By dividing pixel points that have a mapping relationship between corresponding haptic interaction information into the same region, the number of divided regions can be reduced while ensuring the accuracy of representation of haptic dimension information for each region. By making each divided region a video screen region corresponding to each waveform of haptic signals, the number of waveforms of haptic interaction signals that correspond one-to-one to the video screen region can be determined according to the number of video screen regions.

[0037] For example, as shown in FIG. 3, according to the designer's haptic interaction settings, the target video screen may be divided into three video screen areas (i.e., video screen areas 1, 2, and 3 in FIG. 3), and other areas on the target video screen other than the three video screen areas are non-video screen areas for which the designer has not set haptic interaction settings (i.e., consisting of pixel points that do not correspond to haptic interaction information).

[0038] In one possible embodiment, for any two pixel points (e.g., a first pixel point and a second pixel point) in any video screen region, when the attribute information is position information and the mapping relationship is a linear relationship, the haptic interaction signal amplitude of the second pixel point at time t satisfies the following formula:

number

number

number

[0039] In this embodiment, if the linear relationship is a first-order linear relationship, the first set of parameter values ​​can be expressed by a second set of parameter values ​​for expressing the first-order linear relationship. For example, the first set of parameter values ​​and the second set of parameter values ​​corresponding to the first pixel point and the second pixel point can satisfy at least one of the following equations:

number

[0040] In the above embodiment 1, the division basis of different video screen areas is designed, which is advantageous to reduce the overlapping representation of position-related tactile interaction information. Embodiment 2

[0041] This embodiment is intended to illustrate an example of generating a grayscale information representation diagram.

[0042] A set of grayscale values ​​corresponding to the waveform of each haptic interaction signal is determined, and a grayscale information representation diagram corresponding to the target video screen is generated based on the video screen area to which each pixel point on the target video screen belongs and the set of grayscale values ​​corresponding to the waveform of each haptic interaction signal.

[0043] As can be seen, since different haptic interaction signal waveforms (i.e., different video screen areas) correspond to different grayscale value sets, for each pixel point on the target video screen, the grayscale value of the pixel point at the same position in the grayscale information representation diagram can be determined based on the grayscale value set corresponding to the video screen area in which the pixel point is located, so that the grayscale information in the generated grayscale information representation diagram can accurately represent the distribution position of each video screen area on the target video screen, thereby realizing haptic dimensional information representation for each target video screen area.

[0044] In one embodiment, a grayscale information representation diagram corresponding to the target video screen can be generated based on the haptic interaction index map corresponding to the target video screen and the grayscale value set corresponding to the waveform of each haptic interaction signal.

[0045] In this embodiment, an index value of each video screen area in the target video screen is determined based on the identifier of each waveform of each haptic interaction signal. A haptic interaction index map corresponding to the target video screen can be generated based on the index value of each video screen area in the target video screen and the video screen area to which each pixel point in the target video screen belongs. As shown in FIG. 4, the index values ​​(e.g., 0, 1, and 2) corresponding to the video screen area in which each pixel point in the target video screen is located can be determined as the pixel value of the pixel point at the same position in the haptic interaction index map. Then, the haptic interaction index map can be generated based on the index value of each video screen area in the target video screen and the video screen area to which each pixel point in the target video screen belongs. The pixel value of each pixel point in the map can be the grayscale value (i.e., the index value of the video screen area) of the pixel point at the same position in the grayscale information representation diagram, that is, the index value of each video screen area is determined as a set of grayscale values ​​corresponding to the waveform of the tactile interaction signal corresponding to each of the video screen areas, so that after determining the grayscale information corresponding to the tactile interaction operation based on the grayscale information representation diagram, the tactile interaction information corresponding to the tactile interaction operation can be determined based on the grayscale information from the tactile interaction information corresponding to each of the waveforms of the multiple tactile interaction signals corresponding to the target video screen.

[0046] In one possible embodiment, after generating the haptic interaction index map, the pixel values ​​of each pixel point in the haptic interaction index map can be converted into the grayscale values ​​of the pixel points at the same positions in the grayscale information representation diagram based on the grayscale value set corresponding to each pixel point in the haptic interaction index map, thereby realizing the grayscale representation of the distribution positions of each video screen area on the target video screen.

[0047] In a specific implementation, from the set of grayscale values ​​corresponding to the waveform of each haptic interaction signal, one grayscale value is determined as the grayscale value of the video screen area corresponding to the waveform of the haptic interaction signal. Further, a grayscale information representation diagram corresponding to the target video screen may be generated based on the haptic interaction index map corresponding to the target video screen and the grayscale values ​​corresponding to each video screen area in the target video screen, and in this case, the grayscale values ​​of pixel points corresponding to the same video screen area in the grayscale information representation diagram are the same. From the set of grayscale values ​​corresponding to the waveform of each haptic interaction signal, a plurality of grayscale values ​​are determined as the grayscale values ​​of the video screen area corresponding to the waveform of the haptic interaction signal. The haptic interaction index map corresponding to the target video screen and the feature information of each pixel point (e.g., position, haptic interaction information, etc.) are further combined to generate a grayscale information representation diagram, in which pixel points corresponding to the same video screen area in the grayscale information representation diagram have different grayscale values ​​according to different feature information. For example, if the set of grayscale values ​​corresponding to the video screen area includes grayscale values ​​0 and 1, the grayscale values ​​of the pixel points in the grayscale area corresponding to the video screen area in the grayscale information representation diagram are all 0 and 1, and the positions of each pixel point in the grayscale area with a grayscale value of 0 represent the position distribution of the edge area in the video screen area, and the positions of each pixel point in the grayscale area with a grayscale value of 1 represent the position distribution of the center area in the video screen area.

[0048] However, for each pixel point in the non-video screen area, the grayscale value of the pixel point at the same position in the grayscale information representation diagram of these pixel points can be set to a specific value or to a null value.

[0049] In one possible embodiment, determining a set of grayscale values ​​corresponding to the waveform of each haptic interaction signal includes: A-1: Dividing the entire set of grayscale values ​​according to the number of waveforms of the haptic interaction signal, and obtaining a set of grayscale values ​​corresponding to each waveform of the haptic interaction signal; Section A-2: Dividing the entire set of grayscale values ​​according to the attribute values ​​of the video screen areas corresponding to each waveform of each haptic interaction signal, to obtain a set of grayscale values ​​corresponding to each waveform of each haptic interaction signal; A-3: determining a different grayscale value from the entire set of grayscale values ​​for each waveform of the haptic interaction signal, and creating a set of grayscale values ​​corresponding to each waveform of the haptic interaction signal.

[0050] Regarding item A-1, the entire set of grayscale values ​​may be equally divided into the same number of sets of grayscale values ​​based on the number of waveforms of the haptic interaction signal to obtain a set of grayscale values ​​corresponding to each waveform of the haptic interaction signal.

[0051] Specifically, the range value of the grayscale value set corresponding to the waveform of the nth haptic interaction signal may be determined based on the following formula, and the grayscale value set corresponding to the waveform of the nth haptic interaction signal may be determined based on the range value of the grayscale value set:

number

[0052] where Max is the maximum grayscale value in the entire set of grayscale values, and r n is the range value of the grayscale value set corresponding to the waveform of the n-th haptic interaction signal,

number

[0053] In specific implementation, after obtaining the range value of the gray scale value set corresponding to the waveform of the n-th haptic interaction signal according to the above formula, the gray scale value set corresponding to the waveform of the n-th haptic interaction signal is

number

number

number

number

[0054] Regarding item A-2, the attribute value of a video screen area may be information such as the area occupancy rate of the video screen area on the target video screen. By dividing the entire set of grayscale values ​​according to the attribute value of each video screen area, the sets of grayscale values ​​corresponding to the waveforms of each haptic interaction signal can have different range values ​​(i.e., different magnitudes). For example, the set of grayscale values ​​corresponding to a video screen area with a larger area occupancy rate can have a larger range value. In this case, the magnitude of this range value can be used to represent the size of the area occupancy rate of the video screen area. This makes the divided sets of grayscale values ​​more flexible and can represent more dimensional information.

[0055] Regarding item A-3, based on the entire set of grayscale values, one grayscale value can be defined for each waveform of each haptic interaction signal to realize grayscale representation for the related video screen area distribution position, thereby simplifying the algorithm.

[0056] In the above embodiment 2, the grayscale information in the grayscale information representation diagram is used to represent the index values ​​of each video screen area in the target video screen, thereby achieving effective representation of tactile interaction information corresponding to each different video screen area. Embodiment 3

[0057] This embodiment describes an example of a haptic expression information stream.

[0058] Case: When generating a grayscale information representation diagram corresponding to each of a plurality of video screens in the target video, a tactile representation information stream corresponding to the target video is transmitted, and the tactile representation information stream corresponding to the target video includes at least each encoded tactile interaction information corresponding to each of the plurality of video screens and an encoded grayscale information representation diagram corresponding to each of the plurality of video screens, where the plurality of video screens include the target video screen.

[0059] In a specific implementation, the video production equipment may generate a grayscale information representation diagram for each video screen of the target video, and for each video screen after any video screen of the target video (e.g., a target video screen), determine each video screen area included in the video screen, and if each video screen area included in the video screen is different from each video screen area included in the target video screen, generate a grayscale information representation diagram corresponding to the video screen.

[0060] The video production device encodes each piece of haptic interaction information corresponding to each of the multiple video frames in the target video and each grayscale information representation diagram corresponding to each of the multiple video frames, and then obtains a haptic representation information stream. For example, the video production device can encode and compress each piece of haptic interaction information corresponding to each of the multiple video frames and each grayscale information representation diagram, and the binary file format generated after encoding and compression is the haptic representation information stream.

[0061] As can be understood, regional tactile interactions are usually set based on video screen content, but the video screen content of consecutive frames in the target video usually does not change, that is, the video screen area distribution of the video screen areas of consecutive frames does not change. Therefore, after generating a grayscale information representation diagram corresponding to the video screen of a certain frame, only if the video screen area distribution of the video screen after the video screen changes can a grayscale information representation diagram corresponding to the changed video screen be generated, thereby reducing the coding cost.

[0062] In one embodiment, a video playback device (the video playback device and the video production device can be configured in the same hardware device) acquires and decodes a haptic expression information stream corresponding to the target video, and obtains grayscale information expression diagrams corresponding to each of multiple video screens in the target video and encoded haptic interaction information corresponding to each of the multiple video screens.

[0063] wherein the grayscale information representation diagram corresponding to one video screen is for representing the distribution position of each video screen area on the video screen, and the waveform of each haptic interaction signal corresponding to one video screen corresponds to the haptic interaction information of each video screen area on the video screen; In response to detecting a haptic interaction operation on a target video screen among the multiple video screens (e.g., when the touch component detects a user touching the screen, it sends a corresponding request to the video playback device), the video playback device determines a grayscale information representation diagram corresponding to the target video screen from the grayscale information representation diagrams corresponding to the multiple video screens, determines grayscale information corresponding to the haptic interaction operation based on the grayscale information representation diagram corresponding to the target video screen and the video screen area where the haptic interaction operation is located (e.g., the screen played in the screen area touched by the user), determines haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of the multiple haptic interaction signal waveforms corresponding to the target video screen based on the grayscale information corresponding to the haptic interaction operation, and triggers a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation (e.g., play the corresponding haptic interaction information in the screen area touched by the user).

[0064] Case 2: The haptic expression information stream corresponding to the target video is Section B-1: A tactile interaction resolution determined based on size information of a unit tactile interaction area touched by a touch element to perform a tactile interaction operation; Section B-2: a haptic interaction index map corresponding to each of the plurality of video screens, the haptic interaction index map including a respective index value for each pixel point in one video screen; Section B-3: A mapping relationship corresponding to each video screen area included in each of the plurality of videos, the mapping relationship being for determining a haptic interaction signal amplitude of a second pixel point based on attribute information between a first pixel point and a second pixel point and a haptic interaction signal amplitude of the first pixel point; and B-4: the number of waveforms of the haptic interaction signal on the plurality of video screens.

[0065] Regarding item B-1, a unit tactile interaction area (e.g., an area touched by a single finger or a touch pen) describes the smallest area unit touched by a user performing a tactile interaction operation, and the size information of the unit tactile interaction area may be size information of an area that can be touched by a single finger or a touch pen. Considering that tactile interaction resolution is usually larger than physical image pixel resolution, when encoding a grayscale information representation diagram, the video production device first processes the grayscale information representation diagram as a tactile area grayscale representation diagram based on the tactile interaction resolution and then encodes it to reduce encoding costs, and can transmit the tactile interaction resolution and the encoded tactile area grayscale representation diagram to a video playback device. Here, the process of generating a tactile area grayscale representation diagram will be described in detail in embodiment 5, and will not be described here.

[0066] If the information obtained by the video playback device decoding the haptic expression information stream corresponding to the target video further includes haptic interaction resolution, and the multiple grayscale information representation diagrams obtained by decoding the haptic expression information stream corresponding to the target video are specifically haptic area grayscale representation diagrams, the video playback device can determine a unit haptic interaction area corresponding to the haptic interaction operation from the haptic area grayscale representation diagram corresponding to the target video screen (i.e., the video screen on which the haptic interaction operation is detected) based on the haptic interaction resolution. For example, the video playback device can process the haptic area grayscale representation diagram and the target video screen to have the same size based on the haptic interaction resolution, and further find a unit haptic interaction area located at the same position in the haptic area grayscale representation diagram based on the position of the haptic interaction operation in the target video screen, and further determine the grayscale information of the unit haptic interaction area corresponding to the haptic interaction operation as the grayscale information corresponding to the haptic interaction operation.

[0067] Regarding item B-2, the video production equipment transmits the tactile interaction index map corresponding to each of the plurality of video screens to the video playback equipment via the tactile expression information stream, and then the video playback equipment determines an index value corresponding to the tactile interaction operation based on the grayscale information corresponding to the tactile interaction operation and the tactile interaction index map of the target video screen (i.e., the video screen on which the tactile interaction operation is detected), and can determine the tactile interaction information corresponding to the tactile interaction operation from the tactile interaction information corresponding to each of the waveforms of the plurality of tactile interaction signals corresponding to the target video screen based on the index value corresponding to the tactile interaction operation.

[0068] Regarding item B-3, the video production equipment can encode and transmit haptic interaction information corresponding to any pixel point in any video screen area as haptic interaction information corresponding to the video screen area, and simultaneously transmit the mapping relationship corresponding to the video screen area, thereby reducing transmission overhead. Then, the video playback equipment can generate a haptic interaction signal corresponding to the haptic interaction operation based on the mapping relationship and haptic interaction information corresponding to the video screen area where the haptic interaction operation is located (e.g., generate a haptic interaction signal corresponding to each pixel point covered by the haptic interaction operation in the video screen area), and trigger a corresponding haptic actuator to output the haptic interaction signal corresponding to the haptic interaction operation.

[0069] For example, if a video screen area is designed to simulate the feeling of a stone being dropped into water, the video screen area may be characterized by a more intense tactile sensation (i.e., a larger tactile interaction signal amplitude) in the center area describing the stone being dropped, and a lighter tactile sensation (i.e., a smaller tactile interaction signal amplitude) in the edge area, and the wave effect also has a delay (i.e., the timing at which the edge area outputs the tactile interaction signal is later than the timing at which the center area outputs the tactile interaction signal), but the waveforms of the tactile interaction signals output by the center area and the edge area are consistent. That is, depending on the influence of the position factor (center / edge), significant / slight feedback can be fed back by changes in the amplitude of the waveform, and delay is expressed by Δt. Therefore, based on the mapping relationship that expresses the physical position relationship and delay, it is possible to obtain a haptic interaction signal corresponding to each pixel point in the video screen area from the haptic interaction information corresponding to the video screen area (for example, the amplitude of the haptic interaction signal at every time for the edge area pixel point can be obtained based on the amplitude of the haptic interaction signal at every time for the center area pixel point, or the amplitude of the haptic interaction signal at every time for the center area pixel point can be determined based on the amplitude of the haptic interaction signal at every time for the edge area pixel point, so Δt can be either a positive or negative number), thereby reducing redundant information expression.

[0070] In one possible embodiment, if the mapping relationship is a linear relationship, the linear relationship is expressed by a first set of parameter values ​​and a first parameter, and if the mapping relationship is a linear relationship, the first set of parameter values ​​is expressed by a second set of parameter values ​​for expressing the linear relationship, where the definitions of the first set of parameter values ​​and the second set of parameter values ​​have been described in detail in embodiment 1 and will not be further described here.

[0071] Regarding item B-4, after the video production equipment transmits the number of waveforms of the tactile interaction signal in the plurality of video screens to the video playback equipment via the tactile expression information stream, the video playback equipment can assign a touch actuator to each waveform of the tactile interaction signal in the plurality of video screens based on the number of waveforms of the tactile interaction signal in the plurality of video screens, thereby realizing rational assignment to the touch actuators, and triggering the touch actuator assigned to the waveform of the tactile interaction signal corresponding to the video screen area where the tactile interaction operation is located, and outputting tactile interaction information corresponding to the tactile interaction operation.

[0072] In the above embodiment 3, a combined technology scheme for representing and analyzing tactile dimensional information of multiple video screen regions is set up, which can meet the needs of different tactile interaction settings in different regions of the screen. Embodiment 4

[0073] This embodiment describes an example of grayscale information representation diagram coding.

[0074] The coded grayscale information representation diagrams corresponding to each of the plurality of video screens in the tactile representation information stream are: C-1: When the difference between each grayscale information representation in the grayscale information representation sequence consisting of the grayscale information representations corresponding to each of the plurality of video screens is lower than a preset value, adopt an image encoding method to encode the grayscale information representation sequence; Section C-2: Encoding the grayscale information representation diagram sequence using a video encoding method; Clause C-3: Determining sequence parameters based on edge information and / or a sequence bit rate corresponding to the grayscale information representation illustration sequence, and encoding the grayscale information representation illustration sequence based on the sequence parameters.

[0075] Regarding item C-1, if the difference between each grayscale information representation in the grayscale information representation sequence is lower than a predetermined value (for example, each grayscale information representation is the same), an image encoding method can be adopted to encode the grayscale information representation sequence.

[0076] Regarding item C-2, considering that the regions in the grayscale information representation diagram where the waveforms of the haptic interaction signals are located have characteristics of smooth interiors and sharp edges, the grayscale information representation diagram sequence can be encoded using a depth map encoding technique in 3D-HEVC (e.g., wedge encoding technique, contour division technique, etc.). For example, a file encapsulating a grayscale information representation diagram sequence encoded using a video sequence encoding method may include description information of the video sequence (i.e., the grayscale information representation diagram sequence), so that a player can learn the encoding method (e.g., MPEG-H HEVC Video) of the grayscale information representation diagram sequence and detailed information (e.g., detailed information about the video sequence such as resolution, refresh rate, and advanced video encoding standard) through the description information when analyzing the file.

[0077] Regarding item C-3, sequence parameters such as the QP value (the quantization parameter resulting from encoding by the encoder, jointly determined by the video resolution, input frame rate, and bit rate) used in the encoding process can be adjusted based on needs such as whether the edges of each grayscale information representation image in the grayscale information representation image sequence need to be sharpened and whether the bit rate of the grayscale information representation image sequence is required.

[0078] In the above embodiment 4, the grayscale information representation diagram sequence can be coded using the picture / video coding method, but the coding loss accuracy needs to be limited to prevent the decoding index from being disturbed after decoding. Embodiment 5

[0079] This embodiment describes an example of a tactile area grayscale representation.

[0080] A tactile area grayscale representation corresponding to each of the plurality of video screens is generated based on the haptic interaction resolution and the grayscale information representation corresponding to each of the plurality of video screens.

[0081] In this embodiment, considering that the haptic interaction resolution is usually larger than the physical pixel resolution, the generated grayscale information representation diagrams are each reduced based on the haptic interaction resolution to generate a tactile area grayscale representation diagram corresponding to each of the multiple video screens, and then the tactile area grayscale representation diagram corresponding to each of the multiple video screens is coded to obtain a coded grayscale information representation diagram corresponding to each of the multiple video screens, thereby reducing the coding granularity and reducing the coding cost.

[0082] In one embodiment, a grayscale representation of the tactile area corresponding to a single video screen may include: Subsequently, extending each edge of the grayscale information representation diagram corresponding to the video screen according to the haptic interaction resolution and the resolution of the grayscale information representation diagram corresponding to the video screen, so as to ensure that M unit haptic interaction areas of the same size (i.e., matching the haptic interaction resolution) can be divided without overlapping; Dividing the edge-extended grayscale information representation diagram into M unit tactile interaction regions without overlapping, where M is a positive integer greater than 1, based on the tactile interaction resolution; a step of determining grayscale information corresponding to each of the M unit tactile interaction areas based on the grayscale values ​​included in each of the M unit tactile interaction areas, where, for example, an average value of the grayscale values ​​included in each of the M unit tactile interaction areas can be determined as the grayscale information corresponding to each of the M unit tactile interaction areas; and generating a tactile area grayscale representation diagram corresponding to the video screen based on the grayscale information corresponding to each of the M unit tactile interaction areas, for example, by combining the grayscale information corresponding to each of the M unit tactile interaction areas based on their positional relationships in the grayscale information representation diagram of the M unit tactile interaction areas to obtain the tactile area grayscale representation diagram.

[0083] In one embodiment, the grayscale information corresponding to the m-th unit tactile interaction area among the M unit tactile interaction areas is: a step of calculating the grayscale values ​​in the m-th unit tactile interaction area to obtain one grayscale value corresponding to the m-th unit tactile interaction area, wherein the corresponding grayscale value may be the most frequent value or the average value of the grayscale values ​​obtained by the calculation; The step of determining one grayscale value corresponding to the m-th unit tactile interaction area as the grayscale information corresponding to the m-th unit tactile interaction area and realizing the reduction of the grayscale information representation diagram is followed.

[0084] In one embodiment, the step of expanding each edge of the grayscale information representation diagram corresponding to the video screen based on the haptic interaction resolution and the resolution of the grayscale information representation diagram corresponding to the video screen includes: The method includes determining a target resolution based on the haptic interaction resolution and a resolution of a grayscale information representation diagram corresponding to the video screen, and extending each edge of the grayscale information representation diagram corresponding to the video screen based on the target resolution.

[0085] In this embodiment, as shown in FIG. 5 , after determining the target resolution, the pixel points that need to be expanded in the grayscale information representation diagram can be determined. For example, the pixel points on the left, right, top and bottom edges of the grayscale information representation diagram are expanded so that the screen resolution of the grayscale information representation diagram matches the target resolution, and the expanded grayscale information representation diagram can be expanded to M tactile interaction resolutions.

number

[0086] In one possible embodiment, the target resolution can be determined according to the following formula:

number

[0087] where s w , s H are the width and height of the unit tactile interaction area, respectively, W and H are for expressing the resolution of the grayscale information representation diagram, and W new and H new is used to express the target resolution,

number

[0088] In the above embodiment 5, every time the video screen area on the video screen changes, a corresponding grayscale information representation diagram is generated, which can effectively reduce the coding cost; and the grayscale information representation diagram is scaled based on the tactile interaction resolution, which is advantageous for reducing the amount of grayscale diagram information for describing the tactile dimensional information in the tactile representation information stream.

[0089] In a second aspect, as shown in FIG. 6 , an embodiment of the present disclosure provides another haptic interaction information processing method, the method comprising: obtaining and decoding a haptic expression information stream corresponding to a target video, and obtaining grayscale information expression diagrams corresponding to each of a plurality of video screens in the target video and encoded haptic interaction information corresponding to each of the plurality of video screens, Step S201, where the grayscale information representation diagram corresponding to one video screen is for representing the distribution position of each video screen area on the video screen, and the waveform of each haptic interaction signal corresponding to one video screen corresponds to the haptic interaction information of each video screen area on the video screen; a step S202 of determining, in response to detecting a haptic interaction operation on a target video screen among the plurality of video screens, a grayscale information representation diagram corresponding to the target video screen from the grayscale information representation diagrams corresponding to the plurality of video screens; Step S203: determining grayscale information corresponding to the haptic interaction operation according to a grayscale information representation diagram corresponding to the target video screen and a video screen area where the haptic interaction operation is located; In step S204, based on the grayscale information corresponding to the haptic interaction operation, haptic interaction information corresponding to the haptic interaction operation is determined from haptic interaction information corresponding to each of the waveforms of a plurality of haptic interaction signals corresponding to the target video screen; and step S205 of triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation.

[0090] As can be seen from the above steps, by using the grayscale information representation diagram to represent the distribution position of each video screen area on the target video screen in the image grayscale manner, the location areas on the target video screen of haptic interaction signals with different waveforms can be accurately described, which is advantageous for realizing different haptic interaction settings for different areas of the screen.

[0091] In one possible embodiment, the information obtained by decoding the haptic expression information stream corresponding to the target video further includes a haptic interaction resolution determined based on size information of a unit haptic interaction area touched by the touch element when performing a haptic interaction operation; The plurality of grayscale information representation diagrams obtained by decoding the haptic representation information stream corresponding to the target video are specifically haptic area grayscale representation diagrams, and the haptic area grayscale representation diagrams are used to represent distribution positions corresponding to each video screen area on the video screen; determining grayscale information corresponding to the haptic interaction operation based on a grayscale information representation diagram corresponding to the target video screen and a video screen area corresponding to the haptic interaction operation; determining a unit haptic interaction area corresponding to the haptic interaction operation from a haptic area grayscale representation corresponding to the target video screen based on the haptic interaction resolution; and determining grayscale information of a unit tactile interaction region corresponding to the tactile interaction operation as grayscale information corresponding to the tactile interaction operation.

[0092] In one possible embodiment, the information obtained by decoding the haptic expression information stream corresponding to the target video further includes a haptic interaction index map corresponding to each of the plurality of video screens; determining haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of waveforms of haptic interaction signals corresponding to the target video screen based on grayscale information corresponding to the haptic interaction operation; determining an index value corresponding to the haptic interaction operation based on grayscale information corresponding to the haptic interaction operation and a haptic interaction index map of the target video screen; and determining tactile interaction information corresponding to the tactile interaction operation from tactile interaction information corresponding to each of a plurality of tactile interaction signal waveforms corresponding to the target video screen based on an index value corresponding to the tactile interaction operation.

[0093] In one possible embodiment, the information obtained by decoding the haptic expression information stream corresponding to the target video further includes a mapping relationship corresponding to each video screen area included in each of the plurality of videos, and the method includes: generating a haptic interaction signal corresponding to the haptic interaction operation according to a mapping relationship and haptic interaction information corresponding to a video screen region where the haptic interaction operation is located; and triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation, wherein triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation includes: Triggering a corresponding haptic actuator to output a haptic interaction signal corresponding to the haptic interaction operation.

[0094] In one possible embodiment, the information obtained by decoding the haptic expression information stream corresponding to the target video further includes the number of waveforms of the haptic interaction signal on the plurality of video screens, and the method includes: assigning a touch actuator to each of the haptic interaction signal waveforms on the plurality of video screens based on the number of haptic interaction signal waveforms on the plurality of video screens; triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation; triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation; The method includes triggering a touch actuator assigned to a waveform of a haptic interaction signal corresponding to a video screen area where a haptic interaction operation is located, and outputting haptic interaction information corresponding to the haptic interaction operation.

[0095] The haptic interaction information processing method according to the embodiment of the present disclosure may be executed by a haptic interaction information processing device. In the embodiment of the present disclosure, the haptic interaction information processing device according to the embodiment of the present disclosure will be described using an example in which the haptic interaction information processing device executes the haptic interaction information processing method.

[0096] In a third aspect, an embodiment of the present disclosure provides a haptic interaction information processing apparatus, and as shown in FIG. 7, the haptic interaction information processing apparatus 100 includes: a first statistics module 101 for counting the number of waveforms of haptic interaction signals on a target video screen, the target video screen being one frame video screen in the target video, and one waveform of haptic interaction signal corresponding to the haptic interaction information of one video screen area on the target video screen; and a first generation module 102 for generating a grayscale information representation diagram corresponding to the target video screen based on the number of waveforms of the haptic interaction signal, the grayscale information representation diagram for representing the distribution position of each video screen area on the target video screen.

[0097] Optionally, the first statistics module 101 a first statistical sub-module for determining haptic interaction information corresponding to each pixel point in the target video screen; a second statistical sub-module for determining an area formed by a plurality of pixel points having a mapping relationship between corresponding haptic interaction information as one video screen area, and an area formed by a plurality of pixel points not corresponding to haptic interaction information is a non-video screen area, and the mapping relationship is for determining a haptic interaction signal amplitude of the second pixel point according to attribute information between the first pixel point and the second pixel point and the haptic interaction signal amplitude of the first pixel point; and a second statistical sub-module for determining the number of waveforms of the haptic interaction signal based on the determined number of video screen areas.

[0098] Optionally, the attribute information is position information, the mapping relationship is a linear relationship, and the haptic interaction signal amplitude at time t of the second pixel point satisfies the following formula:

number

number

number

[0099] Optionally, the linear relationship is a first-order linear relationship, and the first set of parameter values ​​are represented by a second set of parameter values ​​for representing the first-order linear relationship.

[0100] Optionally, the first set of parameter values ​​and the second set of parameter values ​​satisfy at least one of the following equalities:

number

[0101] Optionally, the first generating module 102: a first generating sub-module for determining a set of grayscale values ​​corresponding to a waveform of each haptic interaction signal; and a second generating sub-module for generating a grayscale information representation diagram corresponding to the target video screen based on a video screen area to which each pixel point in the target video screen belongs and a grayscale value set corresponding to the waveform of each haptic interaction signal.

[0102] Optionally, the first generating sub-module: A first determination module is included that determines an index value for each of the video screen areas as a set of grayscale values ​​corresponding to a waveform of a haptic interaction signal corresponding to each of the video screen areas.

[0103] Optionally, the apparatus comprises: a second generating module for generating a haptic interaction index map corresponding to the target video screen based on the respective index values ​​of each video screen area in the target video screen and the video screen area to which each pixel point in the target video screen belongs; The second generating sub-module: and a third generating sub-module for generating a grayscale information representation diagram corresponding to the target video screen based on the haptic interaction index map corresponding to the target video screen and a grayscale value set corresponding to a waveform of each haptic interaction signal.

[0104] Optionally, the third generating sub-module: a second determination module for determining one grayscale value from the set of grayscale values ​​corresponding to each waveform of the haptic interaction signal as the grayscale value of the video screen area corresponding to the waveform of the haptic interaction signal; and a fourth generating sub-module for generating a grayscale information representation diagram corresponding to the target video screen based on a tactile interaction index map corresponding to the target video screen and a grayscale value corresponding to each video screen area on the target video screen.

[0105] Optionally, the first generating sub-module: a first set determination module for dividing the entire set of grayscale values ​​according to the number of waveforms of the haptic interaction signal, and obtaining a set of grayscale values ​​corresponding to each waveform of the haptic interaction signal; a second set determination module for dividing the entire set of grayscale values ​​according to the attribute values ​​of the video screen regions corresponding to each waveform of each haptic interaction signal, to obtain a set of grayscale values ​​corresponding to each waveform of the haptic interaction signal; and a third set determination module for determining, from the entire set of grayscale values, one different grayscale value for each waveform of the haptic interaction signal, thereby forming a grayscale value set corresponding to each waveform of the haptic interaction signal.

[0106] Optionally, the first set determination module: a first set determination sub-module for determining a range value of a grayscale value set corresponding to the waveform of the n-th haptic interaction signal according to the following formula, and determining a grayscale value set corresponding to the waveform of the n-th haptic interaction signal based on the range value of the grayscale value set;

number

number

[0107] Optionally, the apparatus comprises: a third determination module for determining, for each video screen after the target video screen in the target video, a respective video screen area included in the video screen; and a third generating module for generating a grayscale information representation diagram corresponding to each video screen area included in the video screen when the video screen area is different from the video screen area included in the target video screen.

[0108] Optionally, when generating a grayscale information representation diagram corresponding to each of a plurality of video screens in the target video, the apparatus further comprises: The device further includes a first transmitting module for transmitting a tactile expression information stream corresponding to the target video, the tactile expression information stream corresponding to the target video including at least each encoded tactile interaction information corresponding to each of the plurality of video screens and an encoded grayscale information expression diagram corresponding to each of the plurality of video screens, where the plurality of video screens include the target video screen.

[0109] Optionally, the first transmitting module: a first encoding module for encoding the grayscale information representation illustration sequence by adopting an image encoding method when a difference between each grayscale information representation illustration in the grayscale information representation illustration sequence corresponding to each of the plurality of video screens is lower than a preset value; a second encoding module for encoding the grayscale information representation diagram sequence using a video encoding method; and a third encoding module for determining sequence parameters based on edge information and / or a sequence bit rate corresponding to the grayscale information representation illustration sequence, and encoding the grayscale information representation illustration sequence based on the sequence parameters.

[0110] Optionally, the second encoding module comprises: It includes a first encoding sub-module for encoding the grayscale information representation diagram sequence by adopting depth map encoding technology in 3D-HEVC.

[0111] Optionally, the haptic representation information stream corresponding to the target video comprises: a tactile interaction resolution determined based on size information of a unit tactile interaction area touched by the touch element when performing a tactile interaction operation; a haptic interaction index map corresponding to each of the plurality of video screens, the haptic interaction index map including a respective index value for each pixel point in one of the video screens; a mapping relationship corresponding to each video screen area included in each of the plurality of videos, the mapping relationship being for determining a haptic interaction signal amplitude of a second pixel point based on attribute information between a first pixel point and a second pixel point and a haptic interaction signal amplitude of the first pixel point; and the number of waveforms of the haptic interaction signal on the plurality of video screens.

[0112] Optionally, if the mapping relationship is a linear relationship, the linear relationship is expressed by a first set of parameter values ​​and a first parameter.

[0113] Optionally, if the mapping relationship is a linear relationship, the first set of parameter values ​​is represented by a second set of parameter values ​​for representing the linear relationship.

[0114] Optionally, the first transmitting module: a fourth generation module for generating a tactile area grayscale representation diagram corresponding to each of the plurality of video screens based on a tactile interaction resolution and a grayscale information representation diagram corresponding to each of the plurality of video screens, wherein the tactile interaction resolution is determined based on size information of a unit tactile interaction area touched by a touch element performing a tactile interaction operation; and a fourth encoding module for encoding the tactile area grayscale representation diagram corresponding to each of the plurality of video screens to obtain an encoded grayscale information representation diagram corresponding to each of the plurality of video screens.

[0115] Optionally, the fourth generation module: a first expansion module for expanding each edge of the grayscale information representation diagram corresponding to the video screen based on the haptic interaction resolution and a resolution of the grayscale information representation diagram corresponding to the video screen; a first division module for dividing the grayscale information representation diagram after edge extension into M unit tactile interaction regions without overlapping, where M is a positive integer greater than 1, based on the tactile interaction resolution; a fourth determination module for determining grayscale information corresponding to each of the M unit tactile interaction areas based on the grayscale values ​​included in each of the M unit tactile interaction areas; and a fifth generating module for generating a tactile area grayscale representation corresponding to the video screen based on the grayscale information corresponding to each of the M unit tactile interaction areas.

[0116] Optionally, the fourth decision module: a first determination sub-module for calculating the grayscale values ​​in the m-th unit tactile interaction area to obtain one grayscale value corresponding to the m-th unit tactile interaction area; and a second determination sub-module for determining one grayscale value corresponding to the m-th unit tactile interaction area as grayscale information corresponding to the m-th unit tactile interaction area.

[0117] Optionally, the first expansion module comprises: a first extension sub-module for determining a target resolution based on the haptic interaction resolution and a resolution of a grayscale information representation diagram corresponding to the video screen; and a second expanding sub-module for expanding each edge of the grayscale information representation diagram corresponding to the video screen based on the target resolution.

[0118] Optionally, the first extension sub-module comprises: a fifth determining module for determining a target resolution according to the following formula:

number

number

[0119] The tactile interaction information processing device according to the embodiments of the present disclosure can realize various processes realized by the embodiments of the tactile interaction information processing method described in the first aspect and achieve the same technical effects, and in order to avoid duplication, they will not be further described here.

[0120] In a fourth aspect, an embodiment of the present disclosure provides another haptic interaction information processing apparatus, as shown in FIG. 8 , the haptic interaction information processing apparatus 200 includes: a first processing module 201 for obtaining and decoding a haptic expression information stream corresponding to a target video, and obtaining grayscale information representation diagrams corresponding to each of a plurality of video screens in the target video and encoded haptic interaction information corresponding to each of the plurality of video screens, wherein the grayscale information representation diagram corresponding to one video screen is for representing the distribution position of each video screen area in the video screen, and the waveform of each haptic interaction signal corresponding to one video screen corresponds to the haptic interaction information of each video screen area in the video screen; a second processing module 202 for determining, in response to detecting a haptic interaction operation on a target video screen in the plurality of video screens, a grayscale information representation diagram corresponding to the target video screen from the grayscale information representation diagrams corresponding to the plurality of video screens; a third processing module 203 for determining grayscale information corresponding to the haptic interaction operation according to a grayscale information representation diagram corresponding to the target video screen and a video screen area where the haptic interaction operation is located; a fourth processing module 204 for determining, based on the grayscale information corresponding to the haptic interaction operation, haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of haptic interaction signal waveforms corresponding to the target video screen; and a fifth processing module 205 for triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation.

[0121] Optionally, the information obtained by decoding the haptic expression information stream corresponding to the target video further includes a haptic interaction resolution determined based on size information of a unit haptic interaction area touched by the touch element when performing a haptic interaction operation; The plurality of grayscale information representation diagrams obtained by decoding the haptic representation information stream corresponding to the target video are specifically haptic area grayscale representation diagrams, and the haptic area grayscale representation diagrams are used to represent distribution positions corresponding to each video screen area on the video screen; The third processing module 203 a first processing sub-module for determining, based on the haptic interaction resolution, a unit haptic interaction area corresponding to the haptic interaction operation from a haptic area grayscale representation diagram corresponding to the target video screen; and a second processing sub-module for determining grayscale information of a unit tactile interaction area corresponding to the tactile interaction operation as grayscale information corresponding to the tactile interaction operation.

[0122] Optionally, the information obtained by decoding the haptic representation information stream corresponding to the target video further includes a haptic interaction index map corresponding to each of the plurality of video screens; The fourth processing module 204 a third processing sub-module for determining an index value corresponding to the haptic interaction operation according to grayscale information corresponding to the haptic interaction operation and a haptic interaction index map of the target video screen; and a fourth processing sub-module for determining, based on an index value corresponding to the haptic interaction operation, haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of haptic interaction signal waveforms corresponding to the target video screen.

[0123] Optionally, the information obtained by decoding the haptic representation information stream corresponding to the target video further includes a mapping relationship corresponding to each video screen area included in each of the plurality of videos, and the device: a sixth processing module for generating a haptic interaction signal corresponding to the haptic interaction operation according to a mapping relationship corresponding to the video screen area where the haptic interaction operation is located and haptic interaction information; a seventh processing module for triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation; and an eighth processing module for triggering a corresponding haptic actuator to output a haptic interaction signal corresponding to the haptic interaction operation.

[0124] Optionally, the information obtained by decoding the haptic expression information stream corresponding to the target video further includes the number of waveforms of the haptic interaction signal on the plurality of video screens, and the device: a ninth processing module for assigning a touch actuator to each waveform of the haptic interaction signal on the plurality of video screens based on a number of waveforms of the haptic interaction signal on the plurality of video screens; The eighth processing module is and a fifth processing sub-module for triggering a touch actuator assigned to a waveform of a haptic interaction signal corresponding to a video screen area where a haptic interaction operation is located, and outputting haptic interaction information corresponding to the haptic interaction operation.

[0125] The haptic interaction information processing device according to the embodiment of the present disclosure can realize various processes realized by the embodiment of the haptic interaction information processing method described in the second aspect and achieve the same technical effects, and in order to avoid duplication, it will not be described further here.

[0126] The above-described device is merely an example, and the units described as separate components may or may not be physically separated, and the units shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. The purpose of the solution of this embodiment can be realized by selecting some or all of the modules according to actual needs. Those skilled in the art can understand and implement it without any creative effort.

[0127] Each element of the present disclosure may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will understand that a microprocessor or digital signal processor (DSP) may be used to implement some or all of the functions of some or all of the elements of a computing device according to an embodiment of the present disclosure. The present disclosure may be implemented as an apparatus or device program (e.g., a computer program and computer program product) for executing some or all of the methods described herein. Such a program implementing the present disclosure may be stored on a computer-readable medium or may have the form of one or more signals. Such signals may be downloaded from an Internet site, provided on a carrier signal, or provided in any other form.

[0128] For example, FIG. 9 illustrates a computing device capable of implementing the methods of the present disclosure. The computing device typically includes a processor 1010 and a computer program product or computer-readable medium in the form of a memory 1020. The memory 1020 may be an electronic storage device such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. The memory 1020 includes a storage space 1030 for program code 1031 for executing any of the method steps described above. For example, the program code storage space 1030 may include program code 1031 for implementing each step of the method described above. These program codes may be read from or written to one or more computer program products. These computer program products include program code carriers such as a hard disk, a compact disc (CD), a memory card, or a floppy disk. Such computer program products are typically portable or fixed storage units, as shown in FIG. 10. The storage unit may have a similar configuration of storage segments, storage spaces, etc. as memory 1020 in the computing device of Figure 9. The program code may, for example, be compressed in an appropriate format. Typically, the storage unit includes computer-readable code 1031', i.e., processor-readable code such as 1010, which, when executed by the computing device, causes the computing device to perform the steps of the methods described above.

[0129] An embodiment of the present disclosure further provides a haptic interaction information processing device, which includes a processor and a memory, and the memory stores programs or instructions executable on the processor, and when the programs or instructions are executed by the processor, each process of the embodiment of the haptic interaction information processing method described above can be realized and the same technical effects can be achieved, and to avoid repetition, no further description will be given here.

[0130] An embodiment of the present disclosure further provides a readable storage medium, which stores a program or instruction, and when the program or instruction is executed by a processor, it realizes each process of the above-mentioned haptic interaction information processing method embodiment and achieves the same technical effect, and to avoid repetition, it will not be further described here.

[0131] Wherein, the processor is the processor in the terminal device described in the above embodiment. The readable storage medium includes a computer readable storage medium such as a computer read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0132] An embodiment of the present disclosure further provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor executes a program or instruction to realize each process of the embodiment of the above-mentioned haptic interaction information processing method, which can achieve the same technical effects, and to avoid repetition, will not be further described here.

[0133] It should be understood that a chip according to an embodiment of the present disclosure may also be referred to as a system level chip, a system chip, a chip system, or a system-on-chip chip.

[0134] An embodiment of the present disclosure further provides a computer program / program product, which is stored in a storage medium and can be executed by at least one processor to realize each process of the embodiment of the above-mentioned haptic interaction information processing method, thereby achieving the same technical effects. To avoid repetition, no further description will be given here.

[0135] An embodiment of the present disclosure provides a haptic interaction information processing system including a video production device for performing the steps of the haptic interaction information processing method described in the first aspect above, and a video playback device for performing the steps of the haptic interaction information processing method described in the second aspect above.

[0136] As used herein, the terms "comprise," "include," or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a set of elements includes not only those elements but also other elements not expressly listed or elements inherent in such process, method, article, or apparatus. Absent further limitations, an element defined by the phrase "comprises one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, the scope of the methods and apparatuses in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, but may include performing functions essentially simultaneously or in reverse order based on related functionality. For example, a described method may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to some examples may be combined in other examples.

[0137] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above examples can be realized by adding a required general-purpose hardware platform to software, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the essential or related technical contributions of the technical means of the present disclosure can be embodied in the form of a computer software product, which is stored in one storage medium (e.g., ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing one terminal (which may be a mobile phone, computer, server, air conditioner, network device, etc.) to execute the methods described in each example of the present disclosure.

[0138] Although examples of the present disclosure have been described above using the drawings, the present disclosure is not limited to the above-described embodiments, which are merely illustrative and not limiting, and a person skilled in the art can take many forms that fall within the scope of protection of the present disclosure under the teachings of the present disclosure without departing from the spirit of the present disclosure and the scope protected by the claims.

Claims

1. Counting the number of waveforms of haptic interaction signals in a target video screen, where the target video screen is one frame video screen in the target video, and one waveform of the haptic interaction signal corresponds to the haptic interaction information of one video screen area in the target video screen; A haptic interaction information processing method including generating a grayscale information representation diagram corresponding to the target video screen based on the number of waveforms of the haptic interaction signal, wherein the grayscale information representation diagram is for representing the distribution position of each video screen area on the target video screen.

2. Counting the number of waveforms of the haptic interaction signal on the target video screen determining haptic interaction information corresponding to each pixel point on the target video screen; determining an area formed by a plurality of pixel points having a mapping relationship between corresponding haptic interaction information as one video screen area, and an area formed by a plurality of pixel points not corresponding to haptic interaction information is a non-video screen area, and the mapping relationship is for determining a haptic interaction signal amplitude of the second pixel point according to attribute information between the first pixel point and the second pixel point and the haptic interaction signal amplitude of the first pixel point; and determining a number of waveforms of the haptic interaction signal based on the determined number of video screen areas.

3. The attribute information is position information, the mapping relationship is a linear relationship, and the haptic interaction signal amplitude of the second pixel point at time t satisfies the following formula: [Equation 1] where: [Equation 2] is the haptic interaction signal amplitude at time t of the second pixel point, [Equation 3] 3. The method of claim 2, wherein α is a haptic interaction signal amplitude at a time t+Δt of the first pixel point, α and Δt are a first set of parameter values ​​for expressing the linear relationship, and b is a first parameter for expressing the linear relationship.

4. 4. The method of claim 3, wherein the linear relationship is a first-order linear relationship, and the first set of parameter values ​​is represented by a second set of parameter values ​​for representing the first-order linear relationship.

5. The first set of parameter values ​​and the second set of parameter values ​​satisfy at least one of the following equations: [Equation 4] 5. The method of claim 4, wherein Δx is an offset amount between the first pixel point and the second pixel point in the x direction, Δy is an offset amount between the first pixel point and the second pixel point in the y direction, and c, d, e, f, g, and h are the second set of parameter values.

6. generating a grayscale information representation diagram corresponding to the target video screen based on a number of waveforms of the haptic interaction signal; determining a set of grayscale values ​​corresponding to a waveform of each haptic interaction signal; and generating a grayscale information representation diagram corresponding to the target video screen based on a video screen area to which each pixel point in the target video screen belongs and a set of grayscale values ​​corresponding to a waveform of each haptic interaction signal.

7. Determining a set of grayscale values ​​corresponding to a waveform of each haptic interaction signal includes: The method of claim 6 , comprising determining an index value for each of the video screen areas as a set of grayscale values ​​corresponding to a waveform of a haptic interaction signal corresponding to each of the video screen areas.

8. generating a haptic interaction index map corresponding to the target video screen based on the respective index values ​​of each video screen area in the target video screen and the video screen area to which each pixel point in the target video screen belongs; generating a grayscale information representation diagram corresponding to the target video screen based on a video screen area to which each pixel point in the target video screen belongs and a set of grayscale values ​​corresponding to a waveform of each haptic interaction signal; generating a grayscale information representation diagram corresponding to the target video screen based on a video screen area to which each pixel point in the target video screen belongs and a set of grayscale values ​​corresponding to a waveform of each haptic interaction signal, 7. The method of claim 6, further comprising generating a grayscale information representation diagram corresponding to the target video screen based on a haptic interaction index map corresponding to the target video screen and a set of grayscale values ​​corresponding to waveforms of each haptic interaction signal.

9. generating a grayscale information representation diagram corresponding to the target video screen based on a haptic interaction index map corresponding to the target video screen and a grayscale value set corresponding to a waveform of each haptic interaction signal, determining one grayscale value from the set of grayscale values ​​corresponding to each waveform of the haptic interaction signal as the grayscale value of the video screen area corresponding to the waveform of the haptic interaction signal; and generating a grayscale information representation diagram corresponding to the target video screen based on a haptic interaction index map corresponding to the target video screen and grayscale values ​​corresponding to each video screen region in the target video screen.

10. Determining a set of grayscale values ​​corresponding to a waveform of each haptic interaction signal includes: Dividing the entire set of grayscale values ​​according to the number of waveforms of the haptic interaction signal to obtain a set of grayscale values ​​corresponding to each waveform of the haptic interaction signal; Dividing the entire set of grayscale values ​​according to the attribute values ​​of the video screen areas corresponding to each of the waveforms of the haptic interaction signals to obtain a set of grayscale values ​​corresponding to each of the waveforms of the haptic interaction signals; and determining a different grayscale value for each waveform of the haptic interaction signal from the entire set of grayscale values, thereby obtaining a set of grayscale values ​​corresponding to each waveform of the haptic interaction signal.

11. Dividing the entire set of grayscale values ​​according to the number of waveforms of the haptic interaction signal to obtain a set of grayscale values ​​corresponding to each waveform of the haptic interaction signal; determining a range value of a set of grayscale values ​​corresponding to the waveform of the n-th haptic interaction signal based on the following formula; and determining a set of grayscale values ​​corresponding to the waveform of the n-th haptic interaction signal based on the range value of the set of grayscale values: [Equation 5] where Max is the maximum grayscale value in the entire set of grayscale values, and r ｎ is the range value of the grayscale value set corresponding to the waveform of the n-th haptic interaction signal, [Equation 6] 11. The method of claim 10, wherein: is a rounding function.

12. For each video screen after the target video screen in the target video, determining a respective video screen area included in that video screen; 12. The method of claim 1, further comprising generating a grayscale information representation diagram corresponding to each video screen area included in the video screen if the video screen area differs from the video screen area included in the target video screen.

13. When generating a grayscale information representation diagram corresponding to each of a plurality of video screens in the target video, the method includes: transmitting a haptic expression information stream corresponding to the target video, wherein the haptic expression information stream corresponding to the target video includes at least each encoded haptic interaction information corresponding to each of the plurality of video screens and an encoded grayscale information expression diagram corresponding to each of the plurality of video screens; 13. The method of claim 12, wherein the plurality of video screens includes the target video screen.

14. a coded grayscale information representation diagram corresponding to each of the plurality of video screens, If the difference between each grayscale information representation image in the grayscale information representation image sequence corresponding to each of the plurality of video screens is lower than a preset value, employing an image encoding method to encode the grayscale information representation image sequence; Or, encoding the grayscale information representation image sequence using a video encoding method; Or, The method according to claim 13, obtained by determining sequence parameters based on edge information and / or a sequence bit rate corresponding to the grayscale information representation illustration sequence, and encoding the grayscale information representation illustration sequence based on the sequence parameters.

15. employing a video encoding method to encode the grayscale information representation image sequence; The method of claim 14, including employing a depth map coding technique in 3D-HEVC to encode the grayscale information representation diagram sequence.

16. The haptic representation information stream corresponding to the target video is a tactile interaction resolution determined based on size information of a unit tactile interaction area touched by the touch element when performing a tactile interaction operation; a haptic interaction index map corresponding to each of the plurality of video screens, the haptic interaction index map including a respective index value for each pixel point in one of the video screens; a mapping relationship corresponding to each video screen area included in each of the plurality of videos, the mapping relationship being for determining a haptic interaction signal amplitude of a second pixel point based on attribute information between a first pixel point and a second pixel point and a haptic interaction signal amplitude of the first pixel point; and a number of waveforms of haptic interaction signals on the plurality of video screens.

17. The method of claim 16 , wherein when the attribute information is location information and the mapping relationship is a linear relationship, the linear relationship is expressed by a first set of parameter values ​​and a first parameter.

18. 18. The method of claim 17, wherein if the mapping relationship is a linear relationship, the first set of parameter values ​​is represented by a second set of parameter values ​​for representing the linear relationship.

19. a coded grayscale information representation diagram corresponding to each of the plurality of video screens, generating a tactile area grayscale representation diagram corresponding to each of the plurality of video screens based on a tactile interaction resolution and a grayscale information representation diagram corresponding to each of the plurality of video screens, wherein the tactile interaction resolution is determined based on size information of a unit tactile interaction area touched by a touch element performing a tactile interaction operation; and encoding a tactile area grayscale representation corresponding to each of the plurality of video screens to obtain an encoded grayscale information representation corresponding to each of the plurality of video screens.

20. The grayscale representation of the tactile area corresponding to a single video screen is shown below. expanding each edge of the grayscale information representation diagram corresponding to the video screen based on the haptic interaction resolution and a resolution of the grayscale information representation diagram corresponding to the video screen; Dividing the grayscale information representation diagram after edge extension into M unit tactile interaction regions without overlapping, where M is a positive integer greater than 1, based on the tactile interaction resolution; determining grayscale information corresponding to each of the M unit tactile interaction areas based on the grayscale values ​​included in each of the M unit tactile interaction areas; and generating a tactile area grayscale representation corresponding to the video screen based on grayscale information corresponding to each of the M unit tactile interaction areas.

21. The grayscale information corresponding to the m-th unit tactile interaction area among the M unit tactile interaction areas is Calculating the grayscale values ​​in the m-th unit tactile interaction area to obtain one grayscale value corresponding to the m-th unit tactile interaction area; and determining a grayscale value corresponding to the m-th unit tactile interaction area as the grayscale information corresponding to the m-th unit tactile interaction area.

22. The step of expanding each edge of the grayscale information representation diagram corresponding to the video screen based on the haptic interaction resolution and the resolution of the grayscale information representation diagram corresponding to the video screen includes: determining a target resolution based on the haptic interaction resolution and a resolution of a grayscale information representation diagram corresponding to the video screen; 21. The method of claim 20, further comprising: expanding each edge of the grayscale information representation diagram corresponding to the video screen based on the target resolution.

23. determining a target resolution based on the haptic interaction resolution and a resolution of a grayscale information representation diagram corresponding to the video screen; determining a target resolution according to the following formula: [Equation 7] Here, s ｗ , s Ｈ are the width and height of the unit tactile interaction area, respectively, W and H are for expressing the resolution of the grayscale information representation diagram, and W ｎｅｗ and H ｎｅｗ is used to express the target resolution, [Equation 8] 23. The method of claim 22, wherein: is a rounding operation.

24. Obtain and decode a haptic expression information stream corresponding to a target video, and obtain grayscale information representation diagrams corresponding to each of a plurality of video screens in the target video and encoded haptic interaction information corresponding to each of the plurality of video screens, wherein the grayscale information representation diagram corresponding to one video screen is for representing the distribution position of each video screen area in the video screen, and the waveform of each haptic interaction signal corresponding to one video screen corresponds to the haptic interaction information of each video screen area in the video screen; determining a grayscale information representation diagram corresponding to the target video screen from the grayscale information representation diagrams corresponding to the plurality of video screens in response to detecting a haptic interaction operation for the target video screen in the plurality of video screens; determining grayscale information corresponding to the haptic interaction operation based on a grayscale information representation diagram corresponding to the target video screen and a video screen area where the haptic interaction operation is located; determining haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of waveforms of haptic interaction signals corresponding to the target video screen based on grayscale information corresponding to the haptic interaction operation; and triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation.

25. The information obtained by decoding the haptic expression information stream corresponding to the target video further includes a haptic interaction resolution determined based on size information of a unit haptic interaction area touched by the touch element when performing a haptic interaction operation; The plurality of grayscale information representation diagrams obtained by decoding the haptic representation information stream corresponding to the target video are specifically haptic area grayscale representation diagrams, and the haptic area grayscale representation diagrams are used to represent distribution positions corresponding to each video screen area on the video screen; determining grayscale information corresponding to the haptic interaction operation based on a grayscale information representation diagram corresponding to the target video screen and a video screen area corresponding to the haptic interaction operation; determining a unit haptic interaction area corresponding to the haptic interaction operation from a haptic area grayscale representation corresponding to the target video screen based on the haptic interaction resolution; and determining grayscale information of a unit haptic interaction region corresponding to the haptic interaction operation as the grayscale information corresponding to the haptic interaction operation.

26. determining haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of haptic interaction signal waveforms corresponding to the target video screen based on the grayscale information corresponding to the haptic interaction operation, when each grayscale value in the grayscale information representation diagram corresponding to the target video screen is a respective index value of each video screen area in the target video screen; 25. The method of claim 24, further comprising determining haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of haptic interaction signal waveforms corresponding to the target video screen based on grayscale information corresponding to the haptic interaction operation.

27. the information obtained by decoding the haptic expression information stream corresponding to the target video further includes a haptic interaction index map corresponding to each of the plurality of video screens; determining haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of waveforms of haptic interaction signals corresponding to the target video screen based on grayscale information corresponding to the haptic interaction operation; determining an index value corresponding to the haptic interaction operation based on grayscale information corresponding to the haptic interaction operation and a haptic interaction index map of the target video screen; The method of claim 24, further comprising: determining haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of haptic interaction signal waveforms corresponding to the target video screen based on an index value corresponding to the haptic interaction operation.

28. The information obtained by decoding the haptic expression information stream corresponding to the target video further includes a mapping relationship corresponding to each video screen area included in each of the plurality of videos, and the method includes: generating a haptic interaction signal corresponding to the haptic interaction operation according to a mapping relationship and haptic interaction information corresponding to a video screen region where the haptic interaction operation is located; and triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation, wherein triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation includes:

25. The method of claim 24, comprising triggering a corresponding haptic actuator to output a haptic interaction signal corresponding to the haptic interaction operation.

29. The information obtained by decoding the haptic expression information stream corresponding to the target video further includes the number of waveforms of the haptic interaction signal on the plurality of video screens, and the method includes: assigning a touch actuator to each of the haptic interaction signal waveforms on the plurality of video screens based on the number of haptic interaction signal waveforms on the plurality of video screens; triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation; triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation; 25. The method of claim 24, further comprising triggering a touch actuator assigned to a waveform of a haptic interaction signal corresponding to a video screen region in which a haptic interaction operation is located, and outputting haptic interaction information corresponding to the haptic interaction operation.

30. a first statistics module for counting the number of waveforms of haptic interaction signals in a target video screen, the target video screen being one frame video screen in the target video, and one waveform of the haptic interaction signal corresponding to the haptic interaction information of one video screen area in the target video screen; A haptic interaction information processing device including a first generation module for generating a grayscale information representation diagram corresponding to the target video screen based on the number of waveforms of the haptic interaction signal, the grayscale information representation diagram for representing the distribution position of each video screen area on the target video screen.

31. a first processing module for acquiring and decoding a haptic expression information stream corresponding to a target video, and obtaining grayscale information representation diagrams corresponding to each of a plurality of video screens in the target video and encoded haptic interaction information corresponding to each of the plurality of video screens, wherein the grayscale information representation diagram corresponding to one video screen is for representing the distribution position of each video screen area in the video screen, and the waveform of each haptic interaction signal corresponding to one video screen corresponds to the haptic interaction information of each video screen area in the video screen; a second processing module for determining, in response to detecting a haptic interaction operation with respect to a target video screen in the plurality of video screens, a grayscale information representation diagram corresponding to the target video screen from the grayscale information representation diagrams corresponding to the plurality of video screens; a third processing module for determining grayscale information corresponding to the haptic interaction operation based on a grayscale information representation diagram corresponding to the target video screen and a video screen area where the haptic interaction operation is located; a fourth processing module for determining, based on the grayscale information corresponding to the haptic interaction operation, haptic interaction information corresponding to the haptic interaction operation from haptic interaction information corresponding to each of a plurality of waveforms of haptic interaction signals corresponding to the target video screen; a fifth processing module for triggering a corresponding haptic actuator to output haptic interaction information corresponding to the haptic interaction operation.

32. A haptic interaction information processing device comprising a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and when the program or instructions are executed by the processor, the device realizes the steps of the haptic interaction information processing method described in any one of claims 1 to 23.

33. A haptic interaction information processing device comprising a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and when the program or instructions are executed by the processor, the device realizes the steps of the haptic interaction information processing method described in any one of claims 24 to 29.

34. a memory having computer readable code stored therein; one or more processors; A computing device that, when the computer-readable code is executed by the one or more processors, performs the haptic interaction information processing method of any one of claims 1 to 23 or the haptic interaction information processing method of any one of claims 24 to 29.

35. A computer program comprising computer-readable code that, when executed on a computing device, causes the computing device to execute the haptic interaction information processing method of any one of claims 1 to 23 or the haptic interaction information processing method of any one of claims 24 to 29.

36. 36. A computer readable medium storing the computer program of claim 35.

Images