Method and apparatus for processing haptic signal, and related device

Abstract

The present application relates to the technical field of computers. Disclosed are a method and apparatus for processing a haptic signal, and a related device. The method for processing a haptic signal in the embodiments of the present application comprises: an electronic device acquiring first information; and the electronic device processing a haptic signal on the basis of the first information, wherein the first information comprises at least one of the following: threshold-related information, execution device-related information, and spatial position-related information.

Description

Method, apparatus and related device for processing haptic signal

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 202410887872.8, filed on July 03, 2024, the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application belongs to the technical field of computer, and particularly relates to a method, apparatus and related device for processing haptic signal. BACKGROUND

[0004] In some videos, there may be haptic signals, for example, there are haptic signals in 3D model files. In some related technologies, the haptic signals are often directly compressed and transmitted, which leads to poor processing performance of the haptic signals. SUMMARY

[0005] The embodiments of the present application provide a method, apparatus and related device for processing haptic signal, which can solve the problem of poor processing performance of the haptic signal.

[0006] In a first aspect, a method for processing haptic signal is provided, comprising:

[0007] obtaining first information by an electronic device;

[0008] processing the haptic signal based on the first information by the electronic device;

[0009] wherein the first information comprises at least one of the following:

[0010] threshold-related information, execution device-related information, and spatial position-related information.

[0011] In a second aspect, a device for processing haptic signal is provided, comprising:

[0012] an obtaining module configured to obtain first information;

[0013] a processing module configured to process the haptic signal based on the first information;

[0014] wherein the first information comprises at least one of the following:

[0015] threshold-related information, execution device-related information, and spatial position-related information.

[0016] In a third aspect, a device for processing haptic signal is provided, which is configured to perform the steps of the method for processing haptic signal provided in the embodiments of the present application.

[0017] In a fourth aspect, an electronic device is provided, which includes a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method for processing a haptic signal according to an embodiment of the present application.

[0018] In a fifth aspect, an electronic device is provided, which includes a processor and a communication interface, wherein the processor is configured to obtain first information, and process a haptic signal based on the first information, wherein the first information includes at least one of the following: threshold-related information, execution device-related information, and spatial position-related information.

[0019] In a sixth aspect, an electronic device is provided, which includes a memory configured to store video data, and a processing circuit configured to implement the steps of the method for processing a haptic signal according to an embodiment of the present application.

[0020] In a seventh aspect, a readable storage medium is provided, which stores a program or instructions, the program or instructions, when executed by a processor, implementing the steps of the method for processing a haptic signal according to an embodiment of the present application.

[0021] In an eighth aspect, a codec system is provided, which includes an encoding device and a decoding device, the encoding device being configured to implement the steps of the method for processing a haptic signal according to an embodiment of the present application.

[0022] In a ninth aspect, a chip is provided, which includes a processor and a communication interface, the communication interface being coupled to the processor, the processor being configured to execute a program or instructions, implementing the steps of the method for processing a haptic signal according to an embodiment of the present application.

[0023] In a tenth aspect, a computer program / program product is provided, which is stored in a storage medium, the computer program / program product being executed by at least one processor to implement the steps of the method for processing a haptic signal according to an embodiment of the present application.

[0024] In the embodiments of the present application, an electronic device obtains first information, and processes a haptic signal based on the first information, wherein the first information includes at least one of the following: threshold-related information, execution device-related information, and spatial position-related information. In this way, the haptic signal can be processed based on at least one of the threshold-related information, the execution device-related information, and the spatial position-related information, so as to improve the processing performance of the haptic signal. BRIEF DESCRIPTION OF DRAWINGS

[0025] FIG. 1 is a schematic diagram of a codec system according to an embodiment of the present application;

[0026] FIG. 2 is a structural diagram of an encoder according to an embodiment of the present application;

[0027] FIG. 3 is a structural diagram of a decoder according to an embodiment of the present application;

[0028] FIG. 4 is a schematic diagram of a method for accessing a tactile signal according to an embodiment of the present application;

[0029] FIG. 5 is a flowchart of a method for processing a tactile signal according to an embodiment of the present application;

[0030] FIG. 6 is a structural diagram of a device for processing a tactile signal according to an embodiment of the present application;

[0031] FIG. 7 is a structural diagram of an electronic device according to an embodiment of the present application;

[0032] FIG. 8 is a structural diagram of a terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0034] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" are usually a category, and are not limited to the number of objects, for example, the first object can be one or more. In addition, "or" in the present application means at least one of the connected objects. For example, the protection scope of "A or B" at least covers three schemes, namely, scheme one: including A and not including B; scheme two: including B and not including A; scheme three: including A and B. In addition, the terms "A and / or B", "at least one of A and B", "at least one of A or B" also at least cover the above three schemes, respectively. The character " / " generally represents that the objects before and after are in an "or" relationship.

[0035] FIG. 1 is a schematic diagram of a CODEC system 10 according to an embodiment of the present application. The technical solutions of the embodiments of the present application relate to coding (CODEC) of video data (including encoding or decoding). The video data includes original uncoded video, coded video, decoded (e.g., reconstructed) video, or syntax elements, etc.

[0036] As shown in FIG. 1, the coding system 10 includes a source device 100 that provides encoded video data to be decoded and displayed by a destination device 110. In particular, the source device 100 provides the video data to the destination device 110 via a communication medium 120. The source device 100 and the destination device 110 can comprise any of one or more of a desktop computer, a notebook computer (i.e., a laptop computer), a tablet computer, a set-top box, a mobile telephone, a wearable device (e.g., a smart watch or a wearable camera), a television, a camera, a display device, a vehicle head unit, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, a digital media player, a video gaming console, a video conferencing device, a video streaming device, a broadcast receiver device, a broadcast transmitter device, a spacecraft, an aircraft, a robot, a satellite, etc.

[0037] In the example of FIG. 1, the source device 100 includes a data source 101, a memory 102, an encoder 200, and an output interface 104. The destination device 110 includes an input interface 111, a decoder 300, a memory 113, and a display device 114. The source device 100 represents an example of a video encoding device, while the destination device 110 represents an example of a video decoding device. In other examples, the source device 100 and the destination device 110 can not include some of the components of FIG. 1, or can include other components not shown in FIG. 1. For example, the source device 100 can receive video data from an external data source, such as an external camera. Also, the destination device 110 can interface with an external display device, rather than include an integrated display device. As another example, the memory 102, the memory 113 can be external memory.

[0038] Although FIG. 1 depicts the source device 100 and the destination device 110 as separate devices, in some examples, the source device 100 and the destination device 110 can be integrated in one device. In such embodiments, the corresponding functions of the source device 100 and the destination device 110 can be implemented using the same hardware or software, or using separate hardware or software, or any combination thereof.

[0039] In some examples, the source device 100 and the destination device 110 can engage in one-way video transmission or two-way video transmission. If two-way video transmission, the source device 100 and the destination device 110 can operate in a substantially symmetrical manner, i.e., each of the source device 100 and the destination device 110 includes an encoder and a decoder.

[0040] Data source 101 represents a source of video data (i.e., raw, uncoded video data) and provides encoder 200 with successive pictures containing video data that encoder 200 encodes. Data source 101 of source device 100 can include a video capture device, such as a video camera, a video archive containing previously captured raw video, or a video feed interface to receive video from a video content provider. As another alternative, data source 101 can generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In these cases, encoder 200 encodes the captured, pre-captured, or computer-generated video data. Encoder 200 can rearrange the pictures from the received order (sometimes referred to as "display order") into the encoding order. Encoder 200 can generate a bitstream including encoded video data. Source device 100 can then output the encoded video data via output interface 104 onto a

[0041] Memory 102 of source device 100 and memory 113 of destination device 110 represent general purpose memories. In some examples, memory 102 can store raw video data from data source 101, and memory 113 can store decoded video data from decoder 300. Additionally or alternatively, memory 102, 113 can store software instructions executable by, e.g., encoder 200 and decoder 300, respectively. Although memory 102 and memory 113 are shown separately from, and as distinct from, encoder 200 and decoder 300 in this example, it will be understood that encoder 200 and decoder 300 can also each include an internal memory for a functionally similar or equivalent purpose. If encoder 200 and decoder 300 are deployed on the same hardware device, memory 102 and memory 113 can be one one memory. Furthermore, memory 102, 113 can store encoded video data that is output from encoder 200 and input to decoder 300, for example. In some examples, portions of memory 102, 113 can be allocated to one or more video buffers, e.g., to store raw, decoded, or encoded video data.

[0042] In some examples, source device 100 can output encoded data from output interface 104 to storage 113. Similarly, destination device 110 can access encoded data from storage 113 via input interface 111. Storage 113 or storage 102 can include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, Digital Versatile Discs (DVDs), Compact Discs Read-Only Memories (CD-ROMs), flash drives, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0043] Output interface 104 can include any type of medium or device capable of sending encoded video data from source device 100 to destination device 110. For example, output interface 104 can include a transmitter or a transceiver, e.g., an antenna, configured to transmit encoded video data from source device 100 directly to destination device 110 in real-time. The encoded video data can be modulated according to a communication standard and transmitted to destination device 110.

[0044] Communication medium 120 can include transient media, such as wireless broadcasts or wired networks transmitted over physical transmission lines. For example, communication medium 120 can include radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cable). Communication medium 120 can form a portion of a packet-based network, such as a local area network, a wide area network, or a global network, such as the Internet. Communication medium 120 can also be in a form of storage media, such as a hard drive, flash drive, compact disc, digital video disc, Blu-ray disc, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0045] In some embodiments, communication medium 120 can include routers, switches, base stations, or any other equipment that can be used to facilitate communication from source device 100 to destination device 110. For example, a server (not shown) can receive the encoded video from source device 100 and provide the encoded video data to destination device 110, e.g., via network transmission. The server can include a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery Over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Services (MBMS) or evolved MBMS (eMBMS) server, or a Network-attached storage (NAS) device, etc. The server can implement one or more HTTP streaming protocols, such as the MPEG Media Transport (MMT) protocol, the Dynamic Adaptive Streaming over HTTP (DASH) protocol, the HTTP Live Streaming (HLS) protocol, or the Real Time Streaming Protocol (RTSP), etc.

[0046] Destination device 110 can access the encoded video data from a server, e.g., through a wireless channel (e.g., Wi-Fi connection) or a wired connection (e.g., Digital subscriber line (DSL), a cable modem, etc.) for accessing encoded video data stored on the server.

[0047] Output interface 104 and input interface 111 can represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to the IEEE 802.11 standard or the IEEE 802.15 standard (e.g., ZigBee™), the Bluetooth standard, etc., or other physical components. In examples where output interface 104 and input interface 111 comprise wireless components, output interface 104 and input interface 111 can be configured to transfer data, such as encoded video data, according to a WIFI, Ethernet, cellular (such as fourth generation mobile communication technology (4G), Long Term Evolution (LTE), LTE Advanced, fifth generation mobile communication technology (5G), sixth generation mobile communication technology (6G), etc.), etc.

[0048] The techniques provided by this disclosure can be employed in applications such as one or more of video conferencing, over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions, digital video that is encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0049] Input interface 111 of destination device 110 receives an encoded video bitstream from communication medium 120. The encoded video bitstream can include syntax elements and coded data units (e.g., slices, pictures, groups of pictures, etc.) that, when decoded, reproduce the video data. Display device 114 displays the decoded video data to a user. Display device 114 can comprise a Cathode ray tube (CRT), a liquid-crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other type of display device.

[0050] The encoder 200 and the decoder 300 can be implemented as one or more of various processing circuitry, which can include one or more microprocessors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), discrete logic circuitry, hardware, or any combinations thereof. When the techniques are implemented partially in software, a device can store instructions for the software in a suitable, non- transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure.

[0051] The encoder 200 and the decoder 300 can process based on the following video coding standards: H.263, H.264, H.265 (also known as High Efficiency Video Coding (HEVC)), H.266 (also known as Versatile Video Coding (VVC)), Moving Picture Experts Group 2 (MPEG-2), MPEG-4, VP8, VP9, Alliance for Open Media Video 1 (AV1), Audio Video coding Standard 1 (AVS1), AVS2, AVS3, or a next generation video standard protocol, which are not limited in the embodiments of the present application.

[0052] Generally, the encoder 200 and the decoder 300 can perform block-based coding of pictures. The term “block” generally refers to a structure including data to be processed (e.g., encoded, decoded, or otherwise used in the encoding or decoding process). For example, a block can include a two-dimensional matrix of samples of luma or chroma data. For example, the encoder 200 and the decoder 300 can code video data represented in a YUV format.

[0053] Referring to FIG. 2, which is a structural diagram of an encoder 200 provided by an embodiment of the present application, the encoder 200 can be the encoder 200 in FIG. 1. In the example of FIG. 2, the encoder 200 includes a memory 201, a coding parameter determination unit 210, a residual generation unit 202, a transform processing unit 203, a quantization unit 204, a dequantization unit 205, an inverse transform processing unit 206, a reconstruction unit 207, a filter unit 208, a decoded picture buffer (DPB) 209, and an entropy encoding unit 220.

[0054] The memory 201 can store video data to be encoded, for example, the encoder 200 can receive video data from the data source 101 shown in FIG. 1 and store it. In some examples, the memory 201 can be on the same chip as other components of the encoder 200 (as shown in FIG. 2), or it can be independent of the chip on which the components are located.

[0055] The coding parameter determination unit 210 includes a mode selection unit 211, an inter prediction unit 212, and an intra prediction unit 213. The inter prediction unit 212 is configured to obtain a first prediction block of the current block using an inter prediction mode, the intra prediction unit 213 is configured to obtain a second prediction block of the current block using an intra prediction mode, and the mode selection unit 211 is configured to obtain a target prediction block according to the first prediction block and the second prediction block, and determine a final prediction mode. In addition, the coding parameter determination unit 210 can also include other functional units, such as a functional unit for determining the division manner of a coding unit (CU), a functional unit for determining the transform type of the residual data of the CU, or a functional unit for determining the quantization parameter of the residual data of the CU, etc.

[0056] For ease of description and understanding, the CU to be processed in the current picture is referred to as the current CU, and the image block to be processed in the current CU is referred to as the current block or the image block to be processed, for example, in encoding, it refers to the block currently being encoded; in decoding, it refers to the block currently being decoded.

[0057] The inter prediction unit 212 can include a motion estimation unit and a motion compensation unit. For inter prediction of the current block, the motion estimation unit can perform motion search to identify one or more matching reference blocks in one or more reference pictures (for example, one or more previously coded pictures stored in the DPB 209).

[0058] The motion estimation unit can form one or more motion vectors (MVs) of the position of the reference block in the reference picture relative to the position of the current block in the current picture. The motion compensation unit can obtain a predicted value of the precision indicated by the motion vector by interpolation.

[0059] The encoding parameter determination unit 210 can provide the target prediction block to the residual generation unit 202. The residual generation unit 202 receives original uncoded video data of the current block from the memory 201 and computes a residual between the current block and the target prediction block to obtain a residual block. In some examples, the functions of the residual generation unit 202 can be implemented using one or more subtractor circuits that perform binary subtraction.

[0060] As an example, the encoding parameter determination unit 210 can provide syntax elements representing the encoding parameters to the entropy encoding unit 220 for encoding. The encoding parameters include one or more of a partitioning of the CU, a final prediction mode, a transform type of the residual data of the CU, or a quantization parameter of the residual data of the CU.

[0061] The transform processing unit 203 can perform a transform on the residual block output by the residual generation unit 202 to obtain a transform coefficient block, which can include a discrete cosine transform (DCT), an integer transform, a directional transform, or a Karhunen-Loeve transform, among others. In some examples, the encoder 200 can not include the transform processing unit 203.

[0062] The quantization unit 204 can quantize the transform coefficients in the transform coefficient block according to a quantization parameter (QP) value associated with the current block to produce a quantized transform coefficient block.

[0063] The inverse quantization unit 205 and the inverse transform processing unit 206 can perform inverse quantization and inverse transform, respectively, on the transform coefficient block to obtain a reconstructed residual block. The reconstruction unit 207 can generate a reconstructed block corresponding to the current block based on the reconstructed residual block and the target prediction block generated by the encoding parameter determination unit 210.

[0064] The filter unit 208 can perform one or more filter operations on the reconstructed block. For example, the filter unit 208 can be a deblocking filter (DBF), an adaptive loop filter (ALF), a sample adaptive offset (SAO) filter, among others. In some examples, the encoder 200 can not include the filter unit 208.

[0065] The encoder 200 stores the reconstructed picture resulting from the reconstructed block in the DPB 209. For example, in examples where the operation of the filter unit 208 is not needed, the reconstruction unit 207 can store the reconstructed block to the DPB 209. In examples where the operation of the filter unit 208 is needed, the filter unit 208 can store the filtered reconstructed block to the DPB 209. The inter prediction unit 212 obtains the reconstructed picture from the DPB 209 to perform inter prediction for blocks of a subsequent picture to be encoded. In some examples, the DPB 209 can be replaced by other types of memory.

[0066] The entropy encoding unit 220 can entropy encode syntax elements of other components in the encoder 200 to output encoded video data. For example, the entropy encoding unit 220 can entropy encode quantized transform coefficient blocks from the quantization unit 204. As another example, the entropy encoding unit 220 can entropy encode syntax elements from the coding parameter determination unit 210, such as motion information for inter prediction or intra mode information for intra prediction.

[0067] It can be appreciated that the components of the encoder 200 described in FIG. 2 are only an example and do not constitute a limitation to the embodiments of the present application.

[0068] FIG. 3 is a structure diagram of a decoder 300 according to an embodiment of the present application. The decoder 300 can be the decoder 300 described in FIG. 1. In the example of FIG. 3, the decoder 300 includes a coded picture buffer (CPB) 301, an entropy decoding unit 302, a prediction processing unit 310, an inverse quantization unit 303, an inverse transform processing unit 304, a reconstruction unit 305, a filter unit 306, and a DPB 307.

[0069] The entropy decoding unit 302 can receive encoded video data from the CPB 301 and entropy decode the video data to obtain syntax elements indicative of coding parameters including one or more of a partitioning of a CU, a final prediction mode, a transform type of residual data of the CU, or a quantization parameter of residual data of the CU, etc.

[0070] In the case where the syntax elements include the final prediction mode, the prediction processing unit 310 obtains the final prediction mode. If the final prediction mode is an inter prediction mode, a prediction block of the current CU can be obtained by an inter prediction unit 311 of the prediction processing unit 310; if the final prediction mode is an intra prediction mode, a prediction block of the current CU can be obtained by an intra prediction unit 312 of the prediction processing unit 310. In some examples, the prediction processing unit 310 can further include units for performing prediction functions according to other prediction modes.

[0071] CPB 301 can obtain encoded video data from a communication medium 120 as illustrated in FIG. 1 and store it. The DPB 307 is used to store decoded pictures. Alternatively, the CPB 301 and the DPB 307 can be replaced by other types of memory and the application does not limit in this aspect. In some examples, the CPB 301 can be on the same chip as the other components of the decoder 300 (as illustrated), on a different chip from the other components of the decoder 300.

[0072] The decoder 300 can perform the reconstruction operation separately for each block. The entropy decoding unit 302 can entropy-decode syntax elements for quantized transform coefficients and transform information (e.g., QP or transform mode indication) to obtain quantized transform coefficients. The quantized transform coefficients can be de-quantized by the inverse quantization unit 303 to obtain a transform coefficient block including transform coefficients. The transform coefficient block can be inverse-transformed by the inverse transform processing unit 304 to generate a residual block corresponding to the current block, which is the inverse operation of the above-mentioned transform.

[0073] The reconstruction unit 305 can reconstruct the current block from the prediction block and the residual block. For example, the reconstruction unit 305 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the current block.

[0074] The filter unit 306 can perform one or more filter operations on the reconstructed block. For example, the filter unit 306 can be of the types described with reference to the filter unit 208, which are not repeated here. In some examples, the filter unit 306 can be skipped.

[0075] The decoder 300 can store the reconstructed picture resulting from the reconstructed block in the DPB 307. For example, in examples where the operation of the filter unit 306 is skipped, the reconstruction unit 305 can store the reconstructed block to the DPB 307. In examples where the operation of the filter unit 306 is performed, the filter unit 306 can store the filtered reconstructed block to the DPB 307. The decoder 300 can output decoded pictures (e.g., decoded video) from the DPB 307 for subsequent presentation to a display device, such as the display device 114 of FIG. 1.

[0076] Wherein the encoding end and the decoding end can be realized by software, hardware or combination thereof, when realized by hardware, the encoding end can be referred to as an encoding end device or a video encoding device, and the decoding end can be referred to as a decoding end device or a video decoding device.

[0077] In order for those skilled in the art to better understand the embodiments of the present application, the following is first described.

[0078] 1. The synthesis or rendering process of the scene.

[0079] The rendering engine can synthesize a scene according to a glTF (3D model file format) file, and can also render the scene according to the glTF file. Through the rendering of the haptic scene, the rendering engine can obtain a feedback haptic signal. The haptic signal is played by a haptic signal executor after encoding / transmission / decoding and other operations.

[0080] The way in which the engine obtains the glTF file includes: the server and the client interact with the glTF file through glTF state streaming, or obtain the glTF file through a program file stored on the server and the client.

[0081] 2. The Moving Pictures Experts Group (MPEG) has made a haptic extension to glTF (3D model file format);

[0082] The MPEG_haptic extension provides a method for accessing haptic data, and the MPEG_haptic includes a set of haptic objects (hapticObjects), as shown in Table 1. As shown in Table 2, the accessor attribute in the hapticObjects further points to a haptic data source.

[0083] Table 1

[0084] Table 2

[0085] The specific access method of the haptic data is shown in FIG. 4. The hapticObjects in the MPEG_haptic point to the accessor, and the accessor further points to a circular buffer object. The circular buffer object provides data position information for accessing the circular buffer, and reflects attribute information of the haptic data rendering through a reference to an MPEG_media object.

[0086] 3. The MPEG also makes a haptic material related extension (MPEG_haptic_material extension) to glTF;

[0087] The MPEG_haptic_material extension provides haptic properties for materials. The MPEG_haptic_material contains materials attributes. The MPEG_haptic_material is shown in Table 3, and the MPEG_haptic_material.material is shown in Table 4.

[0088] Table 3

[0089] The materials attribute points to a set of material objects, each of which points to a hapticObject in the MPEG_haptic that provides the data source through the haptic attribute, and indicates the relevant material haptic properties (such as stiffness, friction, etc.) and texture coordinate related information.

[0090] Table 4

[0091] 4. Interactivity extension in glTF (3D Model File Format);

[0092] glTF is used for efficient transmission and loading of 3D scenes and models by engines and applications. glTF defines an extensible publishing format, which simplifies the creation workflow and interactive services by enabling interoperable use of 3D content across the industry.

[0093] The Moving Picture Expert Group (MPEG) has extended glTF for interactivity, and has preliminarily designed the interactivity related standards. For haptic scenes, the interactivity related standards are mainly used to describe the interaction between avatars and haptic sources, based on which haptic signals are triggered and generated.

[0094] Interactivity is defined based on behavior, which includes trigger nodes (i.e., trigger objects) and action nodes (i.e., action objects). The trigger nodes include collision conditions, proximity conditions, and visibility conditions. The action nodes include haptic feedback actions (i.e., ACTION_SET_HAPTIC), activation nodes (ACTION_ACTIVATE), and the like. The ACTION_SET_HAPTIC is implemented based on a HapticActionNode object or a HapticActionMedia object.

[0095] The hapticObject attribute in the HapticActionNode (as shown in Table 5) points to the hapticObject in the MPEG_haptic, and the mediaIndex attribute in the HapticActionMedia (as shown in Table 6) further points to the accessor in the hapticObject to access the haptic data.

[0096] Table 5

[0097] Table 6

[0098] The object (Object) in the embodiments of the present application can include at least one of the following: a scene, a node, a mesh, a material, a vertex, a face, a primitive, and the like.

[0099] The method for processing a haptic signal, the device, and the related equipment provided by the embodiments of the present application will be described in detail in combination with some embodiments and application scenarios and the accompanying drawings.

[0100] Please refer to FIG. 5, which is a flowchart of a method for processing a haptic signal according to an embodiment of the present application. As shown in FIG. 5, the method includes the following steps:

[0101] In step 501, the electronic device acquires first information.

[0102] The first information includes at least one of the following:

[0103] Threshold-related information, execution device-related information, spatial position-related information.

[0104] The electronic device can be a rendering engine, a rendering device, a scene composition device, a scene composition engine, a presentation engine, or a presentation device. Alternatively, it is understood that the electronic device runs a rendering engine, a scene composition engine, or a presentation engine, and the rendering engine, the scene composition engine, or the presentation engine performs the steps in the above method.

[0105] The first information can be received from another device, or the electronic device can obtain predefined first information.

[0106] The threshold-related information is related to a threshold used to process the haptic signal, the execution device-related information is related to an execution device associated with the haptic signal, and the spatial position-related information is related to a spatial position associated with the haptic signal.

[0107] In the embodiments of the present application, the execution device can also be referred to as an actuator, and can be an execution element on a haptic device.

[0108] In the embodiments of the present application, the spatial position can be an absolute coordinate position, a relative coordinate position, a node, a mesh, a vertex, a surface, or a primitive, and the like.

[0109] In step 502, the electronic device processes the haptic signal based on the first information.

[0110] The haptic signal can be one or multiple haptic signals, such as one or more haptic signals associated with an execution device (also referred to as an actuator), or one or more haptic signals associated with a spatial position.

[0111] In the case of multiple haptic signals, the multiple haptic signals can be processed to support compression of the multiple haptic signals, thereby improving the compression degree of the haptic signal.

[0112] In the embodiments of the present application, the haptic signal can also be referred to as haptic data.

[0113] The processing of the haptic signal based on the first information can be pre-processing of the haptic signal based on the first information, or masking processing of the haptic signal based on the first information, or priority reduction processing of the haptic signal based on the first information, or discarding processing of the haptic signal based on the first information, etc.

[0114] In the embodiments of the present application, the haptic signal is processed based on at least one of the threshold value related information, the execution device related information, and the spatial position related information, so as to improve the processing performance of the haptic signal.

[0115] As an optional implementation, the execution device related information includes at least one of the following:

[0116] The related information of the first execution device and the related information of the second execution device, wherein the haptic signal associated with the first execution device has a masking effect on the haptic signal associated with the second execution device.

[0117] The spatial position related information includes at least one of the following:

[0118] The related information of the first spatial position and the related information of the second spatial position, wherein the haptic signal associated with the first spatial position has a masking effect on the haptic signal associated with the second spatial position.

[0119] The threshold value related information is masking effect related threshold value information.

[0120] The related information of the first execution device can be identification information or group information of the first execution device, etc.

[0121] The related information of the second execution device can be identification information or group information of the second execution device, etc.

[0122] The haptic signal associated with the first execution device has a masking effect on the haptic signal associated with the second execution device can be understood as that the haptic signal associated with the second execution device is masked by the haptic signal associated with the first execution device, and the haptic signal associated with the second execution device is affected by the masking effect of the haptic signal associated with the first execution device.

[0123] In some embodiments, the first execution device and the second execution device can be spatially adjacent execution devices.

[0124] In this implementation, at least one of the related information of the first execution device and the related information of the second execution device is included, so as to determine whether the haptic signal associated with the first execution device has a masking effect on the haptic signal associated with the second execution device.

[0125] The information about the first spatial position can be identification information or group information of the first spatial position.

[0126] The information about the second spatial position can be identification information or group information of the second spatial position.

[0127] The masking effect of the tactile signal associated with the first spatial position on the tactile signal associated with the second spatial position can be understood as that the tactile signal associated with the second spatial position is masked by the tactile signal associated with the first spatial position.

[0128] In some embodiments, the first spatial position and the second spatial position can be spatially adjacent spatial positions.

[0129] In this embodiment, the determination of whether the tactile signal associated with the first spatial position has a masking effect on the tactile signal associated with the second spatial position can be made by determining at least one of the information about the first spatial position and the information about the second spatial position.

[0130] The threshold information about the masking effect can be threshold information for determining whether the masking effect exists. Thus, the determination of whether the masking effect exists can be made by using the threshold information about the masking effect.

[0131] In the above embodiment, step 502 can be processing the tactile signal when it is determined that the masking effect exists, such as masking, discarding, or reducing the priority of the tactile signal associated with the second execution device when it is determined that the tactile signal associated with the first execution device has a masking effect on the tactile signal associated with the second execution device, or masking, discarding, or reducing the priority of the tactile signal associated with the second spatial position when it is determined that the tactile signal associated with the first spatial position has a masking effect on the tactile signal associated with the second spatial position, so as to improve the compression efficiency of the multi-channel tactile signal and save signal storage and transmission resources.

[0132] As an optional embodiment, the method further comprises:

[0133] determining that the masking effect exists when the first condition is met, the masking effect including that the tactile signal associated with the first execution device has a masking effect on the tactile signal associated with the second execution device, or that the tactile signal associated with the first spatial position has a masking effect on the tactile signal associated with the second spatial position;

[0134] The first condition includes at least one of the following:

[0135] the signal strength of the haptic signal associated with the first execution device is greater than or equal to the signal strength of the haptic signal associated with the second execution device;

[0136] the signal strength of the haptic signal associated with the first spatial position is greater than or equal to the signal strength of the haptic signal associated with the second spatial position;

[0137] the signal strength of the haptic signal associated with the first execution device or the first spatial position is greater than or equal to a first threshold value;

[0138] the signal strength of the haptic signal associated with the second execution device or the second spatial position is less than or equal to a second threshold value.

[0139] The masking effect is described in the above embodiments and will not be repeated here.

[0140] The first threshold value or the second threshold value can be a threshold value included in the threshold value related information.

[0141] In the above embodiments, step 502 can be processing the haptic signal in the case where it is determined that the masking effect exists, such as masking processing, discarding processing or priority reduction processing of the haptic signal associated with the second execution device in the case where it is determined that the haptic signal associated with the first execution device masks the haptic signal associated with the second execution device, so as to improve the compression performance of the multiple haptic signals. Alternatively, masking processing, discarding processing or priority reduction processing of the haptic signal associated with the second spatial position in the case where it is determined that the haptic signal associated with the first spatial position masks the haptic signal associated with the second spatial position, so as to improve the compression performance of the multiple haptic signals.

[0142] In the above embodiments, the masking effect is determined in the case where the first condition is met, so that the haptic signal with the masking effect can be processed, and the processing performance of the haptic signal is further improved.

[0143] Optionally, the second threshold value includes at least one of a first value, a second value and a reference value.

[0144] The signal strength of the haptic signal associated with the second execution device or the second spatial position being less than or equal to the second threshold value includes at least one of:

[0145] The normalized signal strength of the haptic signal associated with the second execution device or the second spatial position being less than or equal to the first value;

[0146] the signal strength of the haptic signal associated with the second execution device or the second spatial position is less than or equal to the reference value, the reference value being determined based on the signal strength of the haptic signal associated with the first execution device or the first spatial position and a third threshold value;

[0147] the signal strength of the haptic signal associated with the second execution device or the second spatial position is less than or equal to the second value.

[0148] The third threshold value can be a predefined threshold value or a threshold value indicated by the attribute information of the object obtained by the electronic device.

[0149] The reference value determined based on the signal strength of the haptic signal associated with the first execution device or the first spatial position and the third threshold value can be determined based on a mapping relationship between the signal strength, the third threshold value, and the reference value, or calculated based on the signal strength of the haptic signal associated with the first execution device or the first spatial position and the third threshold value using a preconfigured algorithm.

[0150] The normalization can be normalization based on the signal strength of the haptic signal associated with the first execution device or the first spatial position.

[0151] In the above embodiments, it can be determined whether there is a masking effect only based on the signal strength of the haptic signal associated with the second execution device or the second spatial position, so that the relevant information of the first execution device or the first spatial position does not need to be obtained to save computational overhead.

[0152] For example, the electronic device determines whether the signal associated with the second execution device or the second spatial position is masked based on the threshold value related information and the signal strength. The determination condition that the signal associated with the second execution device or the second spatial position is masked includes at least one of the following:

[0153] The signal strength associated with the first execution device or the first spatial position is greater than / less than or equal to the signal strength associated with the second execution device or the second spatial position. The difference between the signal strengths is greater than / less than or equal to a preset threshold value, which is indicated by the attribute information of the glTF object or predefined;

[0154] The signal strength associated with the first execution device or the first spatial position is greater than / less than or equal to a first threshold value, which is indicated by the attribute information of the glTF object or predefined;

[0155] The normalized signal strength associated with the second execution device or the second spatial position is less than / greater than or equal to a first value, which is indicated by the attribute information of the glTF object or predefined;

[0156] The signal strength of the second execution device / second spatial position association is less than / not greater than a reference value, which is determined based on the signal strength of the first execution device / first spatial position association and a third threshold value; wherein the third threshold value can be indicated by the attribute information of the glTF object or predefined.

[0157] The signal strength of the second execution device / second spatial position association is less than / not greater than a second value, which is indicated by the attribute information of the glTF object or predefined.

[0158] If the signal of the second execution device / second spatial position is masked, the signal can be discarded / prioritized.

[0159] As an optional implementation, the attribute information of the object obtained by the electronic device is used to indicate at least one of the following:

[0160] The identification information of the first execution device;

[0161] The information of the first spatial position;

[0162] The identification information of the second execution device;

[0163] The information of the second spatial position;

[0164] The information of the execution device group to which the first execution device belongs;

[0165] The information of the spatial position group to which the first spatial position belongs;

[0166] The information of the execution device group to which the second execution device belongs;

[0167] The information of the spatial position group to which the second spatial position belongs;

[0168] The association relationship between the first execution device and the second execution device;

[0169] The association relationship between the first spatial position and the second spatial position;

[0170] The association relationship between the execution device group to which the first execution device belongs and the execution device group to which the second execution device belongs;

[0171] The association relationship between the spatial position group to which the first spatial position belongs and the spatial position group to which the second spatial position belongs.

[0172] Among them, the attribute information of the object obtained by the electronic device can be the attribute information of the object received by the electronic device from other devices, or the attribute information of the object generated by the electronic device. And it can be the attribute information of one or more objects.

[0173] In some embodiments, the attribute information of the object can be stored by the storage device.

[0174] In some embodiments, the object can be a glTF object, in particular, a glTF object related to a group of spatially adjacent devices or a glTF object related to a spatially masked model.

[0175] The information of the first spatial position can be description information or identification information or group information of the first spatial position.

[0176] The information of the second spatial position can be description information or identification information or group information of the second spatial position.

[0177] The association between the first execution device and the second execution device can indicate whether the haptic signal associated with the first execution device has a masking effect on the haptic signal associated with the second execution device when determining the masking effect.

[0178] The association between the first spatial position and the second spatial position can indicate whether the haptic signal associated with the first spatial position has a masking effect on the haptic signal associated with the second spatial position when determining the masking effect.

[0179] The association between the group of execution devices to which the first execution device belongs and the group of execution devices to which the second execution device belongs can indicate whether the haptic signal associated with the execution devices in the first group of execution devices has a masking effect on the haptic signal associated with the execution devices in the second group of execution devices when determining the masking effect.

[0180] The association between the group of spatial positions to which the first spatial position belongs and the group of spatial positions to which the second spatial position belongs can indicate whether the haptic signal associated with the spatial positions in the first group of spatial positions has a masking effect on the haptic signal associated with the spatial positions in the second group of spatial positions when determining the masking effect.

[0181] In this embodiment, since the attribute information of the object includes the above-mentioned content, the information in the attribute information of the object can be directly used when determining the masking effect, thereby saving transmission overhead.

[0182] For example, the first execution device / first spatial position related information is indicated by the attribute information of the glTF object. For example, the corresponding device identification / position identification is directly indicated; it is indicated whether a certain execution device / spatial position can be used as the first execution device / first spatial position.

[0183] For example, the first execution device / first spatial position belongs to a first execution device group / first spatial position group, and the first execution device group / first spatial position group related information is indicated by the attribute information of the glTF object. For example, the attribute information of the glTF object directly indicates the identifier of the group of devices / the identifier of the group of positions corresponding to the first execution device group / first spatial position group.

[0184] For example, the second execution device / second spatial position related information is indicated by the attribute information of the glTF object. For example, the attribute information of the glTF object directly indicates the identifier of the corresponding device / the identifier of the corresponding position; or indicates whether the execution device / spatial position can be used as the second execution device / second spatial position.

[0185] For example, the second execution device / second spatial position belongs to a second execution device group / second spatial position group, and the second execution device group / second spatial position group related information is indicated by the attribute information of the glTF object. For example, the attribute information of the glTF object directly indicates the identifier of the group of devices / the identifier of the group of positions corresponding to the second execution device group / second spatial position group.

[0186] For example, there is an association between the first execution device / first spatial position / first execution device group / first spatial position group and the second execution device / second spatial position / second execution device group / second spatial position group, and the association is indicated by the glTF object explicitly or implicitly. For example, the second execution device / second spatial position / second execution device group / second spatial position group associated with the first execution device / first spatial position / first execution device group / first spatial position group is indicated by the child object of the glTF object to which the first execution device / first spatial position / first execution device group / first spatial position group belongs, or vice versa. For example, the first execution device / first spatial position / first execution device group / first spatial position group and the second execution device / second spatial position / second execution device group / second spatial position group have an association indicated by the glTF object.

[0187] Taking the above object as an example, the attribute information of the object can include but is not limited to the attribute information shown in at least one table:

[0188] Table 7, Description of MPEG_spatial_masking extension:

[0189] Table 8, Description of actuatorList object:

[0190] In the above table 8, the execution device can be the first execution device or the second execution device.

[0191] Table 9, Description of MPEG_spatial_masking extension

[0192] Table 10, Table x: Description of maskingModel object option 1

[0193] Wherein, the reference execution device in the above table 10 can be the first execution device, and the masking execution device is the second execution device.

[0194] Table 11, Table x: Description of maskingModel object option 2

[0195] Wherein, the reference execution device in the above table 11 can be the first execution device, and the masking execution device is the second execution device.

[0196] Table 12, Description of hapticActuator object

[0197] Wherein, the masking execution device in the above table 12 is the first execution device.

[0198] Table 13, Description of maskingModel object

[0199] Wherein, the masking execution device in the above table 13 is the second execution device.

[0200] It should be noted that the above table is only an example of the attribute information of the object, and the attribute information of the object in the embodiment of the present application is not limited.

[0201] Optionally, in the case that the attribute information of the object indicates the identification information of the execution device, the attribute information of the object further indicates at least one of the mapping position information and the parameter information of the execution device; or,

[0202] In a case where the attribute information of the object indicates the identification information of the execution device, the attribute information of the object associated with the object is used to indicate at least one of the mapping position information and the parameter information of the execution device.

[0203] The parameter information can be hardware parameter information, and can include at least one of:

[0204] maximum frequency, minimum frequency, resonance frequency, maximum amplitude, impedance, maximum voltage, maximum current, maximum displacement, weight, size, type.

[0205] The mapping position information can include at least one of:

[0206] body part mask, vertices, body part target, actuator target.

[0207] The object associated with the object can be a parent-child relationship object of the object.

[0208] In this embodiment, at least one of the mapping position information and the parameter information of the execution device can be indicated in the attribute information of one object, without introducing additional information, thereby saving transmission overhead; and at least one of the mapping position information and the parameter information of the execution device can be indicated by the attribute information of the associated object, thereby avoiding excessive attribute information data and flexibly indicating the mapping position information and the parameter information of the execution device.

[0209] As an optional embodiment, the threshold-related information is predefined, or the threshold-related information is indicated by the attribute information of the object acquired by the electronic device.

[0210] The predefined can be a protocol predefined or an electronic device predefined.

[0211] The attribute information of the object acquired by the electronic device is described in the corresponding description of the above embodiments, which is not repeated here.

[0212] In this embodiment, the threshold-related information can be predefined or indicated by the attribute information of the object, so as to support multiple ways of obtaining the threshold-related information.

[0213] Optionally, the electronic device obtains the first information by:

[0214] The electronic device obtains the threshold-related information corresponding to the at least one frequency range by a predefined or an indication of the attribute information of the object.

[0215] The threshold-related information corresponding to the at least one frequency range can be understood as that the indication of the attribute information of the object has the threshold-related information corresponding to the at least one frequency range.

[0216] The threshold-related information corresponding to the at least one frequency range can be understood as that different frequency ranges have corresponding threshold-related information.

[0217] In this embodiment, the threshold-related information can be configured in the granularity of the frequency range, so as to improve the accuracy of the determination of the masking effect.

[0218] Optionally, the method further includes:

[0219] The signal intensity of the at least one frequency range of the haptic signal associated with the at least one execution device or the spatial position and the corresponding threshold-related information are used to determine whether the masking effect exists.

[0220] The masking effect is described in the corresponding description of the above embodiments, which is not described herein.

[0221] The determination of whether the masking effect exists based on the signal intensity of the at least one frequency range of the haptic signal associated with the at least one execution device or the spatial position and the corresponding threshold-related information can include:

[0222] The haptic signal associated with the at least one execution device or the spatial position is transformed into a frequency domain signal to obtain the signal intensity of at least one frequency range, and then the signal intensity of the at least one frequency range and the corresponding threshold-related information are used to determine whether the masking effect exists. For example, the signal intensity, the average signal intensity, or the maximum signal intensity in the corresponding frequency range is compared with the corresponding threshold to determine whether the masking effect exists. For multiple frequency ranges, the signal intensity, the average signal intensity, or the maximum signal intensity in each frequency range is compared with the threshold of the corresponding frequency range to determine whether the masking effect exists.

[0223] In this embodiment, the masking effect can be determined in the granularity of the frequency range, so as to improve the accuracy of the determination of the masking effect.

[0224] For example, the attribute information of the glTF object indicates / defines a frequency range and a threshold value corresponding to the frequency range by the protocol. When making the masking effect determination, the signal is transformed into a frequency domain signal, and the signal strength / average signal strength / maximum signal strength in the corresponding frequency range is compared with the threshold value.

[0225] For example, the attribute information of the glTF object indicates / defines a frequency range and a threshold value corresponding to the frequency range by the protocol. When making the masking effect determination, the signal is transformed into a frequency domain signal, and the signal strength / average signal strength / maximum signal strength in the corresponding frequency range is compared with the threshold value.

[0226] As an optional implementation, the threshold value related information has a corresponding relationship with the execution device or the spatial position, and the threshold value related information satisfies at least one of the following conditions:

[0227] One threshold value related information corresponds to at least one execution device;

[0228] One threshold value related information corresponds to at least one spatial position;

[0229] One threshold value related information corresponds to at least one execution device group;

[0230] One threshold value related information corresponds to at least one spatial position group;

[0231] One threshold value related information corresponds to a group of interrelated execution devices;

[0232] One threshold value related information corresponds to a group of interrelated spatial positions;

[0233] One threshold value related information corresponds to two groups of interrelated execution devices;

[0234] One threshold value related information corresponds to two groups of interrelated spatial positions;

[0235] One threshold value related information corresponds to one execution device, and corresponds to an execution device group associated with the one execution device;

[0236] One threshold value related information corresponds to one spatial position, and corresponds to a spatial position group associated with the one spatial position.

[0237] Among them, one threshold value related information corresponding to at least one execution device can realize the configuration of threshold value related information with execution device as granularity.

[0238] One threshold value related information corresponding to at least one spatial position can realize the configuration of threshold value related information with spatial position as granularity.

[0239] One threshold-related information corresponds to at least one group of execution devices, which can realize the configuration of threshold-related information at the granularity of a group of execution devices.

[0240] One threshold-related information corresponds to at least one group of spatial positions, which can realize the configuration of threshold-related information at the granularity of a group of spatial positions.

[0241] One threshold-related information corresponds to a group of interrelated execution devices, which can realize the configuration of threshold-related information at the granularity of interrelated execution devices, such as configuring threshold-related information for a first execution device and a second execution device.

[0242] One threshold-related information corresponds to a group of interrelated spatial positions, which can realize the configuration of threshold-related information at the granularity of interrelated spatial positions, such as configuring threshold-related information for a first spatial position and a second spatial position.

[0243] One threshold-related information corresponds to two groups of interrelated execution devices, which can realize the configuration of threshold-related information at the granularity of two groups of interrelated execution devices, such as configuring threshold-related information for a first group of execution devices and a second group of execution devices.

[0244] One threshold-related information corresponds to two groups of interrelated spatial positions, which can realize the configuration of threshold-related information at the granularity of two groups of interrelated spatial positions, such as configuring threshold-related information for a first group of spatial positions and a second group of spatial positions.

[0245] One threshold-related information corresponds to one execution device and a group of execution devices associated with the one execution device, which can realize the configuration of threshold-related information for one execution device and one group of execution devices, such as configuring threshold-related information for a first spatial position and a second group of spatial positions.

[0246] One threshold-related information corresponds to one spatial position and a group of spatial positions associated with the one spatial position, which can realize the configuration of threshold-related information for one spatial position and one group of spatial positions, such as configuring threshold-related information for a first spatial position and a second group of spatial positions.

[0247] In the above embodiments, threshold-related information can be configured at different granularities to improve the accuracy of masking effect judgment.

[0248] For example, the above threshold-related information is indicated / defined for interrelated first execution device / first spatial position and second execution device / second spatial position.

[0249] For example, the above threshold-related information is indicated / defined for interrelated first execution device / first spatial position and second group of execution devices / second group of spatial positions. The advantage of indicating threshold-related information for a group of execution devices / a group of spatial positions is to save signaling overhead.

[0250] For example, the threshold-related information is indicated / defined for a first execution device group / first spatial position group and a second execution device / second spatial position.

[0251] For example, the threshold-related information is indicated / defined for a first execution device group / first spatial position group and a second execution device group / second spatial position group.

[0252] For example, the threshold-related information is indicated / defined for a first execution device / first spatial position.

[0253] For example, the threshold-related information is indicated / defined for a second execution device / second spatial position.

[0254] In some embodiments, the threshold-related information and the corresponding first execution device (group) / first spatial position (group) / second execution device (group) / second spatial position (group) can be indicated by attribute information of the same object, so as to reflect that the threshold-related information and the device / position information are in a corresponding relationship. The threshold-related information and the first execution device (group) / first spatial position (group) / second execution device (group) / second spatial position (group) can also be parent-child objects, since the parent-child objects respectively indicate the threshold information and the device / position information, so as to reflect that the threshold-related information and the device / position information are in a corresponding relationship.

[0255] As an optional implementation, the processing of the haptic signal based on the first information comprises at least one of the following:

[0256] In a case where it is determined based on the first information that there is a masking effect, the haptic signal associated with the second execution device is processed, and the masking effect comprises that the haptic signal associated with the second execution device is masked by the haptic signal associated with the first execution device.

[0257] In a case where it is determined based on the first information that there is a masking effect, the haptic signal associated with the second spatial position is processed, and the masking effect comprises that the haptic signal associated with the second spatial position is masked by the haptic signal associated with the first spatial position.

[0258] The masking effect is described in the corresponding description of the above embodiments, which is not repeated here.

[0259] The processing of the haptic signal associated with the second execution device can be masking processing, discarding processing, or priority reduction processing of the haptic signal associated with the second execution device, so as to improve the compression performance of the multi-channel haptic signal.

[0260] The processing of the haptic signal associated with the second spatial position can be masking processing, discarding processing or priority reduction processing of the haptic signal associated with the second execution device, so as to improve the compression performance of the multi-path haptic signal.

[0261] Optionally, the method further comprises at least one of the following:

[0262] The electronic device selects at least one of the first execution device and the second execution device in a target execution device group.

[0263] The electronic device selects at least one of the first spatial position and the second spatial position in a target spatial position group.

[0264] The target execution device group includes the first execution device and the second execution device, or includes a first execution device group in which the first execution device is located and a second execution device group in which the second execution device is located.

[0265] In some embodiments, the target execution device group can be a spatially adjacent execution device group.

[0266] The target spatial position group includes the first spatial position and the second spatial position, or includes a first spatial position group in which the first spatial position is located and a second spatial position group in which the second spatial position is located.

[0267] In some embodiments, the target spatial position group can be a spatially adjacent spatial position group.

[0268] By selecting at least one of the first execution device and the second execution device in the target execution device group, only the haptic signal associated with at least one of the first execution device and the second execution device in the target execution device group needs to be processed, thereby saving overhead.

[0269] By selecting at least one of the first spatial position and the second spatial position in the target spatial position group, only the haptic signal associated with at least one of the first spatial position and the second spatial position in the target spatial position group needs to be processed, thereby saving overhead.

[0270] Optionally, the electronic device selects at least one of the first execution device and the second execution device in a target execution device group, comprising:

[0271] The electronic device selects the first execution device in a target execution device group; or, the electronic device selects the first execution device in the target execution device group and determines a second execution device corresponding to the first execution device according to the first execution device; or, the electronic device selects the second execution device in the target execution device group; or, the electronic device selects the second execution device in the target execution device group and determines the first execution device corresponding to the second execution device according to the second execution device.

[0272] Optionally, the electronic device selects at least one of the first spatial position and the second spatial position in a target spatial position group, comprising:

[0273] The electronic device selects the first spatial position in a target spatial position group; or, the electronic device selects the first spatial position in the target spatial position group and determines a second spatial position corresponding to the first spatial position according to the first spatial position; or, the electronic device selects the second spatial position in the target spatial position group; or, the electronic device selects the second spatial position in the target spatial position group and determines the first spatial position corresponding to the second spatial position according to the second spatial position.

[0274] The electronic device selects the first execution device in a target execution device group can realize judging whether the above masking effect exists based on the haptic signal of the first execution device, without needing the signal strength of the haptic signal associated with the second execution device, thereby saving the calculation overhead.

[0275] The electronic device selects the second execution device in the target execution device group can realize judging whether the above masking effect exists based on the haptic signal of the second execution device, without needing the signal strength of the haptic signal associated with the first execution device, thereby saving the calculation overhead.

[0276] The electronic device selects the first spatial position in a target spatial position group can realize judging whether the above masking effect exists based on the haptic signal of the first spatial position, without needing the signal strength of the haptic signal associated with the second spatial position, thereby saving the calculation overhead.

[0277] The electronic device selects the second spatial position in the target spatial position group can realize judging whether the above masking effect exists based on the haptic signal of the second spatial position, without needing the signal strength of the haptic signal associated with the first spatial position, thereby saving the calculation overhead.

[0278] In the presence of multiple first execution devices, the multiple first execution devices can be selected in a preset order, such as selecting the first execution devices in descending order of signal strength associated with the first execution devices, or selecting the first execution devices in descending order of the number of second execution devices associated with the first execution devices.

[0279] In the presence of multiple first spatial positions, the multiple first spatial positions can be selected in a preset order, such as selecting the first spatial positions in descending order of signal strength associated with the first spatial positions, or selecting the first spatial positions in descending order of the number of second spatial positions associated with the first spatial positions.

[0280] In the above embodiments, only the haptic signals associated with the second execution devices need to be processed for the target execution device group, thereby saving resources. Only the haptic signals associated with the second spatial positions need to be processed for the target spatial position group, thereby saving resources.

[0281] In some embodiments, only the haptic signals associated with the first execution devices need to be processed for the target execution device group, thereby saving resources. Only the haptic signals associated with the first spatial positions need to be processed for the target spatial position group, thereby saving resources.

[0282] For example, the electronic device obtains information related to the target execution device group / target spatial position group, and the electronic device only needs to process the haptic signals associated with the first execution devices / first spatial positions / second execution devices / second spatial positions in the target execution device group / target spatial position group.

[0283] For another example, the electronic device selects the first execution devices / first spatial positions from the target execution device group / target spatial position group, and performs masking condition determination and processing on the signals on the second execution devices / second spatial positions associated with the first execution devices / first spatial positions. For example, the electronic device selects the first execution devices / first spatial positions in a preset order, and performs masking condition determination and processing on the signals on the second execution devices / second spatial positions associated with the first execution devices / first spatial positions. Specifically, the first execution devices / first spatial positions can be selected in descending order of signal strength associated with the first execution devices / first spatial positions, or the first execution devices / first spatial positions can be selected in descending order of the number of second execution devices / second spatial positions associated with the first execution devices / first spatial positions.

[0284] For example, the electronic device performs signal masking condition determination and processing on signals at the second execution device / second spatial location within the target execution device group / target spatial location group. The electronic device selects the second execution device / second spatial location from the target execution device group / target spatial location group and performs signal masking condition determination and processing on signals at the second execution device / second spatial location according to signals at the first execution device / first spatial location associated with the second execution device / second spatial location. Specifically, the electronic device can select the second execution device / second spatial location in a preset order and perform signal masking condition determination and processing on signals at the second execution device / second spatial location. For example, the electronic device can select the second execution device / second spatial location in descending order of signal intensity of signals associated with the second execution device / second spatial location or in descending order of the number of first execution devices / first spatial locations associated with the second execution device / second spatial location.

[0285] As an optional implementation, the processing of the haptic signal associated with the second execution device includes:

[0286] processing the haptic signal associated with the second execution device according to the haptic signal associated with the first execution device;

[0287] the processing of the haptic signal associated with the second spatial location includes:

[0288] processing the haptic signal associated with the second spatial location according to the haptic signal associated with the first spatial location.

[0289] The processing of the haptic signal associated with the second execution device according to the haptic signal associated with the first execution device can be pre-processing, masking processing, priority reduction processing, or discarding processing of the haptic signal associated with the second execution device according to the haptic signal associated with the first execution device. For example, in a case where the signal intensity of the haptic signal associated with the first execution device is higher than a specific threshold, the haptic signal associated with the second execution device is discarded, and in a case where the signal intensity of the haptic signal associated with the first execution device is not higher than the specific threshold, the priority of the haptic signal associated with the second execution device is reduced.

[0290] The processing of the haptic signal associated with the second spatial position according to the haptic signal associated with the first spatial position can be a preprocessing, a masking processing, a priority reduction processing or a discarding processing of the haptic signal associated with the second spatial position according to the haptic signal associated with the first spatial position. For example, in the case that the signal strength of the haptic signal associated with the first spatial position is higher than a certain threshold, the haptic signal associated with the second spatial position is discarded, and in the case that the signal strength of the haptic signal associated with the first spatial position is not higher than the certain threshold, the priority of the haptic signal associated with the second spatial position is reduced.

[0291] In this embodiment, the processing of the haptic signal associated with the second execution device according to the haptic signal associated with the first execution device can improve the processing reliability of the haptic signal associated with the second execution device. The processing of the haptic signal associated with the second spatial position according to the haptic signal associated with the first spatial position can improve the processing reliability of the haptic signal associated with the second spatial position.

[0292] It should be noted that the processing of the haptic signal associated with the second execution device according to the haptic signal associated with the first execution device in the embodiments of the present application is not limited, for example, the haptic signal associated with the second execution device is directly discarded, or the priority of the haptic signal associated with the second execution device is directly reduced.

[0293] It should be noted that the processing of the haptic signal associated with the second spatial position according to the haptic signal associated with the first spatial position in the embodiments of the present application is not limited, for example, the haptic signal associated with the second spatial position is directly discarded, or the priority of the haptic signal associated with the second spatial position is directly reduced.

[0294] As an optional embodiment, the attribute information of the object acquired by the electronic device further indicates at least one of:

[0295] The identification information of the haptic device to which the at least one execution device belongs;

[0296] The identification information of the haptic device to which the at least one execution device group belongs;

[0297] The identification information of the haptic device to which the at least one spatial position belongs;

[0298] The identification information of the haptic device to which the at least one spatial position group belongs;

[0299] The haptic modality information corresponding to the masking effect.

[0300] The above haptic device can also be referred to as a haptic product. Among them, one haptic device can include one or more execution devices, or one haptic device can include one or more groups of execution devices. One haptic device can include one or more spatial positions, or one haptic device can include one or more groups of spatial positions.

[0301] Since the attribute information of the object indicates the identification information of the haptic device, the electronic device can distinguish different haptic devices when there are multiple haptic devices, further improving the processing performance of the haptic signal.

[0302] The haptic modality information corresponding to the masking effect is understood as the haptic signal corresponding to the haptic modality corresponding to the haptic modality information being affected by the masking effect or existing the masking effect.

[0303] Since the attribute information of the object indicates the haptic modality information corresponding to the masking effect, in the case where the haptic device supports multiple modalities, the masking effects corresponding to different modalities are different, and the different masking effects are distinguished through the modality indication information, so as to improve the accuracy of haptic signal processing.

[0304] In the embodiment of the present application, the electronic device obtains first information; the electronic device processes the haptic signal based on the first information; wherein the first information includes at least one of the following: threshold related information, execution device related information, spatial position related information. This can realize processing of the haptic signal based on at least one of the threshold related information, the execution device related information, and the spatial position related information, so as to improve the processing performance of the haptic signal.

[0305] The processing method of the haptic signal provided in the embodiment of the present application can be executed by the processing device of the haptic signal. As an example, the device can be an electronic device, or a component in the electronic device, such as a chip, a circuit, etc. In the embodiment of the present application, the processing method of the haptic signal executed by the processing device of the haptic signal is taken as an example to illustrate the processing device of the haptic signal provided in the embodiment of the present application.

[0306] Please refer to FIG. 6, which is a structure diagram of a processing device of a haptic signal provided in an embodiment of the present application, as shown in FIG. 6, the processing device of the haptic signal 600 includes:

[0307] The acquisition module 601 is configured to acquire first information;

[0308] The processing module 602 is configured to process the haptic signal based on the first information;

[0309] The first information includes at least one of the following:

[0310] Threshold related information, execution device related information, spatial position related information.

[0311] Optionally, the execution device related information comprises at least one of:

[0312] the related information of a first execution device, the related information of a second execution device; wherein the haptic signal associated with the first execution device has a masking effect on the haptic signal associated with the second execution device;

[0313] The spatial position related information comprises at least one of:

[0314] the related information of a first spatial position, the related information of a second spatial position; wherein the haptic signal associated with the first spatial position has a masking effect on the haptic signal associated with the second spatial position;

[0315] The threshold related information is a threshold information related to the masking effect.

[0316] Optionally, the apparatus further comprises:

[0317] a determining module, configured to determine that there is a masking effect in the case that a first condition is met, the masking effect comprising that the haptic signal associated with a first execution device has a masking effect on the haptic signal associated with a second execution device, or the haptic signal associated with a first spatial position has a masking effect on the haptic signal associated with a second spatial position;

[0318] The first condition comprises at least one of:

[0319] the signal intensity of the haptic signal associated with the first execution device is greater than or equal to the signal intensity of the haptic signal associated with the second execution device;

[0320] the signal intensity of the haptic signal associated with the first spatial position is greater than or equal to the signal intensity of the haptic signal associated with the second spatial position;

[0321] the signal intensity of the haptic signal associated with the first execution device or the first spatial position is greater than or equal to a first threshold value;

[0322] the signal intensity of the haptic signal associated with the second execution device or the second spatial position is less than or equal to a second threshold value.

[0323] Optionally, the second threshold value comprises at least one of: a first value, a second value, a reference value;

[0324] the signal intensity of the haptic signal associated with the second execution device or the second spatial position being less than or equal to a second threshold value comprises at least one of:

[0325] a signal strength of the haptic signal associated with the second execution device or the second spatial position is less than or equal to the first value after normalization;

[0326] a signal strength of the haptic signal associated with the second execution device or the second spatial position is less than or equal to the reference value, the reference value being determined based on the signal strength of the haptic signal associated with the first execution device or the first spatial position and a third threshold value;

[0327] a signal strength of the haptic signal associated with the second execution device or the second spatial position is less than or equal to the second value.

[0328] Optionally, the attribute information of the object acquired by the apparatus is used to indicate at least one of:

[0329] the identification information of the first execution device;

[0330] the information of the first spatial position;

[0331] the identification information of the second execution device;

[0332] the information of the second spatial position;

[0333] the information of a group of execution devices to which the first execution device belongs;

[0334] the information of a group of spatial positions to which the first spatial position belongs;

[0335] the information of a group of execution devices to which the second execution device belongs;

[0336] the information of a group of spatial positions to which the second spatial position belongs;

[0337] the association relationship between the first execution device and the second execution device;

[0338] the association relationship between the first spatial position and the second spatial position;

[0339] the association relationship between a group of execution devices to which the first execution device belongs and a group of execution devices to which the second execution device belongs;

[0340] the association relationship between a group of spatial positions to which the first spatial position belongs and a group of spatial positions to which the second spatial position belongs.

[0341] Optionally, in a case where the attribute information of the object indicates the identification information of the execution device, the attribute information of the object further indicates at least one of the mapping position information and the parameter information of the execution device; or,

[0342] In a case where the attribute information of the object indicates identification information of the execution device, the attribute information of the object associated with the object is used to indicate at least one of the mapping position information and the parameter information of the execution device.

[0343] Optionally, the threshold-related information is predefined, or the threshold-related information is indicated by the attribute information of the object acquired by the electronic device.

[0344] Optionally, the acquiring the first information comprises:

[0345] The threshold-related information corresponding to the at least one frequency range is acquired by predefinition or indication of the attribute information of the object.

[0346] Optionally, the apparatus further comprises:

[0347] The determining module is configured to determine, based on the signal intensity of the at least one execution device or the spatial position associated haptic signal in the at least one frequency range and the corresponding threshold-related information, whether the masking effect exists.

[0348] Optionally, the threshold-related information has a corresponding relationship with the execution device or the spatial position, and the threshold-related information satisfies at least one of the following conditions:

[0349] One threshold-related information corresponds to at least one execution device.

[0350] One threshold-related information corresponds to at least one spatial position.

[0351] One threshold-related information corresponds to at least one group of execution devices.

[0352] One threshold-related information corresponds to at least one group of spatial positions.

[0353] One threshold-related information corresponds to a group of interrelated execution devices.

[0354] One threshold-related information corresponds to a group of interrelated spatial positions.

[0355] One threshold-related information corresponds to two groups of interrelated execution devices.

[0356] One threshold-related information corresponds to two groups of interrelated spatial positions.

[0357] One threshold-related information corresponds to one execution device, and a group of execution devices associated with the one execution device.

[0358] One threshold-related information corresponds to one spatial position, and a group of spatial positions associated with the one spatial position.

[0359] Optionally, the processing module is configured to perform at least one of:

[0360] in a case where it is determined based on the first information that there is a masking effect, the masking effect including the second execution device associated tactile signal being masked by the first execution device associated tactile signal, processing the second execution device associated tactile signal;

[0361] in a case where it is determined based on the first information that there is a masking effect, the masking effect including the second spatial position associated tactile signal being masked by the first spatial position associated tactile signal, processing the second spatial position associated tactile signal.

[0362] Optionally, the apparatus further comprises at least one of:

[0363] a first selection module configured to select at least one of the first execution device and the second execution device in a target execution device group;

[0364] a second selection module configured to select at least one of the first spatial position and the second spatial position in a target spatial position group.

[0365] Optionally, the first selection module is configured to select the first execution device in the target execution device group; or select the first execution device in the target execution device group and determine the second execution device corresponding to the first execution device according to the first execution device; or select the second execution device in the target execution device group; or select the second execution device in the target execution device group and determine the first execution device corresponding to the second execution device according to the second execution device.

[0366] the second selection module is configured to select the first spatial position in the target spatial position group; or select the first spatial position in the target spatial position group and determine the second spatial position corresponding to the first spatial position according to the first spatial position; or select the second spatial position in the target spatial position group; or select the second spatial position in the target spatial position group and determine the first spatial position corresponding to the second spatial position according to the second spatial position.

[0367] Optionally, the processing the second execution device associated tactile signal comprises:

[0368] processing the second execution device associated tactile signal according to the first execution device associated tactile signal;

[0369] the processing the second spatial position associated tactile signal comprises:

[0370] The haptic signal associated with the second spatial position is processed according to the haptic signal associated with the first spatial position.

[0371] Optionally, the attribute information of the object acquired by the device further indicates at least one of the following:

[0372] identification information of the haptic device to which the at least one execution device belongs;

[0373] identification information of the haptic device to which the at least one execution device group belongs;

[0374] identification information of the haptic device to which the at least one spatial position belongs;

[0375] identification information of the haptic device to which the at least one spatial position group belongs;

[0376] haptic modality information corresponding to the masking effect.

[0377] The processing device of the haptic signal can improve the processing performance of the haptic signal.

[0378] The processing device of the haptic signal provided by the embodiments of the present application can implement each process achieved by the method embodiment of FIG. 5 and achieve the same technical effects. To avoid repetition, details are not described herein.

[0379] As shown in FIG. 7, the embodiments of the present application further provide an electronic device 700, which includes a processor 701 and a memory 702, and the memory 702 stores programs or instructions executable on the processor 701. For example, when the electronic device 700 is an encoding end device, the programs or instructions are executed by the processor 701 to implement each step of the above-mentioned haptic signal processing method embodiments and achieve the same technical effects. When the electronic device 700 is a decoding end device, the programs or instructions are executed by the processor 701 to implement each step of the above-mentioned haptic signal processing method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein. Optionally, the memory 702 can be the memory 102 or the memory 113 in the embodiment shown in FIG. 1, and the processor 701 can implement the functions of the encoder 200 or the decoder 300 in the embodiments shown in FIGS. 1-3.

[0380] The embodiments of the present application further provide an electronic device, which includes a memory configured to store video data, and a processing circuit configured to implement each step of the above-mentioned haptic signal processing method embodiments. Optionally, the memory can be the memory 102 or the memory 113 in the embodiment shown in FIG. 1, and the processing circuit can implement the functions of the encoder 200 or the decoder 300 in the embodiments shown in FIGS. 1-3.

[0381] The embodiment of the present application further provides an electronic device comprising a processor and a communication interface, wherein the communication interface is coupled with the processor, and the processor is used to run programs or instructions to realize the steps in the method embodiment shown in FIG. 5. The device embodiment corresponds to the method embodiment described above, and each implementation process and implementation manner of the method embodiment described above can be applied to the terminal embodiment, and the same technical effects can be achieved.

[0382] The processor or processing circuit of the embodiment of the present application can include a general-purpose processor, a special-purpose processor, etc., for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), an artificial intelligent (AI) processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a network processor (NP), a field programmable gate array (FPGA) or other programmable logic devices, a gate circuit, a transistor, a discrete hardware component, etc. The communication interface of the embodiment of the present application can include a transceiver, a pin, a circuit, a bus, etc.

[0383] The electronic device described above can be a terminal, or can be other devices other than the terminal, for example, a server, a network attached storage (NAS), etc.

[0384] The terminal can be a mobile phone, a tablet personal computer, a laptop computer, a notebook computer, a personal digital assistant (PDA), a palm computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile Internet device (MID), an augmented reality (AR) device, a virtual reality (VR) device, a mixed reality (MR) device, a robot, a wearable device, a flight vehicle, a vehicle user equipment (VUE), a shipboard device, a pedestrian user equipment (PUE), a smart home (a home device with a wireless communication function, such as a refrigerator, a television, a washing machine, or furniture), a game console, a personal computer (PC), a teller machine, or a self-service machine, and the like. The wearable device includes a smart watch, a smart bracelet, a smart earphone, smart glasses, smart jewelry (a smart bracelet, a smart necklace, a smart ring, a smart necklace, a smart anklet, a smart necklace, and the like), a smart wristband, smart clothing, and the like. The vehicle-mounted device can also be referred to as a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip, or a vehicle-mounted unit, and the like. It should be noted that the specific type of the terminal is not limited in the embodiments of the present application.

[0385] The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server. The cloud server can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), or cloud computing services based on big data and artificial intelligence platforms.

[0386] For example, the electronic device can include, but is not limited to, a source device 100 or a destination device 110 as shown in FIG. 1.

[0387] Taking the electronic device as an example, FIG. 8 is a schematic diagram of a hardware structure of a terminal according to an embodiment of the present application.

[0388] The terminal 800 includes, but is not limited to, at least part of components such as a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810.

[0389] Those skilled in the art can understand that the terminal 800 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 810 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The terminal structure shown in FIG. 8 does not constitute a limitation on the terminal, and the terminal can include more or fewer components than those shown, or combine certain components, or different component arrangements, which are not described here again.

[0390] It should be understood that in the embodiments of the present application, the input unit 804 can include a graphics processor 8041 and a microphone 8042. The graphics processor 8041 processes image data of a still picture or a video obtained by an image acquisition device (such as a camera) in a video acquisition mode or an image acquisition mode, or can process obtained point cloud data. The display unit 806 can include a display panel 8061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 can include two parts of a touch detection device and a touch controller. The other input devices 8072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc., which are not described here again.

[0391] In the embodiments of the present application, the radio frequency unit 801 can transmit downlink data from the network side device to the processor 810 for processing after receiving the downlink data. In addition, the radio frequency unit 801 can send uplink data to the network side device. Generally, the radio frequency unit 801 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc.

[0392] The memory 809 can be used to store software programs or instructions and various data. The memory 809 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 809 can include a volatile memory or a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 809 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0393] The processor 810 can include one or more processing units; optionally, the processor 810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 810.

[0394] The processor 810 is configured to obtain first information; and process a tactile signal based on the first information.

[0395] The first information includes at least one of the following:

[0396] Threshold-related information, execution device-related information, and spatial position-related information.

[0397] Optionally, the execution device-related information includes at least one of the following:

[0398] the first execution device is associated with a masking effect on the haptic signal associated with the second execution device;

[0399] the spatial position related information comprises at least one of:

[0400] the first spatial position is associated with a masking effect on the haptic signal associated with the second spatial position;

[0401] the threshold related information is threshold information related to the masking effect.

[0402] Optionally, the processor 810 is further configured to:

[0403] determine that the masking effect exists in the case where the first condition is met, the masking effect comprising a masking effect of the haptic signal associated with the first execution device on the haptic signal associated with the second execution device, or a masking effect of the haptic signal associated with the first spatial position on the haptic signal associated with the second spatial position;

[0404] wherein the first condition comprises at least one of:

[0405] the signal intensity of the haptic signal associated with the first execution device is greater than or equal to the signal intensity of the haptic signal associated with the second execution device;

[0406] the signal intensity of the haptic signal associated with the first spatial position is greater than or equal to the signal intensity of the haptic signal associated with the second spatial position;

[0407] the signal intensity of the haptic signal associated with the first execution device or the first spatial position is greater than or equal to a first threshold value;

[0408] the signal intensity of the haptic signal associated with the second execution device or the second spatial position is less than or equal to a second threshold value.

[0409] Optionally, the second threshold value comprises at least one of: a first value, a second value, a reference value;

[0410] the signal intensity of the haptic signal associated with the second execution device or the second spatial position being less than or equal to the second threshold value comprises at least one of:

[0411] the normalized signal intensity of the haptic signal associated with the second execution device or the second spatial position being less than or equal to the first value;

[0412] The signal strength of the haptic signal associated with the second execution device or the second spatial position is less than or equal to the reference value, the reference value being determined based on the signal strength of the haptic signal associated with the first execution device or the first spatial position and a third threshold value.

[0413] The signal strength of the haptic signal associated with the second execution device or the second spatial position is less than or equal to the second value.

[0414] Optionally, the attribute information of the object acquired by the electronic device is used to indicate at least one of the following:

[0415] The identification information of the first execution device;

[0416] The information of the first spatial position;

[0417] The identification information of the second execution device;

[0418] The information of the second spatial position;

[0419] The information of the execution device group to which the first execution device belongs;

[0420] The information of the spatial position group to which the first spatial position belongs;

[0421] The information of the execution device group to which the second execution device belongs;

[0422] The information of the spatial position group to which the second spatial position belongs;

[0423] The association relationship between the first execution device and the second execution device;

[0424] The association relationship between the first spatial position and the second spatial position;

[0425] The association relationship between the execution device group to which the first execution device belongs and the execution device group to which the second execution device belongs;

[0426] The association relationship between the spatial position group to which the first spatial position belongs and the spatial position group to which the second spatial position belongs.

[0427] Optionally, in the case where the attribute information of the object indicates the identification information of the execution device, the attribute information of the object further indicates at least one of the mapping position information and the parameter information of the execution device; or,

[0428] In the case where the attribute information of the object indicates the identification information of the execution device, the attribute information of the object associated with the object is used to indicate at least one of the mapping position information and the parameter information of the execution device.

[0429] Optionally, the threshold-related information is predefined, or the threshold-related information is indicated by attribute information of the object acquired by the electronic device.

[0430] Optionally, the first information acquired by the electronic device includes:

[0431] The threshold-related information corresponding to the at least one frequency range is acquired by the electronic device through predefinition or indication of the attribute information of the object.

[0432] Optionally, the processor 810 is further configured to:

[0433] The signal intensity of the at least one frequency range of the haptic signal associated with the at least one execution device or spatial position is determined based on the threshold-related information to determine whether there is a masking effect.

[0434] Optionally, the threshold-related information has a corresponding relationship with the execution device or the spatial position, and the threshold-related information satisfies at least one of the following conditions:

[0435] One threshold-related information corresponds to at least one execution device.

[0436] One threshold-related information corresponds to at least one spatial position.

[0437] One threshold-related information corresponds to at least one execution device group.

[0438] One threshold-related information corresponds to at least one spatial position group.

[0439] One threshold-related information corresponds to a group of execution devices associated with each other.

[0440] One threshold-related information corresponds to a group of spatial positions associated with each other.

[0441] One threshold-related information corresponds to two groups of execution devices associated with each other.

[0442] One threshold-related information corresponds to two groups of spatial positions associated with each other.

[0443] One threshold-related information corresponds to one execution device, and corresponds to an execution device group associated with the one execution device.

[0444] One threshold-related information corresponds to one spatial position, and corresponds to a spatial position group associated with the one spatial position.

[0445] Optionally, the processing of the haptic signal based on the first information includes at least one of the following:

[0446] in a case where it is determined based on the first information that there is a masking effect, the masking effect including the second execution device associated tactile signal being masked by the first execution device associated tactile signal, processing the second execution device associated tactile signal;

[0447] in a case where it is determined based on the first information that there is a masking effect, the masking effect including the second spatial position associated tactile signal being masked by the first spatial position associated tactile signal, processing the second spatial position associated tactile signal.

[0448] Optionally, the processor 810 is further configured to perform at least one of:

[0449] selecting at least one of the first execution device and the second execution device in a target execution device group;

[0450] selecting at least one of the first spatial position and the second spatial position in a target spatial position group.

[0451] Optionally, the selecting at least one of the first execution device and the second execution device in a target execution device group comprises:

[0452] selecting the first execution device in the target execution device group; or, selecting the first execution device in the target execution device group and determining the second execution device corresponding to the first execution device according to the first execution device; or, selecting the second execution device in the target execution device group; or, selecting the second execution device in the target execution device group and determining the first execution device corresponding to the second execution device according to the second execution device;

[0453] Optionally, the selecting at least one of the first spatial position and the second spatial position in a target spatial position group comprises:

[0454] selecting the first spatial position in the target spatial position group; or, selecting the first spatial position in the target spatial position group and determining the second spatial position corresponding to the first spatial position according to the first spatial position; or, selecting the second spatial position in the target spatial position group; or, selecting the second spatial position in the target spatial position group and determining the first spatial position corresponding to the second spatial position according to the second spatial position.

[0455] Optionally, the processing the second execution device associated tactile signal comprises:

[0456] processing the second execution device associated tactile signal according to the first execution device associated tactile signal;

[0457] the processing the second spatial position associated tactile signal comprises:

[0458] processing the haptic signal associated with the second spatial position according to the haptic signal associated with the first spatial position.

[0459] Optionally, the attribute information of the object acquired by the electronic device further indicates at least one of:

[0460] identification information of a haptic device to which the at least one execution device belongs;

[0461] identification information of a haptic device to which the at least one execution device group belongs;

[0462] identification information of a haptic device to which the at least one spatial position belongs;

[0463] identification information of a haptic device to which the at least one spatial position group belongs;

[0464] haptic modality information corresponding to the masking effect.

[0465] The terminal can improve the processing performance of the haptic signal.

[0466] It can be understood that the implementation process of each implementation manner mentioned in the embodiment can refer to the related description of the method embodiment of the haptic signal processing method, and achieve the same or corresponding technical effects. To avoid repetition, it will not be repeated here.

[0467] The embodiment of the application further provides a readable storage medium, the readable storage medium stores a program or instructions, the program or instructions are executed by a processor to realize each process of the haptic signal processing method embodiment, and the same technical effects can be achieved. To avoid repetition, it will not be repeated here.

[0468] The processor is the processor in the terminal in the above embodiment. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk, etc. In some examples, the readable storage medium can be a non-transitory readable storage medium.

[0469] The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface and the processor are coupled, the processor is used to run a program or instructions, realizes each process of the haptic signal processing method embodiment, and the same technical effects can be achieved. To avoid repetition, it will not be repeated here.

[0470] It should be understood that the chip mentioned in the embodiment of the application can include a system-level chip (also known as a system chip, a chip system or a system-on-chip chip), and can also include a separate display chip, etc.

[0471] The embodiment of the present application further provides a computer program / program product stored in a storage medium, which is executed by at least one processor to implement each process of the above-mentioned processing method embodiment of the tactile signal and achieve the same technical effects. To avoid repetition, details are not described herein.

[0472] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that processes, methods, articles, or apparatuses that comprise a list of elements not only include those elements, but also include other elements that are not expressly listed, or other elements that are inherent in such processes, methods, articles, or apparatuses. Without more limitations, an element defined by the sentence "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element. In addition, it should be pointed out that the scope of the methods and apparatuses in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in a reverse order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0473] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of computer software product and general hardware platform, of course, it can also be realized by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disc, optical disc, etc.), including a plurality of instructions, used to make the terminal or network side device execute the method described in each embodiment of the present application.

[0474] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, not restrictive, and those skilled in the art can make many forms of embodiments under the inspiration of the present application without departing from the scope of the present application and the protection scope of the claims.

Claims

1. A method for processing a haptic signal, comprising: obtaining, by an electronic device, first information; processing, by the electronic device, a haptic signal based on the first information; wherein the first information comprises at least one of: threshold-related information, execution device-related information, spatial position-related information.

2. The method of claim 1, wherein, the execution device-related information comprises at least one of: first execution device-related information, second execution device-related information; wherein a haptic signal associated with the first execution device has a masking effect on a haptic signal associated with the second execution device; the spatial position-related information comprises at least one of: first spatial position-related information, second spatial position-related information; wherein a haptic signal associated with the first spatial position has a masking effect on a haptic signal associated with the second spatial position; the threshold-related information is a threshold value related to the masking effect.

3. The method of claim 1 or 2, further comprising: determining, in a case where a first condition is met, that a masking effect exists, the masking effect comprising a haptic signal associated with a first execution device having a masking effect on a haptic signal associated with a second execution device, or a haptic signal associated with a first spatial position having a masking effect on a haptic signal associated with a second spatial position; wherein the first condition comprises at least one of: a signal strength of the haptic signal associated with the first execution device is greater than or equal to a signal strength of the haptic signal associated with the second execution device; a signal strength of the haptic signal associated with the first spatial position is greater than or equal to a signal strength of the haptic signal associated with the second spatial position; a signal strength of the haptic signal associated with the first execution device or the first spatial position is greater than or equal to a first threshold value; a signal strength of the haptic signal associated with the second execution device or the second spatial position is less than or equal to a second threshold value.

4. The method of claim 3, wherein, the second threshold value comprises at least one of: a first value, a second value, a reference value; the signal strength of the haptic signal associated with the second execution device or the second spatial position being less than or equal to the second threshold value comprises at least one of: a normalized signal strength of the haptic signal associated with the second execution device or the second spatial position being less than or equal to the first value; the signal strength of the haptic signal associated with the second execution device or the second spatial position being less than or equal to the reference value, the reference value being determined based on a signal strength of the haptic signal associated with the first execution device or the first spatial position and a third threshold value; the signal strength of the haptic signal associated with the second execution device or the second spatial position being less than or equal to the second value.

5. The method of any one of claims 2 to 4, wherein, attribute information of an object obtained by the electronic device is used to indicate at least one of: identification information of the first execution device; information of the first spatial position; identification information of the second execution device; information of the second spatial position; information of a group of execution devices to which the first execution device belongs; information of a group of spatial positions to which the first spatial position belongs; information of a group of execution devices to which the second execution device belongs; information of a group of spatial positions to which the second spatial position belongs; an association relationship between the first execution device and the second execution device; an association relationship between the first spatial position and the second spatial position; an association relationship between an execution device group to which the first execution device belongs and an execution device group to which the second execution device belongs; an association relationship between a spatial position group to which the first spatial position belongs and a spatial position group to which the second spatial position belongs.

6. The method of claim 5, wherein, In a case where the attribute information of the object indicates the identification information of the execution device, the attribute information of the object further indicates at least one of the mapping position information and the parameter information of the execution device. Or, In a case where the attribute information of the object indicates the identification information of the execution device, the attribute information of the object associated with the object is used to indicate at least one of the mapping position information and the parameter information of the execution device.

7. The method of any one of claims 1 to 6, wherein, The threshold-related information is predefined, or the threshold-related information is indicated by the attribute information of the object obtained by the electronic device.

8. The method of claim 7, wherein, The electronic device obtains the first information, including: The electronic device obtains the threshold-related information corresponding to at least one frequency range through predefinition or indication of the attribute information of the object.

9. The method of claim 8, further comprising: determining whether there is a masking effect based on the signal intensity of the at least one execution device or spatial position associated tactile signal in the at least one frequency range and the corresponding threshold-related information.

10. The method of any one of claims 1 to 9, wherein, The threshold-related information has a corresponding relationship with an execution device or a spatial position, and the threshold-related information satisfies at least one of the following conditions: one threshold-related information corresponds to at least one execution device; one threshold-related information corresponds to at least one spatial position; one threshold-related information corresponds to at least one execution device group; one threshold-related information corresponds to at least one spatial position group; one threshold-related information corresponds to a group of mutually associated execution devices; one threshold-related information corresponds to a group of mutually associated spatial positions; one threshold-related information corresponds to two mutually associated execution device groups; one threshold-related information corresponds to two mutually associated spatial position groups; one threshold-related information corresponds to one execution device, and corresponds to an execution device group associated with the one execution device; one threshold-related information corresponds to one spatial position, and corresponds to a spatial position group associated with the one spatial position.

11. The method of any one of claims 1 to 10, wherein, The electronic device processes the tactile signal based on the first information, including at least one of the following: in a case where it is determined based on the first information that there is a masking effect, processing the tactile signal associated with the second execution device, the masking effect including the tactile signal associated with the second execution device being masked by the tactile signal associated with the first execution device; in a case where it is determined based on the first information that there is a masking effect, processing the tactile signal associated with the second spatial position, the masking effect including the tactile signal associated with the second spatial position being masked by the tactile signal associated with the first spatial position.

12. The method of any one of claims 1 to 11, wherein, The method further comprises at least one of the following: the electronic device selects at least one of the first execution device and the second execution device in the target execution device group; The electronic device selects at least one of the first spatial position and the second spatial position in the target spatial position group.

13. The method of claim 12, wherein, The electronic device selects at least one of the first execution device and the second execution device in the target execution device group, including: The electronic device selects the first execution device in the target execution device group; or, the electronic device selects the first execution device in the target execution device group, and determines the second execution device corresponding to the first execution device according to the first execution device; or, the electronic device selects the second execution device in the target execution device group; or, the electronic device selects the second execution device in the target execution device group, and determines the first execution device corresponding to the second execution device according to the second execution device. The electronic device selects at least one of the first spatial position and the second spatial position in the target spatial position group, including: The electronic device selects the first spatial position in the target spatial position group; or, the electronic device selects the first spatial position in the target spatial position group, and determines the second spatial position corresponding to the first spatial position according to the first spatial position; or, the electronic device selects the second spatial position in the target spatial position group; or, the electronic device selects the second spatial position in the target spatial position group, and determines the first spatial position corresponding to the second spatial position according to the second spatial position.

14. The method of any one of claims 11 to 13, wherein, The processing of the haptic signal associated with the second execution device includes: Processing the haptic signal associated with the second execution device according to the haptic signal associated with the first execution device; The processing of the haptic signal associated with the second spatial position includes: Processing the haptic signal associated with the second spatial position according to the haptic signal associated with the first spatial position.

15. The method of any one of claims 1 to 14, wherein, The attribute information of the object obtained by the electronic device further indicates at least one of: Identification information of a haptic device to which at least one execution device belongs; Identification information of a haptic device to which at least one execution device group belongs; Identification information of a haptic device to which at least one spatial position belongs; Identification information of a haptic device to which at least one spatial position group belongs; Haptic modality information corresponding to a masking effect.

16. A haptic signal processing apparatus, comprising: an acquisition module configured to acquire first information; a processing module configured to process a haptic signal based on the first information; wherein the first information includes at least one of: threshold-related information, execution device-related information, and spatial position-related information.

17. The apparatus of claim 16, wherein, The execution device-related information includes at least one of: related information of a first execution device and related information of a second execution device; wherein the haptic signal associated with the first execution device has a masking effect on the haptic signal associated with the second execution device; The spatial position-related information includes at least one of: related information of a first spatial position and related information of a second spatial position; wherein the haptic signal associated with the first spatial position has a masking effect on the haptic signal associated with the second spatial position. The threshold-related information is threshold information related to a masking effect.

18. The apparatus of claim 16 or 17, further comprising: a determining module configured to determine, in a case where a first condition is met, that a masking effect exists, the masking effect including a masking effect of a haptic signal associated with a first execution device on a haptic signal associated with a second execution device, or a masking effect of a haptic signal associated with a first spatial position on a haptic signal associated with a second spatial position; wherein the first condition includes at least one of: a signal strength of the haptic signal associated with the first execution device being greater than or equal to a signal strength of the haptic signal associated with the second execution device; a signal strength of the haptic signal associated with the first spatial position being greater than or equal to a signal strength of the haptic signal associated with the second spatial position; the signal strength of the haptic signal associated with the first execution device or the first spatial position being greater than or equal to a first threshold value; the signal strength of the haptic signal associated with the second execution device or the second spatial position being less than or equal to a second threshold value.

19. The apparatus of claim 17 or 18, wherein, The attribute information of the object acquired by the apparatus is used to indicate at least one of: identification information of the first execution device; information of the first spatial position; identification information of the second execution device; information of the second spatial position; information of a group of execution devices to which the first execution device belongs; information of a group of spatial positions to which the first spatial position belongs; information of a group of execution devices to which the second execution device belongs; information of a group of spatial positions to which the second spatial position belongs; an association relationship between the first execution device and the second execution device; an association relationship between the first spatial position and the second spatial position; an association relationship between a group of execution devices to which the first execution device belongs and a group of execution devices to which the second execution device belongs; an association relationship between a group of spatial positions to which the first spatial position belongs and a group of spatial positions to which the second spatial position belongs.

20. The apparatus of any one of claims 16-19, wherein, The processing module is configured to perform at least one of: in a case where it is determined based on the first information that a masking effect exists, processing a haptic signal associated with a second execution device, the masking effect including the haptic signal associated with the second execution device being masked by a haptic signal associated with a first execution device; in a case where it is determined based on the first information that a masking effect exists, processing a haptic signal associated with a second spatial position, the masking effect including the haptic signal associated with the second spatial position being masked by a haptic signal associated with a first spatial position.

21. The apparatus of any one of claims 16-20, wherein, The apparatus further includes at least one of: a first selecting module configured to select at least one of the first execution device and the second execution device in a target group of execution devices; a second selecting module configured to select at least one of the first spatial position and the second spatial position in a target group of spatial positions.

22. An electronic device, comprising a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the method for processing a haptic signal according to any one of claims 1 to 15.

23. A readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps of the processing method of the haptic signal according to any one of claims 1 to 15.

24. A computer program product stored in a storage medium, which when executed by at least one processor implement the steps of the processing method of the haptic signal according to any one of claims 1 to 15.

25. A chip, comprising a processor and a communication interface, the communication interface and the processor coupled, the processor configured to run a program or instructions, which implement the steps of the processing method of the haptic signal according to any one of claims 1 to 15.

Images