Spatial tactile signal processing method and apparatus, and device

Abstract

The present application relates to the technical field of computers, and discloses a spatial tactile signal processing method and apparatus, and a device. The spatial tactile signal processing method in embodiments of the present application comprises: obtaining auxiliary information for processing a one-dimensional spatial tactile signal.

Description

Spatial tactile signal processing methods, devices and equipment

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411052090.9, filed in China on August 1, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of computer technology, and specifically relates to a spatial tactile signal processing method, device and related equipment. Background Technology

[0004] Triggering haptic signals in three-dimensional (3D) scenes can enhance the user's immersive experience. In some related technologies, the processing of spatial haptic signals mainly focuses on two-dimensional (2D) spatial haptic signals, specifically acquiring relevant information about two-dimensional spatial haptic signals. This results in a relatively complex rendering model and limited supported scenarios. Summary of the Invention

[0005] This application provides a spatial tactile signal processing method, apparatus, and related equipment, which can solve the problem that the rendering model is relatively complex and the supported scenarios are relatively limited.

[0006] Firstly, a spatial tactile signal processing method is provided, including:

[0007] Acquire auxiliary information for processing one-dimensional spatial tactile signals.

[0008] Secondly, a spatial tactile signal processing device is provided, comprising:

[0009] The acquisition module is used to acquire auxiliary information for processing one-dimensional spatial tactile signals.

[0010] Thirdly, a spatial tactile signal processing apparatus is provided, the apparatus being configured to perform the steps of the spatial tactile signal processing method as provided in the embodiments of this application.

[0011] Fourthly, a device is provided, the terminal including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the spatial tactile signal processing method provided in the embodiments of this application.

[0012] Fifthly, a device is provided, including a processor and a communication interface, wherein the processor is used to acquire auxiliary information for processing one-dimensional spatial tactile signals.

[0013] In a sixth aspect, an apparatus is provided, comprising: a memory configured to store video data, and processing circuitry configured to implement the steps of the spatial tactile signal processing method provided in the embodiments of this application.

[0014] In a seventh aspect, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the spatial tactile signal processing method provided in the embodiments of this application.

[0015] Eighthly, a chip is provided, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the spatial tactile signal processing method provided in the embodiments of this application.

[0016] In a tenth aspect, a computer program / program product is provided, which is stored in a storage medium and executed by at least one processor to implement the steps of the spatial tactile signal processing method provided in the embodiments of this application.

[0017] In this embodiment of the application, auxiliary information for processing one-dimensional spatial tactile signals is obtained. Since the auxiliary information for processing one-dimensional spatial tactile signals is obtained, it is possible to support the processing of one-dimensional spatial tactile signals. The processing of one-dimensional spatial tactile signals is simpler, which helps to reduce the complexity of the renderer's processing model. Furthermore, since it can process one-dimensional spatial tactile signals, it is applicable to more scenarios. Attached Figure Description

[0018] Figure 1 is a schematic diagram of the encoding and decoding system provided in an embodiment of this application;

[0019] Figure 2 is a schematic diagram of the encoder provided in an embodiment of this application;

[0020] Figure 3 is a schematic diagram of the decoder provided in an embodiment of this application;

[0021] Figure 4 is a schematic diagram of the access method of tactile signals provided in the embodiments of this application;

[0022] Figure 5 is a flowchart of a spatial tactile signal processing method provided in an embodiment of this application;

[0023] Figure 6 is a structural diagram of a spatial tactile signal processing device provided in an embodiment of this application;

[0024] Figure 7 is a structural diagram of an electronic device provided in an embodiment of this application;

[0025] Figure 8 is a structural diagram of a terminal provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0027] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, the scope of protection for "A or B" covers at least three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. In addition, the terms "A and / or B," "at least one of A and B," and "at least one of A or B" also cover at least the above three scenarios. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0028] Figure 1 is a schematic diagram of the encoding / decoding system 10 provided in an embodiment of this application. The technical solution of this application embodiment relates to encoding and decoding video data (CODEC), including CODEC encoding or decoding. The video data includes original unencoded video, encoded video, decoded (e.g., reconstructed) video, or syntax elements, etc.

[0029] As shown in Figure 1, the encoding / decoding system 10 includes a source device 100, which provides encoded video data to be decoded and displayed by the destination device 110. Specifically, the source device 100 provides video data to the destination device 110 via a communication medium 120. The source device 100 and the destination device 110 may include any one or more of the following: desktop computer, laptop computer, tablet computer, set-top box, mobile phone, wearable device (e.g., smartwatch or wearable camera), television, camera, display device, in-vehicle device, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, digital media player, video game console, video conferencing equipment, video streaming equipment, broadcast receiver equipment, broadcast transmitter equipment, spacecraft, aircraft, robot, satellite, etc.

[0030] In the example of Figure 1, source device 100 includes a data source 101, memory 102, encoder 200, and output interface 104. Destination device 110 includes an input interface 111, decoder 300, memory 113, and display device 114. Source device 100 represents an example of a video encoding device, while destination device 110 represents an example of a video decoding device. In other examples, source device 100 and destination device 110 may not include some of the components shown in Figure 1, or they may include components other than those shown in Figure 1. For example, source device 100 may receive video data from an external data source (such as an external camera). Similarly, destination device 110 may interface with an external display device instead of including an integrated display device. As another example, memory 102 and memory 113 may be external memories.

[0031] Although Figure 1 illustrates the source device 100 and the destination device 110 as separate devices, in some examples, they may be integrated into a single device. In such embodiments, the same hardware or software, separate hardware or software, or any combination thereof may be used to implement the functionality corresponding to the source device 100 and the functionality corresponding to the destination device 110.

[0032] In some examples, source device 100 and destination device 110 can perform unidirectional or bidirectional video transmission. If it is bidirectional video transmission, source device 100 and destination device 110 can operate in a substantially symmetrical manner, that is, each of source device 100 and destination device 110 includes an encoder and a decoder.

[0033] Data source 101 represents the source of video data (i.e., raw, unencoded video data) and provides encoder 200 with a series of images containing video data, which encoder 200 encodes. Data source 101 of source device 100 may include a video acquisition device (such as a video camera), a video archive containing previously acquired raw video, or a video feed interface for receiving video from a video content provider. Alternatively, data source 101 may generate computer graphics-based data as source video, or combine live video, archived video, and computer-generated video. In these cases, encoder 200 encodes the acquired, pre-acquired, or computer-generated video data. Encoder 200 may rearrange the images from the received order (sometimes referred to as the "display order") according to the encoding order. Encoder 200 may generate a bitstream including the encoded video data. Source device 100 may then output the encoded video data to communication medium 120 via output interface 104 for reception or retrieval, for example, by input interface 111 of destination device 110.

[0034] The memory 102 of the source device 100 and the memory 113 of the destination device 110 represent general-purpose memory. In some examples, memory 102 may store raw video data from data source 101, and memory 113 may store decoded video data from decoder 300. Additionally or alternatively, memories 102 and 113 may respectively store software instructions executable by, for example, encoder 200 and decoder 300. Although memories 102 and 113 are shown separately from encoder 200 and decoder 300 in this example, it should be understood that encoder 200 and decoder 300 may also include internal memory for functionally similar or equivalent purposes. If encoder 200 and decoder 300 are deployed on the same hardware device, memories 102 and 113 may be the same memory. Furthermore, memories 102 and 113 may store, for example, encoded video data output from encoder 200 and input to decoder 300. In some examples, portions of memories 102 and 113 may be allocated as one or more video buffers, for example, to store raw, decoded, or encoded video data.

[0035] In some examples, source device 100 can output encoded data from output interface 104 to memory 113. Similarly, destination device 110 can access encoded data from memory 113 via input interface 111. Memory 113 or memory 102 can include any of a variety of distributed or locally accessed data storage media, such as hard drives, Blu-ray discs, digital versatile discs (DVDs), compact disc read-only memory (CD-ROMs), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0036] Output interface 104 may include any type of medium or device capable of transmitting encoded video data from source device 100 to destination device 110. For example, output interface 104 may include a transmitter or transceiver, such as an antenna, configured to transmit encoded video data directly from source device 100 to destination device 110 in real time. The encoded video data may be modulated according to the communication standards of a wireless communication protocol and transmitted to destination device 110.

[0037] Communication medium 120 may include transient media, such as wireless broadcasting or wired network transmission. For example, communication medium 120 may include radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cables). Communication medium 120 may form part of a packet-based network (such as a local area network, a wide area network, or a global network such as the Internet). Communication medium 120 may also take the form of a storage medium (e.g., a non-transitory storage medium), such as a hard disk, flash drive, compact disc, digital video disc, Blu-ray disc, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0038] In some implementations, the communication medium 120 may include a router, switch, base station, or any other device that can be used to facilitate communication from source device 100 to destination device 110. For example, a server (not shown) may receive encoded video from source device 100 and provide the encoded video data to destination device 110, for example, via network transmission. The server may include (e.g., a web server for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery Over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Services (MBMS) or Evolved Multimedia Broadcast Multicast Service (eMBMS) server, or a Network-attached storage (NAS) device, etc. The server can implement one or more HTTP streaming protocols, such as MPEG Media Transport (MMT), Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), or Real Time Streaming Protocol (RTSP).

[0039] Destination device 110 can access encoded video data from a server, for example, via a wireless channel (e.g., Wi-Fi connection) or a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.) for accessing encoded video data stored on the server.

[0040] Output interface 104 and input interface 111 can represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to the IEEE 802.11 or IEEE 802.15 standard (e.g., ZigBee™), Bluetooth standard, or other physical components. In an example where output interface 104 and input interface 111 include wireless components, output interface 104 and input interface 111 can be configured to transmit data, such as encoded video data, via Wi-Fi, Ethernet, cellular networks (such as the 4th Generation Mobile Communication Technology (4G), Long Term Evolution (LTE), LTE Advanced, the 5th Generation Mobile Communication Technology (5G), the 6th Generation Mobile Communication Technology (6G), etc.).

[0041] The technology provided in this application can be applied to support video encoding and decoding in one or more multimedia applications such as video conferencing, over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission, digital video encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0042] The input interface 111 of the destination device 110 receives an encoded video bitstream from the communication medium 120. The encoded video bitstream may include syntax elements and encoded data units (e.g., sequences, image groups, images, slices, blocks, etc.), where the syntax elements are used to decode the encoded data units to obtain decoded video data. The display device 114 displays the decoded video data to the user. The display device 114 may include a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0043] The encoder 200 and decoder 300 can be implemented as one or more of various processing circuits, which may include microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. When the technology is implemented wholly or partially in software, the device may store instructions for the software in a suitable non-transitory computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology provided in the embodiments of this application.

[0044] The encoder 200 and decoder 300 can process based on the following video codec standards: H.263, H.264, H.265 (also known as High Efficiency Video Coding, HEVC), H.266 (also known as Versatile Video Coding, VVC), Moving Picture Experts Group 2 (MPEG-2), MPEG-4, VP8, VP9, ​​Alliance for Open Media Video 1 (AV1), Audio Video Coding Standard 1 (AVS1), AVS2, AVS3, or next-generation video standard protocols. This application does not specifically limit the implementation of these protocols.

[0045] Typically, encoder 200 and decoder 300 can perform block-based encoding and decoding of images. The term "block" generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used during encoding or decoding). For example, a block can include a two-dimensional matrix of samples of luminance or chrominance data. For example, encoder 200 and decoder 300 can encode and decode video data represented in YUV format.

[0046] Referring to Figure 2, which is a schematic diagram of the encoder 200 provided in an embodiment of this application, the encoder 200 can be the encoder 200 in Figure 1. In the example of Figure 2, the encoder 200 includes a memory 201, an encoding parameter determination unit 210, a residual generation unit 202, a transform processing unit 203, a quantization unit 204, an inverse quantization unit 205, an inverse transform processing unit 206, a reconstruction unit 207, a filter unit 208, a decoded picture buffer (DPB) 209, and an entropy encoding unit 220.

[0047] The memory 201 can store video data to be encoded. For example, the encoder 200 can receive and store video data from the data source 101 shown in Figure 1. In some examples, the memory 201 can be on the same chip as other components of the encoder 200 (as shown in Figure 2), or it can be on a separate chip from those components.

[0048] The coding parameter determination unit 210 includes a mode selection unit 211, an inter-frame prediction unit 212, and an intra-frame prediction unit 213. The inter-frame prediction unit 212 is used to obtain a first prediction block for the current block using an inter-frame prediction mode. The intra-frame prediction unit 213 is used to obtain a second prediction block for the current block using an intra-frame prediction mode. The mode selection unit 211 is used to obtain a target prediction block based on the first and second prediction blocks and determine the final prediction mode. Furthermore, the coding parameter determination unit 210 may also include other functional units, such as functional units for determining the partitioning method of coding units (CUs), functional units for determining the transformation type of the residual data of the CUs, or functional units for determining the quantization parameters of the residual data of the CUs.

[0049] For ease of description and understanding, in the embodiments of this application, the CU to be processed in the current image is referred to as the current CU, and the image block to be processed in the current CU is referred to as the current block or the image block to be processed. For example, in encoding, it refers to the block currently being encoded; in decoding, it refers to the block currently being decoded.

[0050] Inter-frame prediction unit 212 may include a motion estimation unit and a motion compensation unit. For inter-frame prediction of the current block, the motion estimation unit may perform a motion search to identify one or more matching reference blocks in one or more reference pictures (e.g., one or more previously encoded / decoded pictures stored in DPB 209).

[0051] The motion estimation unit can generate one or more motion vectors (MVs) representing the position of a reference block in a reference image relative to the position of the current block in the current image. The motion compensation unit can then use interpolation to obtain a predicted value with the precision indicated by the motion vectors.

[0052] The encoding parameter determination unit 210 can provide the target prediction block to the residual generation unit 202. The residual generation unit 202 receives the raw uncoded video data of the current block from the memory 201 and calculates the residual between the current block and the target prediction block to obtain the residual block. In some examples, the function of the residual generation unit 202 can be implemented using one or more subtractor circuits that perform binary subtraction.

[0053] As an example, the encoding parameter determination unit 210 can provide the entropy encoding unit 220 with syntax elements representing encoding parameters for encoding. The encoding parameters include one or more of the following: the partitioning method of the CU, the final prediction mode, the transformation type of the residual data of the CU, or the quantization parameters of the residual data of the CU.

[0054] The transformation processing unit 203 transforms the residual block output by the residual generation unit 202 to obtain a transform coefficient block. This transformation may include Discrete Cosine Transform (DCT), integer transformation, direction transformation, or Karhunen-Loeve transformation, etc. In some examples, the encoder 200 may not include the transformation processing unit 203.

[0055] Quantization unit 204 can quantize the transform coefficients in the transform coefficient block according to the quantization parameter (QP) value associated with the current block to generate a quantized transform coefficient block.

[0056] The inverse quantization unit 205 and the inverse transform processing unit 206 can perform inverse quantization and inverse transform on the transform coefficient block, respectively, to obtain the reconstructed residual block. The reconstruction unit 207 can generate a reconstructed block corresponding to the current block based on the reconstructed residual block and the target prediction block generated by the coding parameter determination unit 210.

[0057] Filter unit 208 can perform one or more filter operations on the reconstructed block. For example, filter unit 208 can be a deblocking filter (DBF), an adaptive loop filter (ALF), a sample adaptive offset (SAO) filter, etc. In some examples, encoder 200 may not include filter unit 208.

[0058] Encoder 200 stores the reconstructed image obtained from the reconstructed blocks in DPB 209. For example, in an example where the operation of filter unit 208 is not required, reconstruction unit 207 can store the reconstructed blocks in DPB 209. In an example where the operation of filter unit 208 is required, filter unit 208 can store the filtered reconstructed blocks in DPB 209. Inter-frame prediction unit 212 retrieves the reconstructed image from DPB 209 to perform inter-frame prediction on blocks of subsequent images to be encoded. In some examples, DPB 209 can be replaced with other types of memory.

[0059] Entropy coding unit 220 can entropy code the syntax elements of other components in encoder 200 to output encoded video data. For example, entropy coding unit 220 can entropy code the quantized transform coefficient block from quantization unit 204. As another example, entropy coding unit 220 can entropy code the syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from coding parameter determination unit 210.

[0060] It is understood that the composition of the encoder 200 shown in Figure 2 is only illustrative and does not constitute a limitation on the embodiments of this application.

[0061] Figure 3 is a schematic diagram of the structure of the decoder 300 provided in an embodiment of this application. The decoder 300 can be the decoder 300 described in Figure 1. In the example of Figure 3, the decoder 300 includes a coded picture buffer (CPB) 301, an entropy decoding unit 302, a prediction processing unit 310, an inverse quantization unit 303, an inverse transform processing unit 304, a reconstruction unit 305, a filter unit 306, and a DPB 307.

[0062] The entropy decoding unit 302 can receive encoded video data from the CPB 301 and perform entropy decoding on the video data to obtain syntax elements. The syntax elements indicate encoding parameters, including one or more of the following: CU partitioning method, final prediction mode, transformation type of CU residual data, or quantization parameters of CU residual data.

[0063] When the syntax element includes the final prediction mode, the prediction processing unit 310 obtains the final prediction mode. If the final prediction mode is an inter-frame prediction mode, the prediction block of the current CU can be obtained through the inter-frame prediction unit 311 of the prediction processing unit 310; if the final prediction mode is an intra-frame prediction mode, the prediction block of the current CU can be obtained through the intra-frame prediction unit 312 of the prediction processing unit 310. In some examples, the prediction processing unit 310 may also include a unit for performing prediction functions according to other prediction modes.

[0064] CPB 301 can acquire and store encoded video data from the communication medium 120 shown in Figure 1. DPB 307 is used to store decoded images. Optionally, CPB 301 and DPB 307 can be replaced with other types of memory, which are not specifically limited in this application. In some examples, CPB 301 can be on the same chip as other components of decoder 300 (as shown in the figure), or it can be on a separate chip from those components.

[0065] Decoder 300 can perform reconstruction operations on each block individually. Entropy decoding unit 302 can entropy decode the syntax elements and transform information (e.g., QP or transform mode indication) of the quantized transform coefficients to obtain the quantized transform coefficients. Dequantization unit 303 dequantizes the quantized transform coefficients to obtain a transform coefficient block including the transform coefficients. Inverse transform processing unit 304 performs an inverse transform on the transform coefficient block to generate a residual block corresponding to the current block; this inverse transform is the reverse operation of the above transform.

[0066] Reconstruction unit 305 can reconstruct the current block based on the prediction block and the residual block. For example, reconstruction unit 305 can add samples from the residual block to the corresponding samples from the prediction block to reconstruct the current block.

[0067] Filter unit 306 can perform one or more filter operations on the reconstructed block. For example, the type of filter unit 306 can be referenced to the type of filter unit 208, and will not be described again here. In some examples, the operations of filter unit 306 can be skipped.

[0068] Decoder 300 can store the reconstructed image obtained from the reconstructed blocks in DPB 307. For example, in an example where filter unit 306 is not operated, reconstruction unit 305 can store the reconstructed blocks in DPB 307. In an example where filter unit 306 is operated, filter unit 306 can store the filtered reconstructed blocks in DPB 307. Decoder 300 can output the decoded image (e.g., decoded video) from DPB 307 for subsequent rendering on a display device (such as display device 114 of FIG. 1).

[0069] The encoding end and the decoding end can be implemented by software, hardware or a combination thereof. When implemented by hardware, the encoding end can be called an encoding end device or a video encoding device, and the decoding end can be called a decoding end device or a video decoding device.

[0070] To enable those skilled in the art to better understand the embodiments of this application, the following description will be provided first.

[0071] 1. Scene compositing or rendering process.

[0072] The rendering engine can composite or render a scene based on a glTF (3D model file format) file. By rendering the haptic scene, the rendering engine can obtain haptic feedback signals. These haptic signals are then encoded, transmitted, and decoded before being played by a haptic signal actuator.

[0073] The engine obtains glTF files in several ways: the server and client interact by transmitting glTF files through glTF state streams, or by obtaining glTF files through program files stored on the server and client.

[0074] 2. The Moving Pictures Experts Group (MPEG) has made haptic extensions to glTF (3D model file format) for haptic signal rendering.

[0075] The MPEG_haptic extension provides methods for accessing haptic data. MPEG_haptic includes a set of haptic objects (hapticObjects), as shown in Table 1. As shown in Table 2, the accessor attribute in hapticObjects points to the haptic data source.

[0076] Table 1

[0077] Table 2

[0078] The specific access method for haptic data is shown in Figure 4. In MPEG_haptic, hapticObjects points to the accessor, which in turn points to the circular buffer object. The circular buffer object provides information on the location of the data in the circular buffer and, by referencing the MPEG_media object, reflects the attribute information of the haptic data rendering.

[0079] 3. MPEG has also made haptic material-related extensions to glTF (MPEG_haptic_material extension);

[0080] The MPEG_haptic_material extension provides tactile properties for materials. MPEG_haptic_material contains material properties. MPEG_haptic_material is shown in Table 3, and MPEG_haptic_material.material is shown in Table 4.

[0081] Table 3

[0082] The `materials` property points to a set of material objects. Each material object, through its `haptic` property, points to the `hapticObject` in the `MPEG_haptic` that provides the data source, specifying the relevant tactile properties of the material (such as stiffness, friction, etc.) and texture coordinates. The tactile information carried by the tactile material is represented in the form of textures (such as pixels, taxels, etc.), meaning that the texture value corresponding to the texture coordinates represents the tactile information.

[0083] Table 4

[0084] 4. Interactivity extension in Graphics Language Transmission Forma (glTF), where glTF is a 3D model file format;

[0085] glTF is used for the efficient transfer and loading of 3D scenes and models by engines and applications. glTF defines a scalable publishing format that simplifies the creation workflow and interactive services by enabling interoperability of 3D content across the industry.

[0086] The Moving Picture Experts Group (MPEG) format extends glTF with interactive features and has initially designed relevant standards for interactivity. For haptic scenarios, the interactivity standards primarily describe the interaction between the avatar and the haptic source, thereby triggering and generating haptic signals.

[0087] Interactivity is defined based on behavior, which includes trigger nodes (also described as trigger objects) and action nodes (also described as action objects). Trigger nodes include collision conditions, proximity conditions, and visibility conditions. Action nodes include haptic feedback actions (ACTION_SET_HAPTIC) and activation nodes (ACTION_ACTIVATE). ACTION_SET_HAPTIC is implemented based on haptic action node objects or haptic action media objects.

[0088] The hapticObject property in HapticActionNode (as shown in Table 5) points to the hapticObject in MPEG_haptic, and the mediaIndex property in HapticActionMedia (as shown in Table 6) then points to the accessor in the aforementioned hapticObject, thereby accessing haptic data.

[0089] Table 5

[0090] Table 6

[0091] The object in the embodiments of this application may include at least one of the following: scene, node, mesh, material, vertices, face, primitive, etc.

[0092] The spatial tactile signal processing method, apparatus, and related equipment provided in this application will be described in detail below with reference to the accompanying drawings and through some embodiments and application scenarios.

[0093] Please refer to Figure 5, which is a flowchart of a spatial tactile signal processing method provided in an embodiment of this application. As shown in Figure 5, it includes the following steps:

[0094] Step 501: Obtain auxiliary information for processing one-dimensional spatial tactile signals.

[0095] The aforementioned one-dimensional spatial tactile signal (1-D spatial tactile signal or 1D spatial tactile signal) can be a spatial tactile signal in which the time axis of the time-domain tactile signal is transposed to a one-dimensional spatial axis.

[0096] In some implementations, the aforementioned one-dimensional spatial tactile signal can be a spatial tactile signal with a tactile modality such as stiffness, friction, or tactile texture. For example, when the tactile signal pointed to by the tactile action object corresponds to a specific tactile modality (such as stiffness, friction, tactile texture, etc.), the tactile signal is a one-dimensional spatial tactile signal.

[0097] In some embodiments, the aforementioned one-dimensional spatial tactile signal can be a spatial tactile signal used to represent tactile modalities associated with tactile materials, such as spatial tactile signals used to represent tactile modalities such as stiffness, friction, or tactile texture associated with tactile materials.

[0098] In some implementations, the aforementioned one-dimensional spatial tactile signal can be a media signal, such as a tactile action object pointing to a media signal, which is a one-dimensional spatial tactile signal.

[0099] In this embodiment of the application, the tactile signal can also be referred to as tactile data.

[0100] In some implementations, the above processing can be rendering, i.e., auxiliary information for rendering one-dimensional spatial tactile signals, such as rendering information. This can support the rendering of one-dimensional spatial tactile signals, which is beneficial to improving the tactile feedback effect, i.e., one-dimensional spatial tactile feedback can be realized.

[0101] In some implementations, the above processing can also be other processing besides rendering, such as playback processing.

[0102] The aforementioned auxiliary information for processing one-dimensional spatial tactile signals can be auxiliary information received from other electronic devices, or it can be obtained from attribute information, or it can be auxiliary information obtained by recognizing tactile-related information, etc.

[0103] In some implementations, the processing is based on the one-dimensional spatial tactile signal.

[0104] In this embodiment of the application, the above method is executed by an electronic device or an engine, such as a rendering engine, rendering device, scene compositing device, scene compositing engine, rendering engine, or rendering device. Alternatively, the above method can be executed by an electronic device running a rendering engine, scene compositing engine, or rendering engine.

[0105] In this embodiment, by acquiring auxiliary information for processing one-dimensional spatial tactile signals, processing of one-dimensional spatial tactile signals can be supported. Processing one-dimensional spatial tactile signals is simpler, which helps reduce the complexity of the renderer's processing model. Furthermore, the ability to process one-dimensional spatial tactile signals makes it applicable to more scenarios. Additionally, supporting the processing of one-dimensional spatial tactile signals improves the tactile feedback effect.

[0106] As an optional implementation, the method further includes:

[0107] Identify the one-dimensional spatial tactile signal.

[0108] The above-mentioned identification of one-dimensional spatial tactile signals can be used to determine whether a one-dimensional spatial signal is a one-dimensional spatial tactile signal, such as determining whether the spatial tactile signal to be processed is a one-dimensional spatial tactile signal.

[0109] In this embodiment, since one-dimensional spatial tactile signals can be identified, auxiliary information for processing the one-dimensional spatial tactile signals can be obtained in a timely manner to improve processing efficiency.

[0110] In some implementations, the one-dimensional spatial tactile signal may also be pre-specified, meaning that the method may not include the steps described above for identifying the one-dimensional spatial tactile signal.

[0111] Optionally, the identification of one-dimensional spatial tactile signals includes:

[0112] One-dimensional spatial tactile signals are identified based on tactile material objects, wherein the tactile material object points to the one-dimensional spatial tactile signal, or the tactile material object points to a spatial tactile signal in the form of a texture map, and the texture value of the spatial tactile signal in the form of a texture map is the storage location information of the one-dimensional spatial tactile signal.

[0113] The above-mentioned tactile material object pointing to the one-dimensional spatial tactile signal can be understood as determining the spatial tactile signal pointed to by the tactile material object as a one-dimensional spatial tactile signal.

[0114] The texture value of the spatial tactile signal in the above texture form is the storage location information of the one-dimensional spatial tactile signal. It can be understood that the spatial tactile signal corresponding to the storage location information represented by the texture value of the spatial tactile signal in the texture form is determined as a one-dimensional spatial tactile signal.

[0115] The spatial tactile signals in the above-mentioned texture form can be pixel maps or tactile pixel maps.

[0116] The aforementioned storage location information can be a Uniform Resource Locator (URL), a Uniform Resource Name (URN), a buffer ID, or an accessor ID, etc.

[0117] In this embodiment, it is possible to recognize one-dimensional spatial tactile signals based on tactile material objects, thereby improving the reliability of recognizing one-dimensional spatial tactile signals.

[0118] Optionally, the identification information in the attribute information of the glTF object indicates that the spatial tactile signal pointed to by the tactile material object is the one-dimensional spatial tactile signal.

[0119] The attribute information of the glTF object can be obtained in advance or obtained when performing the above recognition steps.

[0120] In this embodiment, since the identification information in the attribute information of the glTF object indicates that the spatial tactile signal pointed to by the tactile material object is a one-dimensional spatial tactile signal, the one-dimensional spatial tactile signal can be directly determined based on the attribute information of the glTF object, thereby reducing the complexity of recognizing the one-dimensional spatial tactile signal and saving computational overhead.

[0121] It should be noted that the embodiments of this application are not limited to recognizing one-dimensional spatial tactile signals based on tactile material objects. For example, one-dimensional spatial tactile signals can also be recognized based on tactile action objects. See the following embodiments for details.

[0122] In one embodiment, taking the rendering engine executing the above method as an example, the rendering engine determines that the rendered tactile signal is a one-dimensional spatial tactile signal, and the method includes at least one of the following.

[0123] Method 1: The rendering engine identifies tactile material objects, which can point to one-dimensional tactile signals.

[0124] Optionally, the attribute information of the glTF object includes identification information, which is used to indicate that the tactile signal pointed to by the tactile material object is a one-dimensional spatial tactile signal.

[0125] Optionally, the above glTF object is a haptic material object, a sub-object of a haptic material object, or an object associated with a haptic material object.

[0126] Optionally, the above-mentioned glTF object is a tactile action, that is, the attribute information of the tactile action object includes identification information, which is used to indicate the one-dimensional spatial tactile signal corresponding to the tactile material associated with the tactile action object.

[0127] Method 2: The rendering engine identifies tactile material objects, which point to spatial tactile signals in the form of textures, such as pixel maps, taxel maps, etc. The texture value of the spatial tactile signal in the form of textures is the location information of the one-dimensional spatial tactile signal, such as URN, URL, buffer ID, accessor ID, etc.

[0128] Method 3: The rendering engine parses the haptic motion object, which points to a one-dimensional haptic signal. For example, the haptic motion object can point to a media signal, which can be a one-dimensional haptic signal.

[0129] Optionally, when the tactile signal pointed to by the tactile action object corresponds to a specific tactile modality (e.g., stiffness, friction, tactile texture, etc.), the tactile signal is a one-dimensional spatial tactile signal.

[0130] As an optional implementation, the auxiliary information includes at least one of the following:

[0131] Reference object;

[0132] Information about the reference object;

[0133] Reference coordinate system;

[0134] A reference starting position is used to indicate the starting position of the one-dimensional spatial tactile signal in space.

[0135] Reference endpoint position, which is used to indicate the spatial end position of the one-dimensional spatial tactile signal;

[0136] Reference direction information, which is used to represent the spatial mapping direction of the one-dimensional spatial tactile signal;

[0137] Scale factor information, which is used to match the spatial granularity of the haptic signal of the spatial haptic media with the spatial granularity in the scene.

[0138] The aforementioned reference object can be a displacement reference signal, such as a reference node, reference mesh, or reference primitive.

[0139] The aforementioned reference object enables the processing of one-dimensional spatial tactile signals, such as rendering or playing one-dimensional spatial tactile signals based on the reference object.

[0140] In some implementations, the reference object mentioned above includes at least one of the following:

[0141] The object whose properties are related to the reference object of the glTF object;

[0142] The object corresponding to the tactile avatar;

[0143] An object controlled by an external device.

[0144] The attribute information related to the reference object of the aforementioned glTF object can be new attribute information added to the attribute information of the glTF object. In this way, the reference object can be directly determined through the attribute information related to the reference object of the glTF object, thereby reducing computational overhead.

[0145] In this embodiment, multiple reference objects can be used to process one-dimensional spatial tactile signals, thereby improving the processing performance of one-dimensional spatial tactile signals.

[0146] The information from the aforementioned reference object enables the processing of one-dimensional spatial tactile signals, such as rendering or playing one-dimensional spatial tactile signals based on the information from the reference object.

[0147] In some implementations, the information of the reference object includes: the displacement information of the reference object.

[0148] The displacement information mentioned above may include at least one of the following:

[0149] Speed ​​information and direction information.

[0150] This allows for the processing of one-dimensional spatial tactile signals based on the displacement information of a reference object. For example, one-dimensional spatial tactile signals can be rendered or played based on the displacement information of the reference object. Specifically, rendering or playing can be based on velocity information and direction information, thereby improving the processing performance of one-dimensional spatial tactile signals and enhancing the tactile feedback effect.

[0151] In some implementations, the information of the aforementioned reference signal may be the position information of the reference signal, etc.

[0152] The aforementioned reference coordinate system is used to process the aforementioned one-dimensional spatial tactile signal. For example, the relative displacement information of the reference object in the reference coordinate system is used to render the one-dimensional spatial tactile signal.

[0153] In some implementations, the reference coordinate system includes:

[0154] The coordinate system indicated by the reference coordinate system-related attribute information of the glTF object;

[0155] The relative coordinate system corresponding to the reference object.

[0156] The reference coordinate system related attribute information of the glTF object can be new attribute information added to the attribute information of the glTF object. In this way, the reference coordinate system can be directly determined through the reference coordinate system related attribute information of the glTF object, thereby reducing the computational overhead.

[0157] The relative coordinate system corresponding to the aforementioned reference object is pre-configured, meaning that the reference object pre-assigns a relative coordinate system. For example, in the default case or when no explicit indication of the reference coordinate system is provided, the relative coordinate system corresponding to the reference object serves as the reference coordinate system. Alternatively, a flag can be used to indicate whether the relative coordinate system corresponding to the reference object serves as the reference coordinate system.

[0158] The aforementioned reference coordinate system allows for the processing of one-dimensional spatial tactile signals using this coordinate system as a reference, thereby improving the processing performance of one-dimensional spatial tactile signals.

[0159] The starting position of the one-dimensional spatial tactile signal can be determined by the above reference starting position, so that the processing of the one-dimensional spatial tactile signal can start from this starting position, thereby improving the processing performance of the one-dimensional spatial tactile signal.

[0160] The aforementioned reference endpoint position can be used to determine the spatial end position of the one-dimensional spatial tactile signal. This allows the processing of the one-dimensional spatial tactile signal to begin and end at this endpoint, thereby improving the processing performance of the one-dimensional spatial tactile signal and saving computational overhead.

[0161] The above reference direction information can be used to determine the spatial mapping direction of the one-dimensional spatial tactile signal, thereby improving the processing performance of the one-dimensional spatial tactile signal by mapping it in this direction.

[0162] The aforementioned scaling factor information can be used to match the spatial granularity of the tactile signal in the spatial tactile media with the spatial granularity in the scene, thereby enabling the one-dimensional spatial tactile signal to match the tactile signal in the spatial tactile media and the scene, and improving the display effect of the one-dimensional spatial tactile signal.

[0163] In some implementations, at least one of the reference start position, the reference end position, and the scaling factor information is indicated by the corresponding attribute information in the glTF object.

[0164] It can be that at least one of the following indicators—the reference start position, the reference end position, and the scaling factor—is added to the corresponding attribute information of the glTF object.

[0165] Since at least one of the reference start position, reference end position, and scaling factor information is indicated by the corresponding attribute information in the glTF object, it is possible to avoid determining this at least one item through other means, thus saving computational overhead.

[0166] In some implementations, the reference direction information includes at least one of the following:

[0167] The direction indicated by the reference direction related properties of the glTF object;

[0168] The direction is determined based on the normal direction corresponding to the element associated with the reference starting position;

[0169] The direction is determined based on the normal direction corresponding to the element associated with the reference endpoint position;

[0170] The direction indicated by the control information in the haptic media file.

[0171] The reference direction related attribute information of the above glTF object can be new attribute information added to the attribute information of the glTF object.

[0172] The elements associated with the aforementioned reference start position or reference end position can be points, lines, surfaces (circles, triangles, quadrilaterals), grids, grid surfaces, or grid vertices.

[0173] The direction determined based on the normal direction of the element associated with the reference starting position can be either the positive or negative direction of the normal direction of the element associated with the reference starting position. For example, the direction determined based on the normal direction of the element associated with the reference starting position includes either the positive or negative direction of the normal direction of the element associated with the reference starting position, wherein the positive or negative direction is indicated by the relevant attribute information of the glTF object.

[0174] The direction determined based on the normal direction corresponding to the element associated with the reference endpoint position can be either the positive or negative direction of the normal direction corresponding to the element associated with the reference endpoint position. For example, the direction determined based on the normal direction corresponding to the element associated with the reference endpoint position includes either the positive or negative direction of the normal direction corresponding to the element associated with the reference endpoint position, wherein the positive or negative direction is indicated by the relevant attribute information of the glTF object.

[0175] The aforementioned attribute information of the glTF object can be attribute information related to the reference direction of the glTF object, or attribute information related to the reference start position or reference end position.

[0176] The positive or negative direction is indicated by the relevant attribute information of the glTF object. This allows the reference direction information to be indicated by the relevant attribute information of the glTF object, thus saving computational overhead.

[0177] In the above embodiments, multiple directions can be supported as reference direction information to improve the processing flexibility of one-dimensional spatial tactile signals.

[0178] In some implementations, the direction indicated by the attribute information of the glTF object has a higher priority than the direction indicated by the control information in the haptic media file.

[0179] For example, if the control information in the haptic media file indicates the direction, and the attribute information of the glTF object also explicitly indicates the reference direction, then the attribute information of the glTF object has a higher priority than the direction indicated by the control information in the haptic media file. The one-dimensional haptic signal is processed according to the reference direction indicated by the attribute information of the glTF object, so as to improve the processing reliability of the one-dimensional haptic signal.

[0180] It should be noted that at least one of the above auxiliary information may be defaulted. For example, at least one of the following may be defaulted: reference object, information of reference object, reference coordinate system, reference start position, reference end position, reference direction, and scale factor. That is, in this embodiment of the application, at least one of the above may not be obtained, or it can be understood that the above auxiliary information may only include one or more of the above, and does not need to include all of them.

[0181] As an optional implementation, the glTF object in the above embodiments includes at least one of the following:

[0182] Tactile action object;

[0183] Sub-objects of the haptic motion object;

[0184] The object associated with the tactile action object;

[0185] Tactile objects;

[0186] Sub-objects of the haptic object;

[0187] The object associated with the tactile object;

[0188] Tactile source object;

[0189] Sub-objects of the tactile source object;

[0190] The object associated with the tactile source object.

[0191] The aforementioned tactile source objects include at least one of the following:

[0192] Tactile material objects;

[0193] Haptic source node object.

[0194] In this embodiment, it is possible to support the attribute information of multiple glTF objects to indicate the above-mentioned auxiliary information or one-dimensional spatial tactile signals, so as to improve the flexibility of one-dimensional spatial tactile signals and thus improve the processing performance of one-dimensional spatial tactile signals.

[0195] It should be noted that the attribute information indication of the glTF object in the embodiments of this application can be explicit or implicit.

[0196] In one embodiment, taking the rendering engine executing the above method as an example, the rendering engine obtains at least one of the following information a to e (e.g., the rendering engine obtains information by parsing a glTF object) for rendering a one-dimensional spatial tactile signal:

[0197] Information a) Displacement reference object / reference object, such as reference node / reference mesh / reference primitive, etc. The rendering engine renders one-dimensional spatial tactile data based on the displacement information of the displacement reference object;

[0198] Optionally, the displacement information includes at least information such as velocity / direction;

[0199] Optionally, the displacement reference object can be explicitly indicated by the attribute information of the glTF object;

[0200] Optionally, the displacement reference object can be the object corresponding to the haptic avatar. The rendering engine identifies the avatar object by its object identifier / name and uses it as a reference object; for example, by indicating the object corresponding to the haptic avatar in the trigger object or its child objects, the rendering engine can use that haptic avatar object as the displacement reference object for haptic rendering.

[0201] Optionally, the displacement reference object can be an object directly controlled by an external device.

[0202] Information b, reference coordinate system, wherein the relative displacement information of the displacement reference object in the reference coordinate system is used to render one-dimensional spatial tactile data;

[0203] Optionally, the reference coordinate system can be explicitly indicated by the attribute information of the glTF object; for example, the indication method includes indicating information such as nodes / mesh / primitives, and the relative coordinate system corresponding to the indicated node / mesh / primitive is the reference coordinate system;

[0204] Optionally, the reference coordinate system is the relative coordinate system corresponding to the displacement reference object;

[0205] Optionally, by default or when no reference coordinate system indication information is explicitly provided, the relative coordinate system corresponding to the displacement reference object is used as the reference coordinate system.

[0206] Optionally, a flag can be used to indicate whether the relative coordinate system corresponding to the displacement reference object is used as the reference coordinate system.

[0207] Information c, reference start position information and / or reference end position information in the reference coordinate system, wherein the start position represents the position where the rendering of the 1-D spatial tactile signal begins, and the start position represents the position where the rendering of the one-dimensional spatial tactile signal ends;

[0208] Optionally, the start / end position can be explicitly indicated by the property information of the glTF object, for example, by elements such as points, lines, faces (circles, triangles, quadrilaterals), meshes, mesh surfaces, mesh vertices, etc.

[0209] Optionally, the mesh can be a mesh to which a tactile material is attached.

[0210] Information d, reference direction information in the reference coordinate system, wherein the reference direction represents the direction for rendering a one-dimensional spatial tactile signal;

[0211] Optionally, the reference direction information can be explicitly indicated by the property information of the glTF object, for example, by using a direction vector to explicitly indicate the reference direction;

[0212] Optionally, the reference direction is the normal direction corresponding to the face / mesh element associated with the reference start position and / or reference end position;

[0213] Optionally, the property information of the glTF object can indicate whether the reference direction is the positive or negative direction of the normal direction;

[0214] For example, if a one-dimensional tactile signal describes stiffness information, the direction of the signal can be along the normal direction of the grid (which can represent the tactile source object);

[0215] Optionally, the reference direction is the direction indicated by the control information in the haptic media file;

[0216] Optionally, if the glTF object's property information also explicitly indicates the reference direction, its priority is higher than the direction indicated by the control information in the haptic media file, and the rendering engine needs to render according to the reference direction explicitly indicated by the glTF object's property information.

[0217] Information e, scaling factor related information, wherein the scaling factor is used to match the spatial granularity of the tactile signal of the tactile media with the spatial granularity in the scene;

[0218] Alternatively, the scaling factor information can be explicitly indicated by the property information of the glTF object.

[0219] Optionally, if the information obtained by the rendering engine is indicated by the property information of the glTF object, the aforementioned information can be indicated in the property information of the haptic action object, or the triggering object, or its associated child object, so that when haptic feedback is triggered, the rendering engine can correctly identify the aforementioned information used for haptic rendering.

[0220] Optionally, if the information obtained by the rendering engine is indicated by the attribute information of the glTF object, the aforementioned information can be indicated in the attribute information of the haptic source object (e.g., haptic material object, haptic source node object, etc.) or its associated sub-objects, so that when haptic feedback is triggered, the rendering engine can correctly identify the aforementioned information used for haptic rendering.

[0221] Optionally, the rendering engine can monitor the haptic source object and determine when to trigger haptic feedback.

[0222] As an optional implementation, the above method further includes:

[0223] Based on the above auxiliary information processing, one-dimensional spatial tactile signals are processed.

[0224] Taking rendering as an example:

[0225] In the case of rendering one-dimensional spatial tactile signals based on tactile material nodes, the tactile rendering detects that a tactile action has been triggered, and the rendering engine needs to process the tactile material. The tactile modality associated with the tactile material is represented by a one-dimensional spatial tactile signal, such as stiffness. The direction of the one-dimensional spatial tactile signal is along the normal direction of the mesh corresponding to the tactile source.

[0226] The rendering engine can also determine the displacement reference object and judge its movement in the reference frame of the object mapped by the tactile material, including the velocity direction. When the movement of the displacement reference object matches the mapping of the one-dimensional tactile signal (e.g., the displacement reference object moves along the direction of stiffness), the rendering engine generates corresponding tactile feedback.

[0227] When rendering one-dimensional spatial tactile signals directly based on tactile motion nodes, the tactile rendering detects that a tactile motion has been triggered and determines whether to provide tactile feedback based on the one-dimensional spatial tactile signal. First, the rendering engine determines a displacement reference object through the attribute information in the tactile motion object, and determines the movement of the displacement reference object in its coordinate system, including velocity and direction. If the movement of the displacement reference object matches the mapping of the one-dimensional spatial tactile signal, the rendering engine generates the corresponding tactile feedback.

[0228] In this embodiment of the application, auxiliary information for processing one-dimensional spatial tactile signals is obtained. Since the auxiliary information for processing one-dimensional spatial tactile signals is obtained, it is possible to support the processing of one-dimensional spatial tactile signals. The processing of one-dimensional spatial tactile signals is simpler, which helps to reduce the complexity of the renderer's processing model. Furthermore, since it can process one-dimensional spatial tactile signals, it is applicable to more scenarios.

[0229] The spatial tactile signal processing method provided in this application can be executed by a spatial tactile signal processing device. As an example, the device can be an electronic device or a component within an electronic device, such as a chip or circuit. This application uses the execution of the spatial tactile signal processing method by a spatial tactile signal processing device as an example to illustrate the spatial tactile signal processing device provided in this application.

[0230] Please refer to Figure 6, which is a structural diagram of a spatial tactile signal processing device provided in an embodiment of this application. As shown in Figure 6, the spatial tactile signal processing device 600 includes:

[0231] The acquisition module 601 is used to acquire auxiliary information for processing one-dimensional spatial tactile signals.

[0232] Optionally, the device further includes:

[0233] The recognition module is used to recognize the one-dimensional spatial tactile signal.

[0234] Optionally, the identification of one-dimensional spatial tactile signals includes:

[0235] One-dimensional spatial tactile signals are identified based on tactile material objects, wherein the tactile material object points to the one-dimensional spatial tactile signal, or the tactile material object points to a spatial tactile signal in the form of a texture map, and the texture value of the spatial tactile signal in the form of a texture map is the storage location information of the one-dimensional spatial tactile signal.

[0236] Optionally, the identification information in the attribute information of the graphical language transmission format glTF object indicates that the spatial tactile signal pointed to by the tactile material object is the one-dimensional spatial tactile signal.

[0237] Optionally, the auxiliary information includes at least one of the following:

[0238] Reference object;

[0239] Information about the reference object;

[0240] Reference coordinate system;

[0241] A reference starting position is used to indicate the starting position of the one-dimensional spatial tactile signal in space.

[0242] Reference endpoint position, which is used to indicate the spatial end position of the one-dimensional spatial tactile signal;

[0243] Reference direction information, which is used to represent the spatial mapping direction of the one-dimensional spatial tactile signal;

[0244] Scale factor information, which is used to match the spatial granularity of the haptic signal of the spatial haptic media with the spatial granularity in the scene.

[0245] Optionally, the information of the reference object includes: the displacement information of the reference object.

[0246] Optionally, the displacement information includes at least one of the following:

[0247] Speed ​​information and direction information.

[0248] Optionally, the reference object includes at least one of the following:

[0249] The object whose properties are related to the reference object of the glTF object;

[0250] The object corresponding to the tactile incarnation;

[0251] An object controlled by an external device.

[0252] Optionally, the reference coordinate system includes:

[0253] The coordinate system indicated by the reference coordinate system-related attribute information of the glTF object;

[0254] The relative coordinate system corresponding to the reference object.

[0255] Optionally, at least one of the reference start position, the reference end position, and the scaling factor information is indicated by the corresponding attribute information in the glTF object.

[0256] Optionally, the reference direction information includes at least one of the following:

[0257] The direction indicated by the reference direction related properties of the glTF object;

[0258] The direction is determined based on the normal direction corresponding to the element associated with the reference starting position;

[0259] The direction is determined based on the normal direction corresponding to the element associated with the reference endpoint position;

[0260] The direction indicated by the control information in the haptic media file.

[0261] Optionally, the direction indicated by the attribute information of the glTF object has a higher priority than the direction indicated by the control information in the haptic media file.

[0262] Optionally, the direction determined based on the normal direction corresponding to the element associated with the reference starting position includes:

[0263] The positive or negative direction of the normal direction corresponding to the element associated with the reference starting position, wherein the positive or negative direction is indicated by the relevant attribute information of the glTF object;

[0264] The direction determined based on the normal direction corresponding to the element associated with the reference endpoint position includes:

[0265] The positive or negative direction of the normal direction corresponding to the element associated with the reference endpoint position, wherein the positive or negative direction is indicated by the relevant attribute information of the glTF object.

[0266] Optionally, the glTF object includes at least one of the following:

[0267] Tactile action object;

[0268] Sub-objects of the haptic motion object;

[0269] The object associated with the tactile action object;

[0270] Tactile objects;

[0271] Sub-objects of the haptic object;

[0272] The object associated with the tactile object;

[0273] Tactile source object;

[0274] Sub-objects of the tactile source object;

[0275] The object associated with the tactile source object.

[0276] Optionally, the tactile source object includes at least one of the following:

[0277] Tactile material objects;

[0278] Haptic source node object.

[0279] The aforementioned spatial tactile signal processing device is beneficial for improving the tactile feedback effect.

[0280] The spatial tactile signal processing device provided in this application embodiment can implement the various processes implemented in the method embodiment of FIG5 and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0281] As shown in Figure 7, this application embodiment also provides an electronic device 700, including a processor 701 and a memory 702. The memory 702 stores programs or instructions that can run on the processor 701. For example, when the electronic device 700 is an encoding device, the program or instructions executed by the processor 701 implement the various steps of the above-described spatial tactile signal processing method embodiment and achieve the same technical effect. When the electronic device 700 is a decoding device, the program or instructions executed by the processor 701 implement the various steps of the above-described spatial tactile signal processing method embodiment and achieve the same technical effect. To avoid repetition, this will not be described again here. Optionally, the memory 702 can be the memory 102 or memory 113 in the embodiment shown in Figure 1, and the processor 701 can implement the functions of the encoder 200 or decoder 300 in the embodiments shown in Figures 1-3.

[0282] This application also provides an electronic device, including: a memory configured to store video data; and a processing circuit configured to implement the various steps of the above-described spatial tactile signal processing method embodiments. Optionally, the memory may be memory 102 or memory 113 in the embodiment shown in FIG1, and the processing circuit may implement the functions of encoder 200 or decoder 300 in the embodiments shown in FIG1-3.

[0283] This application also provides an electronic device, including a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps in the method embodiment shown in FIG5. This device embodiment corresponds to the above method embodiment, and all implementation processes and methods of the above method embodiments can be applied to this terminal embodiment and can achieve the same technical effect.

[0284] The processor or processing circuit in this application embodiment may include general-purpose processors, special-purpose processors, etc., such as central processing units (CPUs), microprocessors, digital signal processors (DSPs), artificial intelligence (AI) processors, graphics processing units (GPUs), application-specific integrated circuits (ASICs), network processors (NPs), field-programmable gate arrays (FPGAs), or other programmable logic devices, gate circuits, transistors, discrete hardware components, etc. The communication interface in this application embodiment may include transceivers, pins, circuits, buses, etc.

[0285] The aforementioned electronic devices can be terminals or other devices besides terminals, such as servers, network attached storage (NAS), etc.

[0286] The terminal can be a mobile phone, tablet computer, laptop computer, notebook computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile internet device (MID), augmented reality (AR), virtual reality (VR) device, mixed reality (MR) device, robot, wearable device, flight vehicle, vehicle user equipment (VUE), shipboard equipment, pedestrian user equipment (PUE), smart home (home devices with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), game console, personal computer (PC), ATM or self-service machine, etc. Wearable devices include: smartwatches, smart bracelets, smart earphones, smart glasses, smart jewelry (smart bracelets, smart chains, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among these, in-vehicle devices can also be referred to as in-vehicle terminals, in-vehicle controllers, in-vehicle modules, in-vehicle components, in-vehicle chips, or in-vehicle units, etc. It should be noted that the embodiments in this application do not limit the specific type of terminal.

[0287] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server. A cloud server can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), or cloud computing services based on big data and artificial intelligence platforms.

[0288] For example, the aforementioned electronic device may include, but is not limited to, the type of source device 100 or destination device 110 shown in FIG1.

[0289] Taking an electronic device as an example, Figure 8 is a schematic diagram of the hardware structure of a terminal implementing an embodiment of this application.

[0290] The terminal 800 includes, but is not limited to, at least some of the following components: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.

[0291] Those skilled in the art will understand that the terminal 800 may also include a power supply (such as a battery) for powering various components. The power supply can be logically connected to the processor 810 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The terminal structure shown in Figure 8 does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0292] It should be understood that, in this embodiment, the input unit 804 may include a graphics processor 8041 and a microphone 8042. The graphics processor 8041 processes image data of still images or videos obtained by an image acquisition device (such as a camera) in video acquisition mode or image acquisition mode, or it may process the obtained point cloud data. The display unit 806 may include a display panel 8061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include two parts: a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0293] In this embodiment, after receiving downlink data from the network-side device, the radio frequency unit 801 can transmit it to the processor 810 for processing; in addition, the radio frequency unit 801 can send uplink data to the network-side device. Typically, the radio frequency unit 801 includes, but is not limited to, antennas, amplifiers, transceivers, couplers, low-noise amplifiers, duplexers, etc.

[0294] The memory 809 can be used to store software programs or instructions, as well as various data. The memory 809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 809 may include volatile memory or non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 809 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0295] Processor 810 may include one or more processing units; optionally, processor 810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 810.

[0296] The processor 810 or the radio frequency unit 801 is used to acquire auxiliary information for processing one-dimensional spatial tactile signals.

[0297] Optionally, the processor 810 is also used for:

[0298] Identify the one-dimensional spatial tactile signal.

[0299] Optionally, the identification of one-dimensional spatial tactile signals includes:

[0300] One-dimensional spatial tactile signals are identified based on tactile material objects, wherein the tactile material object points to the one-dimensional spatial tactile signal, or the tactile material object points to a spatial tactile signal in the form of a texture map, and the texture value of the spatial tactile signal in the form of a texture map is the storage location information of the one-dimensional spatial tactile signal.

[0301] Optionally, the identification information in the attribute information of the graphical language transmission format glTF object indicates that the spatial tactile signal pointed to by the tactile material object is the one-dimensional spatial tactile signal.

[0302] Optionally, the auxiliary information includes at least one of the following:

[0303] Reference object;

[0304] Information about the reference object;

[0305] Reference coordinate system;

[0306] A reference starting position is used to indicate the starting position of the one-dimensional spatial tactile signal in space.

[0307] Reference endpoint position, which is used to indicate the spatial end position of the one-dimensional spatial tactile signal;

[0308] Reference direction information, which is used to represent the spatial mapping direction of the one-dimensional spatial tactile signal;

[0309] Scale factor information, which is used to match the spatial granularity of the haptic signal of the spatial haptic media with the spatial granularity in the scene.

[0310] Optionally, the information of the reference object includes: the displacement information of the reference object.

[0311] Optionally, the displacement information includes at least one of the following:

[0312] Speed ​​information and direction information.

[0313] Optionally, the reference object includes at least one of the following:

[0314] The object whose properties are related to the reference object of the glTF object;

[0315] The object corresponding to the tactile incarnation;

[0316] An object controlled by an external device.

[0317] Optionally, the reference coordinate system includes:

[0318] The coordinate system indicated by the reference coordinate system-related attribute information of the glTF object;

[0319] The relative coordinate system corresponding to the reference object.

[0320] Optionally, at least one of the reference start position, the reference end position, and the scaling factor information is indicated by the corresponding attribute information in the glTF object.

[0321] Optionally, the reference direction information includes at least one of the following:

[0322] The direction indicated by the reference direction related properties of the glTF object;

[0323] The direction is determined based on the normal direction corresponding to the element associated with the reference starting position;

[0324] The direction is determined based on the normal direction corresponding to the element associated with the reference endpoint position;

[0325] The direction indicated by the control information in the haptic media file.

[0326] Optionally, the direction indicated by the attribute information of the glTF object has a higher priority than the direction indicated by the control information in the haptic media file.

[0327] Optionally, the direction determined based on the normal direction corresponding to the element associated with the reference starting position includes:

[0328] The positive or negative direction of the normal direction corresponding to the element associated with the reference starting position, wherein the positive or negative direction is indicated by the relevant attribute information of the glTF object;

[0329] The direction determined based on the normal direction corresponding to the element associated with the reference endpoint position includes:

[0330] The positive or negative direction of the normal direction corresponding to the element associated with the reference endpoint position, wherein the positive or negative direction is indicated by the relevant attribute information of the glTF object.

[0331] Optionally, the glTF object includes at least one of the following:

[0332] Tactile action object;

[0333] Sub-objects of the haptic motion object;

[0334] The object associated with the tactile action object;

[0335] Tactile objects;

[0336] Sub-objects of the haptic object;

[0337] The object associated with the tactile object;

[0338] Tactile source object;

[0339] Sub-objects of the tactile source object;

[0340] The object associated with the tactile source object.

[0341] Optionally, the tactile source object includes at least one of the following:

[0342] Tactile material objects;

[0343] Haptic source node object.

[0344] The aforementioned terminals are beneficial for improving the haptic feedback effect.

[0345] It is understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of the spatial tactile signal processing method in the method embodiment, and achieve the same or corresponding technical effects. To avoid repetition, it will not be described again here.

[0346] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described spatial tactile signal processing method embodiments and achieve the same technical effects. To avoid repetition, these will not be described again here.

[0347] The processor mentioned above is the processor in the terminal described in the above embodiments. The readable storage medium includes computer-readable storage media, such as ROM, RAM, magnetic disk, or optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

[0348] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described spatial tactile signal processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0349] It should be understood that the chips mentioned in the embodiments of this application may include system-on-a-chip (also known as system chip, chip system, or system-on-a-chip) or discrete display chips, etc.

[0350] This application also provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described spatial tactile signal processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0351] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0352] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes several instructions to cause the terminal or network-side device to execute the methods described in the various embodiments of this application.

[0353] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other implementations under the guidance of this application without departing from the spirit and scope of the claims. All of these implementations are within the protection scope of this application.

Claims

1. A method for spatial haptic signal processing, comprising: obtaining auxiliary information for processing a one-dimensional spatial haptic signal.

2. The method of claim 1, wherein, The method further comprises: identifying the one-dimensional spatial haptic signal.

3. The method of claim 2, wherein, The identifying the one-dimensional spatial haptic signal comprises: identifying the one-dimensional spatial haptic signal based on a haptic material object, wherein the haptic material object points to the one-dimensional spatial haptic signal, or the haptic material object points to a spatial haptic signal in the form of a map, and a map value of the spatial haptic signal in the form of a map is storage location information of the one-dimensional spatial haptic signal.

4. The method of claim 3, wherein, The identification information in the attribute information of the glTF object indicates that the spatial haptic signal pointed to by the haptic material object is the one-dimensional spatial haptic signal.

5. The method of any one of claims 1 to 4, wherein, The auxiliary information comprises at least one of: a reference object; information of the reference object; a reference coordinate system; a reference start position, which is used to represent a start position of the one-dimensional spatial haptic signal in space; a reference end position, which is used to represent an end position of the one-dimensional spatial haptic signal in space; reference direction information, which is used to represent a mapping direction of the one-dimensional spatial haptic signal in space; scale factor information, which is used to match a spatial granularity of the spatial haptic media haptic signal and a spatial granularity in a scene.

6. The method of claim 5, wherein, The information of the reference object comprises displacement information of the reference object.

7. The method of claim 6, wherein, The displacement information comprises at least one of: velocity information, direction information.

8. The method of any one of claims 5-7, wherein, The reference object comprises at least one of: an object indicated by the reference object-related attribute information of the glTF object; an object corresponding to a haptic avatar; an object controlled by an external device.

9. The method of any one of claims 5 to 8, wherein, The reference coordinate system comprises: a coordinate system indicated by the reference coordinate system-related attribute information of the glTF object; a relative coordinate system corresponding to the reference object.

10. The method of any one of claims 5 to 9, wherein, At least one of the reference start position, the reference end position, and the scale factor information is indicated by corresponding attribute information in the glTF object.

11. The method of any one of claims 5 to 10, wherein, The reference direction information comprises at least one of: a direction indicated by the reference direction-related attribute information of the glTF object; a direction determined based on a normal direction of an element associated with the reference start position; a direction determined based on a normal direction of an element associated with the reference end position; a direction indicated by control information in the haptic media file.

12. The method of claim 11, wherein, The priority of the direction indicated by the attribute information of the glTF object is higher than the direction indicated by the control information in the haptic media file.

13. The method of claim 11, wherein, The direction determined based on the normal direction of the element associated with the reference start position comprises: a positive direction or a reverse direction of the normal direction of the element associated with the reference start position, wherein the positive direction or the reverse direction is indicated by the related attribute information of the glTF object. The direction determined based on the normal direction of the element associated with the reference end position comprises: a positive direction or a reverse direction of the normal direction of the element associated with the reference end position, wherein the positive direction or the reverse direction is indicated by the related attribute information of the glTF object.

14. The method of claim 4, 8, 9, 10, 11, 12, or 13, wherein, The glTF object comprises at least one of: haptic action object; sub-object of the haptic action object; object associated with the haptic action object; haptic object; sub-object of the haptic object; object associated with the haptic object; haptic source object; sub-object of the haptic source object; object associated with the haptic source object.

15. The method of claim 14, wherein, The haptic source object comprises at least one of: haptic material object; haptic source node object. 16.A spatial haptic signal processing apparatus, comprising: an obtaining module configured to obtain auxiliary information for processing a one-dimensional spatial haptic signal.

17. The apparatus of claim 16, wherein, The apparatus further comprises: an identifying module configured to identify the one-dimensional spatial haptic signal.

18. The apparatus of claim 17, wherein, The identifying the one-dimensional spatial haptic signal comprises: identifying the one-dimensional spatial haptic signal based on a haptic material object, wherein the haptic material object points to the one-dimensional spatial haptic signal, or the haptic material object points to a spatial haptic signal in a map form, and a map value of the spatial haptic signal in the map form is storage location information of the one-dimensional spatial haptic signal.

19. The apparatus of any one of claims 16-18, wherein, The auxiliary information comprises at least one of: a reference object; information of the reference object; a reference coordinate system; a reference start position, the reference start position being used to indicate a start position of the one-dimensional spatial haptic signal in space; a reference end position, the reference end position being used to indicate an end position of the one-dimensional spatial haptic signal in space; reference direction information, the reference direction information being used to indicate a mapping direction of the one-dimensional spatial haptic signal in space; scale factor information, the scale factor information being used to match a spatial granularity of the spatial haptic media haptic signal and a spatial granularity in a scene. 20.An apparatus comprising a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement steps of the spatial haptic signal processing method according to any one of claims 1 to 15. 21.A readable storage medium, the readable storage medium storing programs or instructions, the programs or instructions being executed by a processor to implement steps of the spatial haptic signal processing method according to any one of claims 1 to 15. 22.A computer program product stored in a storage medium, the computer program product being executed by at least one processor to implement steps of the spatial haptic signal processing method according to any one of claims 1 to 15.

Images